Memory Scene F1

This metric measures outbound/return point-cloud overlap within a completed estimated-pose revisit episode. It shares the score_point_cloud_memory kernel with scene_memory_f1, but uses the episode’s excursion peak rather than a ground-truth action boundary. It is not photometric similarity; pixel recovery is memory_revisit_fidelity.

The comparison is not conditioned on common visibility, and estimator independence is not established: common_visibility_conditioned=false and estimator_independence_verified=false. Newly visible surfaces, occlusions, sampling and ViPE errors can change overlap. Unsupported points alone do not establish forgetting or hallucination.

A failed memory_return_gate no longer blocks the cloud measurement. The separate leaderboard task score still assigns 0 to valid-input gate failures because the model did not complete departure and return. The metric requires ViPE depth_metric.npy, depth_valid_mask.npz, and intrinsics.npy; genuinely missing inputs or unavailable measurements are not imputed as zero. Among detected revisit pairs, choose the longest-gap episode $(i^\star,j^\star)$ whose two segments each contain at least two pose frames (ties prefer earlier endpoints). Selection does not use the scene score. Its turnaround is the earliest internal frame maximizing pose distance from the episode’s first viewpoint:

$$t^\star = \arg\max_{i^\star<t<j^\star}\, d(\mathbf{T}_{i^\star},\mathbf{T}_t) \tag{1}$$

Outbound is $[i^\star,t^\star)$ and return is $[t^\star,j^\star+1)$. Unrelated frames after this completed revisit are excluded; the global trajectory maximum remains a diagnostic, not the split boundary. After a failed gate with no supported episode, split the pose track at its midpoint and compare the two non-overlapping halves, each with at least two pose frames (split_source=temporal_midpoint_fallback, measurement_scope=temporal_halves_point_cloud_overlap). Successful gates retain the episode split (split_source=longest_supported_revisit_episode); insufficient segment or depth support remains unavailable. Each side samples up to max_frames_per_segment frames (default $8$) and back-projects up to max_points_per_frame points per frame (default $4096$) to form $P_{\mathrm{obs}}$ and $P_{\mathrm{rev}}$. The return cloud is aligned with trimmed ICP by default (alignment = trimmed_icp, centroid_align = false). The distance threshold is $\tau_d = 0.02\cdot\mathrm{diag}$.

Backprojection uses $x=(u-c_x)z/f_x$ and $y=(v-c_y)z/f_y$, preserving the supplied signed, nonzero focal lengths (preserve_signed_nonzero_focal). Zero or nonfinite intrinsics cannot be projected. This arithmetic policy neither repairs nor certifies ViPE calibration. Results record split_source=longest_supported_revisit_episode and the selected episode bounds.

Let $d(p,P)$ be nearest-neighbor distance. The legacy retention field is the fraction of outbound points supported by the return cloud; the legacy hallucination field is the fraction of return points lacking outbound support, converted to a precision-like term before F1 aggregation. These are overlap statistics, not causal labels:

$$R_{\mathrm{mem}} = \frac{ |\{p \in P_{\mathrm{obs}}: d(p, P_{\mathrm{rev}}) < \tau_d\}| }{ |P_{\mathrm{obs}}| }, \qquad R_{\mathrm{hall}} = \frac{ |\{q \in P_{\mathrm{rev}}: d(q, P_{\mathrm{obs}}) \ge \tau_d\}| }{ |P_{\mathrm{rev}}| } \tag{2}$$ $$P_{\mathrm{mem}} = 1 - R_{\mathrm{hall}} \tag{3}$$

The comprehensive score combines retention and precision symmetrically:

$$S_{\mathrm{scene}} = \frac{2\, P_{\mathrm{mem}}\, R_{\mathrm{mem}}}{P_{\mathrm{mem}} + R_{\mathrm{mem}}} \tag{4}$$

Higher $S_{\mathrm{scene}}$ means greater overlap between the sampled, aligned point clouds. It does not independently prove memory persistence, absence of hallucination, or metric-scale reconstruction accuracy.

In WorldAtlas this algorithm is implemented as the official metric memory_scene_f1 (point_cloud_f1 backend). Under the default normalization config, raw and normalized outputs lie in $[0,1]$ with higher is better. On memory_loop it is official when supported segments and depth sidecars are present, including after a failed gate. The older probe scene_memory_f1 remains only on experimental: it splits at GT action segments, so on a non-loop trajectory it measures 3D consistency between two halves, not memory.

Leaderboard task score

Let $g_i$ be the fixed return gate and $F_i$ the conditional cloud F1 above. With valid inputs, $T_i=0$ when $g_i=0$; when $g_i=1$ and F1 is measurable, $T_i=F_i$. Missing inputs, backend failures, and passed gates without enough cloud support remain unavailable, not automatic zeros.

$$S_{\mathrm{task}}=\frac{\sum_{i\in M}T_i}{|M|},\qquad \mathrm{coverage}=\frac{|M|}{N}.$$

$M$ includes measured returns and zero-valued gate failures. The leaderboard memory_scene_f1 displays this task mean; conditional F1 and gate pass rate are auxiliary diagnostics. Incomplete coverage displays the measured denominator (for example 479/480), leaves the full-suite score null, and reports bounds $[\sum T_i/N,(\sum T_i+N-|M|)/N]$. This combines task completion and scene overlap, rather than conditioning the ranking on successful returns. Do not mix it with old conditional means or treat its verification as certification of other columns.

60-Second Same-Input Examples

Audit manifest: docs/assets/metric_examples/longseq_memory_60s_examples_20260909.json.

Input image

60s memory_loop · same benchmark case · gap 0.9006

memory-dolly_loop-memory_pool_002

Suite: memory_loopScene: photorealistic / outdoor / city

Benchmark prompt target

Move the camera forward through the scene, then back to where it started.

prompt source: memory_loop_prompt

Compare outbound and return geometry: the better example keeps walls and landmarks; the weaker one replaces the room on the way back.

Model result videos will be available with the ModelScope release. / 模型推理视频将在 ModelScope 方案上线后提供。

Higher metric scoreScore 0.9749 · official

Model result videos will be available with the ModelScope release. / 模型推理视频将在 ModelScope 方案上线后提供。

Lower metric scoreScore 0.0743 · official

Input image

60s memory_loop · same benchmark case · gap 0.9435

memory-double_yaw_loop-memory_pool_348

Suite: memory_loopScene: photorealistic / outdoor / verdant_landscape

Benchmark prompt target

Turn away and back to the start twice in a row.

prompt source: memory_loop_prompt

Compare outbound and return geometry: the better example keeps walls and landmarks; the weaker one replaces the room on the way back.

Model result videos will be available with the ModelScope release. / 模型推理视频将在 ModelScope 方案上线后提供。

Higher metric scoreScore 0.9972 · official

Model result videos will be available with the ModelScope release. / 模型推理视频将在 ModelScope 方案上线后提供。

Lower metric scoreScore 0.0536 · official

Input image

60s memory_loop · same benchmark case · gap 0.9492

memory-yaw_loop_180-memory_pool_249

Suite: memory_loopScene: photorealistic / outdoor / verdant_landscape

Benchmark prompt target

Turn the camera to the right, then turn back to the starting view.

prompt source: memory_loop_prompt

Compare outbound and return geometry: the better example keeps walls and landmarks; the weaker one replaces the room on the way back.

Model result videos will be available with the ModelScope release. / 模型推理视频将在 ModelScope 方案上线后提供。

Higher metric scoreScore 0.9492 · official

Model result videos will be available with the ModelScope release. / 模型推理视频将在 ModelScope 方案上线后提供。

Lower metric scoreScore 0.0000 · official

Input image

60s memory_loop · same benchmark case · gap 0.8678

memory-palindrome_loop-memory_pool_397

Suite: memory_loopScene: stylized / indoor / public_spaces

Benchmark prompt target

Move forward, turn and continue, then retrace the whole path back.

prompt source: memory_loop_prompt

Compare outbound and return geometry: the better example keeps walls and landmarks; the weaker one replaces the room on the way back.

Model result videos will be available with the ModelScope release. / 模型推理视频将在 ModelScope 方案上线后提供。

Higher metric scoreScore 0.9928 · official

Model result videos will be available with the ModelScope release. / 模型推理视频将在 ModelScope 方案上线后提供。

Lower metric scoreScore 0.1250 · official

Input image

60s memory_loop · same benchmark case · gap 0.8650

memory-double_yaw_loop-memory_pool_367

Suite: memory_loopScene: photorealistic / outdoor / terrestrial_landscape

Benchmark prompt target

Turn away and back to the start twice in a row.

prompt source: memory_loop_prompt

Compare outbound and return geometry: the better example keeps walls and landmarks; the weaker one replaces the room on the way back.

Model result videos will be available with the ModelScope release. / 模型推理视频将在 ModelScope 方案上线后提供。

Higher metric scoreScore 0.9172 · official

Model result videos will be available with the ModelScope release. / 模型推理视频将在 ModelScope 方案上线后提供。

Lower metric scoreScore 0.0522 · official