Scene Memory F1

This metric evaluates whether geometry observed before a camera revisit remains supported by 3D point-cloud overlap after the revisit. Unlike image-pair similarity metrics, it scores scene memory from nearest-neighbor coverage between an observation cloud and a revisit cloud. The official closed-loop metric is memory_scene_f1. The scene_memory_f1 on this page stays on experimental: it splits the trajectory at GT action-segment boundaries, so on a non-loop path it measures 3D consistency between two halves rather than memory. Both share the score_point_cloud_memory kernel.

Long-horizon world models must retain stable scene structure across extended rollouts. A model can look visually consistent frame-to-frame while still forgetting walls, furniture, or layout elements once the viewpoint returns. Scene Memory F1 targets that failure mode directly: it asks how much of the earlier observation cloud can still be matched in 3D after revisit, and how much of the revisit cloud is unsupported by the earlier observation.

The runtime first resolves observation and revisit point clouds. When explicit observation/revisit sidecars or a bundled scene_memory NPZ archive are available, those clouds are loaded directly. Otherwise the metric reconstructs both clouds from the action payload and ViPE depth sidecars associated with the prediction. Invalid points are removed, and the clouds may be voxel-downsampled when voxel_size is configured. The revisit cloud is aligned with trimmed ICP by default (alignment = trimmed_icp, centroid_align = false). A distance threshold $\tau_d$ is then chosen from distance_threshold when provided, or from distance_threshold_ratio $\cdot\mathrm{diag}$ otherwise, with default ratio $0.02$.

Let $P_{\mathrm{obs}}$ denote the observation point cloud and $P_{\mathrm{rev}}$ the aligned revisit cloud. For a point $p$ and cloud $P$, let $d(p,P)$ be nearest-neighbor distance. Retention measures how much of the earlier observation survives at revisit:

$$R_{\mathrm{mem}} = \frac{ |\{p \in P_{\mathrm{obs}}: d(p, P_{\mathrm{rev}}) < \tau_d\}| }{ |P_{\mathrm{obs}}| } \tag{1}$$

Hallucination measures revisit geometry that is not supported by the earlier observation. It is converted to a precision-like term before F1 aggregation:

$$R_{\mathrm{hall}} = \frac{ |\{q \in P_{\mathrm{rev}}: d(q, P_{\mathrm{obs}}) \ge \tau_d\}| }{ |P_{\mathrm{rev}}| } \tag{2}$$ $$P_{\mathrm{mem}} = 1 - R_{\mathrm{hall}} \tag{3}$$

The comprehensive score combines retention and precision symmetrically:

$$S_{\mathrm{scene}} = \frac{2\, P_{\mathrm{mem}}\, R_{\mathrm{mem}}}{P_{\mathrm{mem}} + R_{\mathrm{mem}}} \tag{4}$$

Higher $S_{\mathrm{scene}}$ indicates stronger scene memory: more of the previously observed geometry is retained, and fewer unsupported revisit points are hallucinated. The metric evaluates geometric persistence across revisits, not photometric similarity, semantic prompt adherence, or absolute metric-scale reconstruction accuracy.

In WorldAtlas this algorithm is implemented as the official metric scene_memory_f1 (point_cloud_f1 backend). Under the default normalization config, raw and normalized outputs lie in $[0,1]$ with higher is better. This metric uses a different split protocol from the current closed-loop suite, which uses pose-turnaround memory_scene_f1 instead.

60-Second Illustrative Examples

Audit manifest: docs/assets/metric_examples/longseq_memory_60s_examples_20260909.json.

Input image

60s memory_loop · illustrative same input

memory-dolly_loop-memory_pool_002

Suite: memory_loopScene: photorealistic / outdoor / city

Benchmark prompt target

Move the camera forward through the scene, then back to where it started.

prompt source: memory_loop_prompt

This older probe splits at a GT action boundary. Use it to see whether two halves of a rollout still describe one 3D scene.

Model result videos will be available with the ModelScope release. / 模型推理视频将在 ModelScope 方案上线后提供。

Output Ascene_memory_f1

Model result videos will be available with the ModelScope release. / 模型推理视频将在 ModelScope 方案上线后提供。

Output Bscene_memory_f1

Input image

60s memory_loop · illustrative same input

memory-double_yaw_loop-memory_pool_348

Suite: memory_loopScene: photorealistic / outdoor / verdant_landscape

Benchmark prompt target

Turn away and back to the start twice in a row.

prompt source: memory_loop_prompt

This older probe splits at a GT action boundary. Use it to see whether two halves of a rollout still describe one 3D scene.

Model result videos will be available with the ModelScope release. / 模型推理视频将在 ModelScope 方案上线后提供。

Output Ascene_memory_f1

Model result videos will be available with the ModelScope release. / 模型推理视频将在 ModelScope 方案上线后提供。

Output Bscene_memory_f1