Protocol & Eligibility

Sampling Contract

Item	Value	Why it matters
Official protocol	segment_random	headline runs sample reference and prediction on matched temporal support
Compatibility protocol	anchor_compat	legacy anchor-only path for older experiments
Target FPS	24	normalizes evaluation-time temporal assumptions
Anchor positions	0.0, 0.3333, 0.6667, 1.0	anchor-based metrics read these positions from the rollout window
Motion frame count	12	motion-oriented metrics decode a 12-frame window per sample
Segment count	4	deterministic segment selection under segment_random
Seed	17	keeps sampling stable at the sample level

Anchor-based metrics and motion metrics share the same sample-local protocol state, so prediction and reference are compared on aligned temporal support.

Conditioning Contract

Asset type	Conditioning strategy	Default parameters	Evaluation implication
image suites	reference_image	conditioning_frame_count = 1	a still reference image is the conditioning signal; prediction is parsed as a video rollout, so prediction-side video metrics can run
memory_loop	reference_image	conditioning_frame_count = 1, 60-second target duration	still-image closed-loop camera path; scoring is self-referential and needs no ground-truth long video. Revisit pairs are read from estimated poses, not from symmetric time indices
video_static, video_dynamic	none (reference corpus)	indexed as benchmark.reference_corpus under data/video	collected real videos, not a generation track. They are the reference distribution for jedi, fvmd, and trajan. WorldAtlas does not condition on a video prefix, generate a continuation, and score that continuation

The manifest stores this contract per sample, so downstream generation and evaluation do not need to infer it from the file layout.

Public Metric Status

Every metric in the documentation and leaderboard is official. There is no separate diagnostic tier. The diagnostic label in older reports is a historical storage label and does not exclude a validated score from publication.

Measurement state	Treatment
Finite, valid result with complete coverage	Publish as official
not_applicable	Preserve the reason; do not insert a zero
Missing, failed or non-finite result	Keep unavailable until repaired or evaluated

Publication still checks input bindings, scoring provenance, coverage and normalization. Suite selection and published aggregate weights determine which metrics enter a composite score.

Metric Gates

Metric or family	Official gate
prompt_alignment	sample must have explicit instruction supervision and must not be self-conditioned
camera_error	requires pose annotations under sample.annotation_path
motion_accuracy	requires aligned segmentation masks
physical_fps_*	require a working Pulse-of-Motion runtime and usable container FPS or manifest FPS
met3r_consistency	official for readable reference/prediction anchors; an empty overlap mask is recorded and receives the worst normalized score (0) instead of being excluded
reconstruction_consistency	Official measurement; requires video and Depth-Anything-3. Use the configured canonical metric name
memory_return_gate	Official; reports the leave-and-return pass rate
memory_scene_f1, memory_revisit_fidelity, memory_geometric_closure, memory_rendering_recovery	official on memory_loop when scoring inputs and the matching backend are available, regardless of return-gate status. Missing ViPE sidecars or backends yield not_applicable, not a forced zero
memory_half_life	Reliable fits are official; insufficient support, no measurable decay or unreliable fits are not_applicable
keystroke_*, rollout_aesthetic_drift, rollout_imaging_drift	official when measurable. Action-set/trajectory-shape agreement and quality degradation do not constitute independent memory evidence
memory_revisit_consistency, scene_memory_f1, memory_symmetry, revisit_return_gate, entity_reappearance_consistency	Official documented metrics with legacy split protocols; do not mix scores across protocols
long_sequence_*	Official when inputs and backend are valid
all remaining metrics in the formal suites	official whenever the metric returns a numeric value

Two common consequences:

• Self-conditioned samples, or samples without explicit instruction labels, do not compute prompt_alignment; they are listed in not_applicable_summary.json.
• image_static and image_dynamic predictions are I2V videos. Prediction-only video metrics run normally; metrics requiring reference video, pose, or masks return unavailable when those sidecars are missing.

Normalization

All headline values are stored as normalized scores in [0, 1].

Metric family	Raw form	Normalization rule
bounded scalar metrics	one scalar already bounded or naturally mapped to a bounded interval	direct min/max scaling from config/benchmark.yaml
empirical scalar metrics	one scalar where lower is better or higher is better	empirical min/max scaling, optionally inverted
z-score metrics	one scalar calibrated against benchmark-wide statistics	z-score mapping into a configured sub-range of [0, 1]
component metrics	multiple raw components	normalize each component, then reduce

Current benchmark-specific reducers:

• camera_error: normalize rotation and translation separately, then reduce with a geometric mean.
• motion_smoothness: normalize MSE, SSIM, and LPIPS separately, then reduce with an arithmetic mean.

Aggregation and Historical Reports

• Publication reads both per_metric_summary.json and historical diagnostic_metric_summary.json. Validated metrics share the official public status; conflicting scores must not silently overwrite each other.
• Raw historical reports retain their labels and filenames for provenance. These do not define public metric tiers.
• Each metric retains its specified denominator, normalization and validity rules. N/A, failures and missing measurements are not automatically zero-filled.
• Dimension and Overall scores follow the published aggregation contract. The status policy does not change formulas, existing values or weights.
• Compare results only with matching suites, protocols, metric configurations and prediction coverage.