Skip to content
WorldAtlas WorldAtlas WorldAtlas
Home

Run Evaluation

All commands below assume conda activate worldatlas and exported .env variables.

1. Build the benchmark manifest

Section titled “1. Build the benchmark manifest”
Terminal window
worldatlas build-manifest --config config/benchmark.yaml
# memory_loop is expanded by default; turn it off with --memory-loops none

Use worldatlas build-inventory --config config/benchmark.yaml only when you explicitly want the raw asset audit without immediately writing a manifest.

2. Generate predictions

Section titled “2. Generate predictions”

Generate with a bundled adapter:

Terminal window
worldatlas generate \
  --config config/benchmark.yaml \
  --model-config config/models/lingbot_world/lingbot_world.yaml \
  --suites image_static \
  --output-dir artifacts/predictions/lingbot-world

Or run the one-shot path:

Terminal window
worldatlas run-model \
  --config config/benchmark.yaml \
  --model-config config/models/lingbot_world/lingbot_world.yaml \
  --suites image_static

Expected prediction layout:

artifacts/predictions/lingbot-world/
  <prediction_stem>.mp4
  evaluation_manifest.jsonl
  generation_records.jsonl
  generation_manifest.json
  prompt_map.json

Keep the complete output directory, including camera/action sidecars. Generation tracks are image_static, image_dynamic, and memory_loop; video_static and video_dynamic are reference corpora.

3. Evaluate predictions

Section titled “3. Evaluate predictions”

Single-process evaluation of the completed generation run:

Terminal window
worldatlas evaluate \
  --config config/benchmark.yaml \
  --model-config config/models/lingbot_world/lingbot_world.yaml \
  --manifest artifacts/predictions/lingbot-world/evaluation_manifest.jsonl \
  --predictions-root artifacts/predictions/lingbot-world \
  --suites image_static \
  --output-dir artifacts/benchmark/reports/lingbot-world

Use the generated evaluation_manifest.jsonl, which binds each video to its source input. For sharded generation, concatenate all successful shards’ evaluation_manifest.shard<N>.jsonl files before evaluation and preserve their generation records. Videos generated elsewhere need a manifest with verified input-to-video bindings.

Useful options:

  • --suites image_static,image_dynamic selects generated tracks.
  • --limit 32 runs a smoke evaluation; distribution metrics may need more samples.
  • --model-config applies the selected model’s metric overrides.

Evaluation artifacts are written under artifacts/benchmark/reports/:

  • run_manifest.json
  • per_sample_scores.jsonl
  • per_metric_summary.json
  • per_dimension_summary.json
  • leaderboard.csv

For a complete result, cover every requested sample in the input manifest and resolve generation and metric failures. The generated evaluation manifest contains successful predictions only, so missing_predictions = 0 alone does not establish full coverage. Compare runs using the same suites, manifest snapshot, and metric settings. See Submission to submit results or request team evaluation.