TRAJAN

trajan is the official per-video reconstruction protocol from TRAJAN [Allen et al., 2025], implemented through google-deepmind/tapnet. It measures whether the motion field of a generated video can be reconstructed by a trajectory autoencoder trained on real point tracks. The score reflects temporal consistency of object and camera motion—not appearance quality, prompt following, or pixel-level reconstruction.

FVD and other appearance-based video metrics are biased toward frame-level content: a video can look sharp while fingers morph, objects flicker in and out, or interactions fail. TRAJAN first estimates long-horizon 2D point tracks with BootsTAPIR [Doersch et al., 2024], then compresses a support set of tracks into a fixed-size motion latent $\phi_S$ and reconstructs a held-out set of query tracks. Because the autoencoder was trained on real motion, implausible trajectories are harder to reconstruct, so Average Jaccard (AJ) falls.

For a generated video $V$ , WorldAtlas samples $N = 4096$ query points uniformly over space and time on up to 150 frames resized to $256 \times 256$ . BootsTAPIR produces tracks $s_{t,j} = (x_{t,j}, y_{t,j}, o_{t,j})$ . A permutation splits them into 2048 support tracks $S$ and 2048 held-out targets $Q$ . TRAJAN encodes $S$ and, given a query $(x_q, y_q, t_q)$ on a visible target frame, predicts the full track $\hat{q}$ through that point.

The raw score is TAP-Vid Average Jaccard [Doersch et al., 2022] between reconstructed tracks and the BootsTAPIR targets. For threshold $\delta \in \{1, 2, 4, 8, 16\}$ pixels after scaling to $256 \times 256$ :

\mathrm{Jaccard}_\delta = \frac{\mathrm{TP}_\delta}{\mathrm{TP}_\delta + \mathrm{FP}_\delta + \mathrm{FN}_\delta} \tag{1}

A prediction is a true positive if both the reconstructed point and the BootsTAPIR point are visible and $\lVert \hat{p} - p \rVert_2 < \delta$ . False positives include visible predictions that are too far or whose ground-truth point is occluded. False negatives are visible ground-truth points that the decoder misses or displaces. WorldAtlas follows the TAP-Vid first-frame protocol: frame $0$ is excluded from the count (evaluation_query_mode = strided_from_first_frame). The reported raw value is

\mathrm{TRAJAN}(V) = \frac{1}{|\Delta|}\sum_{\delta \in \Delta} \mathrm{Jaccard}_\delta, \qquad \Delta = \{1, 2, 4, 8, 16\} \tag{2}

The raw score already lies in $[0, 1]$ , with higher being better. WorldAtlas applies identity bounded normalization (lower = 0.0, upper = 1.0). Query sampling is deterministic per sample: seed = SHA256(17 : sample_id). Clips shorter than 150 frames are padded and masked through boundary_frame. The metric reads only the prediction video; the reference is unused, so image-conditioned dynamic generation is valid.

The official notebook has an intentional visibility convention that is easy to invert. BootsTAPIR’s postprocessor yields visibility when applied to tracker occlusion logits. Applied to the trajectory decoder’s visible_logits and certain_logits, the notebook supplies the resulting boolean tensor to TAP-Vid as pred_occluded; AJ therefore uses its complement as predicted visibility. The decoder query itself is sampled on a visible frame, while evaluation sets every track’s strided query frame to frame 0. WorldAtlas preserves both details: it excludes frame 0 and scores every other real, unpadded frame.

trajan is configured for image_dynamic and video_dynamic. It is official when a video prediction is available and the JAX / BootsTAPIR / TRAJAN runtime succeeds. Image-only predictions are not_applicable. Missing checkpoints or an unavailable tapnet runtime produce normalized = null rather than a zero. It complements distributional motion metrics such as FVMD (which need a reference set). It does not judge object identity, prompt semantics, or physically impossible but smoothly trackable events.

Legacy aliases trajan_average_jaccard and trajectory_reconstruction_quality resolve to trajan.

Reference

BibTeX

@misc{allen2025directmotionmodelsassessing,
    title = {Direct Motion Models for Assessing Generated Videos},
    author = {Kelsey Allen and Carl Doersch and Guangyao Zhou and Mohammed Suhail and Danny Driess and Ignacio Rocco and Yulia Rubanova and Thomas Kipf and Mehdi S. M. Sajjadi and Kevin Murphy and Joao Carreira and Sjoerd van Steenkiste},
    year = {2025},
    eprint = {2505.00209},
    archivePrefix = {arXiv},
    primaryClass = {cs.CV},
    url = {https://arxiv.org/abs/2505.00209},
}