FVMD

fvmd (Fréchet Video Motion Distance) measures how far the motion-pattern distribution of a set of predicted videos is from the distribution of a set of reference videos. It follows FVMD [Liu et al., 2024]: track keypoint trajectories with a point-tracking model, build explicit motion features from the velocity and acceleration of those trajectories, and take a Fréchet distance over those features.

WorldAtlas drives the official DSL-Lab/FVMD-frechet-video-motion-distance implementation vendored at thirdparty/FVMD-frechet-video-motion-distance through the fvmd backend. It is a suite-level distribution metric, not a per-sample metric: the whole suite yields one score rather than one score per video.

What It Measures

• whether the motion dynamics of a set of predictions match the motion dynamics of the reference set
• temporal inconsistency: abrupt changes in position or pose, sudden deformation or blurring of objects, and jerky movement, all of which show up as disordered keypoint trajectories and therefore as spikes in velocity and acceleration
• not per-sample alignment: there is no notion of “the score for this one video”, only “how far this batch of videos is from the reference distribution”
• not prompt following, object identity, or semantic correctness

Principle

The motivation for explicit motion features is that existing distribution-level video metrics do not look at motion directly. FID-VID compares only frame appearance and ignores motion entirely. FVD does introduce temporal information via I3D, but its features are biased toward per-frame content (content bias) and remain relatively insensitive to temporal distortion. VBench’s temporal flickering and motion smoothness dimensions tend to reward smooth or even static video, which penalizes dancing and sports footage whose large-amplitude motion is exactly what makes it high quality.

FVMD starts from a different observation: motion that obeys physical laws should not exhibit sudden changes in acceleration. So instead of relying on the implicit features of an action-recognition network, write the motion out explicitly — track keypoints, compute velocity and acceleration fields, and take statistics over them. The resulting features are explicit and interpretable, and what they measure is whether motion is coherent rather than whether individual frames look good.

Inside WorldAtlas there is a second motivation: FVMD complements per-sample motion probes like motion_smoothness and motion_magnitude. Those answer how large and how smooth the motion of a single video is; FVMD answers whether the motion distribution of the model’s entire output has drifted away from the reference corpus.

Detailed Algorithm

1. WorldAtlas collects every prediction video in the suite. The reference side is the dynamic reference corpus (see Reference Corpus), not each sample’s own ground truth, so the two sides are independent sets of different sizes.
2. Each video is resampled to sample_fps = 8 and cut into 16-frame $256 \times 256$ clips, giving a fixed 2-second window (clip_frame_count = 16, analysis_size = 256). Predictions take up to 4 non-overlapping windows; corpus videos also take up to 4.
3. Before GPU tracking, a model-independent CPU preflight excludes reference videos with confirmed decode errors, native frame rates below sample_fps, or fewer than 16 sampled frames. The complete content-bound roster and exclusion reasons are saved in reference_selection.json and reused across models. Predictions are never filtered this way: prediction failures, or new failures among already selected references, still fail the evaluation rather than silently shrinking its population. The same OpenCV container off-by-one rule used for per-sample decoding applies here: when the container reports one extra frame, resampling uses the decoded count and the video stays in the population.
4. Each side is tracked separately with $N = 400$ grid keypoints per clip via PIPs++, producing trajectories $\hat{Y} \in \mathbb{R}^{F \times N \times 2}$.
5. A first-order difference over trajectories gives the velocity field $\hat{V}$; a first-order difference over velocity gives the acceleration field $\hat{A}$.
6. Each vector field is cut into $4 \times 5 \times 5$ volumes, and a dense 1D histogram over 8 angle bins is accumulated inside every volume.
7. The velocity and acceleration histograms are concatenated into a 1024-dimensional combined motion feature.
8. A Fréchet distance under a multivariate-Gaussian approximation over the combined features yields the raw score.
9. The WorldAtlas leaderboard retains the linear high-score region and smoothly extends the low-score tail: let s = 1 - raw / 10000 and p = 0.05; use score = s when s ≥ p, otherwise score = p*p / (2*p - s). The leaderboard displays 100 × score, higher is better.

A lower raw score means the two batches of videos have more similar motion distributions.

Leaderboard Score And Raw Distance

p = 0.05 limits the adjustment to the region below 5 points on the original linear scale; scores of at least 5 points remain unchanged. Raw distances of 0, 9500, 10000 and 20000 map to 100, 5, 2.5 and approximately 0.227 points. Both value and slope are continuous at the join, and larger distances yield lower scores without a shared minimum score. Parameters are frozen in config/leaderboard_fvmd.yaml, independent of model additions. This is a display convention, not a quality threshold or real-versus-real baseline. Large distances still receive very low scores rather than mid-range scores.

Existing reports retain their linearly clipped scores for audit. Publication remaps raw from distribution_metric_details.json directly; a clipped normalized zero cannot recover the distance. No feature extraction or GPU rerun is needed. Failed evaluations remain blank rather than receiving fabricated numeric scores. A poor FVMD may reflect insufficient motion, disordered motion or other distribution differences; it alone does not establish that videos are static.

Why The Upstream Entry Point Is Not Used

WorldAtlas drives the upstream subfunctions (tracking_fullseq, calc_hist, calculate_frechet_distance) rather than the top-level fvmd() helper, because that helper cannot express an independently sized reference set:

• its tracking loop is bounded by len(gen_dataset), so a longer reference set is silently truncated and a shorter one silently restarts and repeats samples;
• it then reshapes the reference histograms using the prediction’s window count, which is a shape error whenever the two sides differ.

Note also that upstream’s fvmd() swaps its own gen/gt variables when unpacking build_dataset(). The reported score is unaffected because the Fréchet distance is symmetric, but it means the velocity_gt.npy written into log_dir actually holds the generated side.

Frame-Rate Alignment

FVMD’s features are per-frame pixel displacements, so they scale with the frame rate: the same physical motion sampled at 30 fps produces displacements roughly half the size of the same motion at 16 fps. Comparing a generated video against real footage at a different frame rate would therefore report the frame-rate gap as a motion difference.

sample_fps resamples both sides to a common rate before any clip is cut, which makes the sampling window a duration (2 seconds) rather than a frame count. Upsampling is refused rather than performed: duplicating frames would fabricate zero-motion segments, which FVMD reads as real signal. References below sample_fps are excluded and counted during preflight; predictions cannot be silently excluded.

Reference container frame counts are independently checked using a full ffprobe -count_frames decode without decoder errors. Verified actual counts are used for reference sampling, and tracking-time decoding must still match them. Each probe is allowed at least 600 seconds and scales with file size up to 7200 seconds, so multi-gigabyte references are not aborted as an operational timeout on NFS. Storage errors, timeouts and tool failures abort preparation rather than shrinking the roster. Reference feature caches include the selection digest; historical scores using a different reference population must be recomputed before comparison.

Trajectories, Velocity, And Acceleration

For an $F$-frame clip, PIPs++ returns $N$ trajectories $\hat{Y} \in \mathbb{R}^{F \times N \times 2}$. The velocity field is the first-order difference of positions between consecutive frames, zero-padded on the first frame to keep the shape aligned:

Velocity field $$\hat{V} = \operatorname{concat} \left( \mathbf{0}_{N \times 2}, \; \hat{Y}_{2:F} - \hat{Y}_{1:F-1} \right) \in \mathbb{R}^{F \times N \times 2}$$

The acceleration field is the first-order difference of the velocity field:

Acceleration field $$\hat{A} = \operatorname{concat} \left( \mathbf{0}_{N \times 2}, \; \hat{V}_{2:F} - \hat{V}_{1:F-1} \right) \in \mathbb{R}^{F \times N \times 2}$$

Here $\hat{Y}_{i:j}$ denotes frames $i$ through $j$ inclusive. Both orders are used because velocity describes the motion itself while acceleration describes its rate of change, and unnatural temporal jumps are most visible in acceleration.

Motion Feature Histograms

Let $U$ be either the velocity or the acceleration field. For each frame $i$ and keypoint $j$, compute the magnitude and orientation of the vector:

Magnitude and orientation $$\rho(U)_{i,j} = \sqrt{U_{i,j,1}^2 + U_{i,j,2}^2}, \qquad \phi(U)_{i,j} = \operatorname{arctan2} \left( U_{i,j,1}, \; U_{i,j,2} \right)$$

Difference from the paper

Equation (4) of the paper writes the orientation as $\left|\tanh^{-1}(U_{i,j,1} / U_{i,j,2})\right|$. The upstream implementation uses np.arctan2, which lands in $[-\pi, \pi]$ and covers the full 360 degrees. This page documents the implementation that actually runs.

Orientations are uniformly quantized into 8 bins of 45 degrees each. Magnitudes are first clipped to $[0, 255]$, then log-transformed base 2 and quantized — most vectors have small magnitudes, and taking the log keeps a few large displacements from dominating the histogram:

Quantized magnitude weight $$w(\rho) = \frac{1}{8} \left\lceil \operatorname{clip} \left( \log_2 (\rho + 1), \; 0, \; 8 \right) \right\rceil \in [0, 1]$$

Each vector field is then cut into $f \times k \times k$ volumes, with $f = 4$ and $k = 5$ in the implementation. Following HOG, the quantized magnitude weights inside a volume are summed directly into the matching orientation bin, giving an 8-point histogram per volume. Concatenating and flattening the histograms of all volumes produces the motion feature for that clip, shaped $\frac{F}{f} \times \frac{\sqrt{N}}{k} \times \frac{\sqrt{N}}{k} \times 8$, which is $4 \times 4 \times 4 \times 8$ under the current configuration.

The paper also proposes a quantized 2D histogram (8 orientation bins × 9 magnitude bins, 72 dimensions) and shows through ablation that the dense 1D histogram aligns better with human judgment. Upstream defaults to the 1D path, and WorldAtlas does not override that choice.

Fréchet Distance

After concatenating velocity and acceleration features into the combined feature, FVMD follows FID and approximates both feature distributions with multivariate Gaussians, which admits a closed form:

Fréchet Video Motion Distance $$\operatorname{FVMD} = \left\| \mu_{\mathrm{data}} - \mu_{\mathrm{gen}} \right\|_2^2 + \operatorname{tr} \left( \Sigma_{\mathrm{data}} + \Sigma_{\mathrm{gen}} - 2 \left( \Sigma_{\mathrm{data}} \Sigma_{\mathrm{gen}} \right)^{\frac{1}{2}} \right)$$

$\mu$ and $\Sigma$ are the empirical mean and covariance of the motion features over each clip set. The Gaussian assumption is easier to accept here than in the I3D feature space: motion histograms are bounded low-dimensional statistics rather than high-dimensional semantic embeddings.

Reference Corpus

The reference side is the collected dynamic video corpus under dynamic/{style}/videos/{motion_category}/, configured by benchmark.reference_corpus and indexed by worldatlas build-reference-corpus.

FVMD only applies to the dynamic suites. Static scenes are driven by a camera path that the prompt specifies, whereas the corpus’s static camera motion is whatever was captured, so the two motion distributions are not meant to match and the resulting distance would not be interpretable.

Dynamic assets carry no environment label in the corpus layout and the dynamic corpus is small, so it forms a single unstratified group: the stratum would be identical to the headline and is not scored twice.

Because the corpus is a benchmark constant, its content signature is recorded in run_manifest.json under reference_corpus. Changing the corpus changes what the score means, and values across different corpus versions are not comparable.

Distribution Scoring And Shard Merge

fvmd is registered in SUITE_LEVEL_METRICS together with jedi, so it is never instantiated as a per-sample Metric object. For sharded evaluation this means:

• Shard workers: run_eval_shards.py calls run_benchmark with run_suite_level_metrics=False. Workers only collect prediction videos and never compute FVMD inside a shard.
• Merge: after per-sample scores from all shards are merged and the prediction set is rebuilt, FVMD runs once over the complete set for each suite.

This ordering is required. Computing a score per shard and averaging would give means and covariances that only describe each shard’s local distribution, which cannot represent the model’s full output distribution. Distribution metrics do not decompose across shards.

The corpus-side motion features are cached under benchmark.reference_corpus.cache_dir, keyed by a signature over the corpus file identities and the clip-sampling spec. The tracker therefore runs over the corpus once and every later model reuses it, which is what makes a corpus much larger than the prediction set affordable.

Inputs And Outputs

Item	Current implementation
Prediction input	every prediction video in the suite, up to 4 non-overlapping 2-second clips each
Reference input	the dynamic reference corpus, up to 4 clips per video, independent of the prediction count
Annotation input	none
Suites	image_dynamic, video_dynamic
Aggregation dimension	physics
Raw output	one FVMD scalar for the whole suite, lower is better
Normalized output	Reports retain compatibility linear normalized scores; the leaderboard remaps raw using the piecewise formula above, preserving the linear scale at 5 points and above and smoothly decaying below 5
Result location	<suite>.fvmd in distribution_metric_details.json, carrying raw, normalized, backend, details and groups; also summarised in distribution_group_breakdown.json
Details payload	prediction and reference clip-build reports (candidate_video_count, used_video_count, clip_count, skipped_video_count, skipped_videos_preview, source_fps_min/median/max), prediction_clip_spec, reference_clip_spec, prediction_signature, reference_signature, reference_cache_hit, reference_cache_dir, feature_dim, prediction_window_count, reference_window_count, singular_covariance_sides, raw_direction, repo_root, torch_home
Not-applicable mode	recorded as not_applicable when the suite has no prediction videos, no reference group, or fewer clips than metric_runtime.fvmd.min_prediction_clips / min_reference_clips (1025 in production)
Failure mode	the metric fails as a whole and is recorded in failure_summary when fewer than min_clip_count = 2 valid clips survive; it is never recorded as 0
Runtime requirements	GPU required; the PIPs++ weights pips2_weights.pth resolve through TORCH_HOME into ckpt/torch/hub/checkpoints/

Integration Readiness

Item	Verdict
Current aggregation	official for image_dynamic and video_dynamic whenever a numeric score is produced, contributing to the physics dimension
Publication verdict	Main-table usable, but it must be labelled as a distribution metric, and the window count on both sides must be reported

• It fills a real gap: among WorldAtlas’s motion metrics, only FVMD compares motion distributions directly. In the paper’s five-model human ranking comparison on the TikTok dataset, FVMD reaches a Pearson correlation of 0.8469 with human ratings, ahead of FVD at 0.6708, FID-VID at 0.3402, and the VBench average at 0.7573. Across the broader One Metric Equal and One Metric Diverse studies, FVMD is also the only metric that stays positively correlated with human scores in every setting.
• Sample size is the hard constraint on this score. The combined motion feature is 1024-dimensional and a Fréchet distance estimates a full covariance, so a side with fewer than 1024 windows has a singular covariance. The estimate still returns a number via the upstream eps regularisation, but it is heavily biased. details.singular_covariance_sides names any side in that regime. The dynamic corpus holds roughly 961 videos, which is why max_clips_per_video is 4 on both sides rather than 1. Always confirm prediction_window_count and reference_window_count are comparable before comparing magnitudes across models.
• Multiple clips per video trade independence for sample count. The paper cuts 16-frame segments with stride $s = 1$, that is maximum overlap, and its ablation confirms maximum overlap beats no overlap. WorldAtlas takes up to 4 non-overlapping windows per video, which is denser than one clip per video but still short of the paper’s default, so values from this repository cannot be compared directly against the numbers reported in the paper. Because the reference corpus is fixed, whatever bias this introduces on the reference side is a constant shared by every model.
• Read it against the real-versus-real floor. scripts/calibrate_distribution_metrics.py scores one half of the corpus against the other. Both halves are real video, so the result is the floor a perfect model could reach; a headline score should be read as a distance above that floor, not above zero. A 200-versus-200 probe returned raw 3954, but both sides had only 694 / 648 windows, so both covariances were singular and that number is not a floor. Splitting the 961-video dynamic corpus in half with up to 4 windows per video is what it takes to clear 1024 on each side. Production config raises min_prediction_clips / min_reference_clips to 1025, so a subset run is recorded as not_applicable instead of a biased score.
• It is genuinely sensitive to temporal noise. The paper injects local swap, global swap, interleaving, and switching noise, and FVMD increases monotonically with noise intensity, with the combined velocity-acceleration feature performing best.
• It is still not a motion correctness metric. Two batches can share similar velocity and acceleration distributions while differing entirely in motion direction, causality, or object identity. Report it alongside ground-truth-constrained metrics such as motion_accuracy and camera_error rather than using it alone to claim that motion is modelled correctly.

Reference

Fréchet Video Motion Distance: A Metric for Evaluating Motion Consistency in Videos

Jiahe Liu, Youran Qu, Qi Yan, Xiaohui Zeng, Lele Wang, and Renjie Liao.
First Workshop on Controllable Video Generation at ICML, 2024.
arXiv:2407.16124.
https://arxiv.org/abs/2407.16124

BibTeX

@inproceedings{liu2024fvmd,
    title = {Fr\'{e}chet Video Motion Distance: A Metric for Evaluating Motion Consistency in Videos},
    author = {Liu, Jiahe and Qu, Youran and Yan, Qi and Zeng, Xiaohui and Wang, Lele and Liao, Renjie},
    booktitle = {First Workshop on Controllable Video Generation at {ICML}},
    year = {2024},
    eprint = {2407.16124},
    archivePrefix = {arXiv},
    primaryClass = {cs.CV},
    url = {https://arxiv.org/abs/2407.16124},
}