JEDi

jedi (JEPA Embedding Distance) measures how far the feature distribution of a set of predicted videos is from the distribution of a set of reference videos, and is positioned as a replacement for FVD. It follows Beyond FVD [Luo et al., 2025]: swap the I3D feature space for a V-JEPA feature space, and swap the Fréchet distance for a polynomial-kernel MMD.

WorldAtlas drives the official oooolga/JEDi implementation vendored at thirdparty/JEDi (package videojedi) through the jedi backend. Like fvmd, it is a suite-level distribution metric: the whole suite yields one score.

What It Measures

• whether the feature distribution of a set of predictions matches the reference set, covering per-frame quality, temporal consistency, and content diversity at once
• distribution-level collapse: if a model only produces one kind of appearance or one kind of motion, the distance grows even when individual videos look fine
• not per-sample quality: it cannot answer “what is the score for this one video”, which belongs to per-sample metrics such as perceptual_quality and image_quality
• not prompt following or geometric correctness

Principle

FVD’s reliability rests on two premises, and the Beyond FVD analysis shows that neither holds.

First, I3D features are not multivariate Gaussian. A Fréchet distance compares only the first two moments of a distribution, which is equivalent to comparing the distributions themselves only under a Gaussian assumption. The paper extracts 48,501 features across 11 video datasets and tests them with Mardia’s skewness, Mardia’s kurtosis, and the Henze-Zirkler normality test. The normality null hypothesis is strongly rejected ($p = 0$) on every dataset under every test. PCA and LDA visualizations show features clustering by dataset rather than forming a single Gaussian, and the deviation grows with clip duration.

Second, I3D features are insensitive to temporal distortion. This is the more damaging problem. When the paper adds slight per-frame blur to the test set, FVD drops from 69.0 to 68.1 — it judges the corrupted videos to be closer to the real distribution, reading a quality regression as an improvement. I3D is oversensitive to spatial corruption such as salt-and-pepper noise while remaining relatively blind to temporal corruption.

Third, the Fréchet distance has the worst sample efficiency. A covariance matrix requires estimating $k(k+1)/2$ parameters, with $k = 400$ for I3D and $k = 1280$ for V-JEPA, and the empirical covariance is singular outright when the sample count falls below the dimension. The paper measures that FVD needs 4,350 samples on UCF-101 and 4,700 on SSv2 to converge within a 5% error margin, while many common datasets simply do not contain that many unique videos. The usual workaround — cutting videos into partly overlapping clips — biases the metric because the data repeats.

JEDi replaces one component for each problem: MMD instead of the Fréchet distance, which needs no parametric distribution assumption at all; and V-JEPA features instead of I3D features, because they are more sensitive to temporal distortion and better aligned with human judgment. The measured payoff is reaching a steady value with 16% of the samples (700 versus 4,350 on UCF-101) and a 34% average increase in alignment with human evaluation.

Detailed Algorithm

1. WorldAtlas collects every prediction video in the suite. The reference side is the reference corpus (see Reference Corpus), not each sample’s own reference asset, so the two sides are independent sets of different sizes.
2. Each video is resampled to sample_fps = 8 and cut into 16-frame $224 \times 224$ clips, giving a fixed 2-second window (clip_frame_count = 16, analysis_size = 224). Predictions take 1 window; corpus videos take up to 2.
3. Before GPU encoding, a model-independent CPU preflight excludes reference videos with confirmed decode errors, native frame rates below sample_fps, or fewer than 16 sampled frames. The complete content-bound roster and exclusion reasons are saved in reference_selection.json and reused across models. ffprobe -count_frames is allowed at least 600 seconds and scales with file size up to 7200 seconds; a timeout is an operational failure, not a reason to mark that reference ineligible. Predictions are never filtered this way: prediction failures, or new failures among already selected references, still fail the evaluation rather than silently shrinking its population. The same OpenCV container off-by-one rule used for per-sample decoding applies here: when the container reports one extra frame, resampling uses the decoded count and the video stays in the population.
4. Reference clips are passed as train_loader and prediction clips as test_loader to JEDiMetric.
5. A V-JEPA ViT-Huge/16 encoder plus an SSv2 attentive probe extracts a 1280-dimensional feature per clip; batch_size = 4 and num_samples is the larger of the two clip counts.
6. A degree-2 polynomial-kernel MMD is computed between the two feature sets.
7. The result is multiplied by 100 to give the raw score.
8. WorldAtlas then applies bounded normalization with lower = 0.0, upper = 100.0, lower is better.

A lower raw score means the two batches of videos have more similar feature distributions.

num_samples deserves a note: upstream applies one cap to both sides, and feature_aggregator truncates each side to it. Taking the smaller of the two counts would therefore shrink the reference corpus back down to the number of predictions and undo the point of a fixed corpus, so the larger count is used and neither side is truncated.

Frame-Rate Alignment

Clips are cut on a wall-clock window rather than a raw frame count. Sampling a fixed number of consecutive frames would cover different durations depending on each video’s frame rate, so a generated 16 fps rollout and a 30 fps captured video would be compared over different amounts of elapsed time. sample_fps resamples both sides to a common rate first; upsampling is refused rather than performed, because duplicating frames fabricates motion that is not there. This matters most for fvmd, whose features are per-frame displacements, but it also keeps the V-JEPA temporal receptive field consistent here.

Feature Space

V-JEPA is a self-supervised paradigm that learns by predicting masked parts of a video in an abstract representation space rather than reconstructing pixels. This differs from VideoMAE’s masked-autoencoder objective and is why the paper finds it more sensitive to temporal distortion: the training objective itself requires the model to understand how a video evolves in representation space.

The current configuration uses the paper’s $\text{V-JEPA}_{\text{SSv2}}$ variant, that is the pre-trained encoder plus an attentive classification probe trained on Something-Something-v2, taking pre-logit features for distributional analysis. This matters: the paper’s ablation shows that $\text{V-JEPA}_{\text{PT}}$, the plain pre-trained variant, fails to capture incremental improvement in the training-duration experiment, while $\text{V-JEPA}_{\text{SSv2}}$ succeeds.

Component	Current configuration
Config file	thirdparty/JEDi/configs/vith16_ssv2_16x2x3.yaml
Encoder	V-JEPA ViT-Huge/16, ckpt/JEDi/vith16.pth.tar
Probe	SSv2 attentive probe, ckpt/JEDi/ssv2-probe.pth.tar
Feature dimension	1280
Clip spec	16 frames at $224 \times 224$, tubelet size 2, patch size 16

Polynomial-Kernel MMD

Maximum Mean Discrepancy maps two distributions into a reproducing kernel Hilbert space and compares their mean embeddings. It assumes nothing about distribution shape, which is exactly how it sidesteps FVD’s Gaussianity premise. JEDi uses the polynomial kernel:

Polynomial kernel $$k(x, y) = \left( \gamma \left\langle x, y \right\rangle + c_0 \right)^d, \qquad d = 2, \quad c_0 = 0, \quad \gamma = \frac{1}{1280}$$

$d$ and $c_0$ are set explicitly upstream. $\gamma$ is left unset and falls back to the scikit-learn default of $1 / n_{\text{features}}$, which is $1/1280$ for these features.

Given reference features $X = \{x_i\}_{i=1}^{m}$ and prediction features $Y = \{y_j\}_{j=1}^{n}$, upstream uses the biased estimator that includes diagonal terms:

Biased MMD estimator $$\widehat{\operatorname{MMD}}^2 (X, Y) = \frac{1}{m^2} \sum_{i=1}^{m} \sum_{i'=1}^{m} k(x_i, x_{i'}) + \frac{1}{n^2} \sum_{j=1}^{n} \sum_{j'=1}^{n} k(y_j, y_{j'}) - \frac{2}{mn} \sum_{i=1}^{m} \sum_{j=1}^{n} k(x_i, y_j)$$

The raw score is that value scaled by 100:

JEDi $$\operatorname{JEDi} = 100 \cdot \widehat{\operatorname{MMD}}^2 \left( \Phi(X_{\mathrm{ref}}), \; \Phi(Y_{\mathrm{gen}}) \right)$$

where $\Phi$ is the V-JEPA feature extractor. The factor of 100 carries no statistical meaning; the paper introduces it purely for readability, since raw MMD values sit around $10^{-2}$ (0.017 with no noise and 0.571 under high blur in the paper’s measurements). The normalization ceiling upper = 100.0 matches that scale.

Reference Corpus

The reference side is the collected video corpus configured by benchmark.reference_corpus and indexed by worldatlas build-reference-corpus. Two consequences follow.

Image-conditioned suites now get a distribution score. Previously the reference was each sample’s own asset, and image_static / image_dynamic have a still image there, so those suites were silently excluded. Only the prediction has to be a video now.

The reference distribution is no longer capped by the prediction count. Pairing gave the reference side exactly as many clips as there were predictions, which is far below the sample size an MMD estimate needs — the paper measures roughly 700 samples to converge on UCF-101. The corpus supplies thousands independently of how many samples a run evaluates.

Static predictions are scored twice: once over the whole static corpus (the headline number) and once per (style, environment) stratum, so a distribution shift can be attributed to a look or a scene domain. Stratifying on style matters at least as much as on environment, because the gap between photorealistic and stylized footage in V-JEPA feature space is large enough that a mismatched corpus mix would penalise a model for its style rather than its quality.

Strata are diagnostics, not headline numbers. A stratum with fewer clips than min_prediction_clips or min_reference_clips is reported as not_applicable rather than given an unstable score, and the headline is unaffected.

The corpus content signature is recorded in run_manifest.json under reference_corpus. Scores from different corpus versions are not comparable.

Feature Caching

Upstream JEDiMetric.load_features() checks train.npy and test.npy independently: each side is loaded from disk if present and computed otherwise. WorldAtlas builds on that to cache the corpus features across runs.

Reference features live in benchmark.reference_corpus.cache_dir, keyed by a signature over the corpus file identities (path, size, mtime_ns), the clip-sampling spec, and the checkpoint state of vith16.pth.tar, ssv2-probe.pth.tar and the config file. On a cache hit the file is linked into this run’s feature directory, so V-JEPA runs over the predictions only and the corpus videos are not even decoded. Prediction features are keyed separately, so two models never share them.

The signature is deliberately derived from file identity rather than decoded pixels. Hashing pixel bytes, as an earlier version did, would require decoding the entire corpus just to decide whether a cache hit was possible — several gigabytes of hashing before any work could be skipped.

Distribution Scoring And Shard Merge

jedi is registered in SUITE_LEVEL_METRICS together with fvmd and is never instantiated as a per-sample Metric object:

• Shard workers: workers only collect prediction videos and never compute JEDi inside a shard.
• Merge: after all shards are merged and the prediction set is rebuilt, JEDi runs once over the complete set for each suite.

All three sums in the MMD estimator range over the entire set, so computing per-shard values and averaging them is not mathematically equal to the MMD over the full set. Distribution metrics have to be computed after the merge.

Inputs And Outputs

Item	Current implementation
Prediction input	every prediction video in the suite, one 2-second clip each, passed as test_loader
Reference input	the matching reference corpus group, up to 2 clips per video, passed as train_loader, independent of the prediction count
Annotation input	none
Suites	image_static, image_dynamic, video_static, video_dynamic
Aggregation dimension	quality
Raw output	one scalar for the whole suite, equal to MMD times 100, lower is better
Normalized output	one scalar in [0, 1] with lower = 0.0, upper = 100.0, lower is better
Result location	<suite>.jedi in distribution_metric_details.json, carrying raw, normalized, backend, details and groups; also summarised in distribution_group_breakdown.json
Details payload	prediction and reference clip-build reports (candidate_video_count, used_video_count, clip_count, skipped_video_count, skipped_videos_preview, source_fps_min/median/max), prediction_clip_spec, reference_clip_spec, prediction_signature, reference_signature, reference_selection_sha256, reference_population, reference_cache_hit, reference_cache_dir, feature_dir, reference_feature_count, prediction_feature_count, model_dir, config_path, batch_size, num_samples, raw_direction, repo_root, torch_home
Not-applicable mode	recorded as not_applicable when the suite has no prediction videos, no reference group, or fewer clips than min_prediction_clips / min_reference_clips
Failure mode	the metric fails as a whole and is recorded in failure_summary when fewer than min_clip_count = 2 valid clips survive; it is never recorded as 0
Runtime requirements	GPU required; the V-JEPA wrapper initializes a gloo process group, so it takes the distributed path even on a single device

Integration Readiness

Item	Verdict
Current aggregation	official for image_static and image_dynamic whenever a numeric score is produced, contributing to the quality dimension. video_static / video_dynamic are the reference distribution only
Publication verdict	Main-table usable, the most sample-efficient distribution metric in this repository, but sample size and corpus version still need to be reported

• Among distribution-level quality metrics it is the most trustworthy one available. In the paper’s human study, distances in the V-JEPA feature space clearly outperform I3D and VideoMAE on both UCF-101 and Sky Scene. In the training-duration experiment only JEDi reaches a Spearman correlation of $-1$ with statistical significance, against $-0.6$ for FVD.
• Good sample efficiency is not sample independence. Under the paper’s protocol JEDi needs roughly 700 samples to converge on UCF-101 and 100 to 300 on T2VQA-DB. The corpus supplies enough on the reference side, but the prediction side is bounded by how many samples the run evaluated, and stratifying divides it further. Confirm prediction.clip_count is comparable before comparing models, and treat a stratum as diagnostic rather than headline.
• The role of the reference set needs stating. In the paper, train is a real-video distribution and test is the generated distribution under evaluation. This repository passes the reference corpus as train, so the score measures the distance from the prediction distribution to this corpus, not to a general real-video distribution. Report the reference_corpus.signature from run_manifest.json alongside the number.
• Read it against the real-versus-real floor. scripts/calibrate_distribution_metrics.py scores one half of the corpus against the other. Both halves are real video, so the result is the floor a perfect model could reach and a sanity check on the whole pipeline: a floor far above zero points at the clip sampling or the sample size, not at any model. On corpus signature 7e8b304026f0b043dc53 with 200 videos per half, the measured raw floors (already ×100) are: dynamic 0.096, static photorealistic indoor 0.037 / outdoor 0.078, static stylized indoor 0.062 / outdoor 0.062, and the static union 0.057. That 0.04–0.10 band is the sanity check passing; upper = 100.0 is still the paper-scale ceiling until model-side scores exist to tighten it. The full record is artifacts/benchmark/reports/distribution_calibration/calibration_summary.json.
• It divides labour with fvmd across dimensions: jedi sits in quality and covers overall distribution quality, while fvmd sits in physics and looks only at motion distribution. Their feature spaces and distance measures differ, so neither substitutes for the other.
• It is also not a per-sample diagnostic. A worse score says the distribution as a whole has shifted but does not point at which videos are broken; locating those still requires per-sample metrics and run details such as skipped_pairs_preview.

Reference

Beyond FVD: An Enhanced Metric for Evaluating Video Generation Distribution Quality

Ge Ya Luo, Gian Mario Favero, Zhi Hao Luo, Alexia Jolicoeur-Martineau, and Christopher Pal.
International Conference on Learning Representations (ICLR), 2025.
https://oooolga.github.io/JEDI.github.io/

BibTeX

@inproceedings{luo2025beyondfvd,
    title = {Beyond {FVD}: An Enhanced Metric for Evaluating Video Generation Distribution Quality},
    author = {Luo, Ge Ya and Favero, Gian Mario and Luo, Zhi Hao and Jolicoeur-Martineau, Alexia and Pal, Christopher},
    booktitle = {International Conference on Learning Representations ({ICLR})},
    year = {2025},
}