Skip to content

Keystroke Strict Action Accuracy

Measures exact agreement between recovered and commanded atomic action sets on observed native video-frame intervals. For prediction PiP_i, target GiG_i and interval duration wiw_i:

strict=∑iwi⋅1[Pi=Gi]∑iwi(1)\mathrm{strict} = \frac{\sum_i w_i \cdot \mathbf{1}[P_i = G_i]}{\sum_i w_i} \tag{1}

The decoder uses fixed per-second motion thresholds and actual video FPS; it never selects thresholds using test labels. F frames provide F−1 observable intervals. Missing pose intervals are excluded and reported through observation coverage, not interpolated into successful actions.

Default thresholds are move_speed_threshold=0.06 pose translation units/second and rotation_speed_threshold=6.0 degrees/second. All models use the same thresholds. These are not independently calibrated physical thresholds; ViPE translation units must not be interpreted as meters.

FPS comes from the video or an explicit runtime setting. Only observed consecutive native video-frame intervals are evaluated; malformed frame mappings or unknown FPS do not receive a score. A native action timeline takes precedence. When only ordered action tokens are available, uniform timing is an explicit assumption, not a verified execution schedule.

Higher is better. This is an official action-control metric. Correct labels do not establish correct motion amplitude. Commanded-only accuracy and idle false-action rate are separate details. Requires a usable action timeline and pose sidecar; backend: pose_action.

Audit manifest: docs/assets/metric_examples/longseq_memory_60s_examples_20260909.json.

Input image: memory-dolly_loop-memory_pool_408
Input image
60s memory_loop · same benchmark case · gap 0.9990

memory-dolly_loop-memory_pool_408

Suite: memory_loopScene: photorealistic / indoor / living_spaces
Benchmark prompt target

Move the camera forward through the scene, then back to where it started.

prompt source: memory_loop_prompt

Over this 60s loop, watch whether recovered per-frame actions match the requested keys exactly. Extra motion or a missed turn lowers strict accuracy.

Model result videos will be available with the ModelScope release. / 模型推理视频将在 ModelScope 方案上线后提供。

Higher metric scoreScore 0.9990 · official

Model result videos will be available with the ModelScope release. / 模型推理视频将在 ModelScope 方案上线后提供。

Lower metric scoreScore 0.0000 · official
Input image: memory-palindrome_loop-memory_pool_003
Input image
60s memory_loop · same benchmark case · gap 0.8933

memory-palindrome_loop-memory_pool_003

Suite: memory_loopScene: photorealistic / outdoor / terrestrial_landscape
Benchmark prompt target

Move forward, turn and continue, then retrace the whole path back.

prompt source: memory_loop_prompt

Over this 60s loop, watch whether recovered per-frame actions match the requested keys exactly. Extra motion or a missed turn lowers strict accuracy.

Model result videos will be available with the ModelScope release. / 模型推理视频将在 ModelScope 方案上线后提供。

Higher metric scoreScore 0.9890 · official

Model result videos will be available with the ModelScope release. / 模型推理视频将在 ModelScope 方案上线后提供。

Lower metric scoreScore 0.0957 · official
Input image: memory-yaw_loop_360-memory_pool_028
Input image
60s memory_loop · same benchmark case · gap 0.8297

memory-yaw_loop_360-memory_pool_028

Suite: memory_loopScene: photorealistic / outdoor / city
Benchmark prompt target

Rotate the camera a full turn in place and come back to the starting view.

prompt source: memory_loop_prompt

Over this 60s loop, watch whether recovered per-frame actions match the requested keys exactly. Extra motion or a missed turn lowers strict accuracy.

Model result videos will be available with the ModelScope release. / 模型推理视频将在 ModelScope 方案上线后提供。

Higher metric scoreScore 0.8297 · official

Model result videos will be available with the ModelScope release. / 模型推理视频将在 ModelScope 方案上线后提供。

Lower metric scoreScore 0.0000 · official
Input image: memory-yaw_loop_180-memory_pool_249
Input image
60s memory_loop · same benchmark case · gap 0.8242

memory-yaw_loop_180-memory_pool_249

Suite: memory_loopScene: photorealistic / outdoor / verdant_landscape
Benchmark prompt target

Turn the camera to the right, then turn back to the starting view.

prompt source: memory_loop_prompt

Over this 60s loop, watch whether recovered per-frame actions match the requested keys exactly. Extra motion or a missed turn lowers strict accuracy.

Model result videos will be available with the ModelScope release. / 模型推理视频将在 ModelScope 方案上线后提供。

Higher metric scoreScore 0.8242 · official

Model result videos will be available with the ModelScope release. / 模型推理视频将在 ModelScope 方案上线后提供。

Lower metric scoreScore 0.0000 · official