Vision-language-action (VLA) models and imitation-learning policies do not train on pixels alone. They train on when those pixels happened relative to proprioception, IMU, head pose, and action labels. A dataset that advertises “30 fps stereo” can still have a fabricated timebase. That is the difference between timestamps that were recorded on device and FPS backfill—assigning times after the fact from a nominal frame rate.
This essay explains the conceptual gap, the failure modes that appear when the timebase is fake, and how to verify timing with checks in the spirit of a verify_timing audit. All numbers below come from a public sample you can inspect yourself: two episodes, 4,914 frames, 163.47 s total (EP001 121.57 s; EP002 41.90 s), published as LeRobot v3 under CC BY-NC 4.0. The core claim of that sample—and the one buyers should insist on—is simple: timestamps are recorded, not computed.
Two ways a dataset can “have timestamps”
Recorded timestamps (device-side)
In a recorded-timebase pipeline, each frame carries a timestamp taken from the capture device’s clock at exposure (or a tightly coupled hardware clock). Stereo left/right are hardware-synced. IMU samples are stamped on the same continuous clock. Pose and video share one timeline because the sensors actually shared time, not because a post-process script assumed they did.
On the public sample that means:
- Dual-eye colour at 1920×1200 per eye, 30 fps, hardware-synced stereo
- Per-frame timestamps with 100% device-side coverage and no interpolation
- Frame intervals whose median = p99 = max ≈ 33.28 ms, with 0 frames out of specification
- IMU at approximately 300.48 Hz, 10 samples per exposure window, 18,050 continuous samples, 0 gaps greater than 10 ms
- Per-frame 6-DoF head pose on that same timeline
Nothing in that list requires inventing a time from “we aimed for 30 fps.” The interval statistics are measurements of what the device did.
FPS backfill (computed timebase)
FPS backfill is the opposite move. You have a sequence of frames (and maybe separately logged IMU or actions). You do not have trustworthy per-frame device timestamps—or you have partial coverage—so you fill time offline:
t_i = t_0 + i / fpsfor a nominal fps- Linear interpolation across missing stamps
- Soft alignment of stereo eyes or IMU by nearest-neighbour after recording
- Re-stamping after re-encoding, clipping, or concatenating episodes
The resulting column may still be named timestamp. Loaders may still plot a neat 33.3 ms grid. For VLA training, that column is not evidence of temporal truth; it is a hypothesis about timing. If the real capture dropped frames, drifted, paused, or soft-synced eyes, backfill hides those events behind a uniform clock.
Saying a dataset is “30 fps” therefore answers a different question than saying “timestamps are recorded.” Nominal rate is a capture target. Recorded timestamps are an audit trail.
Why VLA and imitation care about the difference
Modern VLA and behavior-cloning stacks condition action prediction on short visual histories, language, and often proprioceptive or IMU context. Training objectives implicitly assume that consecutive frames are separated by a known Δt, that stereo pairs share an exposure instant, and that IMU / pose samples fall inside the exposure window they claim to describe.
When those assumptions hold, temporal derivatives (optical flow proxies, head angular rate, action velocity) line up with the physics of the demonstration. When the timebase is backfilled, several failure modes appear—often without a loud training crash.
1. Phantom dynamics and wrong action timing
If dropped frames are backfilled as evenly spaced 33.3 ms steps, the policy sees motion that was faster in wall time than the labels suggest—or slower, if duplicate or stretched intervals were normalized away. Imitation then learns the wrong mapping from visual change to action magnitude. On contact-rich tasks (wiring, restocking, insertion), that shows up as early/late contact and brittle rollouts even when offline action MSE looks fine.
2. Stereo that is “synced” only in metadata
Hardware-synced stereo means left and right share an exposure trigger. Soft sync or post-hoc pairing can still ship under a single timestamp after backfill. Depth, triangulation, and any stereo feature used as VLA context then mix two slightly different moments. Residual metrics help catch this when calibration products ship with the data: the public sample reports baseline ≈ 60.7 mm, Sampson median 0.166 px, stereo reprojection 0.800 px, and 15,811 inliers, alongside calibration products K, KB4, R1/R2, P1/P2, Q, IMU↔camera extrinsics, and a calibration version. Geometry QA without a recorded timebase still cannot prove temporal coincidence of the pair—but together, hardware sync + recorded stamps + residuals form a coherent story FPS backfill cannot fake as easily.
3. IMU / pose fusion on a fictional clock
Egocentric demos couple head motion tightly to the image. The public sample’s IMU continuity (~300.48 Hz, 10 samples per exposure window, 18,050 continuous samples, 0 gaps >10 ms) only helps if those samples are on the same recorded clock as the frames. Backfilling video time while leaving IMU on a different log clock—or interpolating both onto a synthetic grid—creates false lag or lead. Policies that condition on 6-DoF head pose then learn spurious correlations: “when the head looks like this, the hand should do that,” with the wrong Δt between pose and pixels.
4. Silent train/eval skew under resampling
Teams often resample demos to a fixed training fps. With recorded timestamps, resampling is a deliberate, auditable transform of real intervals. With FPS backfill, resampling compounds fiction: you resample a uniform grid that never existed. Benchmarks that look stable in-distribution can still fail when deployed against live robot clocks that do not forgive invented timing.
5. Non-reproducible debugging
When a policy fails on a contact event, engineers ask: what did the human do in the 100 ms before contact? Recorded timestamps answer with device time. Backfilled stamps answer with “whatever the exporter assumed.” That difference decides whether you can trust episode surgery, segment labels, and multi-rate fusion in the next data iteration.
None of these modes require malware or malice. They are normal side effects of treating a rate label as a timebase.
How to verify: verify_timing-style checks on public numbers
You do not need to take a vendor’s word for “synced 30 fps.” Run timing verification against measurable distributions. The public sample is sized so those checks are concrete.
Frame-interval audit
- Load every frame timestamp for EP001 and EP002 (4,914 frames; 163.47 s total).
- Compute successive intervals Δt.
- Report median, p99, and max. On this sample they are identical at ≈ 33.28 ms, with 0 frames out of specification.
- Reject batches where median looks like 33.3 ms but p99/max blow out—classic signature of a nominal-rate story with real jitter, or of repaired drops.
Identical median/p99/max is strong evidence the device actually held the period; FPS backfill can force a pretty median while hiding tails unless you inspect them.
Coverage and provenance
- Require 100% of frames to carry a device-side timestamp.
- Forbid interpolation as a delivery method for the primary time column.
- Confirm stereo is hardware-synced, not only “aligned in post.”
If a supplier cannot state “recorded, not computed,” treat the time column as backfill until proven otherwise.
IMU continuity against the same clock
- Measure IMU rate (~300.48 Hz on the sample).
- Count samples per exposure window (10).
- Scan for gaps: the sample shows 18,050 continuous samples and 0 gaps >10 ms.
- Confirm 6-DoF head pose is delivered per frame on that timeline.
Gaps healed by interpolation are still gaps for fusion physics; a hard “0 gaps >10 ms” gate keeps the timebase honest.
Schema load as a gate, not a vibe
Deliver as LeRobot v3, loadable via LeRobotDataset(), and pass Physical-Intelligence/pi-data-sharing checks. Timing columns that only exist in a custom parquet branch are easy to “fix” in a one-off export; schema gates make the recorded-time fields part of the accepted artifact.
Geometry package alongside timing
Ship K, KB4, R1/R2, P1/P2, Q, IMU↔cam extrinsics, and a calibration version. Timing proves when; calibration proves where. VLA stacks that lift stereo or fuse IMU need both. The sample’s Sampson 0.166 px, reprojection 0.800 px, and 15,811 inliers give numeric QA companions to the timing gates—not substitutes for them.
A minimal verify_timing-style script is then: load LeRobot v3 → interval histogram → assert median/p99/max and out-of-spec count → assert timestamp coverage and no-interpolation flag → IMU gap scan → optional stereo residual spot-check. Pass/fail beats brochure fps.
What to put in a SOW so you do not buy backfill by accident
Paste-ready language that mirrors the public sample:
- Dual-eye RGB 1920×1200/eye, 30 fps, hardware-synced stereo.
- Frame intervals: median = p99 = max ≈ 33.28 ms; 0 out of spec.
- Timestamps: 100% device-side, no interpolation—recorded, not computed.
- IMU ≈ 300.48 Hz, 10 samples per exposure window, continuous stream with 0 gaps >10 ms, plus per-frame 6-DoF head pose.
- Calibration products K / KB4 / R1/R2 / P1/P2 / Q / IMU↔cam with version; geometric residuals at or better than the public reference scale where applicable.
- Delivery as LeRobot v3 via
LeRobotDataset(), passing pi-data-sharing checks.
These clauses do not invent a second meaning of “30 fps.” They separate rate target from timebase integrity.
Soft next step
If your VLA or imitation pipeline assumes a real Δt between frames, treat FPS backfill as a different product category from device-recorded timestamps. Inspect the public sample yourself: EP001 and EP002, 4,914 frames, 163.47 s, LeRobot v3, CC BY-NC 4.0, on Hugging Face at tatezhou/ego-stereo-cn-v1. Product and sample context: https://10khoursdata.com. Questions on timing gates or SOW language: tate@10khoursdata.com.
The short version for procurement and research leads alike: nominal fps is a capture setting; recorded timestamps are evidence. For VLA training, only the second one is a timebase.
10K HoursData — egocentric stereo human-demonstration data with timestamps recorded, not computed.