VideoDB

A language model can tell you what usually comes next. A world model should tell you what happens if you act. That is the core of JEPA (Joint-Embedding Predictive Architecture), Yann LeCun's bet on how AI systems should learn. Our essay follows the idea from language models to vision-language-action models and world models for robots. Next-token prediction took us far, but its loss only checks the next visible token. A model can land on the right word while its hidden state drifts away from a coherent future. JEPA changes what gets predicted. It encodes a context view and predicts the embedding of a target view, such as a masked region or a future state, with no pixel reconstruction and no token decoding. Add actions, and the model can predict the consequence of an action before taking it. The early results are already sharp. In one hierarchical world-model paper the essay discusses, a flat planner (VJEPA2-AC) gets 0% success on a Franka pick-and-place task, while the hierarchical version reaches 70% on the cup task. The essay is careful to say these numbers hold under the evaluated setup. It also covers the main risk: everything happens in latent space, and a latent space can collapse or warp while the math still looks elegant. The stack the essay expects keeps language as the interface for instructions and explanations, with a world model holding and predicting state underneath. The full essay is in the first comment. We would love to hear how your team is thinking about world models.

How much of video search comes down to the model, and how much to the system around it? Our team answered that in a technical report, Search over the Visual World (arXiv 2608.08075), with the ideas built and tested in VideoDB. The starting point: video keeps arriving from archives, recordings and live streams, and an answer only helps if someone can press play on it and check it. So the report treats search as a set of system decisions, starting with the analyzer: a model run over the video with its own settings: how it cuts the timeline into scenes, what it is asked, and the shape of its answer. VideoDB keeps analyzer outputs aligned on source time, so a new index can be built without running the model again. Every result stays playable at the original source. Then we tested it. Our pipeline uses general-purpose components, none trained end-to-end for video retrieval. We ran it against a commercial video-native retrieval engine, with both systems indexing the same videos and receiving the same queries: +7.3 points on top-1 accuracy (R@1) across 9,800+ queries on four open datasets It also leads at R@3 and R@10. The practical point for anyone building on video: today, video search is a systems problem. Swapping an analyzer or an index changes what you can find, and a video-native model can join as one more analyzer. The full paper is in the first comment. We would love to hear how you search your own video.

Can a VLM annotate a robot run well enough to train on? We wanted a real answer, so we took WGO-Bench, MacroData's public benchmark of 100 human and robot manipulation videos, and ran our annotation pipeline against a public baseline. Same model on both sides, Gemini 3.7 Flash. Here is where we landed: VideoDB: 28.93% semantic F1 Public baseline: 19.92% Before scoring anything, we wanted a reference we could trust. So our team went through all 100 videos by hand and rebuilt the labels around one rule: one completed manipulation per segment, so a reach stays with the grasp it leads to. That took the reference from 743 segments to 534, with changes in 90 of the videos. Then we scored both pipelines against it. A segment only counted if its timing matched (IoU 0.5) and a judge accepted its label. Most of our lead came from the labels themselves: 149 accepted, compared with 100 for the baseline. This matters because robot policies such as π0.5 learn from labels like these. A clean boundary and the right object name give them a better training signal. We are now taking this further: tightening segment timing on HomER's egocentric video, sharpening labels on DROID, and flagging the annotations that deserve a human review. The full benchmark is in the first comment. We would love to hear how you are annotating your own robot data.

Most robotics labs keep more video than anyone can look at. Teleoperation sessions, test runs on the robot and rollouts from a world model are all recorded, and they all go into a bucket. When a researcher wants to know how often the gripper slips on one part, someone writes a pipeline. The usual approach cuts the archive into clips up front, before anyone knows the next question. Each new question means a new pipeline and the whole archive processed again, so compute is spent per experiment rather than per hour of video. Test runs are worse: they land in a separate bucket and never join the training data. We built a query engine that keeps every episode whole. Any model can run as an analyzer over the archive and write down what happened and when. The index is versioned, so a better model means re-reading the archive instead of re-collecting it. A question in plain English decides the window: a slip needs about five seconds of context, a regrasp ninety, a whole shift twenty minutes. Six question types cover most of what researchers ask: find, count, filter, sample, look closer, watch. The hits from one question become a cohort, the cohort becomes a training manifest, and the next model's runs land in the same archive, searchable the moment they arrive. Full write-up in the first comment.

Multimodal models have a problem that has nothing to do with perception. A model can identify every person in a crowded photograph and still count the group wrong. Not because it missed anyone - because it lost track of who it already counted mid-reasoning. DeepSeek's recent paper calls this the Reference Gap. Their proposed solution is direct: let the model emit bounding boxes and point sequences while it reasons, interleaved with language. A box becomes working memory for an object. A point sequence becomes working memory for a path. The idea matters more than any individual benchmark score. Multimodal reasoning needs a representation for spatial state that survives a long reasoning trace. Language can describe spatial state, but descriptions become ambiguous in crowded scenes, overlapping layouts, and multi-step deductions. Our team wrote a detailed breakdown - covering the math, the five-stage training pipeline, the reward design for counting and maze navigation, and what the paper establishes versus what it leaves open. Runnable Python is included throughout. If you work with vision-language models, this is worth reading carefully. Link in comments.