How many of these Jev demos are false advertising?
The Minecraft post says the movement is “only possible” with Jev’s near-instant decisions. An Ender Dragon kill in 8:43, under $1. Sounds impressive. (Image 1)
Then you look at Feitian Shanke’s breakdown: 129 of 165 Jev calls had exactly one available action. That’s 78.2% of the calls where the code had already made the choice.
He also shows preset routes and action templates, then reports successful runs after replacing Jev with random choices and removing the planner too. His modified setup used Kimi K3 instead of Astra. The remaining 36 calls could still be crucial. You’d need repeated runs with and without Jev to find out.
The other game clips leave me with the same question. Minecraft combat (Image 2), Mario (Image 3), 50 Subway Surfers games at once (Image 4), four Smash characters fighting each other (Image 5). How much worse would each of these actually get if you replaced Jev with a simple policy? Fast responses and a tiny token bill don’t answer that.
And “I rebuilt Tesla Full Self Driving” (Image 6) is a simulator where code generates the candidate paths and handles safety braking. Jev picks a path. A lot of the driving work is already happening outside the model.
It’s easy to give Jev credit for everything happening on screen when its actual job is much smaller. I’d ask each creator to rerun the same setup with Jev replaced and show how the results change.
- How many of these Jev demos are false advertising? The Minecraft post says the movement is “only possible” with Jev’s near-instant decisions. An Ender Dragon kill in 8:43, under $1. Sounds impressive. (Image 1) Then you look at Feitian Shanke’s breakdown: 129 of 165 Jev calls had exactly one available action. That’s 78.2% of the calls where the code had already made the choice. He also shows preset routes and action templates, then reports successful runs after replacing Jev with random choices and removing the planner too. His modified setup used Kimi K3 instead of Astra. The remaining 36 calls could still be crucial. You’d need repeated runs with and without Jev to find out. The other game clips leave me with the same question. Minecraft combat (Image 2), Mario (Image 3), 50 Subway Surfers games at once (Image 4), four Smash characters fighting each other (Image 5). How much worse would each of these actually get if you replaced Jev with a simple policy? Fast responses and a tiny token bill don’t answer that. And “I rebuilt Tesla Full Self Driving” (Image 6) is a simulator where code generates the candidate paths and handles safety braking. Jev picks a path. A lot of the driving work is already happening outside the model. It’s easy to give Jev credit for everything happening on screen when its actual job is much smaller. I’d ask each creator to rerun the same setup with Jev replaced and show how the results change.
The key question is the counterfactual: how much does the demo actually depend on Jev? Controlled A/B runs would make that much clearer.