Joshua Lee

Video models don't just generate. They reason. And now we know exactly how. New research from UC Berkeley, UC San Diego, CMU and SenseTime just overturned a core assumption about how diffusion-based video models think. The old theory: reasoning unfolds frame by frame across a video. The reality: it happens across the denoising steps instead. Early steps explore multiple candidate solutions in parallel. Later steps narrow down to a single answer. When the authors disrupted specific denoising steps, performance collapsed. Disrupting individual frames barely mattered. Three emergent behaviors also showed up:  1. Working memory: Holding onto information even when objects disappear from view  2. Self-correction: Catching and reversing wrong answers mid-process  3. Perception before action: First figure out what and where, then figure out how The practical payoff: a simple, training-free method that ensembles outputs from multiple runs at the right internal layers. No fine-tuning. It outperformed Runway Gen-4 and Kling 2.6 on the VBVR-Bench. Really grateful to Ruisi Wang, Zhongang Cai, and all co-authors for this genuinely illuminating work. Link to Paper: https://lnkd.in/gUwR378V #GenerativeAI #VideoAI #DiffusionModels #AIResearch #DeepLearning

18

📣 NEW VERTEX AI ENGINEERING BLOG: Implementing EAGLE-3 at scale on Vertex AI with SGLang Speculative decoding speeds up inference, but the standard pattern of hosting a separate draft model can be challenging to manage at scale. Today I am so excited to share that we just launched a NEW engineering blog to document and openly share our applied research at Vertex AI. Our first post in collaboration with SGLang details how we implemented EAGLE-3 at scale for all Vertex AI customers and what we learned. Check out the blog in the comments 👇 . We open-sourced the notebook if you want to reproduce benchmark results. _____ ♻️ For more insights and tutorials about Open Models, Ray, Agents and applied research, on Vertex AI, comment, like, and repost with your network. #LLM #Engineering #SGLang #Inference #Optimization

165

3 Comments

Coding benchmarks shape how we evaluate LLMs' coding performance. Some open-source models have been topping SWE-bench with ~80% scores. On paper, they look nearly tied with the strongest closed models. But when evaluated on SWE-rebench, a fresh, decontaminated benchmark built from recent GitHub issues, performance drops sharply. And the leaderboard reshuffles. The difference? Freshness. Once a benchmark is public long enough, models optimize around it. Scores go up. But that doesn’t always mean generalization improves. In my latest blog on To Data & Beyond, I break down: - Why does benchmark saturation happen? - What SWE-rebench changes? - Which models actually generalize best on new tasks? - And what does this mean if you’re building real coding agents? If you’re choosing models based solely on leaderboard scores, this is worth reading. You can read it from the link in the comments!

11

1 Comment

I want AI to work flat-out on curing cancer. The proof-of-concept on solving major mathematics problems is impressive. But there are more important challenges.

8

1 Comment

"Attention Matching" compacts the KV cache of LLMs by 50x with very minimal impact on the accuracy at at a fraction of the time of other compaction methods. It took me a while to figure out how it works, but it makes so much sense. To compact the context of the LLM: - You create a set of "reference queries" for the context. These are proxies for the kind of tasks the LLM is expected to do - You choose a subset of keys in the KV cache (e.g., highest attention values) to preserve - To choose the values (along with an accompanying bias term), you use the reference queries and the keys to compare and adjust your compacted KV cache with the full attention cache. This formulation makes it possible to compact the KV cache using simple and super-fast algebraic optmization tricks (e.g., OLS) as opposed to slow gradient-based learning methods. Result: 50x compaction, very high accuracy, very fast compaction (seconds vs hours for baseline methods that preserve accuracy). There are some caveats, including how you compose the reference queries and the key values, which enable you to adjust the speed-compaction-accuracy tradeoffs. https://lnkd.in/efPDhUfB

4

What happens to all the attempts an AI agent makes on the way to discovering a better solution? Dream-RSI keeps them as a world it can replay, asking what a different search would have found — no new experiments, no retraining. Glad to have been part of this work. Worth a read if you think about discovery loops and self-improvement in agents.

24