Richard Liaw

🔍 High-quality Data: The Hidden Compute Tax, How Do We Scale Past It? 🤖👇 I have recently been exploring why high quality, human-labeled data remains critical (and costly) for building truly intelligent multimodal AI, especially in the wake of Meta's substantial investment in ScaleAI, highlighting just how expensive and critical data can be. Consider this: ScaleAI alone earns over $500M per year, providing preference data and traces. Even a single run of 1M human comparison labels can cost upwards of $6–10M. Smaller multimodal efforts, like LLaVA-RLHF, required about 10k human comparisons (~$3k) just to curb hallucinations. Clearly, the hunger for accurate, preference-style labels grows rapidly as AI expands beyond text into vision, robotics, and other modalities. Enter AI-generated feedback (RLAIF), a powerful and scalable solution for multimodal AI: - > Fact-RLHF [1] : cleverly augments reward models with factual context (captions, ground-truth), drastically cutting hallucinations in multimodal chats with minimal targeted human effort. - > RL-VLM-F [2]: takes this even further, leveraging GPT-4V as automated judges. Robots learn complex manipulation tasks purely from AI-generated feedback, no humans required and scales to pixels. - > Self-Rewarding LMs [3] : push this idea by having models iteratively improve themselves with AI-generated feedback alone, achieving superior results compared to GPT-4 benchmarks. Why does this shift matter right now? ✅ Cost Curve Inversion: Once AI models reliably grade data, labeling costs drop, trending towards compute rather than human labor. ✅ Modality Leap-Frog: Vision-language AI judges already handle images effectively, next up are audio, 3-D, and video. ✅ Safety & Iteration: Continuous AI-driven critique quickly identifies and corrects issues like reward hacking, outperforming periodic human reviews. 🔑 Key takeaway: The future of AI alignment isn't just bigger models, it's a sophisticated data and alignment "stack," where humans set high-level goals, AI provides detailed, scalable feedback across modalities, and models improve iteratively. Post is inspired by listening to Karina Nguyen's (from OpenAI) talk at the ICLAD conference held at Stanford last week!

49

1 Comment

Meet BeTaL: an LLM-in-the-loop framework that automates benchmark design. LLMs reason, simulate, and refine to create dynamic benchmarks that evolve as fast as the models they test—2–4× more precise than traditional methods. 📄 Research: https://lnkd.in/gRqAYtd9

62

1 Comment

🤔 Have you wondered which model size to fine-tune for your Agentic RL effort? Our recent paper (https://lnkd.in/gfFE8aUS) suggests that with the right reward shaping, smaller models is the smarter choice. RL fine-tuned Qwen3-8B matched its 32B counterpart while using 3.5× less compute, 1.5× less memory, and lower variance on TravelPlanner. It reached 56.9% final-pass with only 180 training queries — 2.7× stronger than GPT-5. These models also generalized well across out-of-domain tasks including Multi-IF, NaturalPlan, and Tau-Bench.

217

1 Comment

Excited to share my review of a fantastic new paper that could reshape how we build foundation models! 🧠 For years, AI has relied on a handcrafted bottleneck: fixed tokenization. The paper "Dynamic Chunking for End-to-End Hierarchical Sequence Modeling" by Sukjun Hwang, Brandon Wang, and Albert Gu introduces H-Net, a novel architecture that learns to segment raw data on the fly. Instead of being told how to break down text, code, or even DNA, H-Net discovers meaningful "chunks" on its own, marking a huge step towards truly end-to-end AI and a powerful validation of the famous "bitter lesson." It seems to be the next step compared to BLT (Byte Latent Transformer), that was a very interesting model but not truly end-to-end. For the technical folks, the magic is in "Dynamic Chunking," which uses a clever EMA-based smoothing module to make the discrete chunking process fully differentiable. But the results speak loudest for everyone. H-Net not only outperforms traditional tokenized Transformers but also scales more efficiently, at one point achieving superior results with just half the parameters. 🚀 This breakthrough delivers dramatic improvements in robustness and excels on data where fixed tokenizers struggle, like code, DNA sequences, and various languages. This work provides a powerful, generalizable blueprint for the next generation of foundation models that learn more, with less human preprocessing. And it includes Mamba 2! Full review here: https://lnkd.in/eW4Dw-ga Paper: https://lnkd.in/eZwPxXUr Code: https://lnkd.in/eqPTEmUt #AI #FoundationModels #DeepLearning #MachineLearning #LLM #Tokenization #HNet #Innovation

157

4 Comments

I just read “Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs”, which extends Chinchilla-style laws to include architecture choices like hidden size, MLP/attention balance, and GQA, then uses them to find designs that are both cheaper and more accurate than LLaMA-3.2 under the same training budget. 🔗 https://lnkd.in/gYnqdc6R Insights from the paper: 🔹 Conditional scaling law: Augments classic compute-optimal scaling with architectural variables (hidden size, mlp-to-attention ratio, GQA groups) so loss becomes a function of both scale and shape, not just total params/tokens. 🔹 Architecture–efficiency tradeoffs: Training 200+ models (80M–3B, 8B–100B tokens) shows that shifting parameters from MLP to attention, and using well-tuned GQA, can significantly improve accuracy at the same FLOPs and memory footprint. 🔹 Search framework: Fits the conditional law, then searches over design choices to predict Pareto-optimal points, yielding models with up to 2.1% higher accuracy and 42% higher inference throughput than LLaMA-3.2 for the same training compute. 🔹 Practical takeaway: Instead of “Chinchilla but bigger,” the work argues for architecture-aware scaling as a new axis—optimizing how parameters are wired can matter as much as how many you have. This is a good resource if you’re designing custom LLMs, worrying about serving costs, or exploring how scaling laws and architecture search can be combined for inference-efficient models. #AI #MachineLearning #ScalingLaws #LLM #ModelArchitecture #InferenceEfficiency

3

GEPA over RL. "GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning" link: https://lnkd.in/gJfNJQZc "At the heart of this paper is a singular, transformative claim that challenges the RL paradigm for agent optimization. "The optimal set of instructions governing a compound Al system is discovered more efficiently through an evolutionary process of reflective, language-based self-correction than through numerical policy optimization guided by scalar rewards" #ai

4

Another interesting new RL perspective. They propose Reinforced Attention Learning (RAL) to address the difficulty of applying RL techniques that work well for LLMs but tend to struggle more with multimodal LLMs, although I think the idea is more general than that. They argue that this approach is more robust to reward hacking and less prone to diversity collapse. In practice, they add to the standard RL objective an additional term that looks like: E[ A_t · D(p_t^θ || p_t^old) ] Here, A_t is the advantage, while p_t is the attention distribution policy at generation step t, defined from the final attention layer (averaged across heads): the current one (θ) and the one that generated the batch (old). p_t^θ(i) is simply the attention weight from the generated token at position t to a previous token at position i < t, normalized by the sum of attention weights before t. D is the Jensen–Shannon divergence (JSD), a symmetric and bounded divergence defined via the mixture of the two distributions, and less aggressive than Kullback-Leibler divergence (KL). JSD(P || Q) = 1/2 · KL(P || M) + 1/2 · KL(Q || M) With M = 1/2 · (P + Q). What is happening here is that we incentivize attention patterns associated with high advantage and discourage those associated with low advantage. As the authors put it, when A_t < 0 the model is explicitly pushed to explore new reasoning paths. The full objective becomes: L_RL + λ · L_AttnRL. They then introduce an extension that combines RAL with on-policy knowledge distillation and attention distillation. The idea is similar to on-policy distillation: you generate with the student (the current network), compute the teacher outputs on the same generation, and align them using a reverse KL on the logits. In addition, they also align the attention distributions between teacher and student, again using JSD. Unlike RAL, this attention distillation term does not use an advantage, it is pure structural alignment. Takeaway: RL and on-policy distillation do not have to operate only at the token level. Extending the training signal to internal structures like attention can be quite effective, and I expect this direction to be explored much more in the future. Thanks Petru-Daniel (Dan) Tudosiu for suggesting the paper!

85

1 Comment

Excited to share a new blog post on our "Scalable Framework for Evaluating Health LLMs" 🚀 (https://lnkd.in/gzhuvTfN) Daniel McDuff and I summarize how this novel framework addresses key challenges in evaluating AI models to health queries, especially when user query includes data from wearables, blood biomarkers, and/or user-provided historical health data. This framework is designed to: (1) Significantly reduce evaluation time. (2) Improve inter-rater reliability. (3) Precisely identify quality gaps in LLM responses. A huge thanks to our incredible team, and a special shout-out to my former intern, Neil Mallinar, who was the main driving force behind this project. Great contributions from: Ali Heydari, PhD, Xin Liu, Tony Faranesh, Brent Winslow, Nova H., Benjamin Graef, Cathy Speed, Mark Malhotra, Shwetak Patel, and Xavi Prieto. #AI #LLM #evaluation #health #wearables #biomarkers

172

3 Comments