Ramesh Oswal

Reward models are the unsung heroes of RLHF — but can they scale at inference time like LLMs do? DeepSeek-GRM-27B says yes. The key insight: scalar reward models can't scale at inference because they always output the same number. Generative reward models can sample diverse "principles and critiques," then vote — giving you better rewards with more compute. Their method, Self-Principled Critique Tuning (SPCT), uses rule-based online RL to teach the model to adaptively generate evaluation principles per query. Sample 32 times, aggregate via voting (or a trained meta-RM), and accuracy keeps climbing. Result: a 27B model outperforms scalar RMs up to 671B parameters on comprehensive RM benchmarks. Inference-time scaling beats model-size scaling. This mirrors what DeepSeek-R1 showed for reasoning — compute at inference time can substitute for bigger models. Now the same trick works for reward modeling itself. Here is the research for you to dive deeper into the idea, if interested: https://lnkd.in/eyx-xkrH  

3

You don’t say AI doing what it wants to do ? Ohhhh …….😂 It’s simply because there’s no behavioral science security or true people first ethics in any models 🙄 with no human oversight anywhere for emergencies. Geez who would have seen that coming 🙄 yall are going to be the sacrifice on the way to regulation you do realize this right ?! The web3 economy model isn’t total automation. The premise is community building in small governances , state and locally led , and co creation with tech . Not repeating the same dumb bs for eternity What’s failed globally in web2 business & economy structure died in 2009 & the end of that sliding scale collapse was covid . You can’t just implement all of this with NO MINDSET SHIFT across the board the web3 education awareness gap is too grave . My paper on Behavioral science security as the missing link in AI : https://lnkd.in/ei92tkti #ai #web3 #community #government #blockchain #economy #futureofwork #automation #cybersecurity

1

You don't need a separate reference model for RL. We built an RL pipeline built on a multi-turn-focused modification of Self-Distillation Policy Optimization (SDPO) that uses the same model in two roles: student and teacher. The difference? Context window. → The student generates actions with no knowledge of the outcome. The teacher re-evaluates the same tokens with hindsight: the observation from each action, terminal test output, and a successful rollout from the same prompt. The per-token advantage is just the logit gap between these two views. → This gives you dense, per-token credit assignment instead of a single scalar reward for the whole trajectory. Think tokens get 0.25× weight (loosely guided, not tightly constrained). Action tokens get full gradient. System/user/tool result tokens get nothing. → The "teacher" is not a frozen checkpoint or a separate reward model. It's your current policy, conditioned on richer information. This is well-defined under MoE because same parameters with different routing is normal operating behavior. → We use a tiered reward system with hard gates only for truly unrecoverable failures: sandbox escapes, test corruption, harness crashes, timeouts, infinite loops. Everything else like wrong directory, failed commands, incorrect approaches are learnable signal, not wasted compute. The key insight: most of the information you need for credit assignment is already inside the model. You just need to ask it the right question with the right context. On LiveCodeBench v6 SDPO reached 48.8% vs GRPO's 41.2% beating Claude Sonnet 4 with an 8B model! That's bonkers.

5

Exciting panel at NVIDIA GTC today on what it takes to scale AI systems using next-generation G4 instances for simulation. We discussed everything from infrastructure design and simulation at scale to inference throughput and validation pipelines highlighting how these pieces come together to accelerate real-world AI development. At GM, this directly reflects how we build and operate ML infrastructure for autonomous vehicles, enabling large-scale training, simulation, and faster iteration cycles across the development lifecycle. Appreciate the thoughtful discussion and perspectives from fellow panelists Perry Nightingale and Shane Brauner, and thank you Jason Monden for moderating. #NVIDIAGTC #AIInfrastructure #MachineLearning #AutonomousVehicles #CloudComputing

199

2 Comments

Comparing models.. Finding that a small 1B #llm hallucinates less than a 4B for semantic to structural matching. Small llms for targeted tasks for the win. The new Meta llama3.2 1B is better than expected and cheap on token cost.

3

For the past year the edge AI conversation has mostly been about whether models could run locally at all. Could useful language models fit on devices? Would quantization destroy accuracy? Were mobile NPUs actually capable of meaningful inference? At this point many of those questions have answers. Models run locally. Small language models are surprisingly capable. Modern edge silicon can sustain real inference workloads. The question is shifting. Running a model on one device is easy. Running and operating AI across fleets of devices is something very different. Memory limits, runtimes, quantization strategies, model size, thermal constraints, and lifecycle management all start to shape what is actually practical. I wrote a new piece exploring that layer of the stack. It looks at model tiers that realistically fit on edge hardware, the runtimes that execute them, the role of quantization and distillation, and what it takes to manage models across real device fleets. If you are evaluating edge AI projects, these are the engineering decisions that start to determine whether a proof of concept becomes something operational. AI at the Edge: Models, Runtimes, and the Execution Stack https://lnkd.in/gTR_tiPs

9

1 Comment

Recently delivered a workshop titled “NLP: How AI Understands Us” , Co-Hosted by CPU ISIMM AI Committee and Club Robotech Polytech . We explored how language models parse meaning, context, and intent and what that means for building systems that actually understand human communication Loved the interaction and the questions that came up , always interesting to see the discussion move beyond buzzwords into the mechanics of how these models work

23

1 Comment

Physics-Informed Neural ODEs with Scale-Aware Residuals for Learning Stiff Biophysical Dynamics Kamalpreet Singh Kainth, Prathamesh Dinesh Joshi, Raj Abhijit Dandekar, Rajat Dandekar, Sreedat Panat https://lnkd.in/dtRD_ide This paper tackles a critical challenge in using Neural ODEs for modeling stiff biophysical systems, specifically the difficulty in achieving stable and accurate long-term predictions. The core idea is to improve training stability and accuracy by incorporating physics-informed regularization with scale-aware residual normalization within a Neural ODE framework. The authors propose "PI-NODE-SR," which combines two key elements: 1. Physics-Informed Regularization: They leverage the known governing equations (e.g., Hodgkin-Huxley) to define a physics-informed loss term that penalizes deviations from the expected behavior. This is a standard PINN approach, but the key is how they handle the residual. 2. Scale-Aware Residual Normalization: This is the novel contribution. Stiff systems often involve state variables evolving on vastly different timescales. Directly applying a physics-informed loss can lead to one timescale dominating the training process, hindering convergence and accuracy. To address this, they normalize the residual associated with each state variable by a scaling factor. This factor is chosen to balance the contributions of each variable to the overall loss, preventing the faster dynamics from overwhelming the slower ones. They use a low-order explicit solver (Heun method) to calculate the residual. The authors demonstrate the effectiveness of PI-NODE-SR on the Hodgkin-Huxley equations, showing that it can learn from a single oscillation and extrapolate accurately over longer time horizons. Importantly, they show that the method can recover morphological features in the gating variables that are typically only captured by higher-order solvers. ----- This work directly addresses a significant limitation of Neural ODEs in scientific applications: their struggle with stiff systems. By introducing scale-aware residual normalization, the authors provide a principled way to stabilize training and improve the accuracy of long-term predictions. This is particularly relevant for: - Biophysical modeling: Accurately simulating neuronal dynamics, cardiac electrophysiology, and other complex biological processes. - Chemical kinetics: Modeling reaction networks with widely varying reaction rates. - Multi-physics simulations: Situations where different physical processes evolve on different timescales. The ability to use lower-order solvers with neural correction to achieve accuracy comparable to higher-order methods has significant implications for computational efficiency. While the method's sensitivity to initialization is noted, the overall approach offers a promising direction for developing more robust and efficient Physics-Informed Neural ODEs for a wide range of scientific applications.

147

1 Comment

A Thought Exercise : Context Engineering with LLMs is analogous to Feature Engineering in Classical ML What are the similatieties and differences ? Share in the thread #AImusings courtesy: Jim Hong :)

28

9 Comments

Soumith Chintala, 𝗰𝗿𝗲𝗮𝘁𝗼𝗿 of PyTorch and one of the most respected builders in 𝗺𝗼𝗱𝗲𝗿𝗻 𝗔𝗜, has joined Mira Murati ’s fast-rising startup, Thinking Machines Lab. After 11 years at Meta (Facebook), where PyTorch grew from a 𝗻𝗶𝗰𝗵𝗲 𝗽𝗿𝗼𝗷𝗲𝗰𝘁 𝘁𝗼 𝗽𝗼𝘄𝗲𝗿𝗶𝗻𝗴 𝗴𝗹𝗼𝗯𝗮𝗹 𝗔𝗜 𝗶𝗻𝗻𝗼𝘃𝗮𝘁𝗶𝗼𝗻, Chintala’s move signals a new wave in human–𝗔𝗜 𝗰𝗼𝗹𝗹𝗮𝗯𝗼𝗿𝗮𝘁𝗶𝗼𝗻. Thinking Machines Lab isn’t just gathering top talent from Meta, OpenAI, Anthropic, and leading universities—they’re offering 𝗴𝗮𝗺𝗲-𝗰𝗵𝗮𝗻𝗴𝗶𝗻𝗴 𝘀𝗮𝗹𝗮𝗿𝗶𝗲𝘀 (up to $500K), 𝗮 𝗺𝗮𝘀𝘀𝗶𝘃𝗲 $𝟮 𝗯𝗶𝗹𝗹𝗶𝗼𝗻 𝘀𝗲𝗲𝗱 𝗿𝗼𝘂𝗻𝗱, and are aiming for a $50 billion market valuation. Their new product, Tinker, is already in use at Princeton, Stanford, and by major enterprises. Meta, meanwhile, is regrouping under the 𝗻𝗲𝘄 𝗦𝘂𝗽𝗲𝗿𝗶𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝗰𝗲 𝗟𝗮𝗯𝘀 division led by Alexandr Wang, with other AI leaders reportedly on the move. Chintala’s reasoning: time to pursue “𝙨𝙤𝙢𝙚��𝙝𝙞𝙣𝙜 𝙨𝙢𝙖𝙡𝙡, 𝙨𝙤𝙢𝙚𝙩𝙝𝙞𝙣𝙜 𝙣𝙚𝙬, 𝙨𝙤𝙢𝙚𝙩𝙝𝙞𝙣𝙜 𝙪𝙣𝙘𝙤𝙢𝙛𝙤𝙧𝙩𝙖𝙗𝙡𝙚.” His legacy—the PyTorch tool now thriving in classrooms from 𝗠𝗜𝗧 𝘁𝗼 𝗿𝘂𝗿𝗮𝗹 𝗜𝗻𝗱𝗶𝗮—𝗶𝘀 𝘀𝗲𝗰𝘂𝗿𝗲. #AI #TechNews #PyTorch #Meta #ThinkingMachines #Startup #AIJobs #TalentWar #Innovation #BigFunding

8