Avani Bajaj

If your goal is to truly understand reinforcement learning, here’s a resource you shouldn’t ignore. The "Stanford CS224R Deep Reinforcement Learning" playlist is a complete Stanford University lecture series taught by Prof. Chelsea Finn, covering modern deep reinforcement learning through practical, research-oriented approaches. It is designed to help you understand how intelligent agents learn to make decisions from experience; something that becomes increasingly important as AI moves toward robotics, autonomous systems, and agentic AI. Here’s what you’ll gain from it: 1. A deep understanding of core RL concepts such as: a) Imitation learning and learning from demonstrations b) Policy gradients and actor-critic methods c) Q-learning and value-based methods d) Offline reinforcement learning e) Model-based reinforcement learning f) Reward learning and goal-conditioned RL g) Multi-task learning, meta-RL, and exploration h) Hierarchical reinforcement learning and skill discovery i) Reinforcement learning for LLMs and reasoning 2. Strong intuition about: a) How agents learn effective behavior from experience b) Exploration vs. exploitation c) Sparse rewards and long-horizon decision-making d) Learning from offline datasets e) Generalizing across tasks and environments 3. Exposure to real-world applications in: a) Robotics b) Autonomous systems c) Visual navigation and control d) Large language models and reasoning Here’s how to use this resource effectively: Step 1: Follow the lectures sequentially. Build your understanding from the foundations before jumping into advanced topics. Step 2: Don’t just watch. Write down the key equations, algorithms, and intuitions from each lecture. Step 3: Implement simplified versions of the algorithms you learn using Python and PyTorch. Step 4: Experiment with environments such as Gymnasium or MuJoCo to see how RL algorithms behave in practice. Step 5: Connect each concept to a real application. Ask yourself: What decision-making problem could this solve? Step 6: Revisit the lectures on topics such as offline RL, multi-task learning, and RL for LLMs when working on related projects. Access the playlist here: https://lnkd.in/en3cx6pf Which reinforcement learning topic would you most like to understand? ♻️ Repost to help this reach more aspiring data scientists. PS: I regularly share free data science courses, books, tutorials, and learning resources on my WhatsApp channel. Join here: https://lnkd.in/eJzmRZkQ

35

2 Comments

We used to hide behind binders. Then slide decks. Then dashboards. Now we hide behind chat boxes powered by Agents. Same instinct. Different cognitive bottlenecks. Every enterprise tech shift has been driven by one ruthless obsession: identify the cognitive bottleneck. Eliminate it. PPTs → The bottleneck was manual curation. Dashboards eliminated it. Dashboards → The bottleneck was analytical fatigue — drowning in filters, never finding the root cause. Agents eliminated it. Agents → The bottleneck is prompt-babysitting. Centralized pipelines choking on complexity. Glorified summarizers dressed as intelligence. The next eliminator? The Stigmergic Swarm intelligence in Agents. Not another chatbot. A self-organizing digital ecology (with bidding topologies) that mimics your brain's synaptic plasticity — successful execution paths get reinforced like neural memory, agents leave digital "pheromone trails," and the swarm autonomously picks the optimal route next time. Zero human orchestration. Zero chat box. The catch? Eliminating the human cognitive bottleneck introduces a machine one — rogue feedback loops, state collisions, agents silently colluding to game their own token economies. And as the bottleneck shifts from human fatigue to system behavior, roles aren't disappearing — they're being pruned and forced up the stack: Software Engineers → Forensic Systems Engineers (swarm collisions, runtime memory leakage) Prompters → Context Architects (hyper-dense payloads, strict schema boundaries) Data Analysts → Retrieval Engineers (Graph-Vector networks, real-time context streaming) Functional & Data Experts → Intent Architects (objective functions and guardrails — not approvals) Data Scientist → Outcome champions The uncomfortable truth: If your value is defined by executing a process or clicking approve — you are the cognitive bottleneck. The stack has always eliminated those. It won't stop now. Get ready for a big paradigm shift ! #AI #Agents #NewSkills #NewWayOfWorking

8

5 Comments

There are two common RL training formulations for LLMs: Markov Decision Process (MDP) formulation and bandit formulation. Here’s how they work… Background: We should recall that an LLM generates output via next token prediction; i.e., by sequentially generating each output token. As shown in the image, the next token prediction process maps very easily to an RL setup—we can just model each token as an individual action! MDP formulation: The approach of modeling each token in the LLM’s output as an individual action is called the Markov Decision Process (MDP) formulation. An MDP is simply a probabilistic framework for modeling decision-making that includes states, actions, transition probabilities and rewards—this is exactly the setup used by LLMs for RL! Our initial state is the prompt and our policy acts by predicting tokens. During generation, actions are taken by selecting a token from the LLM’s predicted token distribution—each token is its own action. After a token is predicted, it is added to the current state and used by the LLM to predict the next token—this is just autoregressive next token prediction! Eventually, the LLM predicts a stop token (e.g., <|end_of_text|> or <eos>) to complete the generation process and yield a full trajectory. Outcome vs process reward: In the MDP formulation, we depict a reward being provided to our policy at every step. But, the reward mechanism for an LLM is usually a bit different. Most LLMs are trained using outcome supervision, meaning that a reward is only assigned after the model has generated a complete response (i.e., after the <eos>). Bandit formulation: In an outcome supervision setting, we may begin to question the utility of modeling each token as its own action. How do we handle credit assignment in this scenario? Alternatively, we could model the entire response as a single action that receives an outcome reward. This is the key idea behind the bandit formulation for RL training. This name comes from the idea of a contextual bandit in probability theory. The bandit setup is simple: our agent chooses an action, receives a reward and the episode ends. Our complete trajectory is a single action and reward! For LLMs, our action is the full completion generated for a prompt, which receives an outcome reward. Which formulation is better? In the context of LLMs, we can easily compute the probability of individual tokens or a full completion to a prompt, so we have the ability to model RL using either an MDP or bandit formulation. Given that LLMs usually only receive outcome rewards, however, the bandit formulation—despite being very simple—is quite fitting for LLMs. Plus, it tends to be more simple (and has been argued to perform similarly despite this simplicity). However, MDP formulations are also commonly used. For example, PPO uses an MDP formulation, whereas REINFORCE and RLOO adopt a bandit formulation. In other words, both RL formulations are viable and are used for training LLMs.

181

6 Comments

Wrote an article on Byte Pair Encoding (subword Tokenization) and how it works which was potentially used in the the earlier versions of GPT / traditional Language Learning model paradigms. Took a sample of Hindi Language corpus text data to showcase the same. Article below - https://lnkd.in/gCu7Bgpf

33

Why Metadata-First Retrieval Reduces Hallucinations by 60% "Most RAG systems don't hallucinate because of bad models. They hallucinate because of bad *retrieval*." Here's the hard truth: Vector similarity alone is a lottery. You're asking your LLM to answer from *every document* in your knowledge base, in any language, from any era, without context on what's relevant. That's why we switched to metadata-first retrieval, and it cut hallucinations dramatically. THE SHIFT Traditional RAG: User Query → Embed → Vector Search (all chunks) → Top-K → Generate → Hallucination risk ⬆️ Metadata-First RAG: User Query → Metadata Filters (source, time, status, geography, compliance) → Vector Search (filtered subset) → Generate → Accuracy ⬆️ WHY THIS WORKS 𝟭. 𝗖𝘂𝘁 𝘀𝗲𝗺𝗮𝗻𝘁𝗶𝗰 𝗻𝗼𝗶𝘀𝗲 𝗯𝗲𝗳𝗼𝗿𝗲 𝗶𝘁 𝗿𝗲𝗮𝗰𝗵𝗲𝘀 𝘁𝗵𝗲 𝗺𝗼𝗱𝗲𝗹 Vector search pulls similar-*looking* chunks that are actually wrong: - Old policy that looks semantically close to current policy - Data from wrong region/customer/product - Outdated compliance documents Metadata filters stop these near-misses *before* retrieval. 𝟮. 𝗚𝗿𝗼𝘂𝗻𝗱 𝗮𝗻𝘀𝘄𝗲𝗿𝘀 𝗶𝗻 𝘃𝗲𝗿𝗶𝗳𝗶𝗲𝗱, 𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲𝗱 𝗱𝗮𝘁𝗮 Metadata lets you enforce: 💠 Only approved documents (`is_verified = true`) 💠 Recent data (`last_reviewed >= 2025-01-01`) 💠 Correct access level (`user_role = admin`) 💠 Trusted sources (`source_type = policy` NOT gossip) Structured data + taxonomies = dramatically higher factual accuracy vs. raw text. 𝟯. 𝗘𝗻𝗰𝗼𝗱𝗲 𝗴𝗼𝘃𝗲𝗿𝗻𝗮𝗻𝗰𝗲 𝗶𝗻𝘁𝗼 𝗿𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 Metadata is your contract: - "Only retrieve from Legal-approved docs" - "No data older than 6 months" - "Must come from source of truth, not chat history" No model can argue with hard constraints. REAL IMPACT Implementing metadata-first filters see: → 60%+ reduction in factual hallucinations → 40%+ fewer irrelevant answers → Confidence scores that actually mean something → Compliance and auditability (you can trace every answer) THREE DESIGN RULES WE USE 𝟭. 𝗛𝗮𝗿𝗱 𝗺𝗲𝘁𝗮𝗱𝗮𝘁𝗮 𝗳𝗶𝗹𝘁𝗲𝗿𝘀 𝗳𝗶𝗿𝘀𝘁, 𝗮𝗹𝘄𝗮𝘆𝘀  Time range, geography, product, department, classification, trust level. Pick 3-5 that matter for your domain. 𝟮. 𝗦𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲𝗱 𝘀𝗼𝘂𝗿𝗰𝗲𝘀 𝗼𝘃𝗲𝗿 𝗿𝗮𝘄 𝘁𝗲𝘅𝘁  Tables, taxonomies, ontologies, templated data = cleaner retrieval  Messy PDFs = more hallucination surface 𝟯. 𝗡𝗼 𝗺𝗲𝘁𝗮𝗱𝗮𝘁𝗮 = 𝗻𝗼 𝗿𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹  For high-stakes domains (Finance, Legal, Healthcare), enforce: if it's not tagged, it can't be retrieved. Without metadata-first: LLM searches every document ever → high hallucination risk. With metadata-first: Filtered by region, recency, approval, language → expert-level accuracy. It’s not model size — it’s context quality. Question for you: Is your RAG still searching the entire knowledge base — or filtering smart with metadata first? Your GenAI interview edge: https://lnkd.in/gxyS44n3 #GenAI #RAG

10

2 Comments

Honored to join Padmarchana K. and Ishita Pai Raikar for this month’s NCSU's Ethics Quest Across the Disciplines conversation on AI x Accessibility & the Digital Divide. 🎙️♿🤖 From my work in Responsible AI across NetApp, UNC-CH, and NCSU, I’m excited to share practical approaches to translating principles into action. At the same time, Padma brings deep experience leading AI-driven community learning and championing inclusive innovation. We’ll connect accessibility's human, technical, and societal dimensions to drive equitable impact. 🌐 Event details: 📅 Wednesday, October 22nd 🕛 12 – 1 pm EST 💻 Virtual via Zoom https://lnkd.in/ebadKGWr We’ll explore: Designing AI that’s accessible by default and inclusive in practice. Community-led learning programs that close skills gaps and broaden participation. Responsible AI across industry and academia: governance, evaluation, and transparency. Concrete steps organizations can take now to reduce barriers and widen access. Join us! DM me with accessibility needs or questions so we can make this session work for everyone. 🙌 #Accessibility #DigitalDivide #ResponsibleAI #InclusiveInnovation #AI #TechForGood #AIForAll #CommunityLearning #EquityInTech #IndustryAcademia #PanelDiscussion #NetApp #UNCCH #NCSU NC State Business Analytics and AI Initiative NC State Poole College of Management UNC School of Data Science and Society NetApp

58

1 Comment

Andrej Karpathy saying “AGI is a decade away” means it is not possible with current Transformer models. Ten years is a lifetime in technology. Sam Altman saying “parts of AI are bubbly” means he knows we are in a hype bubble but wants it to last until he exits his stake. The good part is that the GPU infrastructure built during this boom will still be useful for robotics, quantitative models, digital twins, and the next wave of AI that might finally reach AGI.

11

3 Comments

𝗔𝗜 𝗟𝗲𝗮𝗱𝗲𝗿𝘀𝗵𝗶𝗽 𝗥𝗲𝗾𝘂𝗶𝗿𝗲𝘀 𝗜𝗻𝗳𝗿𝗮𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲, 𝗡𝗼𝘁 𝗝𝘂𝘀𝘁 𝗜𝗻𝗻𝗼𝘃𝗮𝘁𝗶𝗼𝗻 If India wants to compete with global AI powers like the USA and China, we must confront the infrastructure reality. Today, India operates at roughly 1 watt of compute per person, while China operates at ~20 watts per person, and the USA at ~150 watts per person. This is not just a statistic. It represents capacity, speed, research advantage, and geopolitical influence. AI is powered by compute. Compute is powered by energy, chips, fabs, and data centers. The scale required to compete is enormous: • 50 million CPUs • 7 million GPUs • 3 billion memory devices • 1 million storage systems • 3nm semiconductor fabs operating at million scale monthly • High Bandwidth Memory scaling from current capacity toward industrial-scale production — with at least 10 memory fabs required This is a national infrastructure challenge, not just a startup problem. The discussion also centered around the five strategic pillars shaping the AI economy: 🔹 Compute 🔹 Content 🔹 Communication 🔹 Cognition 🔹 Community India’s GCCs (Global Capability Centers) were the previous growth engine. The next wave could be sovereign compute, semiconductor capability, and AI-native infrastructure. The question is no longer whether India will adopt AI. The question is whether India will build the backbone that powers it ? #IndiaAI #AIInfrastructure #IndiaAiImpact #ComputeEconomy #Semiconductors #DigitalSovereignty #AIFuture #artificialintelligence

9

🇮🇳 “India Just Got Its Own LLM Leaderboard — and It’s a Game Changer for AI.” Big news from IIT Madras & AI4Bharat 👇 They’ve just launched the Indic LLM-Arena, an open, crowd-sourced leaderboard built to evaluate AI models for Indian languages — by Indians. And honestly, this could redefine how we measure AI intelligence for India’s multilingual reality. --- 💡 Why This Matters Most global leaderboards (like LMSys Chatbot Arena or Open LLM Leaderboard) test only English-centric performance. But India isn’t a single-language country — our daily conversations are a mix of Hindi + English, Tamil + English, Bengali, and more. That’s where Indic LLM-Arena steps in. It evaluates LLMs on three crucial pillars that global benchmarks often ignore: 1. 🗣️ Language Understanding – Can the model truly handle Hinglish, Tanglish, or regional code-mixed queries? 2. 🌏 Cultural Context – Does it understand local nuance, tone, and Indian social behavior? 3. ⚖️ Safety & Fairness – Does it respect India’s sensitivities, norms, and fairness expectations? --- 🧩 How It Works It’s human-in-the-loop, crowd-powered evaluation — simple but brilliant: > You type (or speak) an Indian-language prompt → Two anonymous LLMs respond → You pick the better one. Thousands of these pairwise votes create a statistically strong leaderboard — transparent, open, and shaped by real Indian users. --- 🧠 Why It’s a Big Step Forward This isn’t just a leaderboard — it’s a public benchmark for India’s AI ecosystem. - 🧑‍💻 Developers can use it to improve their Indic models - 🏢 Enterprises can compare models before deployment - 🇮🇳 The public gets to define what “good AI” means for India And with IndiaAI’s sovereign model efforts on the horizon, this benchmark will be key in measuring their impact. --- 🔭 What’s Next AI4Bharat plans to expand into: - 🎨 Multimodal models (vision + audio) - 🧠 Agentic tasks (PDF reasoning, search, tool use) - 🧩 Domain-specific leaderboards And yes — they’ll be open-sourcing everything. Because that’s how you build trustworthy AI for India. --- 🧭 Try It Out You can test, compare, and vote yourself here 👇 🔗 https://lnkd.in/due68NWH Full blog announcement: https://lnkd.in/dPcj562T --- Kudos to the AI4Bharat team and Prof. Mitesh Khapra for leading this initiative. This isn’t just tech — it’s a step toward an India-first AI ecosystem. #AI4Bharat #IndicLLM #IndiaAI #LLM #NLP #AIEngineering #MachineLearning #OpenSource #GenAI #AIinIndia #IITMadras

3

1 Comment