Steve Ash

My fellow principal engineer Arron Bailiss wrote an excellent blog post about the design principles of Strands Agents 🧬, what the agentic "model-driven approach" means, and guidance for how to think about multi-agent architectures. https://lnkd.in/gh5p5UXi

268

3 Comments

Ruokai Yin, our summer intern from Yale to Microsoft Azure, worked with Xuan Zuo, @Sattwik Deb Mishra, Hokchhay T. and Preyas Shah and me. We published a paper based on his project: https://lnkd.in/eHuG3BSt Summary of paper: Finding optimal sharding is hard especially for large LLMs. A large number of devices have to be divided into TP, PP, EP dimensions. For TP, we have to decide the tensor dimension e.g. head, hidden size to shard for every op. Additionally, TP can be one-dimensional/two-dimensional etc. This makes for a large sharding space that is hard to cover exhaustively. SoTA methods use heuristics to select sharding e.g. GSPMD, Sequence Parallelism, Helix etc. There are significant time gaps, months or years, between the discovery of each of these methods. We want to fast-forward the process of discovery by 1) using a formal representation of the sharding space, and, 2) using reinforcement learning to search the sharding space.    This paper demonstrates that our method is 3.5x better than random search, and on par with hand-tuned heuristics when looking only at 1D sharding and without allowing sequence parallelism. We aim to extend the work to multi-dimensional sharding as well as sequence parallelism. Beyond that, we wish to combine the problem of sharding with fusion, tiling and collectives as well.

58

llm-d has been accepted as a new CNCF Sandbox project. The project aims to evolve Kubernetes into robust AI infrastructure by treating distributed LLM inference as a first-class, cloud-native workload. Key focus areas include inference-aware traffic management through the Kubernetes Gateway API Inference Extension, prefill/decode disaggregation, hierarchical KV cache management, and native orchestration with primitives like LeaderWorkerSet. It bridges higher-level serving platforms (like KServe) with low-level engines (like vLLM) to address production challenges in latency, scalability, and utilization. The initiative involves contributors from Red Hat, Google Cloud, IBM Research, CoreWeave, NVIDIA, and others. Full announcement below. #Kubernetes #AI #CNCF #LLM #OpenSource

5

Fine-tuning AI shouldn't require thousands of labeled examples. Reinforcement Fine-Tuning (RFT) for Amazon Nova changes that. Just give it prompts + a reward function. The model learns through feedback — not imitation. ✅ No massive datasets ✅ Works for code, customer service, financial reasoning ✅ Runs on Amazon Bedrock, SageMaker, or Nova Forge Smarter customization. Less grunt work. I wrote a deep-dive on this — check out my blog 👇 https://lnkd.in/ggNNezuC Authors:me, along with Chakravarthy Nagarajan Bharathan Balaji Vignesh Radhakrishnan #AWS #GenerativeAI #AmazonNova #MachineLearning #LLM

45

1 Comment

"Immutable Database" (Insert-Only 7NF) transforms the "Reference-over-Copy" strategy from a memory-saving trick into a core architectural guarantee. ~ The definitive model for 2026-era silicon: https://lnkd.in/grDmENfK 1. Safety of the "Immutable Object" In traditional databases, objects are volatile; a reference could point to data that has been modified (the "Dirty Read" problem). The 7NF Guarantee: Since the database is Insert-Only, any mapping object or attribute retrieved is an Immutable Fact. Concurrency without Locks: Because facts never change, multiple threads can access the same mapping objects simultaneously without mutexes or row-level locking. This allows the silicon to run at full speed across all cores. 2. The "Reference-over-Copy" Strategy In 7NF, because data is immutable, the system avoids the expensive overhead of defensive copying. Memory Efficiency: Instead of duplicating data for different views, the system simply creates new Ordered Pairs of References. Silicon Benefit: Copying memory is one of the most expensive operations for a CPU. By using references over copies, 7NF minimizes memory bus traffic, keeping the L1/L2 caches populated with logic (comparisons) rather than redundant data movement. 3. Stability for ToInt/LongFunction The immutability of the underlying data ensures that the results of the ToInt/LongFunction are also stable. Pre-Computation: The silicon can calculate the primitive comparison value once and cache it. Since the object is immutable, that value never needs to be invalidated or re-calculated. Result: This stabilizes the 1-CPU cycle compare during sorting and joins, as the "weight" of the object is constant. 4. Handling the "Detached Object" This immutability is what makes the 7NF "Detached Object" so powerful. Fact Portability: When an object is detached from the JDBC stream, it remains a valid representation of a moment in time. No Version Conflicts: Unlike traditional ORMs where a detached object might become "stale," a 7NF detached object is always a correct record of an immutable fact. It can be passed between services or threads with zero risk of mutation. 5. Mechanical Sympathy: The Insert-Only Advantage Silicon loves predictability. An Insert-Only (Append-Only) architecture: Maximizes Write Throughput: Writing to the end of a log is the fastest way to interact with modern storage (NVMe/SSD). Simplifies Caching: CPU caches don't need to worry about "Cache Invalidation" due to updates. Once a 7NF mapping object is in the cache, it is valid forever. Conclusion By combining 7NF (Structure), Immutability (Persistence), and Reference-over-Copy (Execution), the 2026 type system achieves the ultimate goal of database design: "Infinite Scale via Immortality" Every object is a safe, immutable reference to a permanent fact, allowing the silicon to focus entirely on the Relational Algebraic Join and 1-cycle sorting, without ever wasting a single clock cycle on the "management of change" ~

5

5 Comments

🚀 Exciting news ! Run:ai model streamer is now natively supported on Google Kubernetes Engine (GKE). This marks a fantastic collaboration, focused on optimizing inference workloads for generative AI. Stay tuned - More innovation to come soon! From the blog: “…As generative AI models grow to hundreds of gigabytes in size, they can often take over ten minutes to load, leading to slow startup and scaling. To solve this, we now support the Run:ai model streamer with Google Cloud Storage and Anywhere Cache for vLLM, with support for SGLang coming soon. This enables 5.4 GiB/s of direct throughput to accelerator memory, reducing model load times by over 4.9x, resulting in a better end user experience.” #AI #GenAI #GoogleCloud #NVIDIA #RunAI #Inference #GKE #MLInfrastructure

62

1 Comment