About
I am the tech lead for the Google Kubernetes Engine (GKE) security team and have been…
Activity
2K followers
-
Greg Castle shared thisI'm super excited to be working on security for agent substrate! It's our oss vendor neutral framework for running agents with extremely fast start/suspend/resume. Run it on any k8s cluster. It feels like the early days of Kubernetes all over again 😀 I'm spending a lot of time talking about identity and policy for agents. If that sounds interesting you should join us for substrate day before kubecon: https://luma.com/48hh3jo5Greg Castle shared thisOver the past few years, we've seen a tremendous shift in the nature of AI from one-shot prompts (original ChatGPT, circa 2022) to reasoning models (o1, circa 2024) to agentic coding (Claude Code, circa 2025) and personal assistants (OpenClaw, circa 2025). Autonomous agents such as long-running productivity agents, personal assistants, and coding agents often spend the vast majority of their time dormant while waiting on model inference, tool execution, or human input. These long periods of idleness create a problem for compute platform teams managing infrastructure. Do you keep the underlying compute running 24/7 (good user experience, but costly since you're paying for idle CPU and RAM) or spin-up agentic workloads on-demand (save money, but results in lengthy startup delays that hurt the user experience)? We're solving this tradeoff. Today, we are announcing the availability of Agent Substrate on Google #Kubernetes Engine. Agent Substrate is an open-source sandbox orchestration engine designed from first principles for the agentic era. Key highlights: • 10x Compute Density: Substrate snapshots guest hypervisor state to local SSD and Google Cloud Storage the moment an agent pauses, greatly increasing the number of agents that can share a single machine. • Sub-500ms Resume: Dormant sessions wake up in milliseconds on pre-warmed workers, sustaining over 500 activations per second. This means the end user experience doesn't suffer with any unnecessary latency. • Hardware Optimization: Native support for Google Axion Arm processors, delivering up to 30% better price-performance for sandbox workloads. • Instant Stateful Workspaces: Integrated with Filestore agent volumes for sub-100ms NFS attach/detach and POSIX-compliant multi-agent collaboration. Scale to millions of agents without ballooning your cluster footprint. https://lnkd.in/gpaRYRhN Alex Zakonov, Tim Hockin, Drew Bradstock, Alex B., Iftach Ragoler, Akshay Ram, Brandon Royal, Dmitry Berkovich, Jason Monden, Neha Bajwa, Karl Weinmeister, Tinsley Shi, Zheng LammertsAgent Substrate available on GKE | Google Cloud BlogAgent Substrate available on GKE | Google Cloud Blog
-
Greg Castle reposted thisGreg Castle reposted this“𝗬𝗼𝘂 𝗰𝗮𝗻'𝘁 𝗵𝗮𝘃𝗲 𝗮 𝘀𝗲𝗰𝘂𝗿𝗲 𝗔𝗜 𝘄𝗼𝗿𝗸𝗹𝗼𝗮𝗱 𝗼𝗻 𝗮𝗻 𝗶𝗻𝘀𝗲𝗰𝘂𝗿𝗲 𝗰𝗹𝘂𝘀𝘁𝗲𝗿.” Today we published the GKE blueprint for securing AI at enterprise scale — the guide I wish every team moving AI from prototype to production had on day one. It covers the full stack, because AI security isn't one control: 🔒 Infrastructure — hardware-attested execution: Confidential GKE Nodes extending encryption and attestation to accelerators, so even infrastructure operators can't reach your model weights 📋 Model provenance — traditional SBOMs don't capture AI artifacts. We built k8s-aibom (open source) to inventory the models, datasets, and frameworks actually running in your clusters 🛡️ The inference path — Model Armor against prompt injection and data leakage, sandboxing for agentic workloads executing generated code Plus a phased adoption model — Deploy → Operate → Govern — so you don't have to boil the ocean to start. The race to deploy AI shouldn't be a race to the bottom on security. Full blueprint: https://lnkd.in/g5UvdusY Blog: https://lnkd.in/gkh3s-gq James Chou Greg Castle Komei Nakamoto Alex B. Iftach Ragoler Nathan Beach Drew Bradstock Alex Zakonov #AISecurity #Kubernetes #GKE #ConfidentialComputingSecuring AI at Enterprise Scale: The Google Kubernetes Engine Blueprint | Google Cloud BlogSecuring AI at Enterprise Scale: The Google Kubernetes Engine Blueprint | Google Cloud Blog
-
Greg Castle reposted thisGreg Castle reposted thisYou can't govern AI you can't see. Today we're excited to share that we've open-sourced k8s-aibom — a lightweight Kubernetes controller that automatically discovers the AI workloads actually running in your Kubernetes clusters and generates standards-based AI Bills of Materials (CycloneDX 1.6 ML-BOM). Shadow AI is real. Developers ship vLLM servers, LangChain agents, and vector databases every day without registering them anywhere. Build-time scanners only tell you what was intended to run. And most runtime tools demand a steep price — privileged DaemonSets, kernel access, or pod-spec changes that slow teams down. k8s-aibom takes a different approach: 🔍 Detects serving runtimes (vLLM, Triton, TGI, Ollama), agent frameworks (LangChain, AutoGen, CrewAI), vector DBs (Milvus, Qdrant, pgvector), training jobs, and eval harnesses 🪶 Runs as a single unprivileged Deployment — no sidecars, no eBPF, no node agents, zero developer friction 📋 Every finding carries a confidence tier (declared / inferred / unresolved), so unknowns get flagged for review instead of silently dropped 🔒 Deterministic, immutable output built for auditors — with mappings to the EU AI Act, NIST AI RMF, and ISO/IEC 42001 It's Apache 2.0, cloud-neutral, and ready to try today. Huge thanks to the team who made this happen. - Blog: https://lnkd.in/eta_Pi5x - GitHub: https://lnkd.in/ebWD5WJG Would love feedback, issues, and contributions. #Kubernetes #AISecurity #AIGovernance #OpenSource #GKE #MLBOMGitHub - GoogleCloudPlatform/k8s-aibom: Know what AI is actually running in your clusters. An unprivileged Kubernetes controller that inventories models, runtimes, agents, and RAG components at runtime - emitting CycloneDX 1.6 ML-BOMs with evidence, confidence tiers, and Sigstore-verified model signaturesGitHub - GoogleCloudPlatform/k8s-aibom: Know what AI is actually running in your clusters. An unprivileged Kubernetes controller that inventories models, runtimes, agents, and RAG components at runtime - emitting CycloneDX 1.6 ML-BOMs with evidence, confidence tiers, and Sigstore-verified model signatures
-
Greg Castle shared thisIf you run containers and freaked out about the copy.fail vuln this week, something is not right. Pull up a chair. It's a cool vuln. It's reliable and easy to exploit. But ultimately for container/K8s users it's just another container breakout. The kernel is absolutely RIDDLED with these vulnerabilities. For the past several years we have patched 1 or 2 fully exploited container breakouts every month via our kCTF program. Not theoretical vulns, these are fully worked exploits. At any given moment since 2021 we have been holding working exploit code for several unpatched container breakouts. They didn't get cool names or websites, but they all got patched on the upstream kernel. See more on kCTF here: https://lnkd.in/g-UsJnuQ Prior to AI tooling, for skilled researchers, building container breakout exploits was O(days) of work. That's not much. Now they are within reach of much less skilled attackers and the timeframes are even shorter. There's lots of cases where relying on the container security "boundary" doesn't pose a ton of risk. This is very common in the industry. For example: you're running fully trusted code with well-sanitized and trusted inputs. Where it *really* doesn't make sense to use the container as a security boundary is when you're running untrusted code, highly vulnerable software with untrusted inputs, or agents that are themselves generating untrusted code and making self-modifications. For those cases you need to do something else, relying on containers is not sufficient. Only taking patching action when a vuln makes it to your social media feed does nothing to contain the real risk. We recommend GKE Sandbox (gVisor if you're not on GKE). It has protected against every container breakout we've ever received in the kCTF, including copy.fail. Gemini uses it, OpenAI uses it, Cloud Run uses it. It's fast and covers most use cases. Also note that hardening the container boundary is just one leg of the stool: you also need to protect network and storage access. In addition to the security of GKE Sandbox, the fast snapshotting is super useful, especially for AI use cases. If you're running agents you should use GKE Agent Sandbox which gives you GKE Sandbox plus performance: sub-second provisioning. That's MUCH faster than standard pod scheduling. https://lnkd.in/gqPk743t If you're not ready to make systematic changes and just need help now, we published a bunch of copy.fail mitigation advice and sample code here (linked from the main security bulletin) including a custom seccomp profile that should be useful on non-GKE platforms as well: https://lnkd.in/g7xVeRUuAbout GKE Agent Sandbox | GKE AI/ML | Google Cloud DocumentationAbout GKE Agent Sandbox | GKE AI/ML | Google Cloud Documentation
-
Greg Castle reposted thisGreg Castle reposted thisIf you use Google #Kubernetes Engine (GKE), then this YouTube playlist is for you: https://lnkd.in/eNY7MEix Gari Singh put together a playlist of the videos from GKE-related sessions at Google Cloud Next '26, including sessions from Drew Bradstock, Bobby Allen, MS, PMP, Yochay Kiriaty, Iftach Ragoler, Greg Castle, Maciej Rozacki, Akshay Ram, Raja Jadeja, Abdel SGHIOUAR, and many others.
-
Greg Castle shared thisIf you're at Next and want to talk about security I'm running a discussion group Friday April 24 at 9:45am with Glen Messenger. Bring your AI, GKE, Kubernetes, containers, vulnpocalypse questions! If you see our talk (BRK 2-114 Thursday April 23 4pm) about how we designed sealed environments on top of K8s with Anthropic this is the place to come to hear more detail. But I'm happy to discuss any security topics that are top of mind.
-
Greg Castle shared thisOSS maintainers are overloaded at the moment with vulnerability reports. I got together with some other CNCF maintainers (Kubernetes, containerd), CNCF Security TAG, Google Project Zero bug finders, and Google's Vulnerability Reward Program folks to produce some advice specifically for OSS maintainers. It covers some advice on how to attack the problem at all stages of finding and fixing vulnerabilities. We also cover what we'd like to see from folks who are using AI to find and report vulns. Thanks to the Cloud Native Computing Foundation (CNCF) and OSS security brainstrust that contributed to this advice: Brandt Keller, Chris Aniszczyk, Evan Anderson, Ivan Fratric, Jordan Liggitt, Michael Lieberman, Monis Khan, Natalie Silvanovich, Rita Zhang, Sam Erb, Samuel Karp https://lnkd.in/gJv7_anJThe AI-driven shift in vulnerability discovery: What maintainers and bug finders need to knowThe AI-driven shift in vulnerability discovery: What maintainers and bug finders need to know
-
Greg Castle shared thisSecurity shouldn't be the bottleneck for AI scale. For the past year with #Anthropic we've been co-designing a system for AI asset protection at massive scale called GKE hypercluster. It uses sealed and attested VMs running inside a GKE cluster to protect model weights and inference query/response using encryption. GKE hypercluster delivers: - Attestation from firmware to Pod - Using standard #Kubernetes configuration - On TPUs and GPUs By only allowing decryption in sealed environments, it can even protect assets from cluster admins who have full privileges over the environment. But it's not just security. Because we've separated some K8s functions out from the workload we've been able to increase scale. GKE clusters could already go big, but we've been able to create some truly gigantic clusters that will bring together accelerator capacity from many regions for companies like Anthropic. I'm super excited to be presenting the work of several teams at Google and a large group of Anthropic design partners at #GoogleCloudNext. Come hear it from Nova DasSarma, Systems Lead from #Anthropic and Glen Messenger, GKE Group Product Manager! Make sure you add the talk to your schedule with Google Cloud so they give us the right size room :) See you there! https://lnkd.in/gRFDWHrP
-
Greg Castle posted thisIt’s been fascinating to watch gVisor’s evolution: from answering the security challenge of “containers don’t contain” to also becoming an industry-leading pod snapshot technology that reduces AI inference start-up by as much as 80% [1]. More security and better performance? Shocking :) The OG: Security gVisor's origins are in providing an opensource, safe sandbox for untrusted code. Without gVisor it is possible to strengthen the container boundary using seccomp, apparmor, and an unprivileged Linux user. But few do. To harden containers against breakout vulnerabilities, gVisor intercepts system calls and re-implements them in an unprivileged user-space kernel. It works: we’ve built the world’s largest library of kernel container exploits through our kCTF program [2] and gVisor has protected against all of them. It runs massive Google-scale workloads that consume untrusted data, and provides a sandbox for GCP serverless products [3]. In GKE it powers GKE Sandbox [4]. But most companies decided they didn’t need a sandbox. They mostly trust the code they run, and only use the container as a weak secondary boundary. Plot twist: AI and Agents Suddenly many companies are running non-deterministic agentic workloads inside Kubernetes that generate and run code that needs to be sandboxed, so there’s massive new use for gVisor sandboxing [5], including for Gemini, OpenAI [6] and other labs. But just as critical for AI is snapshotting—the ability to quickly save and restore a pod's full state, why: Faster Start-up: Puts models/containers/codebases into a ready-to-serve memory state, reducing cold-start and replica scale-up time. Good for agents and good for inference serving. Cost Savings: Allows pods to spin down while waiting on human input, saving expensive GPU/TPU time. Faster Recovery: Improves training and inference goodput by restoring the full pod state quickly after hardware failures. Faster Dev Loop: Accelerates ML dev/test loops with quicker job restarts. gVisor is uniquely positioned to solve these problems because its architecture makes it great at both sandboxing and snapshotting. Why: gVisor’s user-space kernel offers a clean, complete state encapsulation that works better than collecting state from a large number of kernel interfaces; Serialization is a core built-in capability of gVisor’s kernel, not an afterthought that needs to catch up to new features; and It allows us to make significant end-to-end performance optimizations like a speed-optimized serialization/deserialization structure, high-throughput parallel reads from GCS, and lazy loading to only restore memory regions on demand. What about micro VMs? Booting an OS is slower (s) than starting a process (ms) and uses more memory. Nesting VMs degrades performance. On cloud, your nodes are VMs. “Bare Metal” capacity tends to be more limited. May not support GPU [7]. This is why we built GKE Pod snapshots on top of GKE Sandbox. Try it [8] and watch demos [9].
Greg Castle commented on a post
2w
We will miss you! You are one of the most technically deep product managers we've ever had at Google.
-
Greg Castle reacted on thisGreg Castle reacted on thisGave my keynote at BSides Canberra. The only other time I ran into this many Aussies was in Niseko. Which is in Japan. The talk went OK. Made a few people laugh. Made a few think. Physically, I wasn't in great shape. I caught a bug at unprompted.au in Sydney. Not a 0-day. A real bug: sore throat, runny nose, light headache. The trilogy. I was also nervous, as always. What if people found my talk too mundane for a keynote? I don't even know what makes something "keynote level", but I assume it has to be philosophical, or at least insightful. I deliberately made the slides in Keynote, secretly hoping the software would help. Things got easier once I started speaking. I found my flow and got to tell the stories I love sharing. I opened with the story of a car accident I'd been in while in Sydney. Don't worry, nobody was hurt. I'd hesitated to include it because it felt personal. But I needed something to cheer me up, so I added it at the last minute. The accident was absurd, funny and scary in a strange way. Pretty much how I see life: how little control we have, yet how much we can still choose to do with the little time we get. After the keynote, the Honorable Tim Willis took me to find some banh mi. Then he gave me the Aussie starter pack: rabbit-fur Akubra hat, meat pie, blazing sun, dry-ish grass, and eucalyptus woodland. And off we went on a wild hike to track down some kangaroos! I didn't know kangaroos were essentially very big mice. When I got back to the hotel, I checked Ancestry to confirm my new scientific theory. And there you go: I share a common ancestor with these grass-eating dudes. We're essentially cousins. That explains why the entire pack gave me a standing welcome. It was a very successful first day in the capital. I'm looking forward to the second day. I'll be around the conference, please come say hi. We can chat about our furry cousins.
-
Greg Castle liked thisGreg Castle liked thisHello world 👋 I was out for 1 month but likely I still remember all my passwords 😁. I spent the last four weeks knocking out two major bucket list goals: the Sydney Marathon 🇦🇺 and summiting Mount Ararat 🇹🇷 (5,137m). The contrast between running in a t-shirt and climbing in a heavy parka couldn't be sharper. What did I miss in the tech world while I was off the grid? Did AI Took over or not yet? Drop it in the comments so I can catch up! 👇
-
Greg Castle liked thisGreg Castle liked thisBoom! unprompted.au is a wrap. The Aussie AI x Cybersecurity community is well and truly alive and kicking… One of my favorite aspects of the event was the mix of excited OGs and the new generation of researchers and operators that are coming up ❤️ Hugest congrats to Mark Dowd Sianna Holobrodskyj and everyone involved in making this happen 👏
-
Greg Castle liked thisGreg Castle liked thisOver the past few years, we've seen a tremendous shift in the nature of AI from one-shot prompts (original ChatGPT, circa 2022) to reasoning models (o1, circa 2024) to agentic coding (Claude Code, circa 2025) and personal assistants (OpenClaw, circa 2025). Autonomous agents such as long-running productivity agents, personal assistants, and coding agents often spend the vast majority of their time dormant while waiting on model inference, tool execution, or human input. These long periods of idleness create a problem for compute platform teams managing infrastructure. Do you keep the underlying compute running 24/7 (good user experience, but costly since you're paying for idle CPU and RAM) or spin-up agentic workloads on-demand (save money, but results in lengthy startup delays that hurt the user experience)? We're solving this tradeoff. Today, we are announcing the availability of Agent Substrate on Google #Kubernetes Engine. Agent Substrate is an open-source sandbox orchestration engine designed from first principles for the agentic era. Key highlights: • 10x Compute Density: Substrate snapshots guest hypervisor state to local SSD and Google Cloud Storage the moment an agent pauses, greatly increasing the number of agents that can share a single machine. • Sub-500ms Resume: Dormant sessions wake up in milliseconds on pre-warmed workers, sustaining over 500 activations per second. This means the end user experience doesn't suffer with any unnecessary latency. • Hardware Optimization: Native support for Google Axion Arm processors, delivering up to 30% better price-performance for sandbox workloads. • Instant Stateful Workspaces: Integrated with Filestore agent volumes for sub-100ms NFS attach/detach and POSIX-compliant multi-agent collaboration. Scale to millions of agents without ballooning your cluster footprint. https://lnkd.in/gpaRYRhN Alex Zakonov, Tim Hockin, Drew Bradstock, Alex B., Iftach Ragoler, Akshay Ram, Brandon Royal, Dmitry Berkovich, Jason Monden, Neha Bajwa, Karl Weinmeister, Tinsley Shi, Zheng LammertsAgent Substrate available on GKE | Google Cloud BlogAgent Substrate available on GKE | Google Cloud Blog
-
Greg Castle liked thisGreg Castle liked thisHappy birthday hijinks to one and all at ControlPlane! 🎉 9yo today, precipitously close to adolescence whilst nurturing an eternal sense of intrigue and wonder. The last year saw the US business escalate as we shipped supply chain and CVE auto-patching for FSIs and FINOS, our Enterprise offering for OpenBao is the latest addition to the product fold with hardening, performance replication and HSM support, and #fluxcd now enjoys its own Web UI and agentic/MCP extensions! We continue to harden and optimise AI and cloud native systems in the face of agentic-paced FUBAR, and have "AI-natived" high-confidence processes where AI reliably fixes bottlenecks, exclusive of human assurance! And we continue to see "don't read the code" feature vs security dice rolls in sensitive environments, it was ever thus. Strategic, targeted, and precise delivery across emerging threats persists, and oceanic cascades of thanks and gratitude to clients, colleagues, and collaborators. Expect fireworks for the decade next year! 🤖🎆🎡☸️🔒🛡️⚙️🛡️ ⛓️📦🔗☁️ #cloudnative #ainative #securitynative #services #consulting #kubernetes #containers #isolationism #fluxcd #openbao #birthday #hijinks
-
Greg Castle liked thisGreg Castle liked thisFriday was my last day at Google. After 12 years, leaving the company, products, and team was one of the hardest decision of my life. To users and colleagues, thank you for the incredible memories, partnerships, and shared successes over this long journey. My highlight is obviously Google Cloud Run: re-inventing serverless with a developer-beloved product that ended up becoming a major adoption and revenue driver for Google Cloud. We started with a focused scope and relentlessly iterated by listening to our customers. I am forever grateful for the opportunity.
-
Greg Castle liked thisif you're not listening to the Kubernetes podcast, no matter what flavour or cloud of K8s you run, you are missing out. Check out the latest from Tim Hockin and Brandon Royal on the Substrate OSS offering. fun fact - auto complete keeps trying to change K8s to kids for me. rather appropriate I think considering how many releases and Kubernetes birthday parties I have been at now...Greg Castle liked thisCheck out our latest episode: Agent Substrate, with Tim Hockin & Brandon Royal! https://lnkd.in/eGJEWKvG AI agents require terminal access, browser automation, and file system isolation—yet internal benchmarks show they sit idle over 90% of the time waiting on humans or LLMs. Traditional Kubernetes primitives were built for long-running web services and databases, not rapid, ephemeral agent sessions with extreme pod churn. We sat down with Tim Hockin (Principal Software Engineer and one of the original founders of Kubernetes) and Brandon Royal (Product Manager on GKE) to discuss Agent Substrate (https://lnkd.in/eDnAfVYS), a new open-source runtime designed to solve this paradigm shift. We dive deep into the architecture, covering: - The "Idleness" Challenge: Why AI agent workloads require a fundamentally different approach to density and resource allocation. - Workers vs. Actors: How decoupling underlying worker pods from ephemeral agent sessions bypasses control plane limits to achieve >10x density improvements. - Instant Suspend & Resume: How state snapshotting (memory/disk to local storage or cloud buckets) eliminates the cost of idle compute. - Agent Identity & Sandboxing: Why sandbox isolation goes beyond hypervisors to include strict policy enforcement and identity delegation.
Experience & Education
Other similar profiles
Explore more posts
Explore top content on LinkedIn
Find curated posts and insights for relevant topics all in one place.
View top content