Developers & Practitioners

Empower your agents with the Google Cloud CLI remote MCP server

Wed, 30 Sep 2026 16:00:00 +0000

Today, we’re expanding our ecosystem of managed remote MCP servers by introducing the Google Cloud CLI remote MCP server in preview.

Powered by the popular gcloud and bq (BigQuery) command-line tools, this new server gives AI agents immediate, broad access to command-line operations for managing Google Cloud infrastructure and working with advanced BigQuery workflows securely and seamlessly.

Why CLI matters for AI agents

Agents are increasingly performing complex cloud operations, but standardizing how they interact with backend systems remains a challenge. The Google Cloud CLI remote MCP server bridges this gap by packaging the versatility of hundreds of gcloud and bq commands into one single MCP server. This results in two strong benefits for the agent:

Higher-level abstractions: CLI commands package complex multi-step workflows, validation checks, and high-level operations into unified commands rather than requiring multi-step API orchestration.
Leverages model training: LLMs are heavily pre-trained on public command-line documentation, syntaxes, and usage examples, making CLI invocation intuitive and highly accurate for models.

The benefits of putting CLI behind remote MCP

Managing cloud infrastructure with AI agents traditionally requires installing and maintaining Google Cloud CLI binaries inside agent execution environments. The Cloud CLI remote MCP server bridges CLI capabilities with MCP benefits by providing an isolated execution sandbox on Google Cloud infrastructure. This solves key infrastructure challenges:

Simplified dependency and runtime management: For teams building custom agents, maintaining local CLI versions and dependencies across dev, test, and production environments creates operational overhead. Remote MCP eliminates local installations and runtime maintenance.
Access for web-based agent endpoints: Web-hosted agent platforms and web interfaces (such as Gemini Enterprise and other hosted enterprise agent platforms) run in environments where users cannot control or install local packages. Remote MCP enables secure, managed access to Google Cloud CLI operations directly from these surfaces.

Enterprise-grade security and governance

Connecting an AI agent to your infrastructure requires strict, enterprise-ready safeguards. This remote server leverages Google Cloud's standard identity and governance frameworks to keep your environments secure:

Zero ambient credentials: The server isolates execution in a network-restricted proxy boundary with no ambient credentials. Authentication and authorization are handled through Agent Identity, OAuth 2.0, and Identity and Access Management (IAM).
Strict policy enforcement: Every command executed through the remote MCP server is run with the permissions of the authenticated caller identity. Both standard IAM permissions and organization policy service constraints are strictly enforced against downstream target resources.
Advanced protection with Model Armor: To minimize the risks associated with AI tool calling, the Cloud CLI remote MCP server integrates with Model Armor. You can proactively screen LLM prompts and responses to protect against risks like prompt injection and malicious inputs.
Cloud audit logging: The Cloud CLI remote MCP server can be configured to log every tool invocation to Audit Logs (Data Access logs under cloudcli.googleapis.com/mcp). Security teams can gain full visibility into caller identities, OAuth clients, and IAM authorization decisions (mcp.googleapis.com/tools.call) without exposing sensitive command payloads or personally identifiable information (PII).

Connecting to the Google Cloud CLI Remote MCP Server

Integrating cloud management into your agents no longer requires packaging Google Cloud CLI binaries, managing local execution runtimes, or maintaining dependencies inside agent container images. Because the Google Cloud CLI remote MCP server implements the standard Model Context Protocol, any MCP-compatible agent platform or orchestration runtime can connect immediately via standard configuration:

code_block: <ListValue: [StructValue([('code', '{\r\n "mcpServers": {\r\n "google-cloud-cli": {\r\n "uri": "https://cloudcli.googleapis.com/mcp",\r\n ...\r\n }\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d02f6b50>)])]>

Your agent immediately gains access to execute gcloud and bq commands in a secure, network-isolated cloud sandbox.

Authentication is handled via keyless Agent Identity for hosted Google Cloud platforms, or standard OAuth 2.0 for external runtimes. For authentication options, see the MCP Authentication Guide.

Bringing infrastructure management to agents

The Cloud CLI remote MCP server exposes two powerful tools, run_gcloud_command and run_bq_command, giving your AI agents broad, immediate access to Google Cloud operations through natural language.

Managing cloud infrastructure with run_gcloud_command

With run_gcloud_command, agents can execute the full breadth of gcloud operations to manage, diagnose, and secure your Google Cloud environment. An example follows:

Observability and incident diagnostics: An agent streamlines incident diagnostics by automating command execution and reducing context-switching across tools.

Extending BigQuery operations with the run_bq_command tool

While the BigQuery MCP server already helps organizations analyze and explore data using AI agents, with the introduction of run_bq_command, agents can now tackle advanced BigQuery tasks such as resource allocation, job monitoring, and task scheduling by unlocking the full scope of bq CLI functionality. Key capabilities include:

Automating scheduled queries: An agent utilizes BigQuery Data Transfer Service configurations to schedule queries automatically.
Job and resource management: Gain deep insight into query execution details, including processed data volume, slot usage, and execution plans, as well as managing reservations.
Access and permissions control: Data administrators and owners can inspect and update table permissions directly through the agent.

Pricing and availability

The Google Cloud CLI MCP server is available today in public preview. There is no additional charge to use the MCP server itself. You pay only for the GCP resources you create and any applicable data transfer costs.

To get started, enable the Cloud CLI Execution API (`cloudcli.googleapis.com`) in your Google Cloud project, grant the required MCP Tool User (`roles/mcp.toolUser`) IAM role to your agent or user identity, and configure your MCP client to connect to `cloudcli.googleapis.com/mcp`.

Data Agent Kit is now GA: Bring Google Data Cloud to any coding agent

Wed, 30 Sep 2026 13:00:00 +0000

Today, Google Cloud Data Agent Kit is generally available. Data Agent Kit is a free set of Model Context Protocol (MCP) tools and agent skills that lets the coding agent you already use work directly with your Google Cloud data products, whether you're using Antigravity, Claude Code, Codex, or other popular tools.

With GA, we are adding support for BigQuery Graph, Bigtable, and Managed Service for Apache Spark access to your open Lakehouse, along with dozens of quality-of-life improvements that make everyday work faster and smoother.

What is Data Agent Kit?

Coding agents have become remarkably good at writing SQL, PySpark, and pipeline code. What they don't have by default is context about your environment: which tables exist, how they're partitioned, which ones your team trusts, or why last night's job failed. Without that, even a strong agent has to work from assumptions, and you end up pasting schemas and error logs into the chat to fill in the gaps.

Data Agent Kit fills that gap with two things:

MCP tools: Connections to more than 15 Google Data Cloud services, so your agent can inspect schemas, run queries, read job logs, and manage resources in your live environment.
Google-authored skills: Open-source instructions from Google Cloud engineers that teach your agent data best practices, like optimizing BigQuery SQL, designing Bigtable row keys, and building dbt (data build tool) pipelines.

You can use Data Agent Kit wherever you already work: as an IDE extension for VS Code, Antigravity IDE, Cursor, and other VS Code-compatible editors; as a plugin for Antigravity 2.0, Antigravity CLI, Claude Code, and Codex; or in Cloud Shell and Cloud Workstations, where it comes pre-installed. The IDE extension also brings a lightweight version of the Google Cloud console into your editor, so you can browse data, run queries, and review your agent's work without switching windows.

How it works

Say you ask your agent, "Forecast next month's demand for our top-selling products and check whether we have enough inventory to meet it." Data Agent Kit loads the relevant skills, so the agent follows Google's best practices for the task. It searches Knowledge Catalog to find the sales and inventory tables your team trusts. It then uses MCP tools to run a forecast in BigQuery, check current stock levels in AlloyDB for PostgreSQL, and bring the combined answer back to your editor or terminal. Every step runs with your own IAM permissions.

How Data Agent Kit connects to data.

What you can build

Data Agent Kit covers analytics, operational databases, the Lakehouse, and pipelines. Here's what that looks like in practice, starting with what's new at GA.

Analytics and graph: BigQuery and BigQuery Graph (New in GA)

Graphs are a natural way to explore relationships, like which products people buy together or how suppliers connect to your inventory. Building one usually means hand-writing CREATE PROPERTY GRAPH DDL, learning GQL, and working out which keys actually form edges.

Instead, you describe the graph you want and your agent builds it. The bigquery-graph-author skill maps your tables to nodes and edges, checks each proposed relationship against the actual data, and shows you a plan to approve before creating anything. It can even start from an ER diagram or data model you already have. The bigquery-graph-query skill then writes the GQL, and the graph visualizer in the IDE lets you click through the results.

Building and visualizing a BigQuery property graph.

Operational and real-time databases: Spanner, AlloyDB, Cloud SQL, and Bigtable (New in GA)

Some features have to load instantly, like a personalized feed, a live counter, or a "recently viewed" rail on your storefront. Bigtable is built for exactly that, and it rewards a well-designed row key.

With GA, Bigtable joins Spanner, AlloyDB, and Cloud SQL as a fully supported database in Data Agent Kit. The new bigtable-basics skill designs your schema around how the data will be read and flags hotspots and full table scans before you create anything. Your agent can then create the table and query it with GoogleSQL, with column families flattened into readable columns. In the IDE, you can browse Bigtable instances and tables in the catalog explorer and run queries from the SQL editor.

Agent-guided Bigtable schema design and querying.

Lakehouse and Spark: Managed Service for Apache Spark and Lakehouse for Apache Iceberg (New in GA)

Your agent could already query the Apache Iceberg tables in your Lakehouse through BigQuery. Now it can also work with those same tables using serverless Spark on Managed Service for Apache Spark, with no cluster to manage. Each session keeps its state, so temporary views carry across statements, and you get Iceberg's full feature set, including branching, time travel, and schema evolution.

Your tables don't all have to live on Google Cloud, either. The federate-lakehouse-catalog skill connects your Lakehouse to AWS Glue and Databricks Unity Catalog, so your agent can query that data in place without building an ingestion pipeline first.

Branching and querying Iceberg tables with Managed Service for Apache Spark.

Pipelines and orchestration: dbt, Dataform, and Managed Service for Apache Airflow

Once your logic works, your agent can turn it into a pipeline that runs on its own. It writes dbt or Dataform models, then the gcp-pipeline-orchestration skill schedules them together with your notebooks as an Orchestration Pipeline on Managed Service for Apache Airflow. Pipelines can also include Gemini Enterprise Agent Platform steps, like uploading a model or running batch inference.

In the IDE, you can follow each run on a visual pipeline canvas. If a task needs attention, click Diagnose to hand its logs to your agent. Troubleshooting skills for Airflow and Spark trace the root cause and propose a fix for you to approve.

Troubleshooting a failed Airflow DAG with an agent.

Improving the developer experience

GA also streamlines setup and day-to-day workflows. When you get started, you simply sign in once and select the Google Cloud services you use. Data Agent Kit automatically enables the required APIs, installs the matching skills, and configures your MCP servers with no manual setup files. Inside the IDE, the SQL editor and notebooks now support inline code generation, @ references to tables, and diff views for suggested changes. The extension also shares your active project, open file, and the error from the query you just ran with your agent, so asking it to "fix this query" just works.

We also made the core tools faster and more responsive. New Spark notebooks automatically create and select a Spark Connect runtime, and Spark SQL queries in the editor run in isolated sessions with built-in execution metrics. The catalog explorer now loads faster and includes BigQuery public datasets in the sidebar. Tuned notebook skills help your agent finish notebook tasks more quickly while using fewer tokens.

Ready for the enterprise

Data Agent Kit is included at no additional cost; you pay standard pricing only for the Google Cloud services your agent uses. Skills also steer the agent toward cost-aware query patterns, like checking partition keys and running a dry run before executing a BigQuery query to avoid accidental full-table scans. And because the skills are open source on GitHub, your team can audit them, fork them, or write custom skills for your own internal standards.

For access control, the agent connects as you or as a service account you impersonate, so row- and column-level security policies apply automatically. Admins can also govern MCP access with Identity and Access Management (IAM), screen MCP traffic with Model Armor, and scope agents with VPC Service Controls and Principal Access Boundary policies.

Install and get started

You can set up Data Agent Kit in under a minute in either your IDE or your terminal.

Option 1: Install the IDE Extension

IDE extension: Search for "Google Cloud Data Agent Kit" in the Extensions panel of VS Code, Antigravity IDE, Cursor, or any VS Code-compatible editor. You can also install it from the VS Code Marketplace or Open VSX.
Antigravity 2.0: Go to Settings > Customizations > Build with Google Plugins, then download Data Agent Kit.
Cloud Shell and Cloud Workstations: Already installed by default; just open the editor and sign in.

Option 2: Install the CLI Plugin

Run the command for your preferred coding agent using the official GoogleCloudPlatform/data-agent-kit-plugin repository:

code_block: <ListValue: [StructValue([('code', '# Antigravity CLI\r\nagy plugin install https://github.com/GoogleCloudPlatform/data-agent-kit-plugin\r\n\r\n# Claude Code\r\nclaude plugin install data-agent-kit-starter-pack@claude-plugins-official\r\n\r\n# Codex CLI\r\ncodex plugin marketplace add GoogleCloudPlatform/data-agent-kit-plugin\r\ncodex plugin add dak@dak-marketplace'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21cbe60590>)])]>

Then try a first prompt:

code_block: <ListValue: [StructValue([('code', "What are this week's fastest rising search terms in the US that weren't in the last week's top 10? Use the BigQuery public Google Trends dataset."), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d0e0d7d0>)])]>

Data Agent Kit plugin running in Claude Code.

Next steps & hands-on resources

Read the docs: Explore the Data Agent Kit documentation and product overview page.
Explore the skills: Browse, star, and contribute on GitHub.
Build an analytics workflow: Try the Analytics with Data Agent Kit and Antigravity IDE codelab, and read the companion blog, Agentic analytics with the Data Agent Kit.
Build a data science pipeline: Try the Fraud detection pipeline with Data Agent Kit and Antigravity IDE codelab.

Graph Workflows in ADK: Everything You Need to Know

Tue, 29 Sep 2026 16:00:00 +0000

Graph engineering is the design work: breaking a task into nodes, connecting them with edges, and deciding where code, models, or people control the next step. The Agent Development Kit (ADK)'s Workflow turns that design into an executable process, with functions and agents doing the work. Through a refund example, this post shows how to run steps in parallel, route decisions, pause for human review, and process a list of cases. It also explains when to declare the paths in a static graph and when to let Python schedule further work as results arrive.

TL;DR: Using a refund workflow in ADK, we'll cover fan-out and fan-in, deterministic and agent routers, human-in-the-loop pauses, parallel workers, and dynamic orchestration— along with when to use a static graph or let Python decide what runs next.

Start with a single agent

We can give one agent the tools and instructions to handle the refund request from start to finish:

code_block: <ListValue: [StructValue([('code', 'refund_agent = Agent(\r\n name="refund_agent", model=MODEL,\r\n tools=[fetch_order, fetch_payment, fetch_history],\r\n instruction="""You handle refund requests.\r\n 1. Look up the order.\r\n 2. Check the payment record.\r\n 3. Check the customer\'s refund history.\r\n 4. Deny if there\'s an open chargeback or it\'s past 30 days.\r\n Approve if it\'s under $50 and they\'ve had fewer than three\r\n refunds this year.\r\n 5. Write the customer an email explaining the decision.""",\r\n)'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f21d0e86350>)])]>

This puts the model in charge of choosing the tools, applying the policy, and writing the reply. But the prompt already describes distinct pieces of work: three lookups, a decision, and a response. Making those pieces separate nodes lets us decide how each should run.

Assume the customer has selected an order and clicked “Request refund.” The app knows the order ID, so the workflow can begin with the lookups.

Let independent steps run together

The order, payment, and refund-history lookups all need the order ID, but none needs another lookup's result. Although the prompt lists them one after another, there is no reason for them to wait for each other. We can run all three in parallel.

The policy decision is different: it needs all three records. So the workflow splits into three paths, then brings their results together before continuing. These two moves are called fan-out and fan-in.

In ADK, the lookup functions can become nodes directly. A nested tuple starts them together, and a JoinNode waits for their results. First, the imports and a lookup signature:

code_block: <ListValue: [StructValue([('code', 'import asyncio\r\nfrom pydantic import BaseModel, Field\r\n\r\nfrom google.adk import Agent, Context, Event, Workflow\r\nfrom google.adk.events import RequestInput\r\nfrom google.adk.workflow import JoinNode, START, node\r\n\r\nasync def fetch_order(node_input: str) -> dict:\r\n ...'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f21d0f84690>)])]>

Each lookup receives the order ID through node_input and returns a dictionary. We can connect them like this:

code_block: <ListValue: [StructValue([('code', 'join_case = JoinNode(name="join_case")\r\n\r\nedges=[\r\n (START, (fetch_order, fetch_payment, fetch_history), join_case),\r\n]'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f21d17eaad0>)])]>

Read this from left to right: start all three lookups, then continue through join_case once they finish.

The join returns a dictionary keyed by node name. The next node can read node_input["fetch_order"], node_input["fetch_payment"], and node_input["fetch_history"] without a model call to collect the results. The JoinNode guide explains the details.

If the payment lookup needed a transaction ID from the order lookup, those two would run in sequence. Dependencies determine the edges, even when the prompt lists every step in order.

You can learn more about fan out and fan in in this video:

Graph Engineering with ADK

One detail matters when turning tools into nodes: the parameter named node_input receives the previous node's output. Other names bind to ctx.state by default, so order_id would look for ctx.state["order_id"] and raise a ValueError if it is missing. Here, the str annotation also converts START's types.Content input to a string.

Route each request to the right workflow

So far, the customer has explicitly requested a refund. In a broader support conversation, we first need to identify what they want and send the request to the right process. “The shoes are the wrong size. Could you send me a different pair?” should go to an exchange workflow, while a request for money back should enter our refund workflow.

That introduces a router, a node that chooses which branch runs next.

A deterministic router follows explicit rules, the same inputs produce the same route.
A nondeterministic router can choose different branches for the same input.
An agent router uses a model to interpret the request, so its choice can vary.

Routers choose among the paths defined by the workflow. In our support example, we can use an agent to identify the customer's intent, then fixed rules to apply the refund policy.

Identify the intent with an agent router

An agent can interpret the customer's message and classify the request. In one ADK pattern, it returns a structured category, then a small function emits the corresponding Event(route=...) to send the request to the chosen workflow:

code_block: <ListValue: [StructValue([('code', 'Customer message → classification agent → route function\r\n ├─ REFUND → refund workflow\r\n ├─ EXCHANGE → exchange workflow\r\n └─ CLARIFY → ask a follow-up question'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d17eba90>)])]>

The model identifies the intent; the graph defines the available destinations. If the request is unclear, the workflow can ask a follow-up question. ADK's routing sample shows this pattern.

This intent router would sit before our refund workflow. For the selected-order example, the “Request refund” button has already established the intent, so we can enter that workflow directly.

Apply the refund policy with a deterministic router

Inside the refund workflow, the three lookups give us the facts for another decision: approve, deny, or ask a person to review. This time, the policy gives us explicit thresholds, so a function can choose the path:

code_block: <ListValue: [StructValue([('code', 'def refund_policy(amount, days_ago, chargeback_open, priors):\r\n if chargeback_open or days_ago > 30:\r\n return "DENY"\r\n if amount < 50 and priors < 3:\r\n return "AUTO_APPROVE"\r\n return "MANUAL_REVIEW"\r\n\r\ndef route_refund(node_input):\r\n case = {**node_input["fetch_order"], **node_input["fetch_payment"],\r\n **node_input["fetch_history"]}\r\n route = refund_policy(case["amount_usd"], case["placed_days_ago"],\r\n case["chargeback_open"], case["prior_refunds_12mo"])\r\n return Event(output=case, route=route) # the function names the path'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f21d17eb410>)])]>

route_refund combines the records and returns the case with one of three route names: AUTO_APPROVE, DENY, or MANUAL_REVIEW. Manual review handles cases that meet neither automatic rule.

This is a deterministic router: the same case data produces the same decision, and we can test the policy without calling a model.

	Deterministic router	Agent router
Decisions come best from	Rules in code	A model interpreting the input
Best fit	Known facts and explicit policy	Meaning that is hard to capture in rules
Example	Deny an order older than 30 days	Recognize an exchange request
Model call for routing	None	Required

Both routers choose among defined paths. An agent router can sit inside a static graph: the model's choice varies, while the possible connections stay the same.

For an automatic approval or denial, the next step is to explain the decision to the customer. We give each route a notice agent that writes the reply from the case data. ADK passes the dictionary in Event(output=case, ...) to that agent as a JSON user message:

code_block: <ListValue: [StructValue([('code', 'approve_notice = Agent(\r\n name="approve_notice", model=MODEL,\r\n instruction="Tell the customer their refund is approved and when to expect the "\r\n "money, using the case JSON you receive. Short email, warm, no fluff.",\r\n)\r\ndenial_notice = Agent(\r\n name="denial_notice", model=MODEL,\r\n instruction="Tell the customer their refund was declined and exactly why, based "\r\n "on the case JSON you receive. If it carries a reviewer_note, that is "\r\n "the reason. Short email, direct and kind. Do not invent policy.",\r\n)'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f21d17eb210>)])]>

The refund workflow now has a path from the initial lookups to a decision and a reply, with a third branch for cases that need a person:

code_block: <ListValue: [StructValue([('code', 'workflow = Workflow(\r\n name="refund_decision",\r\n edges=[\r\n (START, (fetch_order, fetch_payment, fetch_history),\r\n join_case, route_refund),\r\n (route_refund, {"AUTO_APPROVE": approve_notice,\r\n "MANUAL_REVIEW": escalate_to_human,\r\n "DENY": denial_notice}),\r\n ],\r\n)'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f21d17e8750>)])]>

The first chain fetches the records, joins them, and applies the policy. The second maps the router's decision to a destination. Automatic decisions go straight to a notice agent; MANUAL_REVIEW goes to the human-review node we'll define next.

That third branch needs more than another function call. A reviewer may take minutes or days to answer, so the workflow must pause with the case pending and continue once the person decides.

The review node yields RequestInput, which records the pending request and pauses the run. On resume, rerun_on_resume=True runs the node again, with the answer available in ctx.resume_inputs:

code_block: <ListValue: [StructValue([('code', 'REVIEW = "refund:review"\r\n\r\nclass ReviewDecision(BaseModel):\r\n approve: bool = Field(description="True to refund, False to decline.")\r\n note: str = Field("", description="Why, in the reviewer\'s words.")\r\n\r\n@node(rerun_on_resume=True)\r\nasync def escalate_to_human(ctx: Context, node_input: dict):\r\n answer = ctx.resume_inputs.get(REVIEW)\r\n if answer is None: # first pass: ask, then stop\r\n yield RequestInput(\r\n interrupt_id=REVIEW,\r\n message=f"Refund ${node_input[\'amount_usd\']} on order "\r\n f"{node_input[\'order_id\']}?",\r\n payload=node_input, # what the reviewer is shown\r\n response_schema=ReviewDecision,\r\n )\r\n return\r\n # second pass: the answer is here, so route on it\r\n yield Event(output={**node_input, "reviewer_note": answer.get("note", "")},\r\n route="AUTO_APPROVE" if answer["approve"] else "DENY")'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f21d17ebc50>)])]>

The response schema gives the node an approve value to route on and a note to carry forward. If the reviewer declines, the notice agent receives their reason with the case. One more edge connects the review decision to the reply:

code_block: <ListValue: [StructValue([('code', 'edges=[\r\n ...,\r\n (escalate_to_human, {"AUTO_APPROVE": approve_notice,\r\n "DENY": denial_notice}),\r\n]'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f21d17eb650>)])]>

ADK ships two variants of this. The one above is the single-node pattern: the node reruns and reads ctx.resume_inputs, as in the request_input_rerun sample. The request_input sample shows the two-node variant, where one node yields RequestInput and the reviewer's answer arrives as the next node's node_input — so there is no ctx.resume_inputs to find in that file. The companion scripts linked below include the code that sends the reviewer's answer back to the run.

A Workflow is also a node. This whole refund process can become one step in a larger customer-service workflow.

Apply the same step to a batch of cases

We now have a process for one refund. Suppose a batch of cases arrives with the records already collected. Each needs the same policy check, and the number of cases changes from batch to batch. We can apply one node to every item using a parallel worker.

In ADK, parallel_worker=True runs a node once per item in an input list and collects the results in the original order. We can reuse our policy function:

code_block: <ListValue: [StructValue([('code', '@node(parallel_worker=True)\r\ndef review_case(node_input):\r\n # Each worker receives one case from the input list.\r\n case = node_input\r\n decision = refund_policy(\r\n case["amount_usd"], case["placed_days_ago"],\r\n case["chargeback_open"], case["prior_refunds_12mo"],\r\n )\r\n return {"order_id": case["order_id"], "decision": decision}\r\n\r\ndef collect_decisions(node_input):\r\n # This node receives the list of worker results.\r\n return {"decisions": node_input}\r\n\r\nbatch_review = Workflow(\r\n name="batch_review",\r\n edges=[(START, review_case, collect_decisions)],\r\n)'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f21d17e9290>)])]>

Each worker receives one case, and collect_decisions receives the results as a list. No separate JoinNode is needed. The flag also works on agents—for example, to write an explanation for each case. The parallel_worker sample shows both forms.

The policy calculation here is small. Concurrency is more useful when each item waits on an API or model call, but the way inputs and results move stays the same.

Pattern	Work distributed	Collected output
Fan-out with `JoinNode`	Different notes doing independent jobs	Dictionary keyed by node name
Parallel worker	The same node for every item	List in input order

The batch size can change without changing the graph. A variable amount of work still fits inside a fixed process. See the parallel worker guide for execution details.

Let results shape the next step

Our refund process has known paths, even when a batch contains more cases or a reviewer takes longer to answer. But some work only becomes clear as we investigate.

Suppose a disputed refund reveals a second transaction. Checking it raises a delivery question that needs further investigation. Now each result can create follow-up work. A dynamic node can examine those results and schedule the next checks in Python.

ADK provides ctx.run_node to run another node and await its result. To see how that changes orchestration, let's first express our existing refund flow this way:

code_block: <ListValue: [StructValue([('code', 'HANDLERS = {\r\n "AUTO_APPROVE": approve_notice,\r\n "MANUAL_REVIEW": escalate_to_human,\r\n "DENY": denial_notice,\r\n}\r\n\r\n@node(rerun_on_resume=True)\r\nasync def refund_flow(ctx, node_input):\r\n # Step 1: fetch all three records at once.\r\n order, payment, history = await asyncio.gather(\r\n ctx.run_node(fetch_order, node_input, use_sub_branch=True),\r\n ctx.run_node(fetch_payment, node_input, use_sub_branch=True),\r\n ctx.run_node(fetch_history, node_input, use_sub_branch=True),\r\n )\r\n case = order | payment | history\r\n\r\n # Step 2: apply the refund policy — the same pure function, 0 LLM calls.\r\n decision = refund_policy(\r\n amount=case["amount_usd"],\r\n days_ago=case["placed_days_ago"],\r\n chargeback_open=case["chargeback_open"],\r\n priors=case["prior_refunds_12mo"],\r\n )\r\n\r\n # Step 3: run the chosen handler, and use its output as this node\'s output.\r\n await ctx.run_node(HANDLERS[decision], case, use_as_output=True)'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f21d17e9c10>)])]>

asyncio.gather runs the lookups together. Python combines their results, applies the policy, and runs the selected handler. use_as_output=True makes the handler's result the parent node's output without emitting it twice.

The dynamic version also adjusts the human-review node: after the answer arrives, it calls the chosen notice agent through ctx.run_node. The complete dynamic script (refund_dynamic.py) includes that variation.

The outer graph only needs an entry point:

code_block: <ListValue: [StructValue([('code', 'workflow = Workflow(\r\n name="refund_dynamic",\r\n edges=[(START, refund_flow)],\r\n)'), ('language', 'lang-py'), ('caption', <wagtail.rich_text.RichText object at 0x7f21d17e96d0>)])]>

In the static version, the edge list shows the branches and destinations. In this version, we read refund_flow to see them. The dynamic node guide covers this approach.

The human-review pause still works here. When the answer arrives, refund_flow runs again from the top, but completed ctx.run_node calls return their recorded outputs from session history. The lookups do not repeat, as direct function calls would. Keeping side effects inside child nodes lets completed calls replay their results when the parent resumes. Each child also gets its own trace span, and use_sub_branch=True keeps concurrent children's events on separate branches.

For this fixed refund process, the edge list remains easy to inspect. The Python version gives us a place to add the investigation logic described above: inspect a result, choose a follow-up node, and run independent checks together. A model might suggest what to investigate, while code limits the work—for example, three follow-ups per finding and two levels of investigation before human review.

Loops can still fit in a static graph. A fixed draft → check → revise process can use a conditional back-edge and a router that limits revisions. “Static” describes the possible connections; the actual path and iteration count can vary. The graph guide covers conditional cycles.

You can also place a dynamic node inside a static workflow, using Python for a stage that needs it while keeping the surrounding process visible.

Choose who decides what runs next

Start with a question: can you draw the possible workflow before the input arrives? Include branches, loops, and repeated stages. You do not need to predict the path each request will take.

What you need	Pattern
Independent jobs, then all their results	Fan-out and fan-in
A branch chosen by fixed rules	Deterministic router
A branch chosen by interpreting meaning	Agent router
A person's decision before continuing	`RequestInput`
The same step across a list	Parallel worker
Code that schedules work as results arrive	Dynamic node

Use an edge list when it makes those connections clear. Use dynamic orchestration when results create further work or Python expresses the control more naturally. A small, open-ended task may need only one agent and its tools.

In our refund workflow, the graph coordinates the lookups, code applies the policy, a person handles exceptions, and a model writes the reply. Graph engineering gives each a clear responsibility—and makes it easier to see how the process works.

Get started

Read: the workflow guide.
Explore: the workflow samples.
Build: the hands-on codelab, which applies these patterns to a marathon race-day coach.

Best practices guide for customizing Gemini models via Reinforcement Learning (RL)

Fri, 25 Sep 2026 16:00:00 +0000

Reinforcement learning (RL) has been a keystone of modern LLM post-training, but it demands large training clusters and access to model internals that external customers can't have with proprietary models like Gemini. So here at Google Cloud, we packaged it into a managed RL fine-tuning service (RLFT service) — you bring prompts and a reward function; we handle the infrastructure and the proprietary model internals.

Now, you can adapt Gemini with the service — teaching the model from a reward signal you define, rather than from a fixed set of labeled answers. This unlocks a class of problems that supervised fine-tuning (SFT) struggles with: tasks that are hard to demonstrate but easy to score.

In this guide, we will walk through practical best practices for using RL fine-tuning service. We'll start with a short tour of the RL training loop, how to decide if and when to use RL, and introduce how to get the most value from this approach.

What is RLFT?

RLFT adapts Gemini from a reward signal you define rather than labeled answers. Instead of authoring a large set of gold examples, you write one program that scores a response and the service improves the model against it — unlocking tasks that are hard to demonstrate but easy to verify: you can't hand-write the ideal SQL for every schema, but you can run the query and check the result.

At each training step the service generates multiple candidate responses to your prompts, scores them with your reward, and improves the model so that higher-scoring responses become more likely while it stays close to the original Gemini. The reinforcement learning that makes this work is fully managed — you never configure it. The one thing you own, and the thing that most determines your results, is the reward.

Three properties define what RLFT can and can't do:

It learns from the model's own outputs: It refines what the model already produces rather than copying an external target, so it tends to disturb unrelated capabilities less than SFT.
It rewards outcomes, not paths: Any response that reaches a good result earns reward, which fits open-ended tasks with many valid solutions.
It amplifies existing competence: It makes occasional success reliable, but it can't teach a skill the model never demonstrates.

When to use RLFT

Prompting and SFT handle most adaptation; exhaust them first. RLFT earns its keep when you can grade a response but can't cheaply author it, when SFT has plateaued on the metric that matters (faithfulness, schema validity, tone), or when the task has many equally valid answers a single reference target would wrongly penalize.

SFT and RLFT are complementary, not competing:

Direct RLFT when the base model already succeeds part of the time — enough for the reward to tell better answers from worse ones.
Two-stage SFT → RLFT when you have SFT data or the base success rate is too low for RL to gain traction. Use SFT as a short, cheap warm start — kept light, since over-fitting the demonstrations leaves less room for RL to improve — then continue into RL via Continuous Tuning, which initializes RL from the SFT checkpoint.

Across early adopters, these patterns show where RLFT delivers the most value — each scoring an outcome the business cares about but could never cheaply demonstrate.

Use cases for RLFT

AI-powered NPCs in games

What: In-character, on-brand dialogue held across long, multilingual, multi-turn conversations.
Problem: Off-the-shelf models break immersion — wrong language, hallucinated items, ignored players, repetitive loops.
Objective and reward: A Gemini autorater (LLM-as-a-judge) scores each turn on persona, flow, and game-state syntax, penalizing format and language errors.
Results: Loops and language drift disappeared and state syntax held, making shippable in-game characters viable at scale.

Structured entity extraction

What: Pulling a set of items from unstructured documents, such as supplier invoices and shipping manifests, into structured records automatically.
Problem: The long tail where SFT plateaus — missing required fields (recall) or inventing ones that aren't there (precision).
Objective and reward: A rule-based precision/recall reward forces every field to be grounded in the source text, not imitated from one gold answer.
Results: Field-level accuracy rose on noisy real-world documents where tuning had stalled, turning a manual review step into an automated one.

Content moderation

What: Applying intricate policies and decision trees at scale.
Problem: Models hallucinate false positives or reward-hack with invalid formats to dodge evaluation.
Objective and reward: A Cloud Run reward pairs format validation with a deterministic grader to enforce multi-step policy adherence.
Results: The model handled complex exemption carve-outs, sharply cut false positives, and stopped reward hacking — reducing the human-escalation volume that makes moderation expensive.

Code measured by execution

What: SQL or API calls graded on whether they actually run against customer data.
Problem: SFT mimics one reference query and breaks on unseen proprietary schemas.
Objective & Reward: A code-execution reward runs the code in a secure sandbox and pays out only if it compiles, executes, and returns the correct result.
Results: The model produced first-attempt executable queries at closed-frontier quality and lower inference cost, letting non-technical users query proprietary data in natural language.

Presentation slide generation via HTML

What: Multi-slide decks authored as HTML/CSS.
Problem: Training on text alone is blind to visual quality — overflows, clipped elements, and inconsistent styling slip through unnoticed.
Objective & Reward: A code-execution reward renders the slides and scores visual design, layout integrity, structural completeness, and rubric adherence.
Results: The model emitted modular, well-styled decks with cohesive themes and no layout overflow.

Where to start?

A dataset. A diverse set of prompts with a held-out validation split is enough for a first run — confirm the loop converges and reward moves the right way, then scale. Keep train and eval strictly separated; a contaminated eval hides overfitting.
A reward function. Your task specification as code or configs, and the dominant driver of quality. A good reward correlates with human preference, is robust to malformed output (catch the failed parse and return a clearly negative score rather than crashing), and resists reward hacking — ensemble judges, penalize length, floor degenerate outputs, and prefer a verifiable check over a model's opinion. Validate it offline before launch.

The service handles the rest; start from the defaults, watch reward and eval curves in the console, and take the checkpoint where validation reward saturates rather than the last step.

Get started today

What will you build? The tools are ready and waiting.

Documentation: Reinforcement Learning Fine-Tuning

Agent Factory recap: Agent harnesses, shifting left, and autonomous coding

Thu, 24 Sep 2026 23:00:00 +0000

In this episode of The Agent Factory, we explore the reality of building with autonomous agents alongside Ryan Lopopolo, a software engineer at Google Cloud and the person who coined the term agent harness. From throwing out manual code editors to treating team collaboration like leveling up RPG stats, Ryan breaks down how grounding models in rich context and shifting interventions left unlocks high levels of agent autonomy.

This post guides you through the key ideas from our conversation. Use it to quickly recap topics or dive deeper into specific segments with links and timestamps.

The Agent Harness - What is it?

Timestamp: [00:30]

An AI agent as we're defining it here is a large language model (LLM) plus an agent harness.

Think of the harness as everything wrapped around the LLM that isn't the model itself. For example, if you're working in Google Antigravity using Gemini 3.8 Flash, Gemini Flash is the LLM and Google Antigravity is the harness.

While an unassisted model can answer simple questions out of the box, it can't check live conditions or interact with your workspace on its own. When a user asks a question like "Why is the sky blue?", an unassisted LLM can respond without issue. However, when asked a question like "Should I wear a raincoat today?", the model can't answer on its own because it lacks the necessary data. The harness catches the intent, queries live weather tools, bundles that context back into the prompt, and hands it to the model to produce an informed answer.

Ryan Lopopolo on agent harnesses and autonomous coding

Tilde Thurium sat down with Ryan Lopopolo to discuss what it takes to run fully autonomous coding workflows in production. See the summary below!

Coining the harness and writing zero production code

Timestamp: [02:22]

The term agent harness grew out of Ryan's extensive work on autonomous coding agents, culminating in a February 2026 essay on leveraging coding models in an agent-first world. Ryan shared that he hasn't opened a traditional code editor since May of last year, maintaining that streak through his transition into Google Cloud. In this paradigm, engineers no longer author or review individual lines of syntax; instead, they operate at the level of natural language specifications and inspect the final artifacts, such as pull requests, documents, and spreadsheets. Then they determine whether the end result meets organizational standards.

Harness engineering is the study and the practice of putting a model into an environment where it can succeed. If you don't do that work, you end up doing what I call 'prompt and pray'. Ryan Lopopolo

Context curation and lazy prompting

Timestamp: [03:35]

Upfront harness investment pays off by allowing engineers to become lazy prompters. When the repository contains structured documentation, clear interfaces, and discoverable tools, you do not need to paste walls of text into a prompt box every morning

"I aspire to be an incredibly lazy prompter. If I have done the job to give the model the tools and context it needs to ground itself, I don't need to write a long prompt. It figures it out."

The model uses its harness to pull relevant context, allowing it to navigate large codebases and execute complex tasks without oversight.

Shifting left: engineering best practices as autonomous guardrails

Timestamp: [05:00]

When an agent fails, developers face a whole spectrum of interventions. The most common reflex is to fiddle with the prompt or retry, but that never scales across a team.

"The simplest, smooth-brain, stupidest intervention I can think of is literally just: try my prompt again without changing anything else. But shifting left means moving interventions earlier into the development lifecycle where they are cheapest and automated: from prompts, to repo docs, to linters, to tests, and all the way to upstream evals."

Instead of hoping the model guesses right on the next turn, shifting left embeds standards directly into the environment. Linters, tests, and AGENTS.md files act as durable memory and enforcement of what you think good looks like, making sure the agent stays on the rails without needing constant hand-holding.

Leveraging established tools and determinism

Timestamp: [06:50]

Agents shine when they're handed tools that already mirror patterns heavily represented in pre-training data. Pairing models with standard command-line interfaces moves reasoning into determinism, shifting the burden of context aggregation away from the model and onto reliable tools.

Ryan also shared an environmental design trick for context efficiency: structuring markdown files so link anchors sit directly beneath their corresponding prose blocks rather than inline. This prevents context clutter and mitigates "lost in the middle" retrieval issues. Because Ryan operates exclusively by reviewing end-state artifacts, keeping documentation readable allows him to easily inspect execution runs:

"I want to be able to look at the pull request and review it. If it made a bad decision, I need to know where it went off the rails so I can whack the agent on the head and make sure it does not make that same mistake again."

Long horizons and expanding the agentic loop

Timestamp: [09:45]

The central challenge of harness engineering is ensuring that agents cohere over long time horizons. Because human organizations produce software through iterative refinement rather than single-shot prompts, agent workflows must mirror that cadence. Harness engineering uses tightly scoped, reviewable pull requests to narrow the agent's state space. Stacking these high-confidence changes end-to-end allows supervisors to gradually expand the loop size, building trust until agents can autonomously execute large-scale initiatives, including entire language migrations

Curating agent teams like RPG stats

Timestamp: [11:58]

Rather than divvying up sprint tasks based on individual specialties, having a diverse team contribute to an agent turns it into a central producer of work that carries everyone's strengths. Ryan compared leveling up an agent's capabilities to building out a character sheet:

"[It's like] building out the stats of your RPG character. I get a new person on the team who is a React architect and boom! The attention that they pay is able to bump out the stats in front-end architecture and performance."

With that collective expertise baked into the environment, the agent can autonomously classify incoming work and activate the exact skills it needs on demand, operating as both a backend architect and a front-end specialist.

Accruing leverage in tools, not custom harnesses

Timestamp: [14:38]

For developers wondering whether to build their own custom agent harness, Ryan offered clear advice: don't build one from scratch. Standard harnesses already provide the foundational primitives: file reading, grep search, and command execution. Over-scaffolding an agent with rigid, bespoke frameworks creates technical debt and leads to sunk-cost traps when frontier models advance.

"If you focus all of your efforts on improving quality on tools and context, you can freely adopt the newest models as they come out and you'll be constantly accruing leverage into a bit of the system that will never become obsolete."

Operating Google Cloud and eliminating capability overhang

Timestamp: [16:47]

Discussing his work at Google Cloud, Ryan outlined his motivation to eliminate capability overhang: the delta between what frontier AI models are theoretically capable of and how much useful work is currently extracted in production. Because the cloud functions as a massive, programmable surface, equipping agents with direct interfaces to Google Cloud lets them manage and deploy infrastructure effectively, turning raw model capability into tangible enterprise utility.

Continually updating your priors on AI

Timestamp: [18:13]

The speed of AI development requires engineers and teams to actively unlearn old limitations and constantly reassess what these models can achieve. "It's very important to continually be updating what you think is possible with these lovely tools that we have," Ryan urged. What broke six months ago often runs effortlessly on today's frontier models. Rather than getting locked into rigid workflows, developers should build around the two highly extensible interfaces that will remain relevant across every model upgrade: tools and context.

"Agents will always need context in order to do that last mile adaptation into what you think good is. And as you can continue to... shift it to the left, in terms of increasingly capable tools which act as a form of memory and enforcement of what you think good looks like, you'll continually be amazed as the models are able to do more and more interesting things for you over time."

How To Build A Custom Harness

Next, Billy Jacobson started us off by showing how developers can customize their own agent harnesses for specific tasks.

Under the hood: Why build a custom harness?

Timestamp: [19:44]

Before jumping into code, Billy unpacked why developers should understand the mechanics of a harness rather than treating it like a black box. Recalling advice from an engineering mentor that "You can just use the framework, but a great engineer will really understand the framework", Billy explained that building a harness yourself is the best way to debug what happens when an agent breaks. You can evaluate three core design decisions for every workflow:

Looping: How many iterations should the agent run, and what conditions trigger an exit state?
Tools: What specific tools should the agent access, and when and how should it invoke them?
Memory: How important is conversational and operational memory, and when should it be retrieved or compacted?

Linear Agent Harness: Deterministic Single-Pass Execution

Timestamp: [21:34]

Billy demonstrated a minimalist linear harness designed for deterministic workflows where looping is unnecessary. This pattern is ideal for targeted inspections, file transformations, or single-turn data analyses where you want a high level of determinism and need the agent to perform the exact same execution flow every single time.

Closed-Loop Agent Harness: Iterative Test-Driven Repair

Timestamp: [22:45]

When tasks demand active bug fixing and refactoring, a closed-loop harness provides the iterative reasoning required to reach a verified resolution.

In this demo, Billy showcased an agent that applies an automated code edit to address a failing requirement, and the harness executes the unit test suite against the updated codebase. If the tests fail, the runtime captures standard failure logs and detailed stack traces, feeding those error diagnostics directly back into the agent's working memory. The process repeats continuously until all unit tests pass, backed by a five-iteration ceiling to prevent infinite loops and runaway execution costs.

Guardrail Harness with Google's Agent Development Kit (ADK)

Timestamp: [23:25]

For developers who require custom behavior without rewriting core orchestration plumbing from scratch, Google's Agent Development Kit (ADK) provides scaffolding with automated memory management and execution safeguards.

Billy walked through an example that leverages ADK's native context compaction to summarize older conversational turns, preventing context window bloat during extended debugging runs. Custom interception hooks inspect and filter shell actions before execution, automatically stopping high-risk operations such as recursive file deletions, database drops, or unauthorized remote git pushes. This architecture gives teams fine-grained control over tool execution boundaries while avoiding the maintenance burden of bespoke harness frameworks.

The 3-Layer Agent Dev Stack: Gemini 3.8 Flash, Google Antigravity, and Google Skills

Timestamp: [25:27]

Next up, Smitha Kolan broke down why coding agents do not always require heavier reasoning models, emphasizing that high performance stems from balancing the three layers of the agent stack: Model, Harness, and Knowledge.

"Your coding agent doesn't need a smarter model. It needs a better stack: model, harness, and knowledge. When all three click into place, everything changes."

She then walked through the three tools she's been loving recently, one for each layer of the stack.

Layer 1 | Model | Gemini 3.8 Flash: High-frequency agentic loops run between 20 and 60 sequential hops per task (inspecting files, updating functions, and executing unit tests). Because latency and API costs compound across iterations, a lightweight, responsive model like Gemini 3.8 Flash makes real-time agent loops practical without running up a massive bill.

Layer 2 | Harness | Google Antigravity with /boost: Default Antigravity handles standard navigation and component creation. On top of that, the /boost command spins up an orchestrator that coordinates specialized sub-agents in parallel and concludes with an independent audit pass before modifying files.

Layer 3 | Knowledge | Google Skills Repository: With over 19,000 GitHub stars and 100+ curated domain packages across Google Cloud, Firebase, Flutter, and Maps, this harness-agnostic repository injects precise domain context on demand, preventing agents from guessing cloud configurations

Your turn to build

Building effective coding agents requires moving past the reflex of simply swapping in larger models. As Ryan Lopopolo's philosophy of harness engineering illustrates, true developer leverage is achieved by shifting best practices to the left and investing in rich tools, deterministic verifiers, and well-curated context that survive model upgrades. When combined with fast inference models, structured orchestration harnesses, and modular domain knowledge, agents evolve from conversational novelties into dependable, autonomous engineering partners.

Ready to put it into practice? Explore the tools and resources covered in this episode:

Gemini 3.8 Flash: Fast, low-cost model for multi-hop agent loops.
Google Antigravity (/boost): Orchestrator harness for parallel coding agents and verification.
Google Skills: Modular domain knowledge for Google Cloud, Firebase, and Flutter.
Agent Development Kit (ADK): Custom harness middleware for safety guardrails and memory compaction.
Full episode video: The full interview and live Factory Floor code demos.

Connect with us

Smitha Kolan → LinkedIn | YouTube | X
Luke Schlangen → LinkedIn | website
Ryan Lopopolo → LinkedIn | Hyperbola | X
Tilde Thurium → LinkedIn
Billy Jacobson → LinkedIn | X
Mollie Pettit → LinkedIn | dev.to | X | Bluesky

How Google Cloud Networking Supports Your Fluid Compute Choices for AI Workloads

Thu, 24 Sep 2026 13:00:00 +0000

The availability of resources for AI workloads can be challenging across the industry, especially accelerators. This can slow your AI workload deployment if it’s built around a specific type of accelerator. The concept of fluid compute allows you to design your AI deployment with several options based on available resources that can fit your use case.

In this blog, we will explore how Google Cloud networking supports your AI workloads and considerations that are relevant to your choice of accelerator (GPU or TPU), as the backend networking component configuration is not exactly the same.

The resource options

After deciding the type of work you want to achieve with your AI deployment, another important component is the actual hardware to get this done. In this case, we want to run inference for a private LLM, and the target is the NVIDIA B200 GPU family which is available in the A4 VMs (a4-highgpu-8g).

Now we have identified what we want to get done and a possible compute option, but the challenge is: is this available?

To get access to resources, there are several options which include:

Dynamic Workload Scheduler (Flex-start VM): Queues workloads until all required accelerator nodes are available at the same time, provisioning them together and running non-preemptibly for up to seven days.
Dynamic Workload Scheduler (calendar mode): Enables reserving accelerator capacity 1 to 90 days in advance with guaranteed start and end times, ideal for scheduled pre-training runs and benchmarking.
Future reservations: Guarantees access to committed hardware in a specified zone beginning at a specific future date.
Flex reservations: Offers short-term commitment windows to secure scarce accelerator nodes without multi-year lock-in.
Dynamic node auto-provisioning and ComputeClasses: In Google Kubernetes Engine (GKE), defining multi-family fallback lists within ComputeClasses allows the cluster to automatically attempt provisioning alternative accelerator types if primary pools face regional constraints.
Spot VMs: Delivers surplus compute at substantial discounts for fault-tolerant, checkpointed batch jobs.

Networking your choices

The networking component of the accelerator varies based on your choice, so let's explore four configurations: standard networking, accelerated GPU networking (TCPX/TCPXO and RoCEv2), TPU networking, and Cloud Run.

Standard networking

Supported accelerators: NVIDIA T4 (N1 series), NVIDIA L4 (G2 series), NVIDIA A100 (A2 machine series single-node and multi-node), Cloud TPU v3, and Cloud TPU v5e (single-host/standalone slices).
Architecture: Nodes communicate over the primary Virtual Private Cloud (VPC) network using the Google Virtual NIC (gVNIC) over standard TCP/IP.
Workload fit: Provides straightforward portability across Google Cloud compute environments, supporting distributed data preprocessing, decoupled pipeline stages, independent inference replicas, and computer vision workloads using standard VPC routing and network policies.

Accelerated GPU Networking (TCPX/TCPXO and RoCEv2)

Distributed training and multi-node inference require specialized multi-rail network fabrics to handle massive parameter exchanges and collective communications.

GPUDirect-TCPX and TCPXO Fabrics

Supported accelerators: NVIDIA H100 (A3 High VMs with 4 rails) and NVIDIA H100 Mega (A3 Mega VMs with 8 rails).
Architecture: Uses custom GPUDirect-TCPX (4 dedicated VPCs) and GPUDirect-TCPXO (8 dedicated VPCs) offload engines to achieve high-throughput multi-rail GPU communication over standard Ethernet infrastructure without requiring native RDMA hardware.
Deployment blueprints: These multi-VPC topologies can be deployed in many ways including using pre-built blueprints from the Cluster Toolkit.

RoCEv2 Fabrics (VM and Bare Metal)

Supported accelerators: NVIDIA H200 (A3 Ultra VMs), NVIDIA B200 (A4 VMs), NVIDIA GB200 NVL72 (A4X VMs), and NVIDIA GB300 (A4X Max Bare Metal).
Zonal network profiles: RoCEv2 operates over a dedicated RDMA VPC attached to a specialized zonal network profile: VM instances (A3 Ultra, A4, A4X) use the ZONE-vpc-roce profile, while Bare Metal instances (such as A4X Max) utilize the dedicated ZONE-vpc-roce-metal bare-metal profile.
Rail-aligned fabrics: This dedicated VPC is isolated strictly for GPU communication and contains subnets mapped directly to the accelerator NICs. The backend is rail-aligned, with support for Jumbo Frames (MTU 8896), delivering non-blocking multi-terabit bandwidth with minimal cross-rail interference.
Automated plumbing with GKE Dynamic Resource Allocation Network (DRANET): When deploying these GPUs on GKE, the GKE managed DRANET can be used to automatically provision additional networks and assign drivers that map the RDMA network interfaces to the GPU. These can then be assigned and consumed directly in your workload pods using standard Kubernetes resource claims.
Turnkey deployment: You can deploy this entire end-to-end stack—including RDMA VPCs, MTU tuning, and DRA drivers—using automated blueprints from the Cluster Toolkit.

TPU Networking

Supported accelerators: Cloud TPU v4, Cloud TPU v5p, Cloud TPU v5e (multi-host Pod slices), Cloud TPU v6e (Trillium), and TPU7x (Ironwood).
Inter-chip interconnect (ICI): Inside a TPU Pod or slice, chips communicate directly over dedicated, ultra-low-latency optical links organized in 2D or 3D torus meshes, bypassing traditional network stacks entirely.
Optical circuit switches (OCS): In TPU v4 and TPU v5p SuperPods, software-reconfigurable OCS units dynamically change physical network topologies, route around faulty trays, and provision custom-sized accelerator slices without manual recabling.
Multi-NIC architecture (TPU v6e and Higher): While earlier TPU generations relied on ICI within a slice and single-NIC for host traffic, Cloud TPU v6e (Trillium) and TPU7x introduce a native multi-NIC architecture where worker nodes isolate standard Kubernetes management traffic onto a primary VPC while using secondary dedicated VPCs configured for high-throughput TPU data and cross-slice communication.
DRANET for TPU deployments: When deploying these TPUs on GKE, the GKE managed DRANET can be used to automatically provision additional networks and assign drivers for TPU communication. These can then be assigned and consumed directly in your workload pods using standard Kubernetes resource claims.
Data-center network (DCN) Multislice: For models scaling beyond an individual TPU slice, Cloud TPU Multislice connects multiple independent ICI meshes over Google's high-speed Jupiter Data Center Network utilizing these dedicated multi-NIC paths.

Cloud Run

Supported accelerators: NVIDIA L4 (G2 series) and NVIDIA RTX PRO 6000 (Blackwell) on Cloud Run GPU services.
Direct VPC egress: Binds serverless containers directly to your private VPC network using sub-minute IP allocation via Direct VPC Egress, enabling secure, low-latency access to internal data lakes, databases, and private APIs without traversing the public internet or requiring legacy connector VMs.

Summary

Google Cloud networking options support various accelerator types. When using fluid compute you can adjust your network setup to support the best design to optimise your workloads performance.

Next Steps

Take a deeper dive into Google Cloud AI infrastructure and networking architectures with these resources:

Want to ask a question, find out more, or share a thought? Please connect with me on LinkedIn.

A guide to speeding up your video processing with AlphaEvolve

Wed, 23 Sep 2026 16:00:00 +0000

In real-time streaming, every millisecond counts.

For example, at 30 frames per second (fps), developers have a strict frame budget of just 33.3 ms (and only 16.6 ms at 60 fps) to ingest camera frames, run neural segmentation, apply shaders, and composite output. Exceeding that budget by even a fraction of a millisecond leads to dropped frames and stuttering.

Manual optimization is notoriously tedious — requiring weeks of analyzing flame graphs and hand-tuning low-level code in Swift, C++, or Metal. While standard AI coding assistants can generate boilerplate, they can’t optimize against target hardware, benchmark real-world latency, or ensure optimizations preserve visual fidelity.

Autonomous, closed-loop evolutionary optimization changes this paradigm. Tools like AlphaEvolve pair cloud-scale model reasoning with local hardware execution, and we’re already seeing real-world impact. In partnership with Google, DoIt used AlphaEvolve to autonomously optimize production Swift code in a live macOS streaming app, uncovering performance headroom that manual profiling missed (read the full technical writeup).

While this post focuses on video pipelines, the split-loop pattern applies anywhere performance matters — from microservice throughput and database queries to ML tensor pipelines and embedded systems. In every case, the formula is the same: pair Gemini code generation in the cloud with your domain-specific benchmark harness and automated quality gates.

Today, we’ll show you how to use AlphaEvolve to speed up video processing—and apply these principles to your own performance bottlenecks:

Understanding the split-loop architecture: How AlphaEvolve decouples managed cloud generation (Gemini model ensemble on Google Cloud) from local evaluation (e.g. compiling and timing native Swift/Metal code).
Evaluator craft and quality gates: How to construct scoring functions using metrics like Structural Similarity Index (SSIM) to prevent evolutionary loops from gaming the benchmark (e.g., skipping rendering entirely to go fast).
Autonomous algorithmic discovery: How Gemini-driven evolutionary search can autonomously discover unprompted framework APIs and make intelligent engineering trade-offs (e.g., frame-caching limits).
Setting realistic performance boundaries: How to measure code optimization against physical hardware floors.

1. Understanding AlphaEvolve’s split-loop architecture

AlphaEvolve runs a closed-loop evolutionary process: given a seed program and a custom scoring function, a mixture of Gemini models proposes code variations, executes the scoring function against each candidate, keeps the highest-performing code, and iteratively climbs toward an optimal solution over multiple generations.

A core architectural advantage of AlphaEvolve is its clean separation into two halves:

The generation half (Google Cloud managed service): Contains the prompt sampler, Gemini model ensemble, and program database. Google Cloud handles the scale, prompt orchestration, and generation mechanics.
The evaluation half (customer managed compute): Scoring code quality is strictly domain-specific. You own the evaluator module entirely, running it on your own hardware or target architecture (in this case, macOS running native Swift code).

While AlphaEvolve is Python-first on the cloud generation side, evaluation can be written in any language. The custom evaluator compiles each Swift candidate using swift and executes it against a standard reference webcam clip.

2. Evaluator craft and quality gates

An automated optimization loop like AlphaEvolve never actually "sees" your video stream. It only sees the numeric fitness score your evaluator returns. If your evaluation metric has a blind spot, evolutionary code generation will aggressively exploit it.

In our early runs, a naive fitness score weighted toward raw latency produced an astonishing speedup: the model simply bypassed blur rendering entirely and returned unmodified frames in 0 ms.

Structural Similarity Index Measure (SSIM):

To prevent the model from gaming your benchmark, try building a two-tiered scoring function that pairs throughput with structural fidelity metrics like Structural Similarity Index (SSIM):

code_block: <ListValue: [StructValue([('code', 'speedup = baseline_ms_per_frame / candidate_ms_per_frame\r\nssim = mean_ssim_vs_golden\r\n\r\n#Disqualify any candidate falling below visual threshold\r\n\r\n\r\nif ssim < 0.98 or worst_frame_ssim < 0.95:\r\n return {"speedup": -1e12} # Disqualified\r\n\r\nreturn {"speedup": speedup, "ssim": ssim}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d1c7b990>)])]>

What does this give you?

The ability to test against worst-case clips: Never benchmark on static frames or blank cameras. Candidate code can easily pass an average SSIM gate on static backgrounds while failing completely during quick head turns.
You can track the minimum, not just the mean: Enforce both an average threshold and a per-frame floor to catch dropped frames or delayed mask updates.

Autonomous algorithmic discovery:

Most developers use generative AI for local micro-optimizations (e.g., inlining helper functions, unrolling loops, or tweaking memory pools). But when given architectural room, the evolutionary loop can discover systemic optimizations on its own.

Engineering lessons:

Provide framework context, not isolated loops: Include public SDK headers, interface definitions, or API reference symbols in the prompt or retrieval harness. An LLM cannot adopt a sequence-aware subsystem if its context window only contains an isolated frame-processing callback.
Expose multi-frame lifecycle hooks: Let your candidate code maintain a bounded state across executions (e.g., historical masks or cache timestamps) rather than enforcing pure, stateless functions.
Let quality gates police the trade-offs: When AlphaEvolve introduced temporal mask caching, it initially cached masks too aggressively, causing noticeable trailing artifacts. Because our SSIM gate penalized drift during motion, the search converged on a production-ready cache window without manual parameter tuning.

Setting realistic performance boundaries

A common pitfall in performance engineering is optimizing in the dark. If you achieve a 2x speedup, is that an incredible achievement, or did you leave another 3x on the table?

In real-time media, total frame time splits into two distinct categories:

Mutable software overhead: Memory allocations, buffer format conversions, thread context switches, and API dispatch friction.
Immutable hardware floors: Raw Neural Engine inference latency, GPU shader compute time, and hardware display synchronization.

To make the most of AlphaEvolve, developers should measure against theoretical maximum headroom

Before running optimization loops, here’s a few principles to keep in mind:

Build a "no-op" pipeline: Strip out Swift/C++ orchestration, data marshalling, and frame conversions. Dispatch only the pre-warmed ML model and bare GPU pass on a dummy buffer. The resulting time is your physical hardware lower bound.
Calculate your addressable ceiling: Your total possible optimization potential is:

3. Score against the hardware gap: Instead of arbitrary speedup multiples, measure optimization efficiency:

Get started

All benchmark code, test clips, evaluation scripts, and raw candidate logs are open source:

GitHub repository: AlphaEvolve Camera Background Blur Example
Detailed technical write-up of our case study with DoIt: Running AlphaEvolve on Your Own Code

The DevFest Community Workshop Experience: Building Real Agents Together

Fri, 18 Sep 2026 16:00:00 +0000

This week we kicked off the DevFest season in North America at Google Hudson Square in New York City with 80 engineers packed into the room. Typical technical workshops hand you a finished repo, tell you to blindly paste blocks of code into your terminal, and hope nothing crashes. You walk away with green checkmarks, but your brain stays on autopilot.

We've introduced a completely different experience called Workbench.

Workbench focuses on understanding core ideas and architectural models rather than obsessing over syntax and code snippets. Instead of getting bogged down in boilerplate, engineers spent the day grappling with the actual mental models behind graph engineering, self-evolving architectures, and automated self-patching harnesses.

A glimpse into the Workshop Experience

At the DevFest Community Workshop, we spent one intense day building long-running, self-evolving multi-agent systems powered by Google's agentic stack. Ricky Robinett, Senior Director of Developer Marketing, kicked off the day by diagnosing why so many engineering teams hit a wall with agents. Ricky broke down why prompt engineering fails as a safety mechanism: English is just a probabilistic suggestion, not an execution boundary.

Right after Ricky, Rachel Francois, Google Developer Groups (GDG) North America Program Lead, took the stage alongside GDG Brooklyn organizers to welcome the community and spotlight the power of local developer chapters. They set the tone for the entire day, reminding everyone that building durable software works best as a team sport where engineers share real-world patterns and build local networks that outlast any single framework.

Getting hands on with labs

Annie Wang & Christina Lin, Americas DevRel Team members, led the morning lab that put those runtime ideas to work. Attendees explored Google's Agent Development Kit (ADK), Veo 3.1, Memory Bank on Gemini Enterprise Agent Platform, and RAG Engine on Gemini Enterprise Agent Platform. Through Workbench, developers grasped the principle of separating state from active compute for long running tasks. Workflows paused cleanly mid-execution, waited out asynchronous human approvals, and resumed without running up idle compute costs.

After lunch, Logan Hennessy, Americas Developer Relations Engineer (DRE), and Kartik Derasari, Google Developer Expert (GDE), led a lab using auction history as insight for better bidding strategy. Attendees worked through the architecture by integrating BigQuery data into autonomous data engineering pipelines, reasoning about deterministic bidding logic and adding eval-gated, self-patching harnesses that catch spend anomalies and update runtime execution safely.

Between lab blocks, we ran fast-paced speed quizzes where developers raced to lock in their answers as quickly as possible. Screens flashed, fingers flew across keyboards, and seconds made the difference between topping the leaderboard or dropping five spots. Nothing beats watching a room full of serious engineers completely lose their cool over a live quiz leaderboard.

Join a DevFest Community Workshop this fall

New York was only round one. We are taking this exact experience on tour to five more cities this fall. Find your city and grab your seat before spots fill up:

Sunnyvale on September 30
Washington DC on October 6
Atlanta on October 30 (as a part of DevFest Atlanta)
Seattle on November 4
Boston on November 10

Best practices for handling cloud reliability incidents

Tue, 15 Sep 2026 16:00:00 +0000

Cloud outages can range from global service disruptions to issues isolated to a specific region, zone, or even just your project, workload or application. If you suspect a Google Cloud Platform outage is impacting your services, we recommend you follow a structured “Verify→ Investigate→Report→Resolve→Review" workflow to resolve it. And before that outage occurs, you should also have prepared your environment for an eventual disruption by designing for failure, and actively practicing the steps you need to take to restore service.

In this blog, we summarize the key reliability incident handling best practices to help you design and practice your reliability incident response capabilities and minimize impact. Rather than an exhaustive guide, this is meant as a primer on only the most important practices for advisory purposes. Please note that we do not cover additional practices specific to security incidents here.

Beyond the base steps covered here, you may want to also explore how AI agents and tools are starting to transform incident handling. Check out this episode of the Prodcast, where Googlers explore the latest trends of leveraging agentic AI in Site Reliability Engineering (SRE) to detect issues early and prevent disruptions. Try Cloud Assist investigations, or explore Agent Skills and remote managed MCP servers to give you another set of tools for quickly pinpointing an issue. Before getting into these advanced techniques, we focus below on the foundational steps to good incident handling.

1. Prepare

Long before things start to go sideways, you should have spent significant time preparing for an outage along at least four dimensions: design, data, playbooks and training.

Design: Think ahead and mitigate future incidents by designing automated response actions, like a load balancer shifting traffic away from slow or unresponsive instances, or by automating as much of your incident response playbook as possible. Review designs of all critical applications to automate as many actions as possible to accelerate response and recovery.
Data: When a disruption occurs, having meaningful data at your fingertips vastly improves response capabilities. Use Cloud Logging, Cloud Trace and Cloud Monitoring, or other third-party observability tools, and replicate that data to a redundant stack in a separate location from the systems being observed. Make sure, in advance of any incident, that time stamps are synced across your observability streams for easy correlation, or know how to do that on-demand during an outage, when time is of the essence.
Playbook: A well-thought-out playbook documenting your incident response processes, including crystal clear role and responsibility definitions for all personas, is paramount to efficient incident response. Who is responsible to do what? Who needs to be notified or mobilized for each type of disruption? How can they be reached? What tools and data are available? How are results communicated? How do teams hand over to the next shift during long running incidents? etc. Conduct a simulated incident response and critically review every step to find where your playbook needs clarification. Without clear responsibilities, mitigation inevitably takes longer.
Training: Hopefully, service disruptions are rare events. To ensure your staff knows and remembers how to react, they need to retrain on the process several times per year by running simulated cross-team incident response drills. A retrospective on the simulated exercise will help identify warranted improvements.

2. Verify

Despite your best efforts, sooner or later, a service disruption will occur, which you can detect via any number of mechanisms:

Observability tools (Google tools or third-party tools)
Unified Maintenance Management notifications for planned maintenance
Personalized Service Health notifications managed with alert policies
Proactive customer monitoring by Google

Now, you need to determine what broke and who should ultimately fix the problem:

Google, e.g., a bug, code roll-out, hardware failure, etc.
You, e.g., a configuration change, elevated load, quota ceiling, etc.
Third party, e.g., a directory hosted by a different cloud provider

If Google has declared an incident and started working to fix the problem, estimate whether you can possibly reestablish service sooner, for example by failing over to a secondary stack (see the ‘Typical Causes’ table below). You can determine whether Google has declared an incident and will provide a fix by consulting:

Personalized Service Health: Check this first. Personalized Service Health shows incidents specifically relevant to your projects and regions, distinguishing between incident types:.
- Emerging Incidents: Google has received an alert, on-callers are investigating, impact is yet unknown
- Confirmed Incidents: Google has investigated and found customers are impacted

Located within the Google Cloud console, Personalized Service Health often displays limited-scope incidents that don't appear on the public dashboard. Personalized Service Health also offers a mobile client for Android and iOS smartphones, assuming you can use your work ID and credentials on the phone.

Gemini Cloud Assist, which is integrated with Personalized Service Health, so you can use it to query that information in natural language.
Cloud Service Health dashboard: This is the public-facing non-authenticated web page for broad, severe incidents affecting many customers. Limited blast radius disruptions are not externalized to the public. All its content is available in Personalized Service Health as well. If ever Personalized Service Health goes down, Cloud Service Health serves as an alternative channel built on a separate infrastructure.
Known Issues: In the console, navigate to Support > Cases, view a case, and use the resource selector on the console toolbar to find the specific cloud resource you’re interested in. Then click Known issues. If your issue matches one listed here, you can link a support case to it, so you will receive automatic updates in your case record. If you don’t find a match, open a new support case. Google will automatically match the case to a related incident, as soon as one is declared.
Google declared incidents are updated as new information becomes available, so check back regularly, or set up a Personalized Service Health alert policy to be notified each time new information becomes available.

If you host cloud resources in multiple clouds, a good practice is to check early on whether the problem occurs for multiple cloud providers. If so, the problem is likely external to the providers and caused either by you or by a third-party service that your application interacts with.

3. Investigate

To determine the blast radius within your cloud footprint of Google-declared reliability incidents, first check Personalized Service Health updates for a description of the technical problem. Knowing what to look for will allow you to map your blast radius and decide on suitable contingency actions quicker.

If Google hasn’t declared an incident, try to rule out configuration errors or issues within your environment by checking:

Cloud Monitoring: Look for spikes in error rates (e.g. 5xx errors), increased latency, or drops in traffic in your dashboards.
Cloud Logs: Use Log Explorer to look for specific error messages like DEADLINE_EXCEEDED, SERVICE_UNAVAILABLE, or specific API errors.
Quotas: Ensure you haven't hit a project quota (e.g., CPU, API rate limits), which can often mimic the behavior of an outage.
Change history: Check your log of recently applied changes. Not all problems manifest immediately, but proximity on a timeline can be a powerful indicator of causality, even if it’s not proof. Also check whether Google rolled out any updates just before the symptoms started. See the Unified Maintenance Management interface in Cloud Hub.

Absent a clear culprit, such as a traffic spike or a DDOS attack, and if symptoms manifested immediately after rolling out a change, a good strategy is to back out that change and attempt to return to a last known good configuration.

4. Report

If the Cloud Service Health and Personalized Service Health dashboards are green but your metrics show a failure, you must report it to Google.

Determine priority:

P1 (Critical): Your production service is unusable or severely impacted with no workaround.
P2 (High): Significant impact or degradation, but a workaround may exist.
See guidance on setting priority and guidance on describing your issue.

File a case: Go to Support > Cases > Create Case in the console.

Explain quantifiable business impact to rationalize the submitted priority and prevent it from being reset when Cloud Support prioritizes cases. A clear and accurate rationale helps!

Essential information to include:

Project ID and affected region/zone
Timestamps (when it started and if it's ongoing) with a clearly labeled timezone
Specific error messages or log snippets
Scope: Is it affecting all users/systems, or a specific subset/location?

Escalation for Premium/Enhanced support

If you have a Premium or Enhanced support plan and a P1 case is not receiving the attention it requires, use the Escalate button within the support case in the console. This alerts a support manager to investigate and rectify the situation.

5. Resolve

By taking these steps, you are well on your way to resolving the outage. In the meantime, here are some ways to mitigate the impact of the outage and communicate with impacted stakeholders.

While waiting for a resolution:

Communicate: Notify your stakeholders and customers. Transparency helps manage expectations and reduces duplicate internal reports.
Fail over: If you have a multi-regional architecture, consider shifting traffic to a healthy region. As a best practice, first ensure that the disruption is at the infrastructure level and not at your workload level.
Check for workarounds: While working on a permanent fix, Google often posts temporary workarounds in the Service Health Dashboard updates, or in Personalized Service Health updates.
Consider your regulatory reporting requirements: Know whether your organization is subject to regulatory reporting requirements, and what the required deadlines are for both initial and follow-up reporting. Google Cloud prepares Incident Reports for incidents that meet certain criteria — see details here for how to get those reports. Premium Support customers can also request an Incident Summary, which is an Incident Report customized to your account’s specific hosting location, time stamps, etc.

De-escalation and closure

Once systems are stable, Google downgrades the severity levels and deactivates the active on-call escalation chain. Google only closes an incident in Personalized Service Health when it has taken all the mitigation steps covering all impacted customers. Your specific services might be restored sooner than the incident closure time, if other customers are restored later than you. The incident is officially closed on the Google Cloud Status Dashboard when systems have run stably for a designated auto-close duration. Verify that your services are operating normally at this point. And if your incident responders aren’t compensated for extra time spent on the incident, find a way to thank them.

6. Review

After the problem has been fixed and operations have returned to a normal, steady state, it’s time to conduct a post-mortem analysis to identify how your team can respond better in future service disruptions. A “blameless” approach is essential to surfacing meaningful and impactful improvements that can be made to your incident response process. Ask questions like:

What went well?
What could we have done better?
Where did we get lucky?
Where did we get unlucky?

Then decide what changes can be made to improve your playbook, tools and training.

At Google, we often publish a post-mortem or Incident Report for major outages, available via Personalized Service Health. Review this to understand the root cause and adjust your own disaster recovery plans to prevent or reduce future impact. Customers with a Premium Support plan can request an Incident Summary for a Google-caused incident they were impacted by and for which they opened a P1 case. An Incident Summary is an Incident Report customized for your environment (e.g., start and end times of impact).

Typical causes, comms and prevention strategies

To help you prepare and plan ahead, here’s an overview of some typical incidents based on the symptoms reported in Cloud Service Health and Personalized Service Health along with guidance on what Google communications to expect, and some generic mitigation or prevention strategies you can build into your playbooks.

Blast radius	Typical cause	Comms	Strategy
Single zone or region.Subset of products.	Typical of a software problem triggered by a rollout. Learning points: - Understand the location scope (zones and regions) of your workload - Products can depend on other products	Major incidents are communicated via Cloud Service Health.Major and Minor (by number of customers, not severity) incidents are communicated via Personalized Service Health. Highly localized incidents are not communicated via Cloud Service Health or Personalized Service Health.	Fail over, if so configured, but verify the health of the secondary stack first.
Single zone.Most or all products.	Typical of a power or cooling issue.	Check Cloud Service Health and Personalized Service Health.	Fail over to a different zone, if so configured.
Single region. Most or all products.	Typical of a backbone networking infrastructure issue	Check Cloud Service Health and Personalized Service Health.	Fail over to a different region, if so configured.
Control plane issue for a product	Typical of a late detected issue	Communicated via Personalized Service Health if significant customer impact is verified.	Look for workarounds. Wait for Google to fix. Fail over, if so configured.
Multi-regional issue with a global product	Rare but possible, typically detected quickly. Learnings: Mitigation options can be limited. Try regional variants, alternative products with similar functionality	Check Cloud Service Health and Personalized Service Health.	Wait for Google to fix. In the meantime, verify via Google Comms and your own investigation that this is truly Google’s problem to fix.
Capacity / Stockout issue	System-level demand exceeding capacity in the product/location/model. (Cloud is designed to scale, but limits always exist, so proper planning is advised)	Error message. No incident will be declared.	Place reservations for predicted capacity needs (if cost is acceptable). Flexibility in zone placement can also help.
Quota exhaustion	Difficult / inaccurate prediction of traffic	Error message. No incident will be declared.	Review consumption trends against ceiling regularly.

Go deeper

This document offers only a condensed summary of key points. If you have an active Premium Support contract with Google Cloud, reach out to your account team for a deeper review of your response plans. For a comprehensive treatise on how to build reliable services and how to respond to incidents, we strongly recommend Google’s SRE Book, which is available as a free download. A new version of the SRE book is releasing ~Oct 2026 and will be available for purchase on O’Reilly Media. We’re also working on a future primer that explores AI-supported incident handling in-depth — stay tuned!

Introducing the Google Cloud Developer Plugin for AI Coding Agents

Thu, 10 Sep 2026 10:53:00 +0000

Agent skills fit well alongside documentation and remote MCP servers as ways of enabling the success of your AI workflows. They reduce context window usage for certain use cases, and they're straightforward to install. However, you might have noticed that managing individual skills can be unwieldy, or that some skills are most useful when they act alongside other skills or MCP servers toward the same goal. That's where plugins come in to help.

Today, we're thrilled to announce a new Google Cloud plugin for AI coding agents! Designed as installable bundles, agent plugins equip the AI agent of your choice with skills and tools to be more effective on Google Cloud.

Solving the tool coupling problem

As you expand your usage of coding agents, you might find that they become significantly more capable when they use related skills in tandem or with complementary context and tooling. For example, an agent analyzing infrastructure is more effective when combining domain knowledge, workflow recommendations, and the ability to interact with a live environment together.

Plugins solve this coupling challenge by packaging related capabilities into cohesive, installable bundles. This allows you to take advantage of both broad foundational capabilities and deep, product-specific tools without managing complex dependencies. In the Google Agent Skills repository, you'll start to see the following for Google Cloud popping up over time:

Foundational plugins: Essential platform-wide guidance for things like documentation discovery, project configuration, and architectural design.
Domain-focused plugins: Specialized knowledge and best practices for technical areas in the context of Google Cloud.

For this release, we've started with a foundational plugin that supports agent functionality for all Google Cloud users, focusing on making it easier for agents to retrieve Google Cloud-related skills, make use of official documentation, and handle programmatic interactions with Google Cloud.

Built on an open standard

We've also built our plugin in compliance with the Agent Plugins specification, an open, vendor-neutral standard for packaging Agent Skills and Model Context Protocol (MCP) servers into portable, interoperable units. Rather than requiring developers to maintain different configurations and wrappers for every AI assistant, the Agent Plugins standard provides a unified manifest and directory structure.

Our Google Cloud plugin adopts this standard to ensure that developers across a variety of AI coding environments get consistent, high-quality access to tools that help them succeed with Google Cloud. That includes not only the plugins we talk about today, but all other plugins published to the Google Agent Skills repository as well.

Let's take a look at the flagship plugin that we've just published in the Google Agent Skills repository: google-cloud-developer. This plugin exists to help agents successfully navigate the fundamentals of interacting with Google Cloud: things like authentication, authorization, managing projects, and guardrails for gcloud CLI operations. This plugin also bundles configuration for the Developer Knowledge MCP server, which gives agents up-to-date grounding in Google's official developer documentation.

Plugin in action: Project onboarding and identity authentication

To see how this plugin works, consider a situation where you're bootstrapping a new project as part of working on a script. With the google-cloud-developer plugin installed, you can prompt your agent:

I'm brand new to this platform, and I need to get an account and a first project with billing set up. Then, I need my local machine authenticated so a script that I'm writing can call the APIs as a service identity instead of as me.

Environment awareness: The agent silently runs background checks against your live environment for prerequisites like CLI availability and potential existing projects or organizations.
Review: The agent considers IAM best practices to avoid risks that might be assumed as part of the prompt, like accidental key leaks or git commits.
Interaction with guardrails: The agent outlines a workflow roadmap and offers to act on those steps before modifying any resources.

Installing Google Cloud plugins

Because Google Cloud plugins are available from the open Google Agent Skills repository and adhere to the standard Agent Plugins layout, adding them to your environment is straightforward. For example, here's how you'd install the google-cloud-developer plugin:

Antigravity CLI

Install the plugin directly via the CLI using its path in the Google Agent Skills repository:

code_block: <ListValue: [StructValue([('code', 'agy plugin install https://github.com/google/skills/plugins/cloud/google-cloud-developer'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d0b0c950>)])]>

Enable the Developer Knowledge API in your Google Cloud project by using the gcloud CLI:

code_block: <ListValue: [StructValue([('code', 'gcloud services enable developerknowledge.googleapis.com --project=<YOUR_PROJECT_ID>'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d012bb50>)])]>

Claude Code

Add the Google plugins marketplace, then install the plugin:

code_block: <ListValue: [StructValue([('code', 'claude plugin marketplace add google/skills\r\nclaude plugin install google-cloud-developer@google-plugins'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d012a390>)])]>

Enable the Developer Knowledge API in your Google Cloud project by using the gcloud CLI:

code_block: <ListValue: [StructValue([('code', 'gcloud services enable developerknowledge.googleapis.com --project=<YOUR_PROJECT_ID>'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d0129450>)])]>

Create an API key for the Developer Knowledge API by following the instructions Create and secure the API key. Save the key you create in a secure location.

Export the API key to your environment before starting Claude Code:

code_block: <ListValue: [StructValue([('code', 'export DEVELOPERKNOWLEDGE_API_KEY=<YOUR_API_KEY>'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d012b950>)])]>

Note: The DEVELOPERKNOWLEDGE_API_KEY environment variable needs to be set in the environment before you start Claude Code. Consider adding this export to your shell's startup script (e.g. .bashrc, .zshrc) for convenience.

Codex CLI

Add the Google plugins marketplace, then install the plugin:

code_block: <ListValue: [StructValue([('code', 'codex plugin marketplace add google/skills\r\ncodex plugin add google-cloud-developer@google-plugins'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d0128090>)])]>

Enable the Developer Knowledge API in your Google Cloud project by using the gcloud CLI:

code_block: <ListValue: [StructValue([('code', 'gcloud services enable developerknowledge.googleapis.com --project=<YOUR_PROJECT_ID>'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d0128dd0>)])]>

Create an API key for the Developer Knowledge API by following the instructions Create and secure the API key. Save the key you create in a secure location.

Enable authenticated access to the Developer Knowledge MCP server by updating ~/.codex/config.toml (or your project's .codex/config.toml) to include the following lines:

code_block: <ListValue: [StructValue([('code', '[mcp_servers.developer-knowledge]\r\n url = "https://developerknowledge.googleapis.com/mcp"\r\n env_http_headers = { "X-Goog-Api-Key" = "DEVELOPERKNOWLEDGE_API_KEY" }'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d012a210>)])]>

Export the API key to your environment before starting Codex:

code_block: <ListValue: [StructValue([('code', 'export DEVELOPERKNOWLEDGE_API_KEY=<YOUR_API_KEY>'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21cbe0db50>)])]>

Note: The DEVELOPERKNOWLEDGE_API_KEY environment variable needs to be set in the environment before you start Codex. Consider adding this export to your shell's startup script (e.g. .bashrc, .zshrc) for convenience.

Note: After performing this configuration update, a status of not logged in for the Developer Knowledge MCP server is expected and doesn't block access to the server.

Next Steps

If you're already a Google Cloud user, try the above installation steps to set up your agent for success. We think you'll like what you see! For those who want a more guided approach, our new codelab will walk you through the installation and initial exploration of the plugin in Antigravity.

If you're new to Google Cloud, you can also get started with instructions in our documentation to set yourself up for local development.

The most curious readers can also take a deeper look at the plugins and agent skills available to use today in the Google Agent Skills repository.

Power agent hubs or custom harnesses with the Antigravity SDK in one toolkit

Tue, 08 Sep 2026 16:00:00 +0000

Enterprise agent adoption isn’t one-size-fits-all. While many teams will opt for managed commercial platforms, such as Gemini Enterprise Agent Platform for turnkey agent deployment and governance, developers with bespoke workflows or custom execution engines often choose to build their own lightweight agent hubs.

If you are building a centralized agent hub from the ground up, you need tools that run predictably, log everything, and stay in their sandbox. The Antigravity SDK gives you the exact runtime engine used in Antigravity 2.0 and the Antigravity CLI, adding declarative safety policies, real-time telemetry, and stateful multi-turn persistence straight into your application. When the core runtime updates, your SDK agents get those optimizations automatically. That's why today, we're breaking down how the Antigravity SDK powers a complete multi-agent control plane.

How Antigravity comes together

A multi-agent control plane monitors and manages LLM workloads. It shows you exactly what the agent is thinking, which tools it calls, and how it stores state.

It consists of two critical components:

1. Antigravity SDK agent core: The runtime that manages model interactions (like Gemini 3.1 Pro and Gemini 3.8 Flash), runs tools, generates thinking traces, and executes skills.

2. Observability and telemetry middleware: An event-driven layer powered by Antigravity SDK Lifecycle Hooks. It intercepts agent actions like step starts, thinking updates, and tool calls, and streams telemetry over WebSockets to your dashboard.

Use case: Multi-agent monitoring and interactive control

Let's explore a scenario where an organization is building or maintains a custom agent hub and wants to integrate Antigravity SDK-powered agents.

The problem: An operations engineer needs to monitor multiple active agents (e.g., gemini-pro-agent, github-agent, email-agent) performing background research, document summarization, and task scheduling. Traditionally, observing agent progress requires:

Tailing fragmented console logs across multiple terminal windows
Manually inspecting JSON transcripts to diagnose stuck or failing tool calls
Lack of visibility into which Skills or MCP connectors are loaded for a given agent session
Difficulty tracking cumulative token usage and execution latency

The solution: This post walks through each one: the streaming API for real-time observation, lifecycle hooks for telemetry and interception, the policy engine for steering, skills for capability management, and session state for persistence. With an SDK-powered dashboard, operators get a single view into what every agent is doing.

What happens behind the scenes?
When an operator or dashboard interacts with an Antigravity agent, the runtime coordinates execution through five core mechanisms:

Session initialization and state attachment (save_dir & conversation_id):The runtime initializes or reattaches to a session, binding execution to a root save_dir. Multi-turn trajectory logs, tool receipts, and artifacts are preserved under traj-<conversation_id> for persistent auditability.
Skill resolution (skills_paths):Domain-specific capabilities and instructions are resolved directly from filesystem paths pointing to SKILL.md bundles, dynamically augmenting the agent's system prompt without an external registry.
Concurrent stream generation (ChatResponse):The runtime exposes three concurrent async iterators over the single model response:

response (yields visible text tokens)
response.thoughts (yields internal chain-of-thought reasoning deltas)
response.tool_calls (yields typed ToolCall events containing .name and .args)

Declarative sandboxing and built-in tool execution:When the agent performs workspace operations, built-in tools (list_directory, find_file, search_directory, view_file, create_file, edit_file) execute strictly within configured workspaces directories governed by safety policies (such as policy.workspace_only()).
Telemetry interception via lifecycle hooks:Decorated async hook functions (@hooks.on_session_start, @hooks.pre_tool_call_decide, @hooks.post_tool_call, @hooks.on_session_end) intercept agent transitions in real time, validating or modifying tool calls and broadcasting telemetry payloads over WebSockets to the live dashboard.

The Antigravity SDK organizes these responsibilities into four core building blocks:

1. Modular capabilities with Skills

Skills provide reusable, domain-specific instruction bundles and reference assets that agents load dynamically. Rather than managing an in-memory registry, skills are resolved directly from filesystem directories containing a SKILL.md file:

code_block: <ListValue: [StructValue([('code', 'from google.antigravity import Agent, LocalAgentConfig\r\n\r\n# Pass directory paths containing SKILL.md bundles directly to config.\r\n# The runtime dynamically resolves and injects them into the prompt.\r\nconfig = LocalAgentConfig(\r\n model="gemini-3.8-flash",\r\n system_instructions=(\r\n "You are an enterprise operations assistant equipped with "\r\n "specialized operational skills."\r\n ),\r\n skills_paths=["./skills/research", "./skills/code_review"],\r\n)\r\n\r\nasync with Agent(config) as agent:\r\n response = await agent.chat("Analyze the deployment logs.")'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d0fb2610>)])]>

2. Sandboxed built-in tools and workspace scoping

The SDK provides production-ready file and workspace tools out of the box, which removes the need to write custom filesystem wrappers. When paired with workspaces and declarative safety policies, tools are strictly confined to authorized directories:

code_block: <ListValue: [StructValue([('code', 'from google.antigravity import Agent, LocalAgentConfig, types\r\nfrom google.antigravity.policies import policy\r\n\r\nconfig = LocalAgentConfig(\r\n model="gemini-3.8-flash",\r\n # Selectively enable built-in tools via CapabilitiesConfig\r\n capabilities=types.CapabilitiesConfig(\r\n enabled_tools=[\r\n types.BuiltinTools.LIST_DIR, # "list_directory"\r\n types.BuiltinTools.FIND_FILE, # "find_file"\r\n types.BuiltinTools.SEARCH_DIR, # "search_directory"\r\n types.BuiltinTools.VIEW_FILE, # "view_file"\r\n types.BuiltinTools.CREATE_FILE, # "create_file"\r\n types.BuiltinTools.EDIT_FILE, # "edit_file"\r\n ]\r\n ),\r\n # Enforce filesystem isolation: operations outside these paths are blocked\r\n workspaces=["./workspace"],\r\n policies=[policy.workspace_only()],\r\n)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d1b0fc50>)])]>

3. Session isolation and trajectory persistence (`save_dir` & `conversation_id`)

State persistence in the Antigravity SDK is managed through declarative configuration rather than an external database. Specifying a save_dir establishes a root directory where full turn trajectories, tool receipts, and artifacts are preserved under traj-<conversation_id>:

code_block: <ListValue: [StructValue([('code', 'from google.antigravity import Agent, LocalAgentConfig\r\n\r\nconfig = LocalAgentConfig(\r\n model="gemini-3.8-flash",\r\n # Root directory storing all conversation trajectories\r\n save_dir="./storage/sessions",\r\n # Supply conversation_id to reattach to an existing trajectory;\r\n # omit it to let the SDK mint a new ID on the first turn.\r\n conversation_id="ops-session-20260820-001",\r\n)\r\n\r\nasync with Agent(config) as agent:\r\n # Resumes prior context and continues the multi-turn session seamlessly\r\n response = await agent.chat("Summarize the issues identified in the last turn.")'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d012ad50>)])]>

4. Real-time telemetry and interception with lifecycle hooks

Lifecycle hooks allow dashboards and monitoring engines to observe and steer every stage of execution. Using decorated async functions, you can stream status updates over WebSockets, inspect tool parameters, and enforce human-in-the-loop approvals before tools run:

code_block: <ListValue: [StructValue([('code', 'from google.antigravity import Agent, LocalAgentConfig, types\r\nfrom google.antigravity.hooks import hooks\r\n\r\n# 1. Session start & end telemetry\r\n@hooks.on_session_start\r\nasync def on_session_start():\r\n broadcast_to_dashboard({"type": "STATUS", "status": "RUNNING"})\r\n\r\n@hooks.on_session_end\r\nasync def on_session_end():\r\n broadcast_to_dashboard({"type": "STATUS", "status": "IDLE"})\r\n\r\n# 2. Intercept tool calls before execution (human-in-the-loop / audit gate)\r\n@hooks.pre_tool_call_decide\r\nasync def intercept_tool(tool_call: types.ToolCall) -> types.HookResult:\r\n broadcast_to_dashboard({\r\n "type": "TOOL_CALL",\r\n "tool": tool_call.name,\r\n "args": tool_call.args,\r\n })\r\n # Return HookResult to approve or block execution\r\n return types.HookResult(allow=True)\r\n\r\n# 3. Post-execution tool receipts\r\n@hooks.post_tool_call\r\nasync def record_tool_result(result):\r\n broadcast_to_dashboard({\r\n "type": "TOOL_RESULT",\r\n "tool": result.name,\r\n "error": getattr(result, "error", None),\r\n })\r\n\r\nconfig = LocalAgentConfig(\r\n model="gemini-3.8-flash",\r\n hooks=[on_session_start, on_session_end, intercept_tool, record_tool_result],\r\n)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d012a450>)])]>

Get started

Get started with your own enterprise agent control plane using the following resources:

Spanner migrations: Automating dual-write with Antigravity CLI for minimal disruption

Fri, 04 Sep 2026 16:00:00 +0000

When Google's Finance Engineering team needed to modernize their legacy data layer, they chose Spanner, a globally distributed, strongly consistent, multi-model database with high availability capabilities. But migrating to Spanner without taking production services offline was a daunting engineering challenge: As the internal team responsible for the application, we needed to manually rewrite dual-write logic across dozens of Data Access Objects (DAOs), a process that is slow and prone to human error. Further, doing so without disruption would have required implementing multi-phase dual-write architectures across every DAO in our codebase.

To solve this, we took an alternative approach: We built an automated refactoring pipeline powered by Antigravity CLI in headless mode. This helped us accelerate our migration velocity significantly while maintaining strict data parity in our staging environments as we prepare for production.

The challenge: Anatomy of a dual-write migration

When migrating high-throughput production services where financial accuracy is essential, simple cutover scripts do not work. You must verify that both the legacy datastore and Spanner receive identical writes simultaneously until all the historical data backfills and verifications are complete.

We structured our migration across three distinct phases:

Historical backfill: Copying existing historical records to Spanner while maintaining referential integrity.
Dual-write / dual-read implementation: Modifying every DAO to write mutations to both the primary store and Cloud Spanner in parallel during the migration window.
Automated API verification and parity checking: Intercepting RPC traffic and verifying end-to-end that every write lands with byte-for-byte equivalence across both stores.

The architectural pattern is clean, but at our scale, we began to encounter friction. That’s because each DAO requires:

A dedicated MutationConverter class mapping complex domain models to Spanner schema columns
Dual-write branch handling and rollback or error-reporting logic
A suite of unit tests verifying both primary and Spanner writes using fake time sources and test doubles (FakeTimeSource)

Performing these identical, high-precision code changes across 30+ DAOs by hand would have taken months of engineering time.

The solution: Standardized mutation converter patterns

To verify that our automation pipeline could reliably generate clean code, we first standardized our DAO refactoring pattern around a decoupled MutationConverter interface.

Instead of embedding raw Spanner table names and column assignments directly inside core DAO business logic, we isolate Spanner schema translation into dedicated converter units:

code_block: <ListValue: [StructValue([('code', '// Example of the standardized pattern generated by our pipeline\r\n\r\ntype BpcTransferAmountsMutationConverter interface {\r\n ToInsertMutation(entity *model.BpcTransferAmount) (*spanner.Mutation, error)\r\n ToUpdateMutation(entity *model.BpcTransferAmount) (*spanner.Mutation, error)\r\n}\r\n\r\ntype bpcTransferAmountsMutationConverterImpl struct {\r\n tableName string\r\n}\r\n\r\nfunc (c *bpcTransferAmountsMutationConverterImpl) ToInsertMutation(entity *model.BpcTransferAmount) (*spanner.Mutation, error) {\r\n if entity == nil {\r\n return nil, errors.New("entity cannot be nil")\r\n }\r\n \r\n // Map domain fields to Cloud Spanner table schema\r\n cols := []string{"TransferId", "AmountCents", "CurrencyCode", "LastModifiedTimestamp"}\r\n vals := []interface{}{\r\n entity.TransferId,\r\n entity.AmountCents,\r\n entity.CurrencyCode,\r\n spanner.CommitTimestamp, // Use Spanner commit timestamps\r\n }\r\n \r\n return spanner.Insert(c.tableName, cols, vals), nil\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d132ff50>)])]>

By establishing a rigid, deterministic contract between the DAO and the Spanner SDK (spanner.Mutation), we created an exact target specification that an AI coding agent could reason about and generate reliably.

Why use Antigravity CLI in headless mode?

Interactive AI chat interfaces in IDEs work well for exploratory coding, but they are poorly suited for systematic, multi-file code updates across an entire codebase. When you need to apply repeatable refactoring to dozens of targets without missing edge cases, you need automated workflows.

We addressed this by building an orchestration script (migration_ui.py) that runs Antigravity CLI in headless mode (-p).

Headless mode lets Antigravity run directly inside shell scripts, continuous integration pipelines, and background automation jobs without requiring manual terminal prompts. This approach helped us scale our work in three key ways:

Deterministic prompt architectures: We treated our prompts as version-controlled engineering artifacts. We codified precise rules handling common Spanner edge cases — such as timestamp serialization, nullability conversions, mutation ambiguity, and FakeTimeSource test injection — directly into reusable prompt templates.
Batch execution and automated verification: Our orchestration script takes a target DAO name as input, retrieves the existing single-write source code and schema, and feeds it to headless Antigravity alongside our structural conventions. Antigravity generates the new converter, the refactored dual-write DAO, and corresponding unit tests. The script then runs blaze test. If a linter error or test assertion fails, the error log feeds directly back into Antigravity for self-correction.
Overnight execution at scale: Because the loop runs unattended, engineers can queue up 10 DAOs at the end of the day. By morning, the pipeline generates, tests, and validates 10 clean changelists ready for human code review.

Results and key takeaways for cloud engineers

Combining Spanner's distributed database primitives with Antigravity CLI's headless automation produced clear benefits across our engineering organization:

Significant reduction in migration effort: DAO dual-write migrations that previously required extensive manual coding and testing were completed and reviewed in a fraction of the time
Highly reliable data migration: Because every generated DAO adhered to the exact same tested MutationConverter pattern and underwent automated unit testing against Spanner test doubles, we sustained high data fidelity during our extensive migration testing.
Focus on higher-value engineering: Engineers avoided repetitive boilerplate refactoring, giving them time to focus on data modeling, architectural resilience, and performance optimization.

Three tips for your next database migration

Decouple schema translation first: Before writing migration scripts, define a strict interface (like our MutationConverter) that isolates your new cloud database SDK requirements from your existing business logic. AI agents work best when given clear, bounded design patterns.
Move from interactive chat to headless automation: When executing repetitive refactoring across more than three or four files, invest in scripted, headless workflows. Treating prompt inputs and test verifications as automated build steps help maintain quality and consistency.
Let the build system act as your guardrail: Connect your AI generation loop directly to your build and test harness (bazel test or go test). This lets the model fix compile and assertion errors before a developer reviews the code.

Get started

Whether you’re migrating financial systems or building cloud-native applications from scratch, Spanner and Antigravity provide a foundation for scalable software development.

Explore Cloud Spanner: Learn more about Spanner's distributed architecture Google Cloud Spanner documentation.
Discover Gemini for Developers: See how AI-assisted coding and headless CLI automation can assist your engineering workflows at Google Cloud AI for Developers.

Not All LLM Workloads Are Equal: Benchmarking TPU Performance on Classification vs. Generation

Fri, 04 Sep 2026 15:36:00 +0000

Moving Large Language Models (LLMs) from experimental prototypes into enterprise production exposes a critical truth: your infrastructure dictates both your performance ceilings and your unit economics. Standard hardware benchmarks often ignore a fundamental reality—not all LLM requests stress the silicon in the same way.

In this post, we dive into a comprehensive benchmarking exercise comparing Gemma 3 12B and Gemma 3 27B on Google Cloud TPU v6e to answer a crucial architectural question: How does TPU infrastructure actually perform when tasked with structurally distinct workloads at scale?

Key Findings and Suggestions

Before diving into the methodology, here are the critical takeaways for architects deploying Gemma 3 on TPU v6e:

The Generation Performance Wall

For decode-heavy generation tasks, the Gemma 3 27B model hits a strict performance wall past 64 concurrent users, plateauing at a 4.12x normalized throughput multiplier at 128 users. In contrast, the 12B model scales up to an 8.19x multiplier.

Suggestion: If your workload requires high-concurrency generation, downsize to the 12B model, or set strict pod-autoscaling limits capping concurrent requests at 64 per replica for the 27B model.

The Classification Parity

For prefill-heavy classification tasks, model parameter size matters significantly less. Both the 12B and 27B models achieve similar peak scaling (around 6.0x to 6.4x normalized throughput at 128 users) without saturating the TPUs.

Suggestion: You can safely deploy larger, more capable models for summarization or classification workflows without paying a throughput penalty. The average --max-num-seqs or --max-model-len should be kept judiciously based on the average user load and average tokens per request, without which there might be request drops.

Designing Around the Wall

Hardware saturation manifests as severe latency spikes and silent request dropouts. To mitigate this, do not rely on standard CPU/Memory scaling triggers. Instead, scale based on End-to-End (E2E) latency metrics, and implement aggressive vLLM bucket padding optimizations (VLLM_TPU_BUCKET_PADDING_GAP) to conserve memory.

The Architecture Setup

The inference stack can be divided into three core pillars:

1. Infrastructure: GKE & TPU

The foundation of our deployment is a Google Kubernetes Engine (GKE) Autopilot cluster. Connected to this is a single-host TPU v6e node pool configured with a 2x2 chip topology.

2. Software & Tools: vllm

For the serving framework, we leveraged vllm via vllm-project/tpu-inference.

3. Models: Gemma 3 12B and 27B

We evaluated two highly capable open-weights models: Gemma 3 12B and Gemma 3 27B. These models were accessed via HuggingFace.

The Workloads: Classification vs. Generation

Not all LLM requests stress the system equally. We benchmarked two distinct scenarios: Classification and Generation, across 16, 32, 64, and 128 concurrent users:

Classification (High Input, Low Output): This use case mimics an e-commerce compliance task. The prompt includes large blocks of product rules, item descriptions, and OCR-extracted text. The output is exceptionally small—typically just classifying an item as "Allow" or "Prohibit". Input Sequence Length (ISL) is ~4,000 tokens and Output Sequence Length (OSL) is ~10 tokens.
Generation (Low/Medium Input, High Output): This use case mimics long-form text generation. The prompt requests a detailed, analytical policy brief on the future of AI in the labor market. The model spends the majority of its time decoding and streaming out hundreds of tokens. Input Sequence Length (ISL) is 500 tokens and Output Sequence Length (OSL) is ~1,000 tokens.

Results and Observations

We measured metrics like Throughput (requests/sec), End-to-End Latency and the results provided some fascinating insights into how parameter size and hardware bandwidth interact. To ensure architectural consistency, every benchmark was executed using the vllm-project/tpu-inference hardware plugin, leveraging a standardized global serving configuration of max-model-len=128000, max-num-batched-tokens=8192, and max-num-seqs=512.

Generation Scaling Divergence

In Generation tasks, both models perform similarly up to 64 concurrent users. However, at 128 concurrent users, the Gemma 3 12B model shows significantly better scaling, achieving an 8.19x normalized throughput multiplier compared to a 4.12x plateau for the Gemma 3 27B model (normalized against the Gemma 3 12B baseline at 16 users). This suggests that the larger 27B model hits memory or compute limits much earlier under high generation loads.

Concurrent Users	Gemma 3 12B Throughput (req/s)	Gemma 3 27B Throughput (req/s)
16 users	1.00 x	1.05 x
32 users	1.98 x	1.97 x
64 users	2.96 x	4.00 x
128 users	8.19 x	4.12 x

aside_block: <ListValue: [StructValue([('title', 'Pro Tip → Metrics Inflation at High Concurrency'), ('body', <wagtail.rich_text.RichText object at 0x7f21cbe44a90>), ('btn_text', ''), ('href', ''), ('image', None)])]>

Classification Performance Parity

In Classification tasks, there is negligible difference in scaling behavior between the Gemma 3 12B and Gemma 3 27B models. Both models operate efficiently within the hardware's capacity and scale well, reaching peak normalized throughputs of approximately 6.04x to 6.37x at 128 concurrent users (normalized against the Gemma 3 12B baseline at 16 users).

Concurrent Users	Gemma 3 12B Throughput (req/s)	Gemma 3 27B Throughput (req/s)
16 users	1.00 x	0.76x
32 users	1.18x	1.53x
64 users	2.04x	3.15x
128 users	6.37x	6.04x

Latency Threshold Analysis

End-to-End (E2E) latency exhibits different scaling behaviors depending on the model size and task. When using identical serving hyperparameters (--max-num-seqs=512), the Gemma 3 12B model's Classification latency roughly doubles when moving from 32 users to 64 users, indicating resource contention. However, for the larger Gemma 3 27B model, Classification latency remains relatively flat between 32 and 64 users before doubling at the 128-user mark.

Model	Task	16 Users	32 Users	64 Users	128 Users
Gemma 3 12B	Generation	1.00x	1.13x	1.40x	1.70x
Gemma 3 12B	Classification	1.00x	0.99x	1.79x	2.90x
Gemma 3 27B	Generation	1.20x	1.68x	2.93x	3.33x
Gemma 3 27B	Classification	1.20x	1.95x	1.95x	3.88x

aside_block: <ListValue: [StructValue([('title', 'A Crucial TPU Optimization Technique'), ('body', <wagtail.rich_text.RichText object at 0x7f21d0128290>), ('btn_text', ''), ('href', ''), ('image', None)])]>

Conclusion

Benchmarking Gemma 3 12B and 27B models on Google Cloud TPU v6e architecture reveals that raw parameter count is not the sole predictor of inference performance; rather, the interaction between the serving framework, hardware topology, and workload token ratios dictates efficiency. For generation tasks (low input, high output), the 12B model proves superior at high concurrency, sustaining an 8.19x relative throughput multiplier where the 27B model saturates at 4.12x. Conversely, for prefill-heavy classification tasks, both models perform similarly, allowing organizations to deploy larger models without a severe scaling penalty. Our evaluation also mapped exact hardware saturation thresholds—such as End-to-End latency doubling at 64 users for classification and hitting a cliff at 128 users for generation—enabling precise, data-driven auto-scaling triggers rather than costly over-provisioning. Ultimately, achieving these peak metrics requires aggressive tuning of vllm parameters, such as adjusting batched tokens and configuring TPU-specific bucket padding to prevent compute waste, proving that cost-effective AI infrastructure must strictly align model selection and serving configurations to the unique input/output profiles of production workloads.

Ready to scale your LLM workloads?

Don't let unoptimized infrastructure bottleneck your enterprise AI rollouts. Now that you know how different workload shapes impact hardware saturation, it's time to put these insights into practice:

Use these benchmarks to right-size your production architecture. Safely leverage the larger Gemma 3 27B for prefill-heavy classification tasks without a throughput penalty, but consider switching to the 12B model to maintain linear scaling for decode-heavy generation at high concurrency.
Deploy using Google Kubernetes Engine (GKE) with TPU v6e node pools to build a highly scalable, managed AI foundation and dedicated vllm-project/tpu-inference hardware plugin. Alternatively, you can also deploy via Model Garden on Gemini Enterprise Agent Platform or you can spin up TPU VMs for serving Gemma 3 models.

Have you encountered similar performance walls in your own production deployments? Share your scaling strategies, ask questions, and join the discussion in the Google Cloud Community forums.

Announcing the Google Gen AI SDK for Kotlin 1.0: Idiomatic multiplatform access to Gemini

Thu, 03 Sep 2026 09:00:00 +0000

Integrating modern generative AI capabilities into Kotlin applications shouldn't require juggling raw HTTP clients or bridging disparate Java libraries. Today, we're excited to announce the 1.0 release of the Google Gen AI SDK for Kotlin (google-genai-kotlin). You can dive right into the code, explore runnable samples, and star the project today on GitHub at googleapis/kotlin-genai.

Built from the ground up as a Kotlin Multiplatform (KMP) library, the SDK brings idiomatic Kotlin paradigms (including first-class Coroutines, asynchronous Flow streaming, and immutable data classes with named and default parameters) to developers specifically targeting the JVM (backend services, serverless functions, desktop).

The SDK provides a unified surface to interact with both the Gemini Developer API (Google AI Studio) and the Gemini Enterprise Agent Platform (on Google Cloud) with minimal configuration tweaks.

Although this SDK is a Kotlin Multiplatform library, using it directly from a mobile app is blocked for security reasons. Instead, use Firebase AI Logic for direct access to Gemini models from mobile apps.

1. Getting started: Adding the dependency

The SDK is published to Maven Central under com.google.genai:google-genai-kotlin.

Kotlin Multiplatform (KMP)

For multiplatform applications, add the dependency to your commonMain source set:

code_block: <ListValue: [StructValue([('code', '// build.gradle.kts\r\nkotlin {\r\n sourceSets {\r\n commonMain.dependencies {\r\n implementation("com.google.genai:google-genai-kotlin:1.0.0")\r\n }\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d1a98a10>)])]>

Standard JVM projects

For single-platform Kotlin projects, Gradle automatically selects the optimal variant via Gradle Module Metadata:

code_block: <ListValue: [StructValue([('code', '// build.gradle.kts\r\ndependencies {\r\n implementation("com.google.genai:google-genai-kotlin:1.0.0")\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21cbe91e50>)])]>

2. Unary and streaming text generation and chat

The primary entry point is the Client class. It manages HTTP connections and authentication automatically based on your environment variables (GEMINI_API_KEY or GOOGLE_API_KEY for Google AI Studio, and GOOGLE_GENAI_USE_ENTERPRISE=true with standard Google Cloud Application Default Credentials).

Single prompt request with Gemini Flash

Using Kotlin's use extension ensures the client's underlying network engine and HTTP connections are released cleanly:

code_block: <ListValue: [StructValue([('code', 'import com.google.genai.kotlin.Client\r\nimport kotlinx.coroutines.runBlocking\r\n\r\nfun main() = runBlocking {\r\n Client().use { client ->\r\n val response = client.models.generateContent(\r\n model = "gemini-flash-latest",\r\n text = "Explain quantum entanglement in two sentences."\r\n )\r\n\r\n println(response.text)\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21cbe91890>)])]>

Low-latency streaming with Coroutines `Flow`

For interactive UIs and responsive CLI tools, generateContentStream returns a cold Kotlin Coroutine Flow<GenerateContentResponse>, delivering token chunks in real time:

code_block: <ListValue: [StructValue([('code', 'import com.google.genai.kotlin.Client\r\nimport kotlinx.coroutines.runBlocking\r\n\r\nfun main() = runBlocking {\r\n Client().use { client ->\r\n val responseFlow = client.models.generateContentStream(\r\n model = "gemini-flash-latest",\r\n text = "Outline the key architectural patterns for microservices on Google Cloud."\r\n )\r\n\r\n responseFlow.collect { chunk ->\r\n chunk.text?.let { print(it) }\r\n }\r\n println()\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21cbe93790>)])]>

Multi-turn conversations (chat)

Managing conversation history manually across request turns can become tedious. The SDK includes a dedicated chats service that automatically maintains context, appends turns, formats conversation history, and handles function calling:

code_block: <ListValue: [StructValue([('code', 'import com.google.genai.kotlin.Client\r\nimport com.google.genai.kotlin.types.Content\r\nimport com.google.genai.kotlin.types.GenerateContentConfig\r\nimport kotlinx.coroutines.runBlocking\r\n\r\nfun main() = runBlocking {\r\n Client().use { client ->\r\n val config = GenerateContentConfig(\r\n systemInstruction = Content.fromText("You are an expert Google Cloud Solutions Architect.")\r\n )\r\n\r\n // Create a multi-turn chat session\r\n val chat = client.chats.create(\r\n model = "gemini-flash-latest",\r\n config = config\r\n )\r\n\r\n // Turn 1\r\n val firstResponse = chat.sendMessage("We are designing an event-driven ingestion pipeline on Google Cloud.")\r\n println("Gemini: ${firstResponse.text}\\n")\r\n\r\n // Turn 2: context from the first turn is included automatically\r\n val secondResponse = chat.sendMessage("Which managed messaging service should we choose: Pub/Sub or Kafka?")\r\n println("Gemini: ${secondResponse.text}\\n")\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21cbe92310>)])]>

You can also use chat.sendMessageStream(...) for streaming multi-turn chat responses.

3. Multimodal analysis grounded with Google Search

Gemini's multimodal reasoning is especially effective when combined with external verification. For instance, when analyzing technical, medical, or scientific diagrams, you can attach Google Search Grounding to cross-check factual claims against live web sources.

code_block: <ListValue: [StructValue([('code', 'import com.google.genai.kotlin.Client\r\nimport com.google.genai.kotlin.types.*\r\nimport java.io.File\r\nimport kotlinx.coroutines.runBlocking\r\n\r\nfun main() = runBlocking {\r\n Client().use { client ->\r\n val imageBytes = File("src/main/resources/medical_diagram.png").readBytes()\r\n\r\n val content = Content(\r\n parts = listOf(\r\n Part(inlineData = Blob(mimeType = "image/png", data = imageBytes)),\r\n Part(text = "Is this anatomical diagram accurate? Verify labels against authoritative medical sources.")\r\n )\r\n )\r\n\r\n // Enable Google Search as a grounding tool\r\n val config = GenerateContentConfig(\r\n tools = listOf(Tool(googleSearch = GoogleSearch()))\r\n )\r\n\r\n val response = client.models.generateContent(\r\n model = "gemini-flash-latest",\r\n content = content,\r\n config = config\r\n )\r\n\r\n println("=== Analysis ===")\r\n println(response.text)\r\n\r\n // Inspect citations and search queries\r\n val grounding = response.groundingMetadata\r\n println("\\n=== Search Queries Executed ===")\r\n grounding?.webSearchQueries?.forEach { println("- $it") }\r\n\r\n println("\\n=== Grounding Sources ===")\r\n grounding?.groundingChunks?.mapNotNull { it.web }?.forEach { source ->\r\n println("- ${source.title}: ${source.uri}")\r\n }\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21cbe919d0>)])]>

4. Visual generation and conversational editing: The Gemini 3 image family

The SDK provides full support for Google's latest image generation models (popularly known as the Nano Banana series of models on leaderboards).

Generating and Saving an Image

Generated image bytes are delivered directly in the response parts as a Blob:

code_block: <ListValue: [StructValue([('code', 'import com.google.genai.kotlin.Client\r\nimport java.io.File\r\nimport kotlinx.coroutines.runBlocking\r\n\r\nfun main() = runBlocking {\r\n Client().use { client ->\r\n val response = client.models.generateContent(\r\n model = "gemini-3.1-flash-image", // Nano Banana 2\r\n text = "A photorealistic blueprint of an eco-friendly modern datacenter, isometric view, 4k"\r\n )\r\n\r\n val imagePart = response.parts?.firstOrNull { it.inlineData != null }\r\n imagePart?.inlineData?.data?.let { bytes ->\r\n File("datacenter_blueprint.png").writeBytes(bytes)\r\n println("Image generated and saved successfully.")\r\n }\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21cbe92650>)])]>

Conversational image-to-image editing

You can pass existing images and conversational edit instructions in the same request:

code_block: <ListValue: [StructValue([('code', 'val originalImage = File("input.png").readBytes()\r\n\r\nval editPrompt = Content(\r\n parts = listOf(\r\n Part(inlineData = Blob(mimeType = "image/png", data = originalImage)),\r\n Part(text = "Change the daylight illumination to a dramatic twilight skyline with illuminated windows.")\r\n )\r\n)\r\n\r\nval editResponse = client.models.generateContent(\r\n model = "gemini-3-pro-image", // Nano Banana Pro\r\n content = editPrompt\r\n)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21cbe90110>)])]>

5. Real-time bidirectional interaction with Gemini Live

For low-latency voice, audio, and live multimodal interactions, the SDK supports the Gemini Live API via persistent WebSocket connections using client.live.connect(...):

code_block: <ListValue: [StructValue([('code', 'import com.google.genai.kotlin.Client\r\nimport com.google.genai.kotlin.types.AudioTranscriptionConfig\r\nimport com.google.genai.kotlin.types.LiveConnectConfig\r\nimport kotlinx.coroutines.launch\r\nimport kotlinx.coroutines.runBlocking\r\n\r\nfun main() = runBlocking {\r\n Client().use { client ->\r\n val model = if (client.enterprise) "gemini-live-2.5-flash-native-audio"\r\n else "gemini-3.1-flash-live-preview"\r\n\r\n val config = LiveConnectConfig(\r\n outputAudioTranscription = AudioTranscriptionConfig()\r\n )\r\n\r\n // Establish real-time bidirectional WebSocket session\r\n client.live.connect(model, config).use { session ->\r\n println("Connected to Gemini Live session!")\r\n\r\n // Launch collector for server messages (audio and text transcriptions)\r\n val receiveJob = launch {\r\n session.receive().collect { serverMessage ->\r\n serverMessage.serverContent?.outputTranscription?.text?.let { text ->\r\n print(text)\r\n }\r\n }\r\n }\r\n\r\n // Stream real-time text (or raw PCM audio blobs via session.sendRealtimeInput(audio = ...))\r\n session.sendRealtimeInput(text = "Hello Gemini! Give me a 5-second motivational quote.")\r\n\r\n // When finished, clean up\r\n receiveJob.cancel()\r\n session.closeSession()\r\n }\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d11b8790>)])]>

6. Structured tool and function calling

When building agentic workflows or bridging LLMs with backend microservices, developers can pass structured JSON schemas via FunctionDeclaration. The model will intelligently select when to invoke the tool:

code_block: <ListValue: [StructValue([('code', 'val telemetryTool = FunctionDeclaration(\r\n name = "getDatacenterMetrics",\r\n description = "Fetch real-time CPU and thermal telemetry for a Google Cloud region",\r\n parameters = Schema(\r\n type = Type.OBJECT,\r\n properties = mapOf("region" to Schema(type = Type.STRING)),\r\n required = listOf("region")\r\n )\r\n)\r\n\r\nval response = client.models.generateContent(\r\n model = "gemini-flash-latest",\r\n text = "Check telemetry for europe-west1",\r\n config = GenerateContentConfig(\r\n tools = listOf(Tool(functionDeclarations = listOf(telemetryTool)))\r\n )\r\n)\r\n\r\nresponse.functionCalls?.firstOrNull()?.let { call ->\r\n println("Model triggered tool: ${call.name} with arguments: ${call.args}")\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21cbe93650>)])]>

Additionally, when using the chats service, you can take advantage of Automatic Function Calling (AFC), which means that functions declared in the chat conversation can be invoked automatically and transparently by the SDK on your behalf, as you can see in the following example:

code_block: <ListValue: [StructValue([('code', 'fun main() = runBlocking {\r\n // A mocked function\r\n val getWeather = callableFunction("get_weather", paramName = "city") { city: String ->\r\n "18 degrees and sunny in $city"\r\n }\r\n\r\n Client().use { client ->\r\n val chat = client.chats.create(\r\n model = "gemini-flash-latest",\r\n automaticFunctionCalling = AutomaticFunctionCalling(getWeather),\r\n )\r\n\r\n // SDK calls get_weather if needed in this conversation\r\n println(chat.sendMessage("What is the weather in Zurich?").text)\r\n }\r\n}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d11ba150>)])]>

What's next?

With the 1.0 release of the Google Gen AI SDK for Kotlin, Kotlin developers across backend server ecosystems (Ktor, Spring Boot, Quarkus, Micronaut) now have a clean, multiplatform foundation for building generative AI applications.

To learn more and get started, check out the following resources:

GitHub Repository: Check out the source, stars, and discussions at github.com/googleapis/kotlin-genai.
Documentation and Samples: Explore the Kotlin Gen AI sample suite.
Feedback: File issues, suggest features, or submit pull requests directly on GitHub.

We look forward to seeing what you build with Kotlin and Gemini!

Simplify your resilience testing strategy with Fault Injection Testing

Wed, 26 Aug 2026 16:00:00 +0000

When databases fail and network paths falter, you still need your mission-critical cloud services to stay online. Yet guaranteeing high availability has become increasingly difficult because of the complexity of modern distributed systems.

To help you maintain availability and reliability during adverse events, we’re announcing Fault Injection Testing in preview. Fault Injection Testing is designed to help developers and architects automate failure testing to ensure predictable behavior during disruptions.

By deliberately introducing faults into your environment, you can verify your safety mechanisms before an actual outage impacts your customers.

Why native resilience testing matters

Unlike in self-hosted data centers, cloud applications offer less direct access to underlying infrastructure to facilitate failover testing.

Without native tools to prove your application can survive a failure, you risk a critical gap in your reliability strategy that exposes you to several risks:

Damaged trust and reputation: Frequent failures or poor performance lead to customer dissatisfaction and long-term damage to your brand's image.
Compliance and regulatory penalties: For many industries, particularly financial institutions, failing to prove disaster recovery capabilities can lead to non-compliance, audits, and fines.
Migration delays: Large-scale migrations often stop when teams cannot verify that critical applications will remain stable during a zone failure.

How Fault Injection Testing works

Fault Injection Testing allows you to run experiments by creating experiment templates. These templates act as blueprints, defining the specific fault to be injected and the resources that will be targeted for the experiment.

In this public preview, you can test two primary failure scenarios:

Failover Cloud SQL: This fault triggers a failover of a high availability Cloud SQL instance from the primary zone to a standby zone.
Degrade application traffic: This allows you to selectively add latency and HTTP error codes through an Application Load Balancer.

Before any fault is injected, Fault Injection Testing performs an automated dry run. This read-only simulation checks your permissions and provides an up-to-date list of every resource that will be affected.

Once you verify the scope, you can manually start the injection. The duration you defined in the template will run its course, and the faults will be reverted at the expiration of the timer.

During the experiment, you can verify that your application is behaving as you planned. If things do not go as planned, you can use the stop and revert capability to immediately halt the experiment and begin restoring resources to their normal state.

During preview, we recommend as a best practice to use Fault Injection Testing (FIT) in a non-production environment. Preview is an opportunity to get early access to learn how the service fits and complements your existing testing practices, and to provide us with your feedback to improve the product as well!

Built for the enterprise

Partners like KeyBank and Servier are already using Fault Injection Testing to validate their deployments. By using native fault injection, these organizations can approximate demanding failure scenarios — such as zonal outages — to help ensure their services remain stable.

Get started with Fault Injection Testing

Fault Injection Testing is available through the Google Cloud console, the gcloud CLI, and REST APIs.

Request preview access: Talk to your Google Cloud Account Team to add your project to the preview.
Enable the API: Search for "Fault Testing API" in your Google Cloud console and select enable.
Assign roles: Ensure your team has the roles/faulttesting.operator role to configure and run experiments.
Run your first dry run: Create a template for a Cloud SQL or load balancer resource in a non-production environment and execute a dry run to see the potential impact.

For more details on implementation, talk to your account team, or view the User Guide for Fault Injection Testing.

How Uber improves network reliability while unblocking cloud migration

Wed, 26 Aug 2026 16:00:00 +0000

Uber has a lot in common with the cities it serves. Both are always changing and growing, both must carefully manage the resulting traffic to prevent congestion and sprawl.

Uber has continuously evolved its technical strategies to manage its expanding network, and this careful planning and constant evolution helps ensure that application traffic across its entire platform runs smoothly. Ultimately, maintaining a reliable, high-scale platform that operates seamlessly at any given time is key to preserving user trust.

One important solution in this effort has been application awareness on Cloud Interconnect. An industry-first tool for application prioritization across hybrid networks, application awareness on Cloud Interconnect has helped Uber prioritize critical traffic to ensure business continuity during potential network congestion events.

Uber acted as an early design partner for application awareness on Cloud Interconnect, helping ensure that this capability met the demands of Uber’s global-scale operations. It not only improved Uber’s daily operations, it also gave Uber the confidence to move forward with a Google Cloud migration, with confidence that there would be less risk of service interruptions during switchovers.

In this post, we’ll explain the features Uber most sought and why, the inner workings of application awareness on Cloud Interconnect, and how it can help other organizations as well.

Prioritizing critical traffic

When migrating distributed, hybrid, or multicloud applications at a global scale, network reliability becomes a primary concern. Even the most worthwhile migrations may not seem worth it if such migrations interrupt ongoing service. For organizations like Uber, moving vast amounts of data to support large data analytics workload — including emerging AI use cases — can saturate network links, resulting in increased reliability risk for their critical application traffic.

With standard cloud interconnect approaches, enterprises typically apply simple bandwidth overprovisioning to meet extreme infrastructure needs. But with today's hybrid cloud demands, and given the size of an organization like Uber, overprovisioning network capacity for peak usage is often too costly and unreliable.

The shortcomings of overprovisioning only become magnified with the integration of cutting-edge AI innovations. Uber needs systems in place that can take on massive data transfers without congesting its network and protecting the performance of business-critical applications.

With the benefit of application awareness on Cloud Interconnect, including the four major features of application awareness — traffic handling, congestion response, latency management, and cost efficiency — Uber was able to achieve the networking optimization its modern tech stack requires.

Starting with a private preview, Uber deployed this feature across its infrastructure, beginning with Google Cloud Interconnect deployments in Phoenix, Arizona, and Ashburn, Virginia. Application awareness on Cloud Interconnect allows Uber to classify and prioritize end-user application traffic over less time-sensitive data using DSCP marking and configured queuing profiles.

In the following chart, we look at the four key features of application awareness on Cloud Interconnect, how they differ from legacy approaches, and how they help provide better operational continuity for organizations like Uber.

Feature	Standard interconnect solutions	Application awareness on Cloud Interconnect
Traffic handling	All traffic treated equally (first-in, first-out)	Traffic classified into six distinct traffic classes
Congestion response	High-priority application traffic may be dropped during bursts	Business-critical traffic is protected via strict priority or bandwidth sharing policies
Latency management	Unpredictable latency for high priority applications	Predictable and consistent low-latency for time-sensitive workloads
Cost efficiency	Requires expensive overprovisioning to absorb peaks	Efficient bandwidth utilization and lower TCO

Uber's key takeaways

For Uber, the business value of being able to prioritize business-critical traffic on its networks by deploying application awareness on Cloud Interconnect was immediate. And in doing so, Uber has also created a blueprint that other enterprises with similar hybrid cloud challenges can replicate. The core elements of that blueprint include:

Ensuring business continuity: Uber can decide in real time which application traffic to prioritize during major, high-traffic events. This means that mission critical applications stay up and running during even extreme events (both planned and unplanned). Uber leadership has called application awareness on Cloud Interconnect important for its global operations.
Efficient bandwidth utilization: Instead of blindly overprovisioning bandwidth to prevent congestion, application awareness allows Uber to better utilize their existing Cloud Interconnect capacity aligned with their expected network bandwidth needs. The result is lower total cost of ownership for network infrastructure.
Unblocked workload migration: By protecting critical applications from network congestion, Uber was able to migrate significant workloads to Google Cloud and, in the process, dramatically reduce operational overhead.

"Application awareness on Cloud Interconnect was the key that unlocked our ability to migrate more strategic workloads to Google Cloud and is critical for maintaining service reliability during peak global demand. By allowing us to intelligently prioritize traffic, it helps us ensure that we can protect our higher priority services and make our infrastructure more efficient, lowering our total cost of ownership. This wasn't just a feature deployment; it was a deep engineering partnership that delivered a solution critical to our business." – Harry Liu, Director of Engineering, Uber

Securing network reliability for AI and beyond

As more enterprises integrate cloud-based AI models, distributed applications, and data analytics, it's becoming a business imperative to be ready to handle the massive data transfers that follow. But in doing so, they also have to ensure they never compromise the reliability of their critical applications.

With application awareness on Cloud Interconnect, Uber demonstrated that moving beyond simple bandwidth overprovisioning to protect business-critical traffic was an essential step to building the stability required to embrace modern hybrid and multicloud strategies.

You can read our blog about the potential of Cloud Interconnect across industries to learn more about what the service can bring to your organization, and if you’re ready to explore more, our team of networking and industry experts are ready to help.

Your chance to start building AI agents from the absolute basics

Wed, 26 Aug 2026 09:07:00 +0000

Have you been hearing a lot about "AI agents" lately but aren't sure how to actually start building them? You don't need a background in machine learning or years of software experience to get started. The best way to learn is by doing, which is why we built Agent Valley.

Agent Valley is a free, 5-week live learning series designed to take you from scratch to building your very own hands-on agent systems. And instead of staring at boring terminal lines, you’ll be building and playing inside a tiny, low-poly virtual world!

Meet your instructor

You’ll be learning directly from Annie Wang, one of our top Google DevRel Engineers. She designed this course from the ground up to be fully hands-on, interactive, and beginner-friendly. If you want to learn how AI systems are built by the people actually designing them at Google, this is your chance.

How we'll learn together

You’ll learn by building in a split-screen workspace on your laptop. On Day 1, you'll describe and summon a custom low-poly companion that serves as your play character and save file. As you guide your companion through the valley's five districts, a live Runtime Inspector sits right beside the game, showing you exactly what the AI is thinking, deciding, and costing in real-time. Setup is completely zero-stress. Google will provide the environment for running these exercises, so you can dive straight into building.

Agent 101 Live with 5 modular sessions (Jump in anytime!)

Week 1: The Summoning Grove (CONTROL) · Get started by summoning your companion and learning how to keep its memory and traits consistent across a conversation.
Week 2: The Buildyard (DECOMPOSE) · Learn how to break a big project down so multiple AI assistants can work together in parallel without stepping on each other's toes.
Week 3: Market Street (COORDINATE) · Open up a virtual shop! You'll learn how to write reliable code so transactions and returns go smoothly without crashing.
Week 4: The Archive (REMEMBER) · Give your companion a memory. Learn how to help your agent remember past details without getting confused or making things up.
Week 5: The Night Market (LIVE) · The grand finale. Learn how to make your agent react live to events in the world (like fireworks or stage lights) while keeping the system fast and affordable.

Join the livestream

5 Tue starting Sep 1 · 10:00 AM (Pacific Time)
Anyone new to AI agents who wants to learn by coding and playing.
RSVP Here: goo.gle/agent101

10 questions every startup should answer before moving to production with their AI prototype

Thu, 20 Aug 2026 16:00:00 +0000

It’s never been easier to start an AI-powered startup on Google Cloud.

You grab an API key from Google AI Studio at breakfast, paste it into Antigravity, and by lunch you’ll have a nascent prototype of your product.

But it’s not all one straight line to progress. It's common to bump into these three challenges as you build out your stack:

A leaked API key racks up a large bill in 48 hours.
A "quick" migration from AI Studio to Gemini Enterprise Agent Platform stalls the roadmap for weeks because nobody on the team owns Identity and Access Management (IAM).
The launch works, until the app starts returning HTTP 429 Too Many Requests because of default per-project quotas, and there's no clean path to more capacity without paying a premium.

None of these are unique edge cases. . They're default failure modes of moving fast without a plan, and we've all done it at least once.

Below are the 10 questions every startup should be ready to answer before they scale, grouped into the three phases where decisions can shape your future:

Onboard (setting up your own projects and identities right)
Scale (getting more throughput without breaking the bank)
Govern (keeping costs, keys, and agents from running away).

These ten are scoped to the prototype-to-production transition itself. Each question ends with a short, runnable snippet you can copy into your own project today. Adjacent decisions that matter just as much but aren't specific to that move, your data layer and RAG architecture, CI/CD, network design, are deliberately out of frame here.

Onboard: get the foundation right (in the first hour).

#1 Where should I start: Google AI Studio or Gemini Enterprise Agent Platform?

Both surfaces expose the same Gemini family of models, but they solve different problems.

Google AI Studio (with the Gemini Developer API) is the fastest path from an idea to working code. A browser IDE, an API key, a generous free tier, and no cloud project to configure. It's where most ideas should start, and Google's own guidance says as much.
Gemini Enterprise Agent Platform (formerly Vertex AI) has the same Gemini models (plus 3rd party and OSS ones) with enterprise controls around them: IAM and service-account auth instead of raw keys, VPC Service Controls, Cloud Logging and Monitoring, reserved capacity, regional endpoints, and the compliance surface your first enterprise customer's security review will ask about.

The right answer for most startups is both, sequenced deliberately: first prototype in AI Studio, then migrate before you have real users. The danger for startups is treating them as interchangeable solutions, AI Studio's simple key model does not translate to enterprise controls, and Agent Platform's IAM model might look like overkill until the day it saves you from a stolen-credential incident.

It's less work than it sounds like.

The unified google-genai SDK targets both:

code_block: <ListValue: [StructValue([('code', '# Prototype: Google AI Studio, raw API key\r\nfrom google import genai\r\nclient = genai.Client(api_key="YOUR_AI_STUDIO_KEY")\r\n\r\n# Production: GEAP, no key — uses Application Default Credentials (ADC)\r\nfrom google import genai\r\nclient = genai.Client(\r\n vertexai=True,\r\n project="my-startup-prod",\r\n location="us-central1",\r\n)\r\n\r\nresp = client.models.generate_content(\r\n model="gemini-2.5-pro",\r\n contents="Summarize this contract in three bullets.",\r\n)\r\nprint(resp.text)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d10fa710>)])]>

#2 How do I set up a Google Cloud project without becoming an IAM expert?

The biggest reason startups stall on the migration to Agent Platform isn't the code, it's the operational leap from "here's an API key" to a cloud project with folders, service accounts, org policies, logging, and IAM bindings. If your team doesn't have a dedicated cloud admin, that first project setup can eat a week of engineering time.

Three moves cut that dramatically:

Use an opinionated project template instead of clicking through the console. The Cloud Setup checklist and the Google Cloud Architecture Framework give you a production-grade folder hierarchy (prod / non-prod / dev), a central logging + monitoring project, Security Command Center turned on, and baseline org policies, without you having to design them from scratch.
Enable the APIs you'll actually use, once. Batch it so you're not doing it project-by-project when you need it. The billing-link step is not optional. Every paid API you're about to enable will refuse to activate on a project with no billing account attached, so we handle that first.
Let Gemini pick the roles, but ask it for the narrow ones. You don't have to memorize the roles reference. In the Grant access dialog, Help me choose roles lets you describe the task in plain language, "this service account needs to call Gemini models and read one Cloud Storage bucket", and get predefined roles back with the reasoning shown. One catch worth knowing on day one: by default it suggests roles that cover common journeys, which usually means a service's Admin, Editor, or Viewer. Those are broader than you want. Say "least privileged" or "narrowest access" in the prompt and it returns granular roles instead. Same amount of typing, considerably smaller blast radius when a credential leaks.

Sources: Get predefined role suggestions with Gemini assistance

code_block: <ListValue: [StructValue([('code', '# One-shot: create a Vertex-ready project and turn on the services a\r\n# typical AI startup uses.\r\ngcloud projects create my-startup-prod --name="My Startup (prod)"\r\ngcloud config set project my-startup-prod\r\n\r\n# REQUIRED before enabling billing-dependent APIs (aiplatform, run, etc.).\r\n# Use `gcloud billing accounts list` to find your billing account ID.\r\ngcloud billing projects link my-startup-prod --billing-account=012345-6789AB-CDEF01\r\n\r\ngcloud services enable \\\r\n aiplatform.googleapis.com \\\r\n run.googleapis.com \\\r\n artifactregistry.googleapis.com \\\r\n logging.googleapis.com \\\r\n monitoring.googleapis.com \\\r\n secretmanager.googleapis.com \\\r\n cloudbilling.googleapis.com'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d18ab210>)])]>

Sources: gcloud services enable reference, · gcloud billing projects link (GA), GE Agent Platform environment setup.

If you're a solo founder, resist the urge to build in your personal GCP account. Create a proper organization or self-owned org first, then create the project inside it. That single decision can make everything else, fromIAM to billing and audit, dramatically easier.

#3 I'm on Google Cloud, how should my code actually authenticate: API keys, service accounts, or user credentials?

There's a hierarchy of safety here, and the easiest option is rarely the right one in production.

Raw API keys are fine for local prototyping. They are dangerous in production because they are long-lived, easy to leak into a client bundle or a public repo, and grant unbounded access until you notice.
User credentials via OAuth (application default credentials) are best for interactive tools, CLIs, and any code that runs on a developer's laptop.
Service accounts with least-privilege IAM roles are the right answer for anything running on a server, in a container, or in a scheduled job.

The pattern you're aiming for is one where your code never sees a key at all. It just calls the Google Auth library, which quietly reads Application Default Credentials (ADC) from the environment, a short-lived token minted for whichever service account is attached to your Cloud Run service, GKE workload, or Compute Engine VM. You get enterprise-grade auth without writing any auth code.

code_block: <ListValue: [StructValue([('code', '# On a developer laptop\r\ngcloud auth application-default login\r\n\r\n# On a server (Cloud Run, GKE, etc.) — no login, no key file.\r\n# Attach a service account with just the roles the app needs.\r\ngcloud run deploy my-agent \\\r\n --image=us-docker.pkg.dev/my-startup-prod/agents/api:v1 \\\r\n --service-account=agent-runtime@my-startup-prod.iam.gserviceaccount.com \\\r\n --region=us-central1'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d18a9c50>)])]>

code_block: <ListValue: [StructValue([('code', '# Application code — notice: no keys, no secrets.\r\nfrom google import genai\r\n\r\nclient = genai.Client(\r\n vertexai=True,\r\n project="my-startup-prod",\r\n location="us-central1",\r\n)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d18ab190>)])]>

Do one last favor to your future self: give that service account the minimum IAM role your workload actually needs, usually roles/aiplatform.user for calling models, not the broader admin roles. It takes an extra 30 seconds and prevents the credential from becoming a master key if it leaks.

#4 When should I actually stop procrastinating and migrate from AI Studio's API key to Agent Platform's IAM model?

Sooner than you'd like, and the correct trigger is not when it breaks. It's when any of these is true:

Your key has left your laptop (checked into a repo, pasted into a Slack, shipped in a mobile app).
You have more than one person on the team who needs to call the API.
You're spending more than a few hundred dollars a month.
You're about to onboard paying customers.

A potential pitfall that can catch growing startups off guard is simple: a leaked Gemini API key on an account that normally spends $180 a month gets scraped from a public repo and used to run distillation attacks, accumulating tens of thousands of dollars in charges before the owner even sees the first billing alert. The Google Cloud Shared Responsibility Model is unambiguous: the customer is liable for charges incurred with their own valid credentials.

The migration itself is genuinely smaller than the anxiety around it. In google-genai it's the two-line change shown in #1. What takes real time is the project setup around it, which is exactly why #2 exists.

Practical checklist for cutover day:

code_block: <ListValue: [StructValue([('code', '# 1. Revoke every existing AI Studio key that has ever left a laptop.\r\n# (Go to https://aistudio.google.com/apikey and delete them.)\r\n\r\n# 2. Confirm your production code has no api_key= arguments.\r\ngrep -rn "api_key" src/\r\n\r\n# 3. Enable GEAP and confirm ADC works locally.\r\ngcloud services enable aiplatform.googleapis.com\r\ngcloud auth application-default login\r\npython -c "\r\nfrom google import genai\r\nc = genai.Client(vertexai=True, project=\'my-startup-prod\', location=\'us-central1\')\r\nprint(c.models.generate_content(model=\'gemini-2.5-flash\', contents=\'ping\').text)\r\n"'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d18a9c90>)])]>

If step 3 prints a response, you're on Agent Platform.

Scale: get more capacity without paying a premium.

#5 Now that I'm shipping, why on earth am I getting all these HTTP 429 errors, and how do I make them stop?

429 Too Many Requests from Agent Platform almost always means one of two things:

You've hit the Dynamic Shared Quota (DSQ) ceiling for your project's tier. DSQ is a shared pool sized against your project's history, new projects start with modest limits by design, to prevent abuse across the platform.
You're calling a global endpoint during a global demand spike, competing with worldwide traffic for shared capacity.

The instinctive reaction is to file a quota-increase ticket. You can do that if you must, but two architectural moves usually solve the problem faster and cheaper.

Pin to a regional endpoint. Over half of startup traffic on Agent Platform defaults to global routing. Pinning to a specific region (say us-central1) sidesteps global contention and typically improves latency at the same time. (One narrow exception, which we'll get to in the next question: if you specifically want Priority PayGo, that feature currently only ships on the `global` endpoint. For everything else, pin regionally.):

code_block: <ListValue: [StructValue([('code', 'from google import genai\r\n\r\n# Global (default): competes against worldwide demand.\r\n# Regional: routes only to the regional cluster, less contention.\r\nclient = genai.Client(\r\n vertexai=True,\r\n project="my-startup-prod",\r\n location="us-central1", # <-- this is the one-line fix\r\n)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d182a450>)])]>

Add real retry and backoff. A 429 is a retryable signal, not a fatal error. Any production client should have exponential backoff with jitter. The modern google-genai SDK ships this behavior built in, but only if you actually enable it. This is easy to overlook. Don't reach for the classic `google.api_core.retry.if_transient_error` decorator you may have seen on older Vertex code. It's designed for the legacy exception classes and does not recognize the new `google.genai.errors.APIError, so it will silently pass 429s through without retrying. Use the SDK's built-in retry options instead:

code_block: <ListValue: [StructValue([('code', 'from google import genai\r\nfrom google.genai import types\r\n\r\nclient = genai.Client(\r\n vertexai=True, project="my-startup-prod", location="us-central1",\r\n http_options=types.HttpOptions(retry_options=types.HttpRetryOptions(\r\n attempts=5, initial_delay=1.0, max_delay=60.0, exp_base=2.0, jitter=1.0,\r\n http_status_codes=[408, 429, 500, 502, 503, 504],\r\n ))\r\n)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d1829050>)])]>

How do you see this coming? Preferably not from a user telling you. Agent Platform publishes serving metrics to Cloud Monitoring, and there is a prebuilt dashboard you don't have to assemble: Console → Agent Platform → Dashboard → Model observability. It gives you requests per second, token throughput, first-token latency, and error rates out of the box.

The metric to actually alert on is aiplatform.googleapis.com/publisher/online_serving/model_invocation_count. It carries an error_category label with values of user, system, or capacity. Alerting on capacity isolates genuine throttling from your own bad requests, which a raw 429 count won't do.

One thing worth internalizing, because it trips people up: you cannot build a "warn me at 80% of my quota" alert for Standard PayGo. Under Dynamic Shared Quota there is no fixed per-project number to be at 80% of. A 429 means transient contention for shared capacity, not that you crossed a line. Percent-of-limit alerting only becomes meaningful once you're on Provisioned Throughput, which does expose real limit metrics.

code_block: <ListValue: [StructValue([('code', 'gcloud monitoring policies create --policy-from-file=capacity-alert.yaml'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d182ad90>)])]>

Sources: Agent Platform metrics list, Model observability dashboard, RetryOptions source, core retry_base.py, genai errors.py, reduce 429 errors, gcloud monitoring policies create, Dynamic Shared Quota.

Follow the Agent Platform rate limits documentation to understand what your project's current ceiling actually is before you assume you've outgrown it.

#6 Which consumption mode do I pay for: Standard PayGo, Priority PayGo, or Provisioned Throughput?

Three consumption models, three completely different workload shapes, and three completely different ways to proceed. Picking the right one can help startups see meaningful savings on AI bills. First let’s define them and then see when they are, or aren’t, a good fit:

Standard PayGo (DSQ): Pay per token from a shared pool; cheap, no guarantees.
Priority PayGo: Pay per token at a premium to jump the queue.
Provisioned Throughput (PT): Prepay for reserved capacity; predictable, use it or lose it.

Consumption type	Best for	Watch out for
Standard PayGo (DSQ)	Early-stage, low-QPS, spiky prototype traffic	429s during spikes; no reliability SLO
Priority PayGo	Bursty, revenue-critical traffic that can't tolerate 429s	Roughly 1.8x the standard token price
Provisioned Throughput (PT)	Steady, predictable, high-volume production traffic	Wasted spend if utilization is under ~40%; overflow to PayGo on spikes

The dominant startup mistake is buying PT too early. Usually this happens the week after a big launch when it feels like traffic will only ever go up. PT is reserved capacity. You pay whether you use it or not, and it only starts paying you back once your baseline is genuinely predictable, not just aspirational.

Here’s a pragmatic sequence:

Weeks one through four on Standard PayGo. Use it to measure your real request shape (tokens per minute at p50 and p99, request bursts, batchable vs. real-time split).
When you get your first bad 429 storm, flip on Priority PayGo for the traffic that actually matters. It's a config change, not a purchase order, nobody in procurement needs to be involved:

code_block: <ListValue: [StructValue([('code', '# Priority PayGo request: use the global endpoint + two extra headers.\r\nfrom google import genai\r\nfrom google.genai import types\r\n\r\nclient = genai.Client(vertexai=True, project="my-startup-prod", location="global")\r\nresp = client.models.generate_content(\r\n model="gemini-2.5-pro",\r\n contents="Rank these support tickets by urgency: ...",\r\n config=types.GenerateContentConfig(\r\n # Priority PayGo headers, per current GEAP docs.\r\n http_options=types.HttpOptions(headers={"X-Vertex-AI-LLM-Request-Type": "shared", "X-Vertex-AI-LLM-Shared-Request-Type": "priority"}),\r\n ),\r\n)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d18280d0>)])]>

3. Once you can predict your baseline TPM, buy PT to cover the flat baseline and let anything above it overflow to PayGo. That's the combined pattern Google recommends for exactly this reason. Best of both worlds, not marketing spin.

Sources: Priority PayGo docs, google-genai HttpOptions source, GEAP REST reference.

#7 Which of my requests actually need to be live, and which should be batch jobs?

Most startup workloads are secretly batch jobs pretending to be real-time. Every one you move off the interactive path frees up DSQ headroom for the traffic that genuinely needs to be fast, the traffic where a user is actually watching a spinner.

Three questions to help you sort your traffic:

Does a human have to see the result within a second? That means: Live inference.
Can the user wait a few seconds and see a spinner? That means: Still live, but a candidate for streaming.
Would the user tolerate "we'll email you when it's ready" or "check back in a bit"? That means: Batch prediction.

Batch prediction on Agent Platform runs in a completely separate queue, does not consume your interactive DSQ, and is typically about half the price of on-demand inference. That's a rare double win: faster live traffic and a lower bill.

code_block: <ListValue: [StructValue([('code', '# Kick off a batch prediction job from a JSONL file in Cloud Storage.\r\n# Each line is one prompt; results land in another Cloud Storage prefix.\r\nfrom google import genai\r\nfrom google.genai import types\r\n\r\nclient = genai.Client(vertexai=True, project="my-startup-prod", location="us-central1")\r\n\r\njob = client.batches.create(\r\n model="gemini-2.5-flash",\r\n src="gs://my-startup-prod-batch/inputs/nightly-summaries.jsonl",\r\n config=types.CreateBatchJobConfig(\r\n dest="gs://my-startup-prod-batch/outputs/",\r\n ),\r\n)\r\nprint(job.name, job.state)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d1829310>)])]>

Common candidates: nightly document summarization, background classification of new signups, bulk translation, embedding backfills, evaluation runs against your test set. If any of those are on your live path today, moving them is often the single highest-leverage change you can make this week.

Govern: Keep costs, keys, and agents under control.

#8 How do I set spend caps that actually reduce cost, and not just send me polite emails while my bill triples?

Until recently the honest answer was that budgets only notify, and you had to build your own brake pedal. That changed in July. There are now three mechanisms, and you should think of them as layers.

A spend cap budget (Preview). Cloud Billing budgets can now enforce rather than just email. Set a spend cap on a project and, when usage costs cross 100% of the budget, Google pauses the service until you manually lift it. Agent Platform is explicitly on the eligible list, alongside the Gemini API, Cloud Run, and Cloud Run functions. Alerts still fire at 50% and 80%, so the pause isn't a surprise.

Three things to know before you rely on it:

Each cap covers one project and one eligible service. It is not account-wide protection. If you want Agent Platform and Cloud Run both capped, that's two caps.

Enforcement is not instant and is based on estimated costs. Overages past the cap are billed as normal, so set the number below your real ceiling. Lifting it is manual, and service resumption can take up to an hour. It also pauses Provisioned Throughput usage, so if you've prepaid for capacity, a cap hit stops that too.

It's in Preview as of publication, and the eligible-service list is documented as growing. Check the current list before you design around it.

2. A billing budget with a Pub/Sub trigger that disables billing. Still the right tool when you need blast radius the spend cap can't give you: multiple services at once, an entire project, or a service that isn't eligible yet. When the budget hits a threshold, Pub/Sub fires a Cloud Function that detaches the billing account, which stops all billable activity within minutes. Blunter and more dangerous than the native cap — it can leave resources unrecoverable — so reach for it second, not first. Full walkthrough: Automatically respond to budget notifications.

code_block: <ListValue: [StructValue([('code', '# Sketch: create a budget SCOPED TO ONE PROJECT that publishes to Pub/Sub at 50%, 90%, 100%.\r\ngcloud billing budgets create \\\r\n --billing-account=012345-6789AB-CDEF01 \\\r\n --display-name="my-startup-prod hard stop" \\\r\n --budget-amount=2000USD \\\r\n --filter-projects=projects/my-startup-prod \\\r\n --threshold-rule=percent=0.5 \\\r\n --threshold-rule=percent=0.9 \\\r\n --threshold-rule=percent=1.0,basis=current-spend \\\r\n --notifications-rule-pubsub-topic=projects/my-startup-prod/topics/budget-alerts'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d182a890>)])]>

Sources: Manage spend cap budgets, Set up programmatic notifications gcloud billing budgets create reference, Cloud Billing budgets concepts, Disable billing with notifications walkthrough, Programmatic notification payload schema.

Two things to get ahead of for, as the defaults can cause unexpected issues:

Limit your budget scope: Without --filter-projects, your budget applies to your entire billing account. A spike in any project will trigger the kill switch for everything.
Deploy locally: The budget notification doesn't specify which project is affected. To ensure the kill switch only affects the intended project, deploy your Cloud Function in the same project you're protecting (e.g., my-startup-prod).

Then wire up a tiny Cloud Function to that topic that calls projects.updateBillingInfo to unlink the billing account when the 100% threshold fires. That is your circuit breaker.

Mechanical ceilings via quota overrides. Even if you never set up the above kill switch, you can cap the rate at which cost can accumulate by setting explicit per-model, per-region quotas below the platform default. If your app never legitimately needs more than 500 requests per minute for gemini-2.5-pro, cap it there in the Cloud Quotas console; a leaked key can't burn what the quota flatly refuses to serve.

#9 Where should I actually keep secrets? (Not in .env files!)

The short answer is: Secret Manager. Not in environment variables, not in .env files, and never in your repo. Grant read access via IAM only to the service account that needs it.

code_block: <ListValue: [StructValue([('code', '# Store a third-party API key (Stripe, OpenAI, whatever).\r\necho -n "sk_live_xxx" | gcloud secrets create stripe-live-key --data-file=-\r\n\r\n# Grant only the runtime service account access to read it.\r\ngcloud secrets add-iam-policy-binding stripe-live-key \\\r\n --member=serviceAccount:agent-runtime@my-startup-prod.iam.gserviceaccount.com \\\r\n --role=roles/secretmanager.secretAccessor'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d1828910>)])]>

code_block: <ListValue: [StructValue([('code', '# Application code fetches it at startup; nothing lives on disk.\r\nfrom google.cloud import secretmanager\r\nsm = secretmanager.SecretManagerServiceClient()\r\nresp = sm.access_secret_version(\r\n name="projects/my-startup-prod/secrets/stripe-live-key/versions/latest"\r\n)\r\nstripe_key = resp.payload.data.decode("utf-8")'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d182a590>)])]>

Then two little disciplines that pay for themselves the first time you need them:

Rotation on a schedule and on suspicion. Secret Manager versions are cheap; treat them as immutable and roll forward.
Detection when a secret leaks. Secret Manager notifications and Google Cloud's Sensitive Data Protection can catch keys checked into a repo or pasted into a log stream, before an attacker does.

For any AI application that acts on a user's behalf, calls Gmail on their behalf, reads a Drive folder, hits a third-party SaaS with the user's credentials, do not store a long-lived token. Use OAuth 2.0 with short-lived access tokens and a refresh flow, so that when a user rage-quits or a compromised account gets revoked, the agent loses access at the same time.

#10 How do I stop my brand new AI agent from doing something it absolutely shouldn't?

An agent that can call tools, browse the web, or execute code needs the same defense-in-depth thinking as any other production service, arguably more, because it makes decisions that neither you nor the model can fully predict in advance.

Four layers, none optional once you have real users:

1. Identity for the agent itself. Give the agent its own service account, scoped only to the resources and tools it genuinely needs, the exact same least-privilege principle as any other workload. Agent Engine supports first-class agent identity so every action can be attributed to a specific agent instance in your audit logs.

2. Sandboxed code execution. If your agent runs generated code, a common pattern for data-analysis or "run this Python for me" flows, do not run it in your application process. Use an isolated sandbox so a bad combination can't touch your production data.

code_block: <ListValue: [StructValue([('code', '# Enable server-side code execution inside a sandbox for a request.\r\nfrom google import genai\r\nfrom google.genai import types\r\n\r\nclient = genai.Client(vertexai=True, project="my-startup-prod", location="us-central1")\r\nresp = client.models.generate_content(\r\n model="gemini-2.5-pro",\r\n contents="Compute the correlation between these two columns: ...",\r\n config=types.GenerateContentConfig(\r\n tools=[types.Tool(code_execution=types.ToolCodeExecution())],\r\n ),\r\n)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d1829550>)])]>

3. Prompt and response filtering. Model Armor sits in front of your model calls and screens for prompt injection, jailbreaks, sensitive-data exfiltration, and off-brand output, all of which are essentially guaranteed the moment you have real users being real users.

4. Behavioral monitoring. Security Command Center with threat detection flags anomalies in agent behavior, a service account suddenly calling an API it's never touched before, an agent reaching out to an unfamiliar external host, an unexpected spike in privileged operations. In near-real-time.

None of these are optional once your agent is acting on behalf of a real user or handling real money.

Your homework, so to speak:

Audit for raw API keys in your repo, your notebooks, and your production runtime. Rotate anything that shouldn't be there.
Move any workload that doesn't need a synchronous response to the Batch API.
Turn on the Model observability dashboard and put one alert on capacity errors, so the next 429 reaches you before it reaches a customer.
Set a spend cap on the project, and keep an eye out for 50% and 80% alerts. If usage crosses 100% of the budget, Google will pause the service until you manually lift it.

Do those four things this week and you're already ahead of most startups shipping AI features.

Have a scenario you'd like us to cover next? Reach us at Google Cloud for Startups.

Introducing the Developer Device Platform for agentic mobile app development

Mon, 10 Aug 2026 22:00:00 +0000

Most enterprises connect with their customers through a device. Whether it’s using a mobile app to order a product, contact customer service, view content, or manage their account, the customer experience depends on how well an app can run locally on the customer’s device.

For this reason, building and testing applications across a wide variety of devices is critical for any enterprise launch that involves locally running components. However, procuring and hosting devices at scale is expensive and complex, and tests are often flaky, inconclusive, or just difficult to debug. This leaves many developers to test launches on the physical phones in their pockets and hope the results apply to most devices.

To solve this challenge, today we are excited to announce the public preview launch of Developer Device Platform (DDP) on Google Cloud. DDP is a fully managed cloud platform that provides instant, on-demand access to multiple hardware profiles across real physical devices and high-concurrency virtual emulators. DDP represents an evolution of Firebase Test Lab for Cloud developers, and is also the first device platform built for agentic development. With DDP, developers can now utilize their preferred agents to vibe code apps, run tests, debug, and optimize performance across devices efficiently and quickly.

Build and test your apps to guarantee performance

In the standard mobile development lifecycle, developers iteratively build new features, run QA tests to ensure performance across a variety of target devices, and ship the optimized and debugged feature to production for their users. Developer Device Platform offers two main functions to accelerate this cycle:

Interactive debugging with Device Streaming: With our Device Streaming API, developers can directly access an emulator or physical device of their choice, and vibe code, iteratively test, debug and interact with the app remotely. Device streaming makes it simple to dive into your customer experience, and scroll and click in real time, all while also monitoring performance on the real device hardware.

Parallel testing with Device Run: With our Device Run API, developers can write tests as part of their CI/CD pipelines and run them in parallel across hundreds of different devices at once. With the results, developers can pinpoint and debug specific device issues, and ship code to production with confidence that it will run across device tiers.

Accelerate mobile app development with DDP agent skills and efficient tests

The rise of coding agents in mobile development specifically opens up new possibilities when paired with physical devices. Coding agents can interact and test on the real hardware, helping them take advantage of unique phone screen sizes (e.g., foldable phone UI) and specialized hardware (e.g., CPUs vs GPUs).

Developer Device Platform will soon integrate with Android Studio and Android CLI, giving you direct access to physical devices via Device Streaming API. The DDP agent skill will also allow you to work with the AI coding agents of your choice to accelerate development. With DDP agent skill, coding agents can:

Execute multi-step user journeys independently
Spot visual artifacts
Analyze real-time chip performance on-device
Validate fixes to hardware specific bugs and/or optimize for unique phone features

In addition to these agentic capabilities, DDP also enables developers to package apps and launch parallelized tests with smart sharding, giving you access to results across hundreds of devices in minutes. With smart auto-retries, DDP also retries specific tests that fail within your shards, helping you get past errors faster without rerunning your entire suite of tests.

Start building with Developer Device Platform today

Starting August 12, Developer Device Platform is available in public preview to all Google Cloud users. During public preview, users are charged based on a pay-per-minute model so you pay only for the active testing minutes you consume, with rates differing for emulator vs physical devices.

We can’t wait to see how Developer Device Platform can help mobile developers across Google Cloud accelerate their development and take advantage of the growing number of unique device features and on-device AI possibilities.

ClusterNetworkPolicy in GKE: Balancing control and autonomy for your microservices

Mon, 10 Aug 2026 16:00:00 +0000

Managing network security in a multi-tenant Kubernetes environment typically requires balancing two distinct needs: developers need their microservices to communicate effectively, while platform and security teams must maintain compliance, prevent lateral movement, and establish cluster-wide guardrails.

Historically, the standard Kubernetes NetworkPolicy has been the primary tool for this. While effective for single-namespace isolation, standard NetworkPolicy is scoped strictly to individual namespaces and designed around developer self-service. When cluster administrators attempt to use it for global security enforcement, it can lead to policy conflicts and operational challenges.

To address this, we introduced ClusterNetworkPolicy (CNP), an open-source standard developed by the Kubernetes SIG-Policy Working Group (WG), to Google Kubernetes Engine (GKE). Designed for scale, CNP is a cluster-wide resource that allows administrators to manage network security centrally, providing a mechanism for those responsible for global security to implement consistent, non-bypassable policies.

Read on for technical details about CNP, some common use cases, an example policy, and how to get started.

Structuring policies with tiers

A core capability of ClusterNetworkPolicy is its hierarchical tier system. Rather than attempting to reconcile flat, conflicting peer rules simultaneously, CNP establishes a deterministic, top-to-bottom evaluation hierarchy:

The admin tier: The highest precedence level. Rules here are enforced before any other policies.
The network policy tier: The standard namespace level, where developers manage their specific application policies.
The baseline tier: The lowest precedence, establishing the cluster’s default behavior when no other policies apply. This can be overridden using namespace scoped policies.

This tiered structure helps align network security with organizational roles. Using standard role-based access control (RBAC), you can manage the admin tier to enforce compliance mandates, while platform teams can use the baseline tier to set a default "deny-all" zero-trust posture across the cluster. At the same time, developers can write standard network policies for their applications without overriding core security mandates.

This deterministic, top-to-bottom evaluation method resolves conflicts between different teams' policies. The admin tier introduces an explicit Pass action. This allows security teams to inspect traffic against global rules and then delegate the final Accept or Deny decision down to the developer's namespace policy, facilitating both central oversight and distributed management.

Common network security scenarios

This tiered architecture translates complex security requirements into centralized rules. Here are common scenarios where ClusterNetworkPolicy provides a practical solution:

Isolating sensitive workloads: You can apply an admin-tier global deny rule to isolate specific namespaces — such as those used for payment processing or compliance data — from the rest of the cluster. This action overrides any permissive developer policies that might otherwise expose these environments.
Protecting core services: To prevent configurations that might disrupt internal operations, administrators can create an admin-tier global allow rule for critical services like kube-dns. This allows these services to remain accessible regardless of any misconfigured namespace policies.
Managing external egress: By utilizing IP address range matching, egress traffic can be controlled at the cluster level. This functionality allows you to explicitly restrict or permit access to corporate intranets or external IP ranges, serving as a safeguard against unauthorized data exfiltration.

Example scenario

Consider a common enterprise requirement: Application workloads across all namespaces must be permitted to reach central platform infrastructure (such as shared authentication and telemetry services), while access to sensitive environments — like a restricted vault namespace — is strictly prohibited. Meanwhile, routine microservice traffic is delegated to developer-managed, namespace-scoped policies.

ClusterNetworkPolicy makes this straightforward. A platform administrator simply defines an admin-tier guardrail centrally:

code_block: <ListValue: [StructValue([('code', 'apiVersion: policy.networking.k8s.io/v1alpha2\r\nkind: ClusterNetworkPolicy\r\nmetadata:\r\n name: platform-isolation-guardrail\r\nspec:\r\n tier: Admin\r\n priority: 10\r\n subject:\r\n # Target all application tenant namespaces, excluding system and core infrastructure\r\n namespaces:\r\n matchExpressions:\r\n - key: kubernetes.io/metadata.name\r\n operator: NotIn\r\n values: ["kube-system", "shared-services", "restricted-vault"]\r\n egress:\r\n # 1. Mandate access to central shared platform services\r\n - name: allow-shared-services\r\n action: Accept\r\n to:\r\n - namespaces:\r\n matchLabels:\r\n kubernetes.io/metadata.name: shared-services\r\n\r\n # 2. Enforce strict block on accessing the restricted vault namespace\r\n - name: block-restricted-vault\r\n action: Deny\r\n to:\r\n - namespaces:\r\n matchLabels:\r\n kubernetes.io/metadata.name: restricted-vault\r\n\r\n # 3. Explicitly delegate all remaining traffic to developer namespace policies\r\n - name: delegate-remaining-egress\r\n action: Pass\r\n to:\r\n - namespaces: {}\r\n - networks:\r\n - 0.0.0.0/0\r\n - ::/0'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f21d0893fd0>)])]>

Extending open-source foundations

Instead of building this functionality as proprietary extensions, we worked with the Kubernetes community to design the ClusterNetworkPolicy API (policy.networking.k8s.io), distinguishing it from the namespace-scoped NetworkPolicy API (networking.k8s.io). Furthermore, we collaborated closely with the Cilium community to build its implementation of the API.

Because it is built on open-source standards, GKE helps ensure that security configurations remain portable across different environments. The ClusterNetworkPolicy API natively supports tier selection, enabling clear and deterministic policy evaluation. This approach lets administrators enforce robust security guardrails while maintaining the operational flexibility that development teams depend on.

ClusterNetworkPolicy on GKE elevates workload network security — shifting operations from namespace-scoped rules to unified, cluster-wide governance. It is currently in preview in version 1.36 and later. To learn more and get started, check out: