ateapi: add ActorSnapshot lifecycle APIs by EItanya · Pull Request #570 · agent-substrate/substrate

Eitan Yarmush (EItanya) · 2026-07-28T18:46:12Z

Implements #529.

This branch is rebased onto #568's merged storage-layout commit.

creates a first-class immutable ActorSnapshot on every successful suspend
captures the exact source Actor version when suspension starts
exposes snapshots through Actor.latest_snapshot, GetActorSnapshot, and paginated listing
introduces separate Atespace-owned tags addressed as atespace/name
permits identical tag names in different Atespaces with O(1) indexed lookup
puts reuse policy on each tag: ATESPACE (the zero/default value) or PUBLISHED
keeps a published tag's Atespace-owned identity stable; there is no global tag namespace
blocks Atespace deletion while Actors or ActorSnapshotTags remain
initializes new Actors only through tag references and enforces tag scope
requires tags to belong to the source snapshot's Atespace
requires the exact source ActorTemplate UID for full and data snapshots
rejects cloning templates with external volumes until CSI/provider snapshot support exists
retains snapshot metadata while explicit deletion and automatic GC/retention remain deferred
removes embedded snapshot tags, snapshot-level visibility, and DeleteActorSnapshot
extends kubectl-ate to list/get snapshots, create/update/delete snapshot tags, and create Actors from tags
adds a live demo E2E covering suspend, snapshot lookup/listing, tag publication, clone restore, and tag deletion

Validation:

focused ActorSnapshot/tag, Atespace deletion, external-volume clone, and CLI tests
go test ./...
make verify
hack/run-e2e-kind.sh ./internal/e2e/suites/demo -v -args --no-color

Dmitry Berkovich (dberkov) · 2026-07-28T21:49:12Z

  string worker_pool_name = 12;

+  // The latest durable snapshot created for this Actor.
+  ObjectRef latest_snapshot = 13;


An additional hop to persistence will be required for every resume operation. It introduces latency on the hot path. For 100M+ agents it will increase memory for redis or memory for indexing data in postgress.

Will it be simpler to save ActorSnapshotScope for tagged versions only? It will be cheaper solution and will not be used very often.

Good point. We can add more data to the Actor so we don’t need to query on the hot path. However this feels like a hack around redis issues vs a real requirement, this could be a single query in Postgres

Dmitry Berkovich (dberkov) · 2026-07-29T00:30:13Z

 		}
 		return nil, fmt.Errorf("while getting ActorTemplate: %w", err)
 	}
+	if sourceSnapshot.GetContentScope() == ateapipb.SnapshotContentScope_SNAPSHOT_CONTENT_SCOPE_FULL && sourceSnapshot.GetActorTemplateUid() != string(template.GetUID()) {


Template ID for data supposed to match too.

I still dont know how exactly Zoe Zhao (@zoez7) will implement the template snapshoting but most likely what needs to be validated is a common layer of snapshot. We need to use data part of the snapshot, If template is full and template version does not match.

For gVisor it still not possible to extract durDir from the snapshot, however it is possible for microVM.

I would validate right now just template ID and create a TODO to fix it, once Zoe added support for multi versioning and Fabricio Voznika (@fvoznika) will add support for gVisor to extract data part from the memory.

Dmitry Berkovich (dberkov) · 2026-07-29T00:40:43Z


 	setSpanActorAttributes(ctx, actor)
-	return &ateapipb.SuspendActorResponse{Actor: actor}, nil
+	return &ateapipb.SuspendActorResponse{Actor: actor, Snapshot: snapshot}, nil


The actor already has field "latest_snapshot", what is a reason to return full snapshot as a separate field?

Dmitry Berkovich (dberkov) · 2026-07-29T01:18:56Z

+			location := latestActor.InProgressSnapshot
+			prefix := strings.TrimSuffix(state.ActorTemplate.Spec.SnapshotsConfig.Location, "/") + "/snapshots/"
+			snapshotID := strings.ToLower(strings.NewReplacer(":", "-", "+", "-").Replace(strings.TrimPrefix(location, prefix)))
+			lock, err := s.store.AcquireLock(ctx, "lock:actor-snapshot:"+input.Atespace+":"+snapshotID)


out of curiosity, why lock is required? The code supposed to add a new instance of actorSnapshot ?

Dmitry Berkovich (dberkov)

I have completed an initial review pass and wanted to share high-level feedback before doing a deep dive into the PR.

My primary concern is generalizing "tagged" snapshots alongside golden and untagged ones. The current design introduces an extra persistence read on the hot path, which offers no advantage in 99% of cases.

Typically, an actor's lifecycle follows a simple sequence: run -> suspend -> run -> suspend. Since tagging is rarely needed, snapshot details can remain internal to the actor rather than being exposed to an external entity.

There are two distinct types of tagging:
Per-actor: Used to roll back to a well-defined state. These snapshots should not be garbage-collected by the retention policy, but they must be deleted when the actor itself is deleted.
Per-atespace: The snapshot lifecycle is tied directly to the atespace and cleaned up when the atespace is deleted. This type is only used during actor creation, and I am not aware of any use case for reverting an actor to this kind of snapshot.

Regarding golden snapshots: the current layer requires a full redesign to support management by CPU type. While the new ActorSnapshot entry could potentially facilitate this, I suggest excluding it for now. We can extend the ActorSnapshot layer to handle golden snapshots once we initiate that broader redesign.

By scoping down the tagging use cases, the entire PR can be significantly simplified.

What do you think?

Eitan Yarmush (EItanya) · 2026-07-29T13:19:45Z

My primary concern is generalizing "tagged" snapshots alongside golden and untagged ones. The current design introduces an extra persistence read on the hot path, which offers no advantage in 99% of cases.

If the main design is the extra read then I can solve that as per my comment here: #570 (comment)

Typically, an actor's lifecycle follows a simple sequence: run -> suspend -> run -> suspend. Since tagging is rarely needed, snapshot details can remain internal to the actor rather than being exposed to an external entity.

I'm not sure we know that tagging is rarely needed, we're making that assumption

There are two distinct types of tagging:
Per-actor: Used to roll back to a well-defined state. These snapshots should not be garbage-collected by the retention policy, but they must be deleted when the actor itself is deleted.
Per-atespace: The snapshot lifecycle is tied directly to the atespace and cleaned up when the atespace is deleted. This type is only used during actor creation, and I am not aware of any use case for reverting an actor to this kind of snapshot.

It's not clear to me that these are the only ones. We specifically discussed having globally scoped snapshots which can be shared outside of the atespace.

By scoping down the tagging use cases, the entire PR can be significantly simplified.

I think there are tradeoffs to both approaches. The ways that I can imagine scoping this down would also build us into corners. I think the other viable option would be tracking n-1 snapshots directly on the actor, and then the tagging operation would actually create the ActorSnapshot object. What do you think?

Haven Xia (HavenXia) · 2026-07-29T18:55:14Z

+message ActorSnapshotRef {
+  oneof reference {
+    ObjectRef snapshot = 1;
+    string tag = 2;


Should we make snapshot tag also atespaced? Otherwise we have no limitation to restore a snapshot that doesn't belong to me.

Hmm the CreateActor has a scope chheck to prevent that.

But UpdateActor doesn't have, so bascially I can do

guess a tag

UpdateActorSnapshot(snapshot: {tag}, scope: GLOBAL)

CreateActor(source_snapshot: {tag})

to create an actor from others' snapshot

I'm redoing this system as we speak, so will push soon and then we can chat about it

Haven Xia (HavenXia) · 2026-07-29T18:58:08Z

 		return err
 	}
 	state.Actor = actor
+	if actor.GetStatus() == ateapipb.Actor_STATUS_SUSPENDED && actor.GetLatestSnapshot() == nil {


When an actor get just created, it will satisfy

actor.GetStatus() == ateapipb.Actor_STATUS_SUSPENDED && actor.GetLatestSnapshot() == nil

So it will error out if suspending it again - I think this makes this step not idempotent anymore.

Haven Xia (HavenXia) · 2026-07-29T19:11:34Z

We don't have snapshot retention for now - But this reminds me that for future cascade deletion of atespace(#396), , but we should let it

Delete all ACTOR_SNAPSHOT_SCOPE_ACTOR /ACTOR_SNAPSHOT_SCOPE_ATESPACE snapshots belongs to actors in that atespace.
Keep GLOBAL scope snapshots belongs to actors in that atespace.

Julian Gutierrez Oschmann (juli4n) · 2026-07-29T21:10:39Z

  string in_progress_snapshot = 8;
  string ateom_pod_uid = 9;
-  SnapshotInfo latest_snapshot_info = 10;
+  reserved 10;


No need to worry reserving ids as the change is breaking anyways.

Eitan Yarmush (EItanya) · 2026-07-30T13:45:38Z

Updated this PR to reflect the design we converged on:

Tags are now Atespace-owned resources addressed as atespace/name; the same name can exist in multiple Atespaces.
Publishing changes a tag's reuse policy, not its identity. Published tags remain owned by and addressed through their original Atespace, so there is no separate global namespace.
Deleting an Atespace deletes all of its tags, including published ones. Snapshot metadata remains for future GC/retention work.
Tags are stored separately from snapshots with O(1) lookup. Snapshot-level scope, embedded tags, and explicit snapshot deletion were removed.
Explicit Actor creation from a snapshot accepts only tag references and enforces ATESPACE/PUBLISHED reuse policy.
Cloning templates with external volumes is rejected until CSI/provider snapshot support is designed.
Suspension now records the source Actor version when checkpointing begins, so snapshot metadata identifies the version actually captured.

This deliberately defers tiered Atespaces/higher-level grouping and snapshot GC until concrete requirements justify those concepts.

Dmitry Berkovich (dberkov) · 2026-07-30T14:15:22Z

+  string actor_template_name = 6;
+  string actor_template_uid = 7;
+  SnapshotContentScope content_scope = 8;
+  reserved 9, 10;


out of curiosity what it is a reason to have reserved fields for a completely new message

Removed the reserved numbers and names. This is a new message, so there is no compatibility history to protect.

Dmitry Berkovich (dberkov) · 2026-07-30T14:21:58Z

 		return nil, store.ErrFailedPrecondition
 	}
-
+	if err := s.deleteMatching(ctx, actorSnapshotTagScanPattern(name)); err != nil {


Currently, deletion fails if at least one actor exists. Should this prevent us from following the same pattern for releasing tags?

Updated to follow the existing non-empty Atespace behavior: deletion now returns FailedPrecondition while any ActorSnapshotTag remains instead of deleting tags.

Dmitry Berkovich (dberkov) · 2026-07-30T15:24:00Z

+	if err := validateActorSnapshotRef(req.GetSnapshot(), "snapshot"); err != nil {
+		return nil, err
+	}
+	if err := validateActorSnapshotTag(req.GetTag(), "tag"); err != nil {


There is no cross-namespace validation. Do we allow tag snapshots for cross atespace?

Cross-Atespace tag creation is now rejected with FailedPrecondition. A tag must be owned by the source snapshot's Atespace; PUBLISHED controls cross-Atespace reuse.

Dmitry Berkovich (dberkov) · 2026-07-30T15:27:19Z

-    LocalSnapshotInfo local = 3;
-  }
+enum SnapshotContentScope {
+  // Defaults to FULL for compatibility with existing snapshot configuration.


nit:
The proto comment on ACTOR_SNAPSHOT_TAG_SCOPE_UNSPECIFIED says "Defaults to ATESPACE", but validateActorSnapshotTagScope (actor_snapshot.go:231) rejects it with InvalidArgument, and the
CreateActor scope switch returns FailedPrecondition for it. Either default needs to be applied or comment needs to be fixed.

Made ATESPACE the zero/default enum value and removed UNSPECIFIED. Omission now intentionally means Atespace-local.

Dmitry Berkovich (dberkov) · 2026-07-30T15:28:36Z

+	return result, nextToken, nil
+}
+
+func (s *Persistence) TagActorSnapshot(ctx context.Context, atespace, name string, tag *ateapipb.ActorSnapshotTag) (*ateapipb.ActorSnapshotTag, error) {


Idempotent re-tag ignores scope — the method returns the existing tag when it points at the same snapshot, even if the requested scope differs. A caller asking for PUBLISHED can silently get back an
ATESPACE tag. Consider treating a scope mismatch as AlreadyExists.

Fixed. Idempotent re-tagging now requires both the same snapshot and the same scope; a scope mismatch returns AlreadyExists and callers use UpdateActorSnapshotTag to change it.

Dmitry Berkovich (dberkov) · 2026-07-30T15:31:39Z

+		return nil, err
+	}
+	ref := &ateapipb.ActorSnapshotRef{Reference: &ateapipb.ActorSnapshotRef_Tag{Tag: req.GetTag()}}
+	lock, _, _, _, err := s.lockActorSnapshot(ctx, ref)


nit: Unmapped ErrNotFound — after lockActorSnapshot, a concurrent DeleteAtespace (which only needs the atespace lock, acquired later by the handler) can delete the tag (please take a look on comment in the delete, it might be worth preventing the atespece deletion of tags assigned to it) ; the subsequent store call's ErrNotFound is wrapped
as a generic error (gRPC Unknown) instead of NotFound

Fixed the error mapping: ErrNotFound from the update is now returned as gRPC NotFound. Atespace deletion also blocks while tags remain, reducing this race while preserving the correct status.

Eitan Yarmush (EItanya) · 2026-07-30T20:39:08Z

Updated this PR after #568 merged:

rebased and squashed the branch to one commit on top of ateapi: store snapshots under stable UUID paths #568's merged commit
preserved ateapi: store snapshots under stable UUID paths #568's actor_volumes = 13 wire field and moved the new Actor snapshot fields to 14–16
preserved the asynchronous external-volume creation flow while combining it with snapshot-based Actor creation
added kubectl-ate commands to list/get snapshots, create/update/delete snapshot tags, and create an Actor with --snapshot-tag <atespace>/<name>
documented the new commands
added a demo E2E that suspends a real Actor, gets/lists and tags its snapshot, publishes the tag, creates/restores a clone, verifies restored state, and deletes the tag
fixed ListActorSnapshots for clustered Valkey by replacing cross-slot MGET with pipelined per-key GETs; the live E2E caught this

Validation passes: go test ./..., make verify, and hack/run-e2e-kind.sh ./internal/e2e/suites/demo -v -args --no-color.

Eitan Yarmush (EItanya) requested review from Haven Xia (HavenXia), Dmitry Berkovich (dberkov) and Julian Gutierrez Oschmann (juli4n) and removed request for Dmitry Berkovich (dberkov) and Julian Gutierrez Oschmann (juli4n) July 28, 2026 18:49

Eitan Yarmush (EItanya) force-pushed the issue-529-actor-snapshots branch from a97f6dd to 78cd8e3 Compare July 28, 2026 19:27

Eitan Yarmush (EItanya) mentioned this pull request Jul 28, 2026

Add first-class ActorSnapshot lifecycle APIs #529

Closed

10 tasks

Dmitry Berkovich (dberkov) reviewed Jul 28, 2026

View reviewed changes

Julian Gutierrez Oschmann (juli4n) reviewed Jul 28, 2026

View reviewed changes

Comment thread pkg/proto/ateapipb/ateapi.proto Outdated

Dmitry Berkovich (dberkov) reviewed Jul 29, 2026

View reviewed changes

Eitan Yarmush (EItanya) force-pushed the issue-529-actor-snapshots branch 2 times, most recently from d5cb8b1 to 81c3d37 Compare July 29, 2026 16:14

Haven Xia (HavenXia) reviewed Jul 29, 2026

View reviewed changes

Julian Gutierrez Oschmann (juli4n) reviewed Jul 29, 2026

View reviewed changes

Eitan Yarmush (EItanya) force-pushed the issue-529-actor-snapshots branch from 81c3d37 to 6bc5c1a Compare July 30, 2026 13:43

Eitan Yarmush (EItanya) force-pushed the issue-529-actor-snapshots branch from 6bc5c1a to 317c71d Compare July 30, 2026 13:49

Dmitry Berkovich (dberkov) reviewed Jul 30, 2026

View reviewed changes

Eitan Yarmush (EItanya) force-pushed the issue-529-actor-snapshots branch 4 times, most recently from a892685 to 067e731 Compare July 30, 2026 20:38

ateapi: add ActorSnapshot lifecycle APIs

0a1bf9e

Eitan Yarmush (EItanya) force-pushed the issue-529-actor-snapshots branch from 067e731 to 0a1bf9e Compare July 30, 2026 20:55

Dmitry Berkovich (dberkov) approved these changes Jul 31, 2026

View reviewed changes

Dmitry Berkovich (dberkov) merged commit ef7b29d into agent-substrate:main Jul 31, 2026
11 checks passed

Eitan Yarmush (EItanya) mentioned this pull request Jul 31, 2026

Add retention policy and garbage collection for ActorSnapshots #664

Closed

8 tasks

Navigation Menu

ateapi: add ActorSnapshot lifecycle APIs - #570

ateapi: add ActorSnapshot lifecycle APIs#570
Dmitry Berkovich (dberkov) merged 1 commit into
agent-substrate:mainfrom
kagent-dev:issue-529-actor-snapshots

Eitan Yarmush (EItanya) commented Jul 28, 2026 •

edited

Loading

Dmitry Berkovich (dberkov) Jul 28, 2026

Eitan Yarmush (EItanya) Jul 29, 2026

Uh oh!

Dmitry Berkovich (dberkov) Jul 29, 2026

Dmitry Berkovich (dberkov) Jul 29, 2026

Dmitry Berkovich (dberkov) Jul 29, 2026 •

edited

Loading

Dmitry Berkovich (dberkov) left a comment

Eitan Yarmush (EItanya) commented Jul 29, 2026

Haven Xia (HavenXia) Jul 29, 2026

Haven Xia (HavenXia) Jul 29, 2026 •

edited

Loading

Eitan Yarmush (EItanya) Jul 29, 2026

Haven Xia (HavenXia) Jul 29, 2026

Haven Xia (HavenXia) commented Jul 29, 2026 •

edited

Loading

Julian Gutierrez Oschmann (juli4n) Jul 29, 2026

Eitan Yarmush (EItanya) commented Jul 30, 2026

Dmitry Berkovich (dberkov) Jul 30, 2026

Eitan Yarmush (EItanya) Jul 30, 2026

Dmitry Berkovich (dberkov) Jul 30, 2026

Eitan Yarmush (EItanya) Jul 30, 2026

Dmitry Berkovich (dberkov) Jul 30, 2026

Eitan Yarmush (EItanya) Jul 30, 2026

Dmitry Berkovich (dberkov) Jul 30, 2026

Eitan Yarmush (EItanya) Jul 30, 2026

Dmitry Berkovich (dberkov) Jul 30, 2026

Eitan Yarmush (EItanya) Jul 30, 2026

Dmitry Berkovich (dberkov) Jul 30, 2026

Eitan Yarmush (EItanya) Jul 30, 2026

Eitan Yarmush (EItanya) commented Jul 30, 2026 •

edited

Loading

Uh oh!

Labels

4 participants

Conversation

Eitan Yarmush (EItanya) commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Dmitry Berkovich (dberkov) Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Dmitry Berkovich (dberkov) left a comment

Choose a reason for hiding this comment

Eitan Yarmush (EItanya) commented Jul 29, 2026

Choose a reason for hiding this comment

Haven Xia (HavenXia) Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Haven Xia (HavenXia) commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Eitan Yarmush (EItanya) commented Jul 30, 2026

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Eitan Yarmush (EItanya) commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Labels

4 participants

Eitan Yarmush (EItanya) commented Jul 28, 2026 •

edited

Loading

Dmitry Berkovich (dberkov) Jul 29, 2026 •

edited

Loading

Haven Xia (HavenXia) Jul 29, 2026 •

edited

Loading

Haven Xia (HavenXia) commented Jul 29, 2026 •

edited

Loading

Eitan Yarmush (EItanya) commented Jul 30, 2026 •

edited

Loading