Skip to content

feat(git-sync): the container pulls origin on its own (trinity-enterprise#703) - #3021

Open
dolho wants to merge 15 commits into
devfrom
feature/ent703-pull-heartbeat
Open

dolho wants to merge 15 commits into
devfrom
feature/ent703-pull-heartbeat

Conversation

@dolho

@dolho dolho commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

⚠️ Merge order: after #3020, which itself follows #3015 → #3016 / #3017 / #3018. This branch is stacked on #3020, so until those merge the diff against dev includes their commits. This PR's own changes run from ee892984 to the latest fix commit; the merge commits bring in #3020 and dev.

Summary

Invariant G3: human and fleet work reaches the agent within a bound. Nothing inside the container ever pulled, and the 2026-09-24 fleet audit found agents up to 31 commits behind.

  • Pull loop in the agent server beside the push loop, gated per cycle on the owner's pull_sync_enabled. The flag is read live from GET .../git/pull-sync with the agent's own key, with GIT_SYNC_PULL as the fallback (the bug(git-sync): the auto-sync toggle is not authoritative — OFF never takes effect (the baked env is OR'd forward at recreate), ON waits for the next recreate #3010 pattern). Interval GIT_SYNC_PULL_INTERVAL_SECONDS, defaulting to the push interval (900 s).
  • Safe by construction (_run_pull_once, under _REPO_LOCK):
    • Never starts while an execution runs or is queued (list_running() + list_pending_ids()). Checked from the process registry, and again right before the tree is touched; an unreadable registry counts as busy. This is check-then-act: turn admission does not wait on a pull, so a turn admitted during the integrate window can see HEAD move (documented; holding admission is a follow-up).
    • Nothing committed locally → fast-forward. A local commit → rebase (--rebase-merges), aborted on conflict.
    • A trinity/* working branch also merges origin/main in (_integrate_source), so human pushes to main arrive. A conflict is aborted and recorded as diverged: merge conflict with main (<files>).
    • Refuses to run over unmerged paths; the push cycle refuses to commit them. Stale lock litter is reaped first. Every exit, a timed-out git child included, puts local edits back or names the stash.
    • Uncommitted edits are stashed explicitly, not with --autostash. When incoming commits touch the same files, autostash's re-apply conflicts, leaves the edits only in the stash, and still reports success. Here a colliding pull is undone (back to the pre-pull HEAD, where the stash applies cleanly) and recorded.
    • Local work is never discarded.
  • Flag (operator ruling 2026-09-25, recorded on ent#703):
    • new column pull_sync_enabled on both migration tracks
    • backfill on only where auto_sync_enabled is already on
    • new github: agents get it at creation, source mode included (pull-only agents are the ones that most need it); ghosts excluded
    • GIT_SYNC_PULL derived from the DB flag alone at recreate
  • Observability: last_pull_at / last_pull_status / last_pull_error / behind_after_pull / last_successful_pull_at / consecutive_pull_failures / consecutive_pull_skips in sync-state.json, persisted on agent_sync_state (status bounded to its vocabulary, counter coerced). A failed pull never touches the push's consecutive_failures.
  • UI: Settings → Git sync gains "Pull changes from GitHub on every sync cycle".
  • Open-core (operator ruling).
  • Now in scope (PR review ruling): merging main into a working branch, every cycle.
  • Out of scope:

Changes

  • Agent server:
    • auto_sync.py: run_pull_loop / run_one_pull_cycle / resolve_pull_sync_enabled; the live flag read is generalised (_resolve_flag) and shared with the push loop
    • routers/git.py: _run_pull_once, _with_stash, _integrate_remote, _integrate_source, _unmerged_paths, _executions_in_flight, _record_pull; _rebase_onto_remote uses --rebase-merges; /api/git/status reports pull_sync_enabled
  • Backend:
    • schema/tables/migrations: SQLite pull_sync + Alembic 0081_pull_sync ← 0080_agent_skill_sets
    • db/schedules/git_config.py, db/sync_state.py, db_models.py, database.py
    • routers/git.py + models.PullSyncToggle: GET/PUT .../git/pull-sync
    • crud.py: on at creation + env
    • lifecycle.py: env from the DB flag
    • sync_health_service.py: persists the pull outcome
  • Frontend: GitSyncSettingsPanel.vue, stores/agents.js
  • Tests:
    • test_ent703_pull_cycle.py (33, real repos)
    • test_1484 (+2: creation turns pull on for agents and deployments)
    • test_ent109_git_env_seam (+2: env follows the DB flag alone; the owned-key guard fixture now writes every owned var)
    • test_sync_health_service (+2: persisted, and agent-written garbage dropped)
    • gitSyncSettingsPanel.spec.js (+2, mounted)
  • Docs:
    • requirements/github.md §11.18
    • feature-flows/git-sync-health.md §1d
    • architecture/agent-lifecycle.md
    • architecture/api-endpoints.md

Test Plan

  • New and touched suites green in random order:
    • pull cycle 33
    • 1484 62, ent109 24, sync_health_service 49
    • route/dependency pairing guard, schema parity, migrations, models-centralized
    • auto-sync / 1595 / 2742 / 3010 / 3011
    • sync-state / 1595-signals / 73-bulk / fleet audit / 1596
    • alembic heads + ids, fork-to-own, ent123
  • Frontend vitest run 3632/3632 (ratchets included); check:tokens OK
  • Mutations:
    • replacing _integrate_remote with a naive rebase --autostash fails exactly the colliding-edits test
    • reverting the agent side breaks all 16 cycle tests
  • lint_sys_modules, root placement, enterprise-docs guard clean; single Alembic head (0081_pull_sync)
  • Live on local dev (not run yet)

Related to abilityai/trinity-enterprise#703 (cross-repo; close at release)

🤖 Generated with Claude Code

…rise#703)

Invariant G3: human and fleet work reaches the agent within a bound.
Nothing inside the container ever pulled; the 2026-09-24 audit found
agents up to 31 commits behind.

- agent server: a pull loop beside the push loop, gated per cycle on
  the owner's pull_sync_enabled (read live, GIT_SYNC_PULL fallback),
  interval GIT_SYNC_PULL_INTERVAL_SECONDS (defaults to the push one).
  Under _REPO_LOCK, never while an execution runs (checked again right
  before the tree is touched). Fast-forward, or rebase aborted on
  conflict. Uncommitted edits are stashed explicitly and a pull that
  collides with them is undone - not --autostash, which strands them
  in the stash while reporting success.
- pull_sync_enabled + last_pull_at/_status/behind_after_pull on both
  migration tracks; backfill on only where auto-sync is on; new github
  agents get it at creation (source mode included), GIT_SYNC_PULL env
  derived from the DB flag alone at recreate.
- GET/PUT /api/agents/{name}/git/pull-sync; sync-health persists the
  pull outcome (bounded); Settings -> Git sync gains the toggle.

Stacked on #3020 (+#3015-#3018): merges after it.

Related to Abilityai/trinity-enterprise#703

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@dolho dolho added the ui PR touches the frontend UI — triggers Playwright e2e tests label Sep 25, 2026
…eSQL (trinity-enterprise#703)

The inline comment on `pull_sync_enabled` sat between the comma and the
FOREIGN KEY clause. `_PG_TABLE_SUBS` strips that clause only when it directly
follows the comma, so the clause survived without its REFERENCES and
PostgreSQL rejected the table ("syntax error at or near ')'"), failing 18
requires_postgres tests in schema-parity. The comment moves above the column.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@vybe

vybe commented Sep 25, 2026

Copy link
Copy Markdown
Contributor

merge-train: ejected from this train because it is stacked on #3020, which is ejected pending a ruling (see #3020). On its own delta this is READY: single Alembic head (0076_pull_sync), auth matches auto-sync, and tests pass (154 own, 2598 related, 29 vitest incl. ratchets).

One intent question to settle while it waits: _run_pull_once pulls whichever branch is checked out. For working-branch agents (the #3020 default and every agent #3017's backfill enables), that's the agent's own trinity/<agent>/<id> branch, so human work pushed to main never arrives, and goal G3 isn't met for that population.

@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown

⚠️ Nightly unit-suite check skipped — merge conflict against dev.

Resolve by running git merge dev locally and pushing the result. The next nightly run will re-test once the conflict is gone.

@github-actions

Copy link
Copy Markdown

⚠️ Live-instance suite skipped — merge conflict against dev.

Resolve by merging dev locally and pushing the result; the next nightly re-tests.

@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

⚠️ Alembic head fork if this PR is merged into dev.

dev has advanced since this PR's checks last ran. GitHub recomputes the merge ref when the base moves but does not re-trigger workflows, so a green schema-parity here can describe a base that no longer exists (#2533).

scripts/ci/check_alembic_heads.py against dev + this PR, merged in memory
alembic-heads: FAIL — src/backend/migrations/versions resolves to 2 heads across 87 revision(s); exactly 1 is required.
  • 0083_pull_sync  (0083_pull_sync.py)
  • 0085_ent720_email_identity  (0085_ent720_email_identity.py)

They fork at: 0082_agent_sync_state_divergence  (0082_agent_sync_state_divergence.py)

`alembic upgrade head` is singular and resolves its target BEFORE applying anything,
so this graph applies ZERO revisions — every revision since the fork stops arriving,
not only the one that forked. Fix by chaining the newer revision off the real head,
or — if the forked revision may already be applied somewhere — by adding a merge
revision (`alembic merge -m "…" <head-a> <head-b>`), whose tuple `down_revision`
converges the line from any starting state. See Architectural Invariant #3.
alembic-heads: src/backend/enterprise/backend/migrations/versions — version directory absent (submodule not initialised) — skipped.

Evaluated on the version line only — this PR also conflicts with dev in 1 unrelated file(s), which do not change this verdict but must be resolved before merge:

src/backend/db/migrations.py

Fix: rechain this PR's revision off 0085_ent720_email_identity (the current dev head), or — if the forked revision may already be applied somewhere — add a merge revision (alembic merge -m "…" 0083_pull_sync 0085_ent720_email_identity), whose tuple down_revision converges the line from any starting state. See Architectural Invariant #3.

Push the fix and schema-parity re-checks it against the merge ref immediately; this comment clears on the next push to dev touching src/backend/migrations/versions/**.

Advisory — this check does not block merge. · head_sha: f1bee1340eb2c9206017385b8504a0dda064ed39 · run

@AndriiPasternak31 AndriiPasternak31 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes. The design is right: a separate loop, the live flag, an explicit stash instead of --autostash, and push fields kept apart. The real-repo tests are good. I ran them (19/19) plus a few probes against _run_pull_once. Blocking items:

  1. alembic-head-watch is red because of this PR. dev now has 0076_operator_queue_ask_object through 0079_telegram_group_context off 0075, and 0076_pull_sync still revises 0075, which gives two heads. Rebase, renumber to 0080_pull_sync with down_revision = "0079_telegram_group_context", and re-resolve db/migrations.py. The PR is also CONFLICTING in 12 files.
  2. _integrate_remote can strand the stash, which is the case the docstring exists to prevent (routers/git.py:876-903). run_registered raises TimeoutExpired, so a timed-out merge --ff-only, stash pop or reset --hard skips every restore. Repro: fast-forward times out → failed, the edit is gone from the tree and sits in stash@{0}, and the error doesn't mention the stash. Please put a try/finally around the post-stash section, and make the error say when a stash entry is left.
  3. The return code of reset --hard is ignored at :898. If it fails (e.g. index.lock held by the backend's docker-exec gitignore sweep, which runs outside _REPO_LOCK), the tree is left UU with conflict markers. The next auto-sync cycle's git add -A commits them and pushes them to origin with status success. I reproduced this end to end. Please check the return code and stop loudly. I'd also make the push cycle refuse to commit while diff --diff-filter=U is non-empty.
  4. The execution gate misses queued turns (:830). list_running() excludes register_pending entries (#2433), so a turn waiting on the chat lock or the headless pool counts as idle. Please count list_pending_ids() as busy too.

Non-blocking, but worth resolving before merge:

  • Where the gate actually protects anyone. For auto-sync agents, the #3011 push cycle already fetches and rebases every cycle with no execution gate. So for the backfilled population the pull is mostly redundant, and the UI line "Never runs while the agent is working" isn't true. Existing source-mode agents, the ones that need the pull, stay off until toggled. Worth saying so in the PR, and probably gating #3011's rebase the same way.
  • Pull failures are write-only. last_pull_error isn't persisted, there is no last_successful_pull_at, nothing feeds sync_failing or /sync-health, and repo-busy skips aren't recorded. ent#706/#707 will need last_successful_pull_at, a persisted last_pull_error, and consecutive pull failure/skip counts. Cheaper to add them to this migration pair than to open a second one on the same table.
  • behind_after_pull is own-branch behind. On a trinity/* agent it reads 0 while behind_main is 1. Please document it.
  • Docs. architecture/agent-runtime.md:27 still says "two loops". The UI label hard-codes 15 minutes, but GIT_SYNC_PULL_INTERVAL_SECONDS makes it configurable. The pull cycle also never reaps a stale index.lock, which matters for pull-only agents.
@AndriiPasternak31

Copy link
Copy Markdown
Contributor

Small correction to item 1 of my review: don't hard-code 0080. #2984 carries 0080_seat_ask_class_state and is mergeable, so 0080_pull_sync on 0079 would fork with it. Take the number and parent from the live head at rebase time, and run scripts/ci/check_alembic_heads.py just before merge.

dolho and others added 3 commits September 28, 2026 10:56
…nto feature/ent703-pull-heartbeat

# Conflicts:
#	src/backend/db/migrations.py
#	tests/registry.json
…ns, brings main to working branches (trinity-enterprise#703)

PR #3021 review:
- _with_stash wraps every post-stash step: a git child that times out
  (run_registered raises TimeoutExpired) goes back to the pre-pull HEAD
  and re-applies the stash; when that is impossible the error says the
  edits are kept in `git stash`.
- reset --hard's return code is checked. A failed undo stops with an
  error naming it instead of leaving UU files behind silently.
- Both cycles refuse to run over unmerged paths. The push cycle checks
  before `git add -A`, which would stage conflict markers and push them
  to origin as a successful sync.
- The execution gate counts register_pending entries (#2433), so a turn
  queued on the chat lock or the headless pool is busy, not idle.
- The pull reaps stale lock litter first, as the push cycle does; a
  pull-only agent has no push cycle to do it.
- sync-state gains last_pull_error streak fields: consecutive pull
  failures / skips and last_successful_pull_at.

Ruling on the intent question: a trinity/* working branch only ever
pulled itself, so human pushes to main never arrived (G3). The cycle
now also merges origin/main (via _get_pull_branch) into the working
branch — a merge, not a rebase, since the branch is already pushed; a
conflict is aborted and recorded.

Nine real-repo tests, all red before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… + UI copy (trinity-enterprise#703)

- Alembic 0076_pull_sync -> 0080_pull_sync <- 0079_telegram_group_context
  (one head, 81 revisions); SQLite entry re-appended after dev's.
- The same migration pair adds agent_sync_state.last_pull_error,
  last_successful_pull_at, consecutive_pull_failures and
  consecutive_pull_skips, persisted by the sync-health poller (bounded,
  coerced; a success clears the error). ent#706/#707 read these, so they
  land here rather than as a second migration on the same table.
- agent-runtime.md says three loops; the flow and requirement describe the
  main merge, the queued-turn gate, the unmerged-path refusal, what
  behind_after_pull measures, and that auto-sync agents already rebase in
  the push cycle.
- Settings copy no longer hard-codes 15 minutes
  (GIT_SYNC_PULL_INTERVAL_SECONDS) and no longer claims the push cycle
  never touches the tree while the agent works.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dolho

dolho commented Sep 28, 2026

Copy link
Copy Markdown
Contributor Author

All four blocking items, the intent question and most of the non-blocking ones are in. The branch re-merges #3020, which now carries dev.

Blocking (Andrii)

  1. Head fork: 8c1282456 renumbers to 0080_pull_sync ← 0079_telegram_group_context, the live head at push time, per the correction. The SQLite entry is re-appended after dev's, and the test's pinned pair and the PR body are updated. check_alembic_heads.py: 81 revisions, 1 head.
  2. Stranded stash: edf8edd22 moves every post-stash step into _with_stash.
    • A git child that times out (TimeoutExpired) goes back to the pre-pull HEAD and re-applies the stash.
    • Where that isn't possible, the error ends with "local edits are kept in git stash".
  3. reset --hard return code: it is now checked. A failed undo stops with "…the pull could not be undone (…); local edits are kept in git stash".
    • Both cycles now refuse to run over unmerged paths (git diff --diff-filter=U). The push cycle checks before git add -A, so conflict markers can't be staged and pushed as a success.
    • Your end-to-end repro is a test: the reset fails, the tree is left UU, and the next push cycle returns refused: unmerged paths (notes.md) with origin/main unchanged.
  4. Queued turns: _executions_in_flight now counts list_running() + list_pending_ids(), so a queued turn is busy.

Intent (vybe): a trinity/* branch only ever pulled itself, so pushes to main never arrived and G3 wasn't met for that population. The cycle now also merges origin/main into the working branch (_integrate_source, source branch from _get_pull_branch). It merges rather than rebases because the branch is already pushed, and rewriting it would make the next push cycle rebase it back. A conflict is aborted and recorded as merging main: ….

Non-blocking

  • Pull failures persisted: last_pull_error, last_successful_pull_at, consecutive_pull_failures and consecutive_pull_skips are added to this same migration pair and persisted by the poller: bounded, coerced, and a success clears the error.
  • Stale index.lock: the pull cycle now reaps stale lock litter first, the same way the push cycle does.
  • behind_after_pull: documented as the branch's own lag. behind_main is the lag behind main.
  • Docs: agent-runtime.md now says three loops. The Settings copy no longer hard-codes 15 minutes and no longer claims the push cycle stays out of the tree while the agent works. With auto-sync on, the push cycle rebases first, and the flow doc now says so.
  • Not done: gating bug(git-sync): the auto-sync heartbeat pushes without fetch or rebase — on a shared branch it fails non-fast-forward forever after the first foreign push #3011's push-cycle rebase on executions. That changes the push cycle's own semantics (skip vs. fail vs. non-fast-forward), so it's better as its own change.

Tests

  • test_ent703_pull_cycle.py: 28 passed. The 9 new real-repo tests all fail against the previous git.py.
  • 949 across every agent-server git/auto-sync test.
  • 2,945 across the git, sync, pull, schema, migration and create suites (seed 12345).
  • Full vitest passes. CI green.
@dolho

dolho commented Sep 28, 2026

Copy link
Copy Markdown
Contributor Author

Ownership moves to @AndriiPasternak31 (ent#703 handover, per the Mon–Wed plan). The branch is ready for your re-review; I won't push to it further unless you ask.

@AndriiPasternak31 AndriiPasternak31 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve. All four blocking items from my 09-27 review are fixed, each with a real-repo test, and the "bring main into working branches" follow-up is implemented. I found nothing new that blocks. There are three should-fix items below: two are small and cheap to land before merge, the third is at least a wording fix. I reviewed only this PR's own commits (ee8929840..8c1282456, diffed against #3020's head 8b20c1714). #3020 is reviewed separately, and this PR merges after it.

Previous blocking items

  • ✅ 1. Alembic head fork. 0080_pull_sync.py now revises 0079_telegram_group_context, and the SQLite pull_sync entry comes after dev's. check_alembic_heads.py reports 81 revisions and 1 head (0080_pull_sync). alembic-head-watch is green, and the PR is MERGEABLE. #2984 and #3005 both carry an 0080_* on 0079 and are both mergeable, so whichever of the three lands later has to re-chain. Re-run the head check right before merge.
  • ✅ 2. Stash stranded on timeout. Every step after the stash now runs inside _with_stash (routers/git.py:891-957). On TimeoutExpired it resets to pre_head and pops. If that can't be done, the error ends with "local edits are kept in git stash". Tests: test_a_timed_out_fast_forward_puts_the_edits_back and test_a_stash_left_behind_is_named_in_the_error.
  • ✅ 3. reset --hard return code ignored / markers pushed. The return code is checked now: a failed reset returns "…could not be undone…". Both cycles refuse to run while _unmerged_paths is non-empty, and the push cycle checks before git add -A (:1159). My end-to-end repro is now test_a_failed_undo_stops_loudly_and_the_push_refuses_the_markers: the push returns refused: unmerged paths (notes.md), origin/main is unchanged, and no markers reach it.
  • ✅ 4. Queued turns counted as idle. _executions_in_flight returns list_running() + list_pending_ids() (:830-845). Test: test_a_queued_turn_counts_as_busy.
  • Non-blocking items from last time:
    • Done: pull health persisted (last_pull_error, last_successful_pull_at, pull failure/skip streaks), stale-lock reaping before the pull, behind_after_pull documented, "three loops" in agent-runtime.md, and the UI copy no longer hard-codes 15 minutes.
    • Deferred with a stated reason: execution-gating #3011's push-cycle rebase. That's fine as its own change.
  • ✅ "Pull main into working branches." _integrate_source (:975-994) merges origin/<main> into a trinity/* branch every cycle and aborts on conflict. Tests: test_main_is_merged_into_the_working_branch and test_a_conflicting_main_is_aborted_and_recorded. The PR body still lists this as out of scope, so please update it.

New findings

  1. [should-fix] A conflict merging main is recorded as "Auto-merging <file>". docker/base-image/agent_server/routers/git.py:991

    • Problem: on a conflict, git prints to stdout and leaves stderr empty. _summarize_git_error(merge.stderr or merge.stdout) therefore takes stdout's first line, which is the auto-merge notice, not the conflict.
    • Reproduced on a real repo: the agent commits notes.md on its working branch, a human commits notes.md on main, and _run_pull_once returns {'status': 'failed', 'error': 'merging main: Auto-merging notes.md'}. That exact string is also persisted as last_pull_error.
    • Failure scenario: every 15 minutes the same failure repeats with an error that reads like progress. ent#706/#707 will surface this field directly, and an operator can't tell it is a conflict.
    • Also here: the merge --abort return code is ignored (:990). The unmerged-paths guard does stop the push, but the error doesn't say the abort failed.
    • The current test only asserts startswith("merging main: "), so it doesn't catch this.
    • Fix: mirror _rebase_onto_remote. Read --diff-filter=U (or look for CONFLICT in the output) before aborting, and return diverged: merge conflict with main (notes.md). Check the abort's return code and say so if it failed. Make the test assert the conflict wording.
  2. [should-fix] Any rebase of a working branch silently flattens the merge commit the pull just made. routers/git.py:803 (git rebase --autostash origin/<branch>), reached from _integrate_remote (:970) and the push cycle (:1205)

    • Problem: plain git rebase drops merge commits. It replays main's commits one by one onto the working branch as new SHAs.
    • Reproduced: auto-sync is off, cycle 1 merges main into trinity/a/1 (1 merge commit, not pushed), then a human pushes to origin/trinity/a/1. Cycle 2 returns success, but git log --merges is empty, main's commit reappears as a copy, and origin/main is no longer an ancestor of HEAD.
    • Failure scenario: behind_main keeps reporting the agent as behind even though the content is there. The next cycle merges main again on top of the duplicates, and the history the agent pushes carries copied commits.
    • The PR's own reasoning for merging rather than rebasing ("rewriting it would make the next push cycle rebase it back") is exactly what this undoes.
    • Rare today: it needs an unpushed merge plus someone else pushing to the agent's own branch. It gets likelier once pull stays on while auto-sync is toggled off.
    • Fix: use rebase --rebase-merges in _rebase_onto_remote, or merge instead of rebasing when the branch is trinity/*. Add a test for this sequence.
  3. [should-fix] "Never while a turn runs" is check-then-act. routers/git.py:1072-1082

    • Problem: the gate is read before _integrate_remote / _integrate_source run, but nothing on the admission side waits for it. register_pending in chat.py:53/204, claude_code.py:750 and result_callback.py:418 never looks at _REPO_LOCK or at a pull in progress.
    • Failure scenario: a turn admitted during the integrate window (a fast-forward or merge of up to 60 s, a rebase up to 120 s, plus the stash operations) reads files while HEAD is moving. That is the case the issue's acceptance criterion rules out ("serialize on _REPO_LOCK and the execution lock").
    • The claim appears as an absolute in three places: the Settings copy (GitSyncSettingsPanel.vue:75, "never runs while the agent is working"), requirement §11.18 and agent-lifecycle.md.
    • Fix: hold admission briefly while a pull is integrating (for example, a pull-in-progress event that register_pending callers wait on, with a bound). Or, at minimum, change the copy to "never starts while…" and document the window.
  4. [nit] A failed ahead/behind count reads as "up to date." routers/git.py:1065-1070

    • _compute_ahead_behind returns (0, 0) on any failure, including its 10 s timeout. The pull then records success with behind_after_pull=0 and moves last_successful_pull_at forward.
    • Fix: give the pull cycle a strict variant that fails the pull instead.
  5. [nit] A timed-out step with nothing stashed is left as the kill left it. routers/git.py:946-957

    • The reset back to pre_head only happens when stashed is true.
    • Failure scenario: a killed merge --ff-only or merge on a clean tree can leave a half-updated tree or a MERGE_HEAD. The next push cycle can then commit that partial state as the agent's own work.
    • Fix: reset to pre_head on timeout whether or not anything was stashed. If an index.lock from the killed child blocks the reset, the error should say so.
  6. [nit] The PR description is stale.

    • "Out of scope: merging the source branch into a working branch" is now in scope.
    • The test count still says 19; there are 28 now.
    • "Live on local dev" is still unchecked. The loop's startup wiring (schedule_auto_sync_if_enabled) and the real process-registry import have only run under unit tests, never inside an agent container.

What I checked

  • Read this PR's own diff against #3020's head. That covers the agent server (auto_sync.py, routers/git.py), both migration tracks and the schema, the pull-sync routes, creation/recreate env (crud.py and lifecycle.py), the sync-health poller, the UI panel and the docs.
  • Architecture checks:
    • Creation and recreate don't touch the PAT gate or the push blackhole; the pull only fetches.
    • Errors pass through _summarize_git_error, which redacts URL userinfo.
    • The pull never writes the push's consecutive_failures, so the freeze still keys on push health. The new refused: unmerged paths counts toward the freeze, which is the right direction.
    • The reaper reuses the #1595 1-hour age gate. It never removes a live index.lock.
    • Shutdown cancels both loop tasks.
  • Tests: 16 touched and related unit files (pull cycle, #3010, #3011, auto-sync, #1595, #2742, sync health/state, ent109 env seam, #1484, fork-to-own, pull branch, dual ahead/behind). 429 passed + 1 skipped on seed 12345, 430 passed on seed 99999.
  • Real-repo probes (bare origin, agent clone, human clone) for findings 1 and 2.
  • check_alembic_heads.py: 1 head. gh pr checks 3021: every check passing or skipped, none failing.
#3005 landed Alembic 0080_agent_skill_sets on 0079_telegram_group_context,
the same parent this branch's 0080_pull_sync used, so merging as-is would
leave two heads and `upgrade head` would apply nothing.

Renumber the revision to 0081_pull_sync, chained on 0080_agent_skill_sets
(file, revision, down_revision, docstring, the SQLite mirror note and the
revision test). SQLite MIGRATIONS keeps dev's agent_skill_sets entry first,
pull_sync after it. tests/registry.json keeps both entries. No logic change.
@AndriiPasternak31

Copy link
Copy Markdown
Contributor

Merged dev into this branch to fix the migration fork. #3005 landed 0080_agent_skill_sets on 0079_telegram_group_context, the same parent as this PR's 0080_pull_sync, so merging this would have left two Alembic heads.

  • Renamed 0080_pull_sync → 0081_pull_sync, now chained on 0080_agent_skill_sets. The SQLite pull_sync entry now comes after dev's agent_skill_sets, and tests/registry.json keeps both entries. There's no logic change.
  • check_alembic_heads.py finds 1 head (0081_pull_sync). check_alembic_parity.py origin/dev HEAD passes. The unit files this PR touches, plus the schema/alembic/migration guards and test_ent530_skill_sets.py, pass at 452 passed, 2 skipped on seeds 12345 and 99999.

Merge order: #3035 adds 0081_agent_sync_state_divergence on the same parent (0080_agent_skill_sets). Whichever of #3035 and #3021 merges second will need re-chaining again.

@vybe could you review? I approved this earlier today, but I've pushed to it since, so it needs an approver other than me (SOC 2).

@AndriiPasternak31
AndriiPasternak31 dismissed their stale review September 28, 2026 23:29

Dismissing my own approval: I also pushed commits to this branch (the dev merge fixing the 0080 migration fork), so this approval would cover my own code (SOC 2 separation of duties). Needs an independent review.

dolho and others added 2 commits September 29, 2026 10:11
…nto feature/ent703-pull-heartbeat

# Conflicts:
#	tests/unit/test_1484_create_agent_characterization.py
…p merges (trinity-enterprise#703)

PR #3021 re-review:
- A conflict merging main was recorded as "Auto-merging <file>": git
  prints it on stdout with an empty stderr. Unmerged paths are read
  before the abort; the error is "diverged: merge conflict with main
  (<files>)", and a failed merge --abort is named.
- `git rebase` flattened the merge the pull made into copies of main's
  commits; both cycles now rebase with --rebase-merges.
- A failed strict ahead/behind count fails the pull instead of reading
  as up to date.
- A timed-out step is reset to the pre-pull HEAD even with nothing
  stashed; a reset blocked by the killed child's index.lock says so.
- "Never runs while the agent is working" is check-then-act: the copy
  and docs now say "never starts" and document the admission window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dolho

dolho commented Sep 29, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the 09-28 re-review: 689a97fc0 merges #3020's review fixes, and the fixes themselves are in 8a4346cc9. The six new real-repo tests all failed before the fix.

  1. [should-fix] Conflict recorded as "Auto-merging <file>".
    • _integrate_source now reads --diff-filter=U before the abort (and falls back to CONFLICT in the output), then records diverged: merge conflict with main (notes.md).
    • The merge --abort return code is checked. A failed abort appends ; merge --abort failed (…).
    • Tests: test_a_conflicting_main_is_aborted_and_recorded now asserts the exact wording and the persisted last_pull_error. test_a_failed_merge_abort_is_named_in_the_error is new.
  2. [should-fix] A rebase flattened the pull's merge.
    • _rebase_onto_remote is now git rebase --autostash --rebase-merges, used by both cycles. A linear history rebases the same as before.
    • test_a_rebase_keeps_the_merge_the_pull_made runs your sequence: cycle 1 merges main without pushing, a human pushes to trinity/agent/1, then cycle 2 runs. It checks that the merge survives, origin/main is an ancestor, and main's commit appears once.
  3. [should-fix] Check-then-act. I took the minimum option.
  4. [nit] A failed count read as up to date. The pull cycle now uses the strict _ahead_behind_vs. When it can't count, it records failed: could not count commits on <ref> against origin and last_successful_pull_at doesn't move. Test: test_an_uncountable_branch_fails_the_pull_instead_of_reading_up_to_date.
  5. [nit] A timed-out step with nothing stashed. _with_stash now runs merge --abort and reset --hard <pre_head> on every timeout, whether or not anything was stashed. A reset that can't run appends ; the tree could not be reset to <sha> (… index.lock …). Tests: test_a_timed_out_step_on_a_clean_tree_is_reset and test_a_reset_blocked_by_the_killed_childs_lock_says_so.
  6. [nit] PR body. Updated.
    • "Merging main into a working branch" moves to in-scope.
    • The pull-cycle test count is now 33.
    • The pull fields and the gate wording match the code.
    • "Live on local dev" is still unchecked, because it hasn't been run in an agent container.

Verification

  • Alembic after the merge: check_alembic_heads.py reports 82 revisions and 1 head (0081_pull_sync). check_alembic_parity.py origin/dev HEAD passes. dev has moved since, by refactor(db): typed parameter objects for the four widest writers (#1482) #3042, which has no migration, and the branch still merges cleanly.
  • 37 related unit files (pull cycle, 3010, 3011, auto-sync, 1595, 2742, sync health/state, ent109, 1484, fork-to-own, pull branch, 2107, schema/alembic/migrations): 865 passed and 3 skipped on seeds 12345 and 99999.
  • Every agent-server git/rebase test file: 744 passed.
  • Frontend npm run test:unit: 3702/3702, ratchets included.

🤖 Generated with Claude Code

…nto feature/ent703-pull-heartbeat

# Conflicts:
#	tests/registry.json
@vybe

vybe commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

merge-train: not on train #3070 — the code validated READY (both migration tracks consistent, single Alembic head 0081_pull_sync on dev+this, dismissed-review points all addressed), but this branch is stacked on #3020, which is CONFLICTING with dev and has an open changes-request. Squash-merging this first would land #3020's content unreviewed. Rides the train after #3020.

Non-blocking notes: backend GET/PUT .../git/pull-sync routes have no test; the # mcp: header in routers/git.py and the _patch_sync_state docstring ("one write per boot") are stale; Related to for ent#703 is right while turn admission doesn't wait on a pull (check-then-act) — file the follow-up or have the operator accept it.

@obasilakis obasilakis left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve on content.

  • Security: the new endpoints use the same auth as auto-sync (reading the flag needs agent access, changing it needs ownership). The pull refuses to run over unresolved conflicts, and the push refuses to commit them. Status text written by the agent is limited to the known values and truncated before it is stored.
  • Both migration tracks are present. Merged into dev, the Alembic history has a single head (0081_pull_sync).
  • The PR's tests pass locally (178). Two deliberate code breaks were each caught by a test: removing the push cycle's refusal to commit unresolved conflicts, and dropping --rebase-merges.

Follow-ups (non-blocking):

  1. File the issue for the window between the idle check and the pull: a turn admitted during a pull can see files change mid-read.
  2. Add backend tests for GET/PUT .../git/pull-sync: a non-owner is rejected, and 404 when git is not configured.
  3. The # mcp: header in routers/git.py does not mention pull-sync.
  4. Release notes: agents with auto-sync already on get pull turned on at upgrade and will merge main into their working branch every cycle.

Merge after #3020. tests/registry.json will conflict once #3020 lands; resolving it is a small by-hand merge.

…eartbeat

Brings the updated base (with current dev). Resolve: registry = base file
plus the ent#703 entry; agent-lifecycle keeps the pull-cycle paragraph beside
the base's updated agent-side paragraph; git-sync-health keeps both new
sections (1c ent#708, 1d ent#703).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…/ent703-pull-heartbeat

dev gained 0081_portal_messages_unread_idx (#3076). Resolve:
- db/migrations.py: keep dev's portal_messages_unread_index entry, then pull_sync.
- Alembic: 0081_pull_sync -> 0082_pull_sync, down_revision
  0081_portal_messages_unread_idx, so the version line keeps a single head.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vybe

vybe commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

merge-train: not on this train — rides the next train once fixed.

Race in the pull cycle: the "no execution running" check happens before the pull, but nothing holds turn admission during it. _with_stash in docker/base-image/agent_server/routers/git.py runs git reset --hard pre_head on the conflict-undo and timeout paths, which discards tracked-file edits made by a turn that started mid-pull. The PR body ("Local work is never discarded") and GitSyncSettingsPanel.vue ("never discards its own changes") claim more than the code guarantees. Either hold admission for the duration of the pull or narrow both claims — that's your call, hence the ejection.

Also noted, not blocking: GET/PUT /api/agents/{name}/git/pull-sync have no endpoint test (non-owner / git-not-configured); the PR body names 0081_pull_sync ← 0080 but the file is 0082_pull_sync ← 0081; --rebase-merges now applies to the shared push rebase too. Heads check, both migration tracks and the pull-cycle suite (33 tests) are all fine.

Heads-up for sequencing: #3035 also adds 0082_* off 0081 — whichever lands second must re-parent.

@AndriiPasternak31

Copy link
Copy Markdown
Contributor

Migration clash with current dev: two Alembic heads.

This branch adds 0082_pull_sync with down_revision = "0081_portal_messages_unread_idx". dev now has 0082_agent_sync_state_divergence (from #3035), and it also has down_revision = "0081_portal_messages_unread_idx". If this PR merges as it is, the revision graph has two heads. alembic upgrade head resolves its target before applying anything, so on PostgreSQL it would apply zero revisions, not only this one (Invariant #3, #2068). Git reports no conflict, because each file is valid on its own.

Fix before merge:

#3022 carries this migration and will need a re-merge of #3021 after that.

dolho and others added 2 commits September 30, 2026 10:38
… revision to 0083

dev gained 0082_agent_sync_state_divergence (#3035), which adds its own
agent_sync_state columns. Union both column sets across schema/tables/
sync_state/sync_health_service, chain 0083_pull_sync off dev's head, and
order the SQLite entry after agent_sync_state_divergence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#2996's route census (merged to dev) requires every new route to be
classified. The agent's pull loop reads GET .../git/pull-sync with its
own key each cycle (AGENT_CALLABLE); the PUT is a setting write, so it
takes Depends(require_person) like the other #2996 settings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vybe

vybe commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

merge-train: not on this train. It rides the next one once fixed. The migration renumber is right: 0083_pull_sync ← 0082_agent_sync_state_divergence gives a single head merged with dev, and both tracks are present.

Yesterday's ejection reason is still open. The two commits since then (28cd3d8ce renumber, f1bee1340 person-only write) don't touch it.

  • The reset: _with_stash (docker/base-image/agent_server/routers/git.py:1088-1098, and the timeout path at :1111) runs git reset --hard pre_head on any stash pop failure.
  • The race: a turn admitted mid-pull that rewrites a stashed file makes the pop fail. The reset then deletes that turn's write. This was reproduced in a scratch repo.
  • The gate doesn't prevent it: admission never waits on _REPO_LOCK (process_registry.py:124), and web-terminal docker exec sessions (services/agent_service/terminal.py:195-227) aren't in the registry at all.
  • The claims are unchanged: "Local work is never discarded" still appears in the PR body, git.py:1123 and :1196, requirements/github.md:886, and GitSyncSettingsPanel.vue:75-77.
  • Your call: either hold admission or refuse the reset when the tree changed since the stash, or narrow all five claims.

Also noted, not blocking:

dev moved past 0082 (0083 to 0085 landed), so 0083_pull_sync forked the
Alembic graph into two heads, which alembic-head-watch flagged. It is now
0086_pull_sync on 0085_ent720_email_identity, and the SQLite list keeps both
sides, with dev's entries first and pull_sync after. The merge also brings
#3107, the agent-server boot fix whose absence failed journey-smoke.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dolho added a commit that referenced this pull request Oct 1, 2026
dev moved past 0082 (0083 to 0085 landed), so 0083_seat_ask_class_state
forked the Alembic graph into two heads, which alembic-head-watch flagged.
It is now 0087_seat_ask_class_state on 0085_ent720_email_identity. 0087
rather than 0086 because #3021's pull_sync takes 0086; whichever of the two
merges second re-parents onto the other. The SQLite list keeps both sides,
with dev's entries first. The merge also brings #3107, the agent-server boot
fix whose absence failed journey-smoke.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vybe vybe added the status-needs-fix PR has an unaddressed review/validation finding; cleared by the author's next push (#2815) label Oct 1, 2026
@vybe

vybe commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

merge-train: not on today's train. I've labelled this status-needs-fix so the next train skips it until you push. The finding from 09-29/09-30 is still in the current head (3c2dc687e): _with_stash still runs git reset --hard pre_head at docker/base-image/agent_server/routers/git.py:1093 and :1111, and :1196 still says "Local work is never discarded". There are two ways to fix it: hold turn admission for the duration of the pull (or refuse the reset when the tree changed after the stash), or narrow all five claims. #3022 stays blocked behind this PR.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

status-needs-fix PR has an unaddressed review/validation finding; cleared by the author's next push (#2815) ui PR touches the frontend UI — triggers Playwright e2e tests

4 participants