Skip to content

feat(verify): offline / air-gapped verification (--insecure-ignore-tlog) - #1798

Merged
lockwobr merged 2 commits into
mainfrom
feat/1154-offline-verify
Jul 17, 2026
Merged

lockwobr merged 2 commits into
mainfrom
feat/1154-offline-verify

Conversation

@lockwobr

Copy link
Copy Markdown
Contributor

Summary

Add an offline / air-gapped verification mode to aicr verify via a --insecure-ignore-tlog flag (bool, default false). When set (and only with --key), verification skips the Rekor transparency-log and observer-timestamp requirement and verifies the signature against the public key with no network calls. This is the verification counterpart to #409's aicr bundle --signing-key ... --tlog-upload=false.

Motivation / Context

Air-gapped signing (#409) produces a bundle with no Rekor entry. The default verify path requires a transparency-log inclusion proof, so such a bundle cannot be verified in a disconnected environment. This flag drops that requirement for key-based verification, mirroring cosign verify --insecure-ignore-tlog.

Fixes: #1154
Related: #409 (air-gapped signing, the sign-side counterpart), #407 (KMS public-key verification)

Land-together dependency: the acceptance case ("a bundle signed with bundle --signing-key ... --tlog-upload=false verifies offline") needs the --tlog-upload=false flag from #409. This PR's docs/help reference that flag, so the two should merge as a pair. Neither changes the other's code; the only shared file is docs/user/cli-reference.md, edited in non-overlapping sections.

Type of Change

  • New feature (non-breaking change that adds functionality)
  • Documentation update

Component(s) Affected

  • CLI (cmd/aicr, pkg/cli)
  • Bundlers (pkg/bundler, pkg/component/*)
  • Docs/examples (docs/, examples/)

Implementation Notes

  • Adds the offline sibling policy the codebase reserved for this issue: noTLogVerifyPolicy / NewNoTLogVerifyPolicy() in verifytransparency.go, the verify-side dual of feat(bundle): air-gapped signing with --tlog-upload=false for restricted networks #409's noTLogPolicy.
  • Uses sigstore-go verify.WithNoObserverTimestamps() (confirmed in vendored pkg/verify/signed_entity.go), used exclusively — the vendored VerifierConfig.Validate() forbids combining it with WithTransparencyLog/WithObserverTimestamps, and it is only valid for key-based (not certificate) verification, exactly the air-gapped path.
  • Security boundary: IgnoreTLog reaches only verifyKeySignedBundle. The keyless path (verifySigstoreBundle) and the NVIDIA-CI binary attestation (VerifyBinaryAttestation) stay hardcoded to NewRequireTLogPolicy() and cannot be relaxed. Defense-in-depth: WithKey sets requireSigningKey, so a Fulcio-cert bundle fed to the key path is rejected.
  • Fail-closed: IgnoreTLog && Key=="" is rejected with ErrCodeInvalidRequest in both the library (verifier.Verify) and the CLI.
  • Content binding preserved: VerifyStatementWith still applies WithArtifactDigest and refuses an empty digest, so offline mode cannot accept a signature over unrelated content.

Testing

go test -race ./pkg/cli/... ./pkg/bundler/verifier/... ./pkg/bundler/attestation/...
golangci-lint run -c .golangci.yaml ./pkg/cli/... ./pkg/bundler/verifier/... ./pkg/bundler/attestation/...
  • New tests: TestNoTLogVerifyPolicy_VerifierOptions (exactly one option), TestNoTLogVerifyPolicy_RoundTrip (sign offline with NewNoTLogPolicy(), verify with NewNoTLogVerifyPolicy() → success; same bundle with NewRequireTLogPolicy() → failure, proving the relaxation is load-bearing; runs hermetically via an injected trust source), TestVerify_IgnoreTLogRequiresKey, and the CLI --insecure-ignore-tlog-without---key guard.
  • golangci-lint: 0 issues on all changed packages.

Risk Assessment

  • Low — Additive flag, default false preserves current behavior. The relaxation is scoped to the key-signed bundle attestation and gated on --key; the keyless and binary-attestation trust paths are unchanged.

Rollout notes: Backwards compatible. "insecure" is intentional in the name: with no transparency log there is no trusted timestamp of when the signature was made, so it is only for air-gapped bundles you signed yourself.

Checklist

  • Tests pass locally (make test with -race)
  • Linter passes (make lint)
  • I did not skip/disable tests to make CI green
  • I added/updated tests for new functionality
  • I updated docs if user-facing behavior changed
  • Changes follow existing patterns in the codebase
  • Commits are cryptographically signed (git commit -S)
@lockwobr
lockwobr requested a review from a team as a code owner July 16, 2026 20:37
@lockwobr lockwobr added the theme/supply-chain SLSA, SBOM, Sigstore, and provenance verification label Jul 16, 2026
@lockwobr lockwobr self-assigned this Jul 16, 2026
@lockwobr

Copy link
Copy Markdown
Contributor Author

Paired with #1797 (air-gapped signing counterpart). #1797 adds bundle --tlog-upload=false (the tlog-less KMS bundle); this PR adds verify --insecure-ignore-tlog to verify it offline. They should land together; this PR's docs/help reference the #1797 flag.

@github-actions

Copy link
Copy Markdown
Contributor
@github-actions

Copy link
Copy Markdown
Contributor

Recipe evidence check

No leaf overlays affected by this PR.

This gate is warning-only and never blocks merge.

@coderabbitai

coderabbitai Bot commented Jul 16, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Adds key-based offline bundle verification that skips transparency-log and observer-timestamp requirements. The verifier rejects IgnoreTLog without a key, while the CLI exposes --insecure-ignore-tlog with matching validation. Tests cover policy behavior, tlog-less verification, key validation, and CLI registration. Documentation adds an air-gapped signing and verification example.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related issues

  • Issue 409 — Adds verification support for bundles produced with --tlog-upload=false.

Suggested labels: area/tests, area/security

Suggested reviewers: mchmarny

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title is concise and clearly summarizes the new offline verification mode and its flag.
Description check ✅ Passed The description is directly about the added air-gapped verification mode and its expected behavior.
Linked Issues check ✅ Passed The PR implements the #1154 offline tlog-skip path with --key gating, no-network intent, tests, and docs.
Out of Scope Changes check ✅ Passed The changes shown are all in scope for offline verification support, tests, and related documentation.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/1154-offline-verify

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/user/cli-reference.md`:
- Line 2637: Correct the `--insecure-ignore-tlog` documentation and its offline
example so the claimed no-network verification uses a local exported PEM file,
not `awskms://alias/my-key`. Update the example and surrounding text to
reference the PEM path recommended by the existing note, and clarify that KMS
URI keys still require network access to resolve the public key.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 69d2d944-9e54-40d9-b0d9-6bb77583da93

📥 Commits

Reviewing files that changed from the base of the PR and between f0e2273 and 49ef3ba.

📒 Files selected for processing (7)
  • docs/user/cli-reference.md
  • pkg/bundler/attestation/verifying_test.go
  • pkg/bundler/attestation/verifytransparency.go
  • pkg/bundler/verifier/verifier.go
  • pkg/bundler/verifier/verifier_test.go
  • pkg/cli/bundle_verify.go
  • pkg/cli/bundle_verify_test.go
Comment thread docs/user/cli-reference.md Outdated
@github-actions

github-actions Bot commented Jul 16, 2026 •

Copy link
Copy Markdown
Contributor

Coverage Report ✅

Metric Value
Coverage 79.7%
Threshold 75%
Status Pass
Coverage Badge
![Coverage](https://img.shields.io/badge/coverage-79.7%25-green)

Merging this branch changes the coverage (1 decrease, 2 increase)

Impacted Packages Coverage Δ 🤖
github.com/NVIDIA/aicr/pkg/bundler/attestation 79.72% (+1.83%) 👍
github.com/NVIDIA/aicr/pkg/bundler/verifier 75.57% (-0.52%) 👎
github.com/NVIDIA/aicr/pkg/cli 72.31% (+0.03%) 👍

Coverage by file

Changed files (no unit tests)

Changed File Coverage Δ Total Covered Missed 🤖
github.com/NVIDIA/aicr/pkg/bundler/attestation/verifytransparency.go 100.00% (ø) 4 (+2) 4 (+2) 0
github.com/NVIDIA/aicr/pkg/bundler/verifier/verifier.go 74.07% (-0.58%) 297 (+5) 220 (+2) 77 (+3) 👎
github.com/NVIDIA/aicr/pkg/cli/bundle_verify.go 87.88% (+0.58%) 66 (+3) 58 (+3) 8 👍

Please note that the "Total", "Covered", and "Missed" counts above refer to code statements instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code.

@lockwobr
lockwobr force-pushed the feat/1154-offline-verify branch from 49ef3ba to 7e57f3d Compare July 16, 2026 21:16

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/bundler/verifier/verifier.go`:
- Around line 286-293: Move the IgnoreTLog and Key validation from its current
location to immediately after the context check in the verifier entry point,
before os.Stat(bundleDir) or any bundle-path inspection. Normalize the options
first, then return InvalidRequest for IgnoreTLog with an empty Key so this
validation consistently takes precedence over NotFound errors.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: cba3f1a1-e737-408c-bc27-10973ee2fa08

📥 Commits

Reviewing files that changed from the base of the PR and between 49ef3ba and 7e57f3d.

📒 Files selected for processing (7)
  • docs/user/cli-reference.md
  • pkg/bundler/attestation/verifying_test.go
  • pkg/bundler/attestation/verifytransparency.go
  • pkg/bundler/verifier/verifier.go
  • pkg/bundler/verifier/verifier_test.go
  • pkg/cli/bundle_verify.go
  • pkg/cli/bundle_verify_test.go
Comment thread pkg/bundler/verifier/verifier.go
@lockwobr
lockwobr force-pushed the feat/1154-offline-verify branch from 7e57f3d to 5e9f409 Compare July 16, 2026 21:27

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/bundler/verifier/verifier.go`:
- Around line 158-166: The IgnoreTLog contract must not promise fully offline
verification when KMS key resolution remains remote. In
pkg/bundler/verifier/verifier.go lines 158-166 and 726-730, clarify that it
suppresses transparency-log network calls only and that fully offline
verification requires a local PEM key; in pkg/cli/bundle_verify.go lines 68-70,
replace the KMS URI example with ./bundle-signer.pub, and in lines 105-108 state
the local-PEM requirement.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: acc4d30f-4b45-42e1-b120-4dbdc4f4c6ec

📥 Commits

Reviewing files that changed from the base of the PR and between 7e57f3d and 5e9f409.

📒 Files selected for processing (7)
  • docs/user/cli-reference.md
  • pkg/bundler/attestation/verifying_test.go
  • pkg/bundler/attestation/verifytransparency.go
  • pkg/bundler/verifier/verifier.go
  • pkg/bundler/verifier/verifier_test.go
  • pkg/cli/bundle_verify.go
  • pkg/cli/bundle_verify_test.go
Comment thread pkg/bundler/verifier/verifier.go
@lockwobr
lockwobr force-pushed the feat/1154-offline-verify branch from 5e9f409 to 9efe67c Compare July 16, 2026 22:16

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/bundler/attestation/verifytransparency.go`:
- Around line 42-59: Update the noTLogVerifyPolicy comments and
NewNoTLogVerifyPolicy documentation to promise only no transparency-log network
calls, not zero network calls overall. Explain that fully offline verification
additionally requires offline verification identity and trusted-material
sources, while preserving the policy’s requirement for no transparency-log proof
or observer timestamp.

In `@pkg/cli/bundle_verify_test.go`:
- Around line 297-343: The “ignore-tlog with key” case does not verify
successful offline verification or that the no-tlog policy is selected. Update
TestBundleVerifyCmd_IgnoreTLogRequiresKey to use a tlog-less signed fixture and
its local PEM, then assert verification succeeds with --insecure-ignore-tlog and
fails without that flag while retaining the existing missing-key validation.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 0a0d2762-2824-4238-b17e-2b0cd5c012ed

📥 Commits

Reviewing files that changed from the base of the PR and between 5e9f409 and 9efe67c.

📒 Files selected for processing (7)
  • docs/user/cli-reference.md
  • pkg/bundler/attestation/verifying_test.go
  • pkg/bundler/attestation/verifytransparency.go
  • pkg/bundler/verifier/verifier.go
  • pkg/bundler/verifier/verifier_test.go
  • pkg/cli/bundle_verify.go
  • pkg/cli/bundle_verify_test.go
Comment thread pkg/bundler/attestation/verifytransparency.go
Comment thread pkg/cli/bundle_verify_test.go
Signed-off-by: Brian Lockwood <lockwobr@gmail.com>
@lockwobr
lockwobr force-pushed the feat/1154-offline-verify branch from 9efe67c to d3bf424 Compare July 16, 2026 22:24
@github-actions

Copy link
Copy Markdown
Contributor
@lockwobr

Copy link
Copy Markdown
Contributor Author

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Jul 17, 2026 •

Copy link
Copy Markdown
Contributor
✅ Action performed

Full review finished.

@njhensley njhensley left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall assessment — ✅ Approve

Multi-persona review (supply-chain-security, correctness, test-coverage), each finding independently confirmed or refuted by a senior meta-reviewer against the resolved code; the security boundary was additionally re-derived by hand at head d3bf424.

Implements offline key-based verification (#1154) by adding noTLogVerifyPolicy as the clean dual of the signing-side noTLogPolicy, threading IgnoreTLog through VerifyOptions → verifyStagedSnapshot → verifyKeySignedBundle, and fail-closing on --key at both the CLI and library layers. The claimed security boundary holds: the relaxation reaches only verifyKeySignedBundle (gated on Key != ""); the keyless path (verifySigstoreBundle) and the NVIDIA-CI binary attestation (VerifyBinaryAttestation, hardcoded NewRequireTLogPolicy()) cannot be relaxed. Content binding (artifact digest) is preserved.

✅ Confirmed non-issues (examined and cleared)

  • Trust-downgrade / path confinement — IgnoreTLog cannot reach keyless or binary attestation (re-derived at head SHA).
  • Fail-closed guard — IgnoreTLog && Key=="" rejected with ErrCodeInvalidRequest in verifier.Verify and the CLI; the guard correctly precedes os.Stat so it isn't masked by NotFound.
  • sigstore-go usage — WithNoObserverTimestamps() used exclusively per the vendored VerifierConfig.Validate() constraint; valid only for key-based verify, which is exactly this path.
  • Docs — cli-reference.md + flag help updated in-PR; "insecure" naming is accurate.

Summary

Tier Count
🔴 Blocker 0
🟠 Major 0
🟡 Minor 0
🔵 Nitpick 1

Recommendation: Approve. The single nitpick is optional test hardening.

}
signer, err := attestation.VerifyStatementWith(ctx, data, id, attestation.NewRequireTLogPolicy(), artifactDigest)
tlogPolicy := attestation.NewRequireTLogPolicy()
if ignoreTLog {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Nitpick — Offline-verify passing path proven only at the attestation layer, not the verifier layer

This if ignoreTLog { tlogPolicy = NewNoTLogVerifyPolicy() } branch and its threading from VerifyOptions are exercised in the verifier package only by the reject guard; the one positive verifier-package case uses an awskms://test/key URI that fails remote key resolution before reaching the policy selection (the test intentionally tolerates that downstream failure). So no verifier-package test drives Verify() end-to-end with IgnoreTLog=true to a successful offline verification. The relaxation itself IS fully proven one layer down by TestNoTLogVerifyPolicy_RoundTrip (sign offline -> verify with the no-tlog policy succeeds; the same bundle under NewRequireTLogPolicy() fails), so a policy regression is caught — only this trivial 3-line verifier delegation lacks an in-package end-to-end test.

Blast radius: Coverage only; the underlying behavior is covered at the attestation layer. Optional hardening.

Fix: Add a verifier-level test: sign with a local ECDSA PEM, then assert Verify(ctx, dir, &VerifyOptions{Key: pemPath, IgnoreTLog: true}) succeeds and a companion with IgnoreTLog:false fails — proving the threading, not just the guard.

@lockwobr
lockwobr merged commit 6548f17 into main Jul 17, 2026
167 checks passed
@lockwobr
lockwobr deleted the feat/1154-offline-verify branch July 17, 2026 20:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/bundler area/cli area/docs size/L theme/supply-chain SLSA, SBOM, Sigstore, and provenance verification

2 participants