Skip to content

refactor: redact PII with core's guard engine, and restore tool arguments too - #28

Merged
fylorn merged 1 commit into
devfrom
refactor/pii-on-tw-guard
Sep 23, 2026
Merged

fylorn merged 1 commit into
devfrom
refactor/pii-on-tw-guard

Conversation

@fylorn

@fylorn fylorn commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

The server edition had its own copy of everything the desktop gateway does for redaction: matching, one placeholder per value, holding back a split placeholder in a stream, and restoring frame by frame. The two copies had drifted, each missing something the other had. tw-guard (core v0.37.0, ThinkWatchProject/ThinkWatch-Core#149) is now the one engine; the patterns and the {{EMAIL_1}} scheme stay ours.

What changes

  • think_watch_common::pii is the single home of the pattern config and the at-rest redactor. PiiPatternConfig existed twice (common and gateway), and so did redact_blob. Both second copies are deleted.

  • PiiRedactor keeps only what an in-flight redactor needs:

    • which parts of a decoded request to look at;
    • apply_to, which carries the values onto the raw request;
    • restore_body for a whole response.

    Matching is core's scan_text: patterns run on the decoded text as written, and restoring into JSON escapes what it puts back.

  • Tool-call arguments are restored now. The stream shaper uses core's FrameRestorer, with one lane per content block or tool call. Before, a model asked to email a@x.com called the tool with {{EMAIL_1}} as the address. New unit test: a_tool_calls_arguments_get_the_callers_pii_back.

  • Saving a pattern compiles it exactly as the redactor will, with NFA and DFA capped the same way. The placeholder prefix must be 1–32 letters, digits or underscores; a brace in it would make a placeholder indistinguishable from text.

  • The admin "try patterns" endpoint reads the label up to the last underscore, so CUSTOM_EMAIL is no longer reported as CUSTOM.

  • PiiRedactor::new() hard-coded the six seed patterns a second time, only for tests. Tests now build from the list db/seeds.sql ships.

  • GatewayError::PolicyBlocked (new in core) maps to 403 policy_blocked.

Verification

  • cargo fmt --check and cargo clippy --workspace --all-targets -D warnings are clean.
  • cargo nextest run --workspace --lib --bins --tests: 630 passed. 9 stream-restorer unit tests moved to core with the code, and 1 is new.
  • Integration suite, run with core as a local path patch before tagging (same code as v0.37.0):
    • 224 passed, 23 failed.
    • 22 of the failures are identical to dev's baseline.
    • The 23rd, successful_login_decays_subnet_failure_counter, is a proof-of-work grinder hitting its 10M-iteration cap at difficulty 21 (~1% per run). It passed when rerun.
    • pii_streaming and content_filter_pii pass.

No release.

🤖 Generated with Claude Code

…ents too

The server edition had its own copy of everything the desktop gateway
already does for redaction: matching, one placeholder per value,
holding back a split placeholder in a stream, restoring it frame by
frame. It had drifted twice over. tw-guard (core v0.37.0) is now the
one engine, with our patterns and our `{{EMAIL_1}}` scheme.

- `think_watch_common::pii` is the single home of the pattern config
  and the at-rest redactor. `PiiPatternConfig` existed twice (common
  and gateway) and `redact_blob` twice; both copies are gone.
- `PiiRedactor` keeps what only an in-flight redactor needs: which
  parts of a decoded request to look at, `apply_to` to carry the
  values onto the raw request, `restore_body` for a whole response.
  Matching is `scan_text`: patterns run on the decoded text as
  written, and restoring into JSON escapes what it puts back.
- `PiiRedactor::new()` hard-coded the six seed patterns a second time
  for tests; tests now build from the same list `db/seeds.sql` ships.
- The stream shaper restores through core's `FrameRestorer`, one lane
  per content block or tool call. **Tool-call arguments are restored
  now**: a model asked to email `a@x.com` used to call the tool with
  `{{EMAIL_1}}` as the address.
- Saving a pattern compiles it exactly as the redactor will, and the
  placeholder prefix must be letters, digits or underscores; a brace
  in it would make a placeholder indistinguishable from text.
- The admin "try patterns" endpoint reads the label up to the last
  underscore, so `CUSTOM_EMAIL` is no longer reported as `CUSTOM`.

`GatewayError::PolicyBlocked` (new in core) maps to 403
`policy_blocked`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@fylorn
fylorn merged commit a6678cf into dev Sep 23, 2026
@fylorn
fylorn deleted the refactor/pii-on-tw-guard branch September 23, 2026 19:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant