feat(pii): export PII/audit events as a Prometheus counter#10641
Open
walcz-de wants to merge 1 commit into
Open
feat(pii): export PII/audit events as a Prometheus counter#10641walcz-de wants to merge 1 commit into
walcz-de wants to merge 1 commit into
Conversation
The PII EventStore ring buffer is capacity-bound and meant for
recent-audit browsing via /api/pii/events; operators also want a
monotonic, scrape-friendly signal on /metrics — how many
detections/masks/blocks per hour, per origin, and whether the filter
stopped firing after a deploy (silent-failure class).
EventStore.Record is the single choke point every producer already goes
through (request middleware, response scrubbing, MITM proxy
connects/intercepts), so one lazily-initialised counter there covers all
paths without touching any producer:
localai_pii_events_total{kind, origin, action, direction}
Same lazy otel.Meter pattern as core/services/routing/billing, so the
counter lands on the Prometheus-backed global MeterProvider installed by
the monitoring service. No behaviour change; label cardinality is
bounded (enum-like fields only, no pattern IDs or user IDs).
Assisted-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: stefanwalcz <stefan.walcz@walcz.de>
Contributor
Author
|
Live verification as promised: deployed this patch on our LocalAI instance (v4.5.6 + this commit, PII middleware active on cloud-proxy models). First request containing PII (name + email, masked before egress) produced: Counter lands on /metrics via the existing Prometheus-backed MeterProvider, series appears on first event, labels populate as designed ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a single Prometheus counter for the PII middleware / MITM audit pipeline:
Why
The PII
EventStorering buffer is capacity-bound and meant for recent-audit browsing via/api/pii/events. For operations you also want a monotonic, scrape-friendly signal on/metrics: how many detections/masks/blocks per hour, per origin — and most importantly whether the filter stopped firing after a deploy (silent-failure class; alertable with a simplerate()).How
EventStore.Recordis the single choke point every producer already goes through (request middleware, response scrubbing, MITM proxy connects/intercepts), so one lazily-initialised counter there covers all paths without touching any producer. Same lazyotel.Meterpattern ascore/services/routing/billing, so the counter lands on the Prometheus-backed global MeterProvider installed by the monitoring service.No behaviour change. Label cardinality is bounded — enum-like fields only (no pattern IDs, no user IDs).
Verification
go build ./core/services/routing/pii/+go test ./core/services/routing/pii/...green. Deployed on a LocalAI instance with the PII middleware + MITM proxy active; counter series appear on /metrics after the first detection/connect event.