Fix stale heartbeat process PIDs cleanup issue

[aditya-shantanu]

What this PR does / why we need it:

Fixes a logic vulnerability in resourcectl cleanup where stale heartbeat process PIDs remained permanently trapped in the local state file (~/.local/resourcectl/state.json) if the heartbeat process exited prematurely. When the OS eventually recycled that PID and assigned it to a new innocent process, the subsequent execution of resourcectl cleanup would violently terminate that unrelated process via SIGKILL.

To fix this, we:

• Safely verify the heartbeat process identity on Linux by reading and parsing /proc/<pid>/cmdline to ensure it contains the "heartbeat" and "--name" <resource> arguments before attempting to kill it.
• Guarantee that stale PIDs are zeroed in state.json even if the kill operation fails.
• Prune fully released resources completely from the state file rather than leaving empty structs, preventing unbounded file growth.
• Ignore ESRCH errors if a heartbeat process is already dead.
• Added comprehensive unit tests in main_test.go to reproduce the conditions and verify safety and cleanup behavior.

Which issue(s) this PR is related to:

Fixes b/511322601

Release Note

Fixed a logic issue in `resourcectl` where stale heartbeat PIDs could lead to arbitrary process termination due to PID recycling.

[netlify]

✅ Deploy Preview for agent-sandbox canceled.

Name	Link
🔨 Latest commit	13ca5cf
🔍 Latest deploy log	https://app.netlify.com/projects/agent-sandbox/deploys/6a1f18112f29420008f19375

[Copilot]

Pull request overview

Fixes a PID-recycling hazard in resourcectl cleanup where stale heartbeat PIDs persisted in ~/.local/resourcectl/state.json could cause SIGKILL to be delivered to an unrelated, recycled process. The fix verifies the target process's identity via /proc/<pid>/cmdline on Linux before killing, always zeroes stale PIDs, prunes fully-released resources from the state file, and tolerates ESRCH. Unit tests are added covering matching, innocent (non-heartbeat), and dead-PID scenarios.

Changes:

• Verify /proc/<pid>/cmdline contains heartbeat and --name <resource> before killing (Linux only); ignore ESRCH.
• Unconditionally clear HeartbeatPID after the kill attempt and remove fully-released resources from state instead of leaving empty entries.
• Add main_test.go with a helper-process pattern and a runCleanup integration test using a mock Boskos server.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
dev/tools/resourcectl/main.go	Adds isHeartbeatProcess PID verification, ESRCH tolerance, and prunes released resources from state.
dev/tools/resourcectl/main_test.go	New tests covering isHeartbeatProcess and the full runCleanup flow with matching, innocent, and dead PIDs.

[barney-s]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aditya-shantanu, barney-s

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• dev/OWNERS [barney-s]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment