Skip to content

[ci][doc] Fail when a BUILD exclude, CODEOWNERS entry, or rule names a missing doc path - #66462

Draft
dstrodtman wants to merge 3 commits into
masterfrom
djs-260924-doc-literal-path-guard
Draft

[ci][doc] Fail when a BUILD exclude, CODEOWNERS entry, or rule names a missing doc path#66462
dstrodtman wants to merge 3 commits into
masterfrom
djs-260924-doc-literal-path-guard

Conversation

@dstrodtman

@dstrodtman dstrodtman commented Sep 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Why are these changes needed?

Three config surfaces name doc files by literal path. In each of them, a path that no longer exists is silently ignored instead of raising an error:

  • A glob exclude in a BUILD file under doc/. A missing path excludes nothing, so a moved page silently rejoins the doctest or example target it was excluded from.
  • .github/CODEOWNERS. A missing path matches no file, so the owning team stops being requested on the moved page.
  • .buildkite/*.rules.txt. A missing path matches no changed file, so the moved file falls through to a later, usually broader, rule.

A page rename, a directory move, or an .rst-to-.md conversion changes a path without touching any of these. The docs are in the middle of a full RST-to-MyST conversion, with a round of page and directory renames coming next, so this failure mode is about to get much more common. It has already happened six times on master:

Entry What happened
doctest[serve] excludes deploy-many-models/multi-app.md and production-guide/deploy-vm.md Both pages moved, so they're back inside the target's glob. Neither has testcode or a >>> prompt today, so nothing runs from them.
The serve doc_code test excludes doc_code/llm/llm_yaml_config_example.py and doc_code/llm/qwen_example.py Both moved to source/llm/doc_code/serve/qwen/, outside the target's include, so the entries do nothing.
CODEOWNERS names doc/source/data/working-with-llms.rst #66375 converted the page to .md, so @ray-project/ray-llm hasn't been requested on it since.
test.rules.txt routes doc/test_llms_txt.py to doc The test moved to doc/source/_ext/ in #65461. It now matches the _ext/ directory rule and also emits doc_api, which runs both API-consistency checks for a hermetic Sphinx test that imports no Ray code.

This PR fixes all six and adds a check that makes the next one fail CI.

The check

doc/test_literal_paths.py collects every wildcard-free entry under doc/ from those three surfaces and fails on any path that doesn't exist. It currently checks 129 entries. Details:

  • It reads BUILD files with Python's ast, not regexes, so inline comments and nested glob() calls don't confuse it.
  • It checks only exclude lists. A missing path in srcs, main, or files already fails the Bazel build.
  • Glob patterns that match nothing are left alone, because a pattern is often written before the files it covers.
  • Paths written ahead of their files can go in an ALLOWLIST. It's empty today, because the one pre-routed pair in test.rules.txt ([doc] Add in-repo llms.txt generator, replacing sphinx-llms-txt #64458) has since landed.
  • It needs no Bazel or Ray install and checks the tree rather than a diff.

It runs in the always lint matrix, next to doc_no_new_rst. That matrix is where checks a prose-only diff can fail belong, and a rename PR is prose-only, so it triggers no test steps. The script routes to doc in test.rules.txt alongside test_no_new_rst.py.

It's scoped to paths under doc/. Outside doc/, test.rules.txt has 14 entries that name missing CI files (.buildkite/pipeline.*.yml, ci/docker/*.wanda.yaml, examples/, and others), and CODEOWNERS has one for /python/ray/_private/usage/. Those belong to the CI owners, and some may be deliberate, so this PR leaves them alone.

Commits

  1. Drop the four dead doc/BUILD.bazel excludes. This doesn't change what any target runs. The same commit is in [doc] Convert the doctest-wired Ray Core, Data, RLlib, Train, and observability pages to MyST #66461, so the two PRs merge cleanly in either order.
  2. Point CODEOWNERS at working-with-llms.md. Give doc/source/_ext/test_llms_txt.py its own doc rule ahead of the _ext/ directory entry, the same way api_sidebar.py has one, and move its test.rules.test.txt case to the new path.
  3. Add the check and wire it into the lint matrix.

Related issue number

None. No open PR adds a check like this (gh pr list --search "CODEOWNERS stale path" and "BUILD exclude").

Checks

  • I've signed off every commit (git commit -s).
  • Pre-commit hooks passed on every commit.
  • python doc/test_literal_paths.py passes on this branch. It failed on master with exactly the six entries above.
  • I simulated four failures, restored each afterward, and the check exited 1 on all four: renaming a page doctest[serve] excludes, converting a doctest[data-gpu] page from .rst to .md, moving the CODEOWNERS directory doc/source/serve/llm/, and moving the rules-routed doc/rtd_doctor.py.
  • rayci test-rules, built from ray-project/rayci at c890d24, passes. With only the test.rules.test.txt case moved to the new path, it failed with +doc_api (unexpected), which confirms the rule change is what fixes it.
  • ./ci/lint/lint.sh doc_literal_paths passes.

AI assistance (Claude Code) was used to write the check and find the stale entries.

🤖 Generated with Claude Code
dstrodtman and others added 3 commits September 24, 2026 12:59
@dstrodtman @claude
[doc] Drop four doc/BUILD.bazel excludes that name files that no long…
2594051 
…er exist

A glob exclude naming a missing file is silently a no-op, so these entries
stopped applying when their files moved:

- doctest[serve] excluded deploy-many-models/multi-app.md and
  production-guide/deploy-vm.md. They now live at serve/multi-app.md and
  serve/advanced-guides/deploy-vm.md, inside the target's glob again.
  Neither has testcode or a >>> prompt, so nothing runs from them either way.
- The serve doc_code test excluded doc_code/llm/llm_yaml_config_example.py
  and doc_code/llm/qwen_example.py, which moved to
  source/llm/doc_code/serve/qwen/, outside that target's include.

No change to what any target runs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Douglas Strodtman <douglas@anyscale.com>
@dstrodtman @claude
[ci][doc] Point two stale doc paths in CODEOWNERS and test rules at t…
56dbb23 
…heir files

Both entries name files that have moved, so each has silently stopped
applying:

- CODEOWNERS names doc/source/data/working-with-llms.rst, which #66375
  converted to working-with-llms.md. ray-project/ray-llm hasn't been
  requested on the page since.
- test.rules.txt routes doc/test_llms_txt.py to `doc` alone, but the test
  moved to doc/source/_ext/ in #65461. There it matches the _ext/ directory
  rule and also emits doc_api, which runs both API-consistency checks for a
  hermetic Sphinx test that imports no Ray code. It now gets its own rule
  ahead of the directory entry, the same way api_sidebar.py does, and the
  test.rules.test.txt case follows it to the new path.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Douglas Strodtman <douglas@anyscale.com>
@dstrodtman @claude
[ci][doc] Fail when a BUILD exclude, CODEOWNERS entry, or rule names …
30b4d94 
…a missing doc path

Three config surfaces name doc files by literal path and ignore a path that
no longer exists instead of erroring: glob `exclude` lists in the BUILD
files under doc/, .github/CODEOWNERS, and the .buildkite rules files. A page
rename, move, or .rst-to-.md conversion changes the path without touching
them, so the exclusion, ownership, or routing quietly stops applying. Four
dead doc/BUILD.bazel excludes and the two entries fixed in the previous
commit accumulated that way.

doc/test_literal_paths.py checks every wildcard-free entry under doc/ in
those three surfaces and fails on any that doesn't exist, with an ALLOWLIST
for paths written ahead of their files. It reads BUILD files with ast, needs
no Bazel or Ray install, and checks the tree rather than a diff. It runs in
the `always` lint matrix next to doc_no_new_rst, because a prose-only rename
can fail it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Douglas Strodtman <douglas@anyscale.com>
@dstrodtman dstrodtman mentioned this pull request Sep 24, 2026
Merged
2 tasks done
dstrodtman added a commit that referenced this pull request Sep 28, 2026
@dstrodtman @claude
[doc] Convert the doctest-wired Ray Core, Data, RLlib, Train, and obs…
445d70a 
…ervability pages to MyST (#66461)

## Why are these changes needed?

Converts the last non-API-reference RST pages under `ray-core/`,
`data/`, and `rllib/` to MyST Markdown, plus the `train/` and
`ray-observability/` pages that `doc/BUILD.bazel` names by path. The API
reference trees (`*/api/`, `rllib/package_ref/`, and
`ray-core/compiled-graph/compiled-graph-api`) stay RST for now.

Each page gets its own commit, so you can review one file at a time.
Every commit renames the page's literal `.rst` paths in
`doc/BUILD.bazel` to `.md`. These paths are in `exclude` lists and in
the explicit `files` lists of `doctest[data-gpu]` and
`doctest[train-gpu]`, and they don't glob. Without the edit, an excluded
page would silently join its target once it's `.md`, and an included
page would silently drop out of it.

Conversions are format-only: labels, headings, and prose are unchanged.
A final whitespace-only commit soft-wraps the prose. `ray-soft-wrap`'s
`verify.py` confirms non-whitespace bytes, rendered HTML, and
idempotence for all 11 files.

### `rllib/getting-started`: doctest target removed

The first commit deletes `doctest[rllib2]`, the only target over
`rllib/getting-started.rst`, before the page converts. The target is
tagged `manual` with a hang TODO, so no premerge or postmerge step has
run it, and nothing else references it (`git grep rllib2` is empty). Its
15 `testcode` blocks had no `testoutput`, so they asserted nothing. The
12 visible blocks become display-only `python` code blocks. The three
`:hide:` blocks held `.stop()` cleanups that never rendered, so they're
deleted. `doctest[rllib]` still excludes the page. If these examples
should be validated later, a `doc_code/` script is the better route than
reviving the hanging target. cc @elliot-barn, who owns the TODO.

This commit isn't buildable on its own. It turns a `.. testcode::` that
had no blank line before its body into a `.. code-block:: python`, still
with no blank line, and docutils reads the body as extra arguments to
the directive. The later conversion commit fixes it in the `.md`, and a
squash merge never puts the intermediate commit on master.

### Doctest coverage

Docs-only PRs don't run doctests premerge. This PR edits
`doc/BUILD.bazel`, so every library's docs example step runs, but
premerge can't show the conversion kept the same blocks. For that, I
parsed each `.md` and the `.rst` it replaced with Ray's `pytest-sphinx`
fork, the same parser the doctest macro runs, and asserted identical
sections (directive, body, `:options:`, `:skipif:`) and examples.

| Page | BUILD role | Examples before → after |
| --- | --- | --- |
| `ray-core/handling-dependencies` | `doctest[core]` exclude | 17 → 17 |
| `ray-core/tasks/nested-tasks` | `doctest[core]` exclude | 0 → 0
(`literalinclude` of a tested `doc_code/` file) |
| `data/batch-inference` | `doctest_each` exclude, `doctest[data-gpu]`
files | 7 → 7 |
| `data/transforming-data` | `doctest_each` exclude, `doctest[data-gpu]`
files | 24 → 24 |
| `data/contributing/contributing` | `doctest_each` glob | 0 → 0
(toctree only) |
| `rllib/getting-started` | `doctest[rllib]` exclude, `doctest[rllib2]`
removed | 15 → 0, by the removal commit |
| `ray-observability/user-guides/cli-sdk` | top-level exclude | 16 → 16
|
| `ray-observability/user-guides/ray-tracing` | top-level exclude | 3 →
3 |
| `train/horovod` | `doctest[train]` exclude | 2 → 2 |
| `train/user-guides/data-loading-preprocessing` | `doctest[train]`
exclude, `doctest[train-gpu]` files | 6 → 6 |
| `train/user-guides/using-accelerators` | `doctest[train]` exclude,
`doctest[train-gpu]` files | 10 → 10 |

To show the check would catch a broken conversion, I mutated each page
with examples three ways: dropping a `testcode` block, altering a line
in one, and turning one into a four-backtick fence. The last renders
fine but is never collected. The check failed on all 24 mutations.

### Rendering changes

I ran `render_diff.py` on all 11 pages against `/en/master`. Six render
identically: `nested-tasks`, `data/contributing/contributing`,
`cli-sdk`, `ray-tracing`, `horovod`, and `using-accelerators`. Every
difference on the other five is listed here.

- `rllib/getting-started`: on master, one `.. testcode::` line had no
blank line after it. docutils read the block's first six lines, the
`best_result = results.get_best_result(...)` call, as the directive's
argument and dropped them. So master shows code that uses `best_result`
without showing where it comes from. The converted page renders all six
lines. This is the one place the conversion deliberately doesn't
reproduce master.
- `rllib/getting-started`: the "Farama gymnasium" link now has
`https://`. On master it's a relative link to `gymnasium.farama.org`
that 404s.
- `ray-core/handling-dependencies`: MyST drops the apostrophe when it
slugifies the "can't import the packages" heading, so the section id
changes. That heading had no label, so the page adds a target carrying
the old docutils slug, and existing `#...-can-t-import-...` links still
resolve. An auto-numbered span id, which nothing links to, also changes
from `id4` to `id2`.
- `data/batch-inference` and
`train/user-guides/data-loading-preprocessing`: each renders an RST
comment as an HTML comment, which readers never see.
- `data/transforming-data`: the "When using" definition list's `<dl>`
gains the unstyled `myst` class.

The first preview diff also caught three regressions that the green
build didn't. MyST's `replacements` extension turned two `(c)` list
markers into `©` and an `etc..` into `etc…`. The anchor above had
changed too. A follow-up commit fixes all three. A sweep of the 11 pages
found no other text that extension rewrites.

### Dead `doc/BUILD.bazel` excludes

A final commit drops four `exclude` entries that name files that have
moved: two in `doctest[serve]` and two in the serve `doc_code` test. A
glob exclude naming a missing file excludes nothing, so these had
silently stopped applying. This doesn't change what any target runs.
#66462 carries the same commit, along with a lint check that fails on
this class of stale path, so the two PRs merge cleanly in either order.

### Open PRs that edit these files

Each of these open PRs edits one of the converted `.rst` files, so each
will need to move its change to the `.md`: #66279, #66191, #65956,
#65933, and #65830 (`handling-dependencies`), #65693 (`cli-sdk`), and
#66419, #66420, #66422, and #66423 (`data/batch-inference`,
`train/user-guides/data-loading-preprocessing`). #66096, #64053, and
#63881 also edit `doc/BUILD.bazel`, and only #66096 edits nearby lines.

## Related issue number

None. No open PR converts these pages. I checked `gh pr list` for open
PRs touching each file.

## Checks

- [x] I've signed off every commit (`git commit -s`).
- [x] Pre-commit hooks passed on every commit.
- Doctest parity: `python3 doctest_parity.py <the 11 .md files>` with
`pytest==7.4.4` and `pytest-sphinx @
git+https://github.com/ray-project/pytest-sphinx`. All OK, run again
after the soft-wrap commit.
- Static checks on every `.md`: fences balance, no leftover RST roles or
directives, every label from the `.rst` is present, and directive counts
match.
- Read the Docs preview builds green under `fail_on_warning`, and the
render diff against `/en/master` is in Rendering changes. The first
build hit the known autosummary-import segfault and passed on rebuild.
- Local doctest run: not done. The GPU targets need hardware, and the
CPU targets run in this PR's premerge because of the `BUILD.bazel`
change.

AI assistance (Claude Code) was used for the conversion and the
verification scripts.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Douglas Strodtman <douglas@anyscale.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant

@dstrodtman