[doc][DOC-908] exclude Jupyter notebooks from llms-full.txt

[dstrodtman]

Why

After #63130 shipped, the generated llms-full.txt corpus is polluted by raw Jupyter notebook source. sphinx-llms-txt reads each docname's source from _sources/ verbatim, so for the 127 .ipynb pages under doc/source/ it appends the full notebook JSON (cells, outputs, embedded base64 images, metadata) into the file. That's the largest source of low-signal bytes in the corpus targeted at agents.

What

Append computed notebook docnames to the existing llms_txt_exclude list in doc/source/conf.py.

llms_txt_exclude matches docnames (extension stripped) via fnmatch.fnmatch — see sphinx_llms_txt/collector.py. A pattern such as **/*.ipynb can't match because the docname carries no extension. The change enumerates *.ipynb files under the source directory at conf-load time and converts each path to its docname (relative to the source dir, suffix stripped, posix separators).

Scope:

• Affects only llms.txt / llms-full.txt. The Sphinx HTML build is governed by the separate exclude_patterns list (line 351 of conf.py), which is untouched. All 127 notebooks remain fully rendered on docs.ray.io.
• Notebooks already in exclude_patterns (e.g. serve/tutorials/video-analysis/*.ipynb) aren't built, so adding their docnames to llms_txt_exclude is a harmless no-op.

Verification

After RtD builds the PR preview:

• Fetch llms-full.txt from the PR build and confirm no "cell_type": "code" / "output_type": "stream" / base64 image data appears in the corpus.
• Confirm llms.txt (the summary index) still resolves and looks sane.
• Spot-check that a couple of notebook pages still render normally in the HTML preview.

Context

Tracked under [DOC-908]. Follow-up to #63130 (DOC-875). Part of [DOC-844] (Agent Ray docs umbrella).

Tuning this list further (other low-signal page types) is deferred until the rebuilt corpus can be inspected directly.

[gemini-code-assist]

Code Review

This pull request updates doc/source/conf.py to dynamically exclude all Jupyter notebooks from the llms-full.txt corpus. This is achieved by scanning the documentation directory for .ipynb files and adding their relative docnames to the llms_txt_exclude list, preventing raw JSON content from being included in the LLM agent corpus. I have no feedback to provide.

[ronny-anyscale]

after while, 🐊