Utility, testing, data-generation, and CI scripts that support Phoenix development. Most Python scripts can be run with uv run scripts/<path>; some declare PEP 723 inline dependencies and others assume the project venv is active.

GitHub + PyPI usage analytics. See analytics/README.md. Requires GITHUB_TOKEN.

• hallucination_eval_benchmark.ipynb — notebook benchmarking the hallucination evaluator.

Scripts run from CI workflows.

• compile_openapi_schema.py — emit Phoenix's OpenAPI schema to a file.
• ensure_graphql_mutations_have_permission_classes.py — enforce IsNotReadOnly / IsNotViewer on Strawberry mutations and subscriptions.
• test_helm.py — comprehensive Helm chart validation with async concurrency.
• json-canonicalization-schema/ — fixtures/schema for canonicalization tests.

Data wrangling and corpus building (LangChain / LlamaIndex / HaluEval / MS MARCO / WikiQA / Wiki Toxic).

• build_langchain_vector_store.py, build_llama_index_*.py — build vector stores over the Arize docs.
• convert_arize_docs_query_csv_to_jsonl.py, fetch_arize_documentation.py — corpus prep.
• generate_traces.py — emit synthetic LLM traces.
• wrangle_*.ipynb — dataset preparation notebooks.

• generate_ddl_postgresql.py — extract DDL from a PostgreSQL Phoenix DB into src/phoenix/db/ddl/postgresql_schema.sql, validated with pglast. PEP 723 script.
• generate_ddl_sqlite.py — extract DDL from a SQLite Phoenix DB into src/phoenix/db/ddl/sqlite_schema.sql, validated by replaying it into a fresh in-memory database and re-rendering it. PEP 723 script.
• The checked-in canonical schema assets live in src/phoenix/db/ddl/.
• compare_schemas.py — assert both dialects describe the same tables, columns, explicitly created indexes, and CHECK/UNIQUE/FOREIGN KEY constraint names; each generator only validates against its own database, so nothing else catches the two files drifting apart. Names are normalized for PostgreSQL's 63-byte identifier cap, which SQLite does not share. Constraint-backed indexes and PRIMARY KEY names are excluded because the dialects legitimately differ there. PEP 723 script.

Local docker-compose stack for development: Phoenix, OIDC, LDAP, SMTP, Grafana, Prometheus, Toxiproxy, Vite dev server, k8s manifests. See docker/devops/README.md.

GraphQL smoke tests for the chat / playground / evaluator surface.

• test_chat_mutation.py, test_chat_subscription.py — chatCompletion mutation/subscription.
• test_chat_over_dataset.py, test_chat_over_dataset_mutation.py — chatCompletionOverDataset.
• test_create_llm_evaluator.py — createDatasetLlmEvaluator mutation.

• generate_multipage_experiment_data.py — seed an experiment with 300 examples for pagination testing.

Notebook fixtures used during demos and manual testing: ChatRAG-Bench.ipynb, multi-turn_chat_sessions.ipynb, vision.ipynb.

PL/pgSQL-backed bulk data generation for performance testing. See generate_data_via_plpgsql/README.md.

• generate_spans_deeply_nested.py — emit deeply nested span trees.
• generate_spans_for_time_series.py — emit spans spread over time for time-series charts (PEP 723 script).

• generate_spans_for_cost_calculations.py — emit spans across many models/providers to exercise cost calculation.

• generate_spans_for_large_session.py — emit a single very large session for stress-testing the session view.

• litellm_model_prices.py — fetch LiteLLM model pricing JSON, output model_prices.csv and model_prices_by_token_type.csv.

TypeScript mock for OpenAI / Anthropic / Google GenAI APIs with a real-time dashboard. See mock-llm-server/README.md.

• get_spans_dataframe_for_random_conversation_id.py — sample a random conversation_id from PG, then time Client.spans.get_spans_dataframe.
• postgres/postgres_explain_analyze.py — run EXPLAIN ANALYZE over queries in paste_queries_here.sql (PEP 723 script).

• compile_python_prompts.py — compile YAML prompts into Python.
• compile_typescript_prompts.py — compile YAML prompts into TypeScript.

• plotresults.py — plot helper for RAG retrieval evaluation.

Smoke tests intended to be run against a live Phoenix instance.

• dataset_upsert_smoke.py / .ts — exercise the dataset upsert/update flow end to end.
• experiment_runs_filters.ipynb — interactive filter exploration.
• send_spans.py — emit synthetic OpenInference spans.

TypeScript generator for deterministic, realistic agent trace data — an incident-investigation conversation with growing LLM message histories (2→20 messages), tool calls with JSON arguments/results, and search-friendly near-miss vocabulary. Sends to a live Phoenix via OTLP using @arizeai/phoenix-otel. See ts-span-generator/README.md.

• type_check — wrapper invoked by Make targets for typecheck.

# In the project venv
uv run python scripts/<path>/<script>.py

# PEP 723 scripts (declare their own deps inline) work standalone
uv run scripts/ddl/generate_ddl_postgresql.py
uv run scripts/ddl/generate_ddl_sqlite.py
uv run scripts/generate_spans/generate_spans_for_time_series.py
uv run scripts/perf/postgres/postgres_explain_analyze.py

Most span-generation scripts assume Phoenix is reachable at http://localhost:6006; DB-direct scripts read PHOENIX_SQL_DATABASE_URL.