When running the debug command below:
time data-validation -ll DEBUG find-tables -sc bq -tc bq --allowed-schemas=pso_data_validator
I see tables listed for all target schemas, not just pso_data_validator. They are then matched in Python and filtered out.
...
09/27/2026 02:51:50 PM-DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/proj-01/datasets/ACME1/tables?prettyPrint=false HTTP/1.1" 200 None
09/27/2026 02:51:50 PM-DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/proj-01/datasets/ACME2/tables?pageToken=B&prettyPrint=false HTTP/1.1" 200 None
09/27/2026 02:51:50 PM-DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/proj-01/datasets/ACME3/tables?pageToken=D&prettyPrint=false HTTP/1.1" 200 None
09/27/2026 02:51:51 PM-DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/proj-01/datasets/ACME4/tables?pageToken=F&prettyPrint=false HTTP/1.1" 200 None
09/27/2026 02:51:51 PM-DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/proj-01/datasets/acme1/tables?pageToken=S&prettyPrint=false HTTP/1.1" 200 None
09/27/2026 02:51:51 PM-DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/proj-01/datasets/acme2/tables?pageToken=V&prettyPrint=false HTTP/1.1" 200 None
09/27/2026 02:51:51 PM-DEBUG: https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/proj-01/datasets/acme3/tables?prettyPrint=false HTTP/1.1" 200 None
...
09/27/2026 02:51:51 PM-DEBUG: Find tables elapsed: 8.37s
...
real 0m15.494s
user 0m6.058s
sys 0m1.549s
For a target with 100s of schemas (such as a BigQuery project) this can add a large overhead.
If we pass --allowed-schemas then we should be able to casefold filter target schemas before listing tables.
When running the debug command below:
I see tables listed for all target schemas, not just
pso_data_validator. They are then matched in Python and filtered out.For a target with 100s of schemas (such as a BigQuery project) this can add a large overhead.
If we pass
--allowed-schemasthen we should be able to casefold filter target schemas before listing tables.