Possible performance enhancement in ConfigManager build #1836

While generating column validation configurations for a batch of 20 tables I noticed simple YAML file generation was quite slow.

Profiling build_config_managers_from_args() showed that the majority of this latency comes from redundant database metadata round-trips and repeated Ibis table/builder compilations per table. I was testing using SQL Server therefore the timings may not be equal to other SQL engines.

1. Unconditional source table reflection in cli_tools.get_pre_build_configs() that is immediately discarded (~0.5s/table)

In data_validation/cli_tools.py (get_pre_build_configs), DVT iterates over tables_list and unconditionally reflects the source table schema via _get_pre_build_configs_base_columns():

# data_validation/cli_tools.py
for table_obj in tables_list:
    casefold_source_columns = _get_pre_build_configs_base_columns(
        source_client, table_obj, query_str
    )

• _get_pre_build_configs_base_columns() calls clients.get_ibis_table_schema(), which executes full SQLAlchemy table reflection (client.table(table_name, schema=schema_name).schema()) against the source database for every table.
• However, casefold_source_columns is only referenced inside the if pre_build_configs[consts.CONFIG_ROW_CONCAT] or pre_build_configs[consts.CONFIG_ROW_HASH]: branch (cli_tools.py:1749-1759).
• Impact:
  1. For validate column and validate schema, casefold_source_columns is never used and immediately discarded.
  2. Even for COUNT(*)-only validations (where get_aggregate_config() needs zero column metadata), DVT still performs 20 sequential source table reflections.
  3. Because get_ibis_table_schema() does not pass or cache the reflected Ibis table on ConfigManager, ConfigManager.get_source_ibis_table() reflects every source table a second time during build_config_from_args().

2. Redundant DB metadata queries and repeated ValidationBuilder compilations in get_aggregate_config() (+0.95s/table)

When --count '*' --sum '*' --min '*' --max '*' are supplied, get_aggregate_config() (data_validation/__main__.py:91-135) invokes ConfigManager.build_config_column_aggregates() once per aggregate type (4 calls per table):

1. Duplicate Source Table Reflection (ConfigManager.get_source_ibis_table):
   Because the reflection in get_pre_build_configs() was discarded, self.get_source_ibis_table() (config_manager.py:420-426) reflects all tables on the source engine a second time.
2. Unnecessary raw_column_metadata queries on non-matching engines (ConfigManager._is_raw_data_type):
   Inside the column loop of build_config_column_aggregates(), DVT checks engine-specific raw types (_is_sql_server_image, _is_db2_zos_xml, _is_db2_xml, _is_oracle_lob, _is_sql_server_text), which all delegate to _is_raw_data_type() (config_manager.py:746-770):

   def _is_raw_data_type(
       self,
       client_name: str,
       source_column_name: str,
       target_column_name: str,
       type_list: List[str],
   ) -> bool:
       """Returns True when either source or target column is of a client & type."""
       raw_source_types = self.get_source_raw_data_types()
       raw_target_types = self.get_target_raw_data_types()
       return bool(
           (
               self.source_client.name == client_name
               and raw_source_types
               and raw_source_types.get(source_column_name.casefold(), [None])[0]
               in type_list
           )
           or (
               self.target_client.name == client_name
               and raw_target_types
               and raw_target_types.get(target_column_name.casefold(), [None])[0]
               in type_list
           )
       )

   Because self.get_source_raw_data_types() and self.get_target_raw_data_types() are called before checking self.source_client.name == client_name or self.target_client.name == client_name, DVT executes raw_column_metadata() queries against both source and target for every table—even when an engine (such as PostgreSQL or BigQuery) is never matched by any client_name ("mssql", "oracle", "db2", "db2_zos").
3. 8x ValidationBuilder + table.mutate() rebuilds per table (get_source/target_ibis_calculated_table):
   Each call to build_config_column_aggregates() (count, sum, min, max, etc.) starts by calling:

   source_table = self.get_source_ibis_calculated_table()
   target_table = self.get_target_ibis_calculated_table()

   Each call constructs a new ValidationBuilder(self) and compiles/mutates all calculated fields accumulated so far (4 agg types × 2 sides = 8 full ValidationBuilder compilations per table), even though build_config_column_aggregates() only inspects base table column names and types (self.get_source_ibis_table() / self.get_target_ibis_table()).

Proposed Fixes

1. Lazy-evaluate _get_pre_build_configs_base_columns() in cli_tools.get_pre_build_configs():
   Only call _get_pre_build_configs_base_columns() inside the if pre_build_configs[consts.CONFIG_ROW_CONCAT] or pre_build_configs[consts.CONFIG_ROW_HASH]: block (and only when cols == "*" or args.exclude_columns is set).
2. Short-circuit ConfigManager._is_raw_data_type() by client_name:
   Check self.source_client.name == client_name before calling self.get_source_raw_data_types(), and check self.target_client.name == client_name before calling self.get_target_raw_data_types().
3. Use cached base Ibis tables in build_config_column_aggregates():
   Use self.get_source_ibis_table() and self.get_target_ibis_table() (which are already cached on self._source_ibis_table and self._target_ibis_table) instead of rebuilding ValidationBuilder(self) via get_source_ibis_calculated_table() / get_target_ibis_calculated_table() on every aggregate type pass.