[Bazel Migration] Master Tracking Issue (Make parity) #3530

Goal

Replace Make with Bazel without regressing supported platforms, compiler/ISA behavior, release contents, ABI/public symbols, downstream use, performance, or developer workflows.

Objective is a cutover, not indefinite coexistence at 100% flag parity. Obsolete or irrelevant Make knobs are to be explicitly de-scoped with maintainer approval rather than reimplemented. Items below are ordered by whether they block the cutover.

Last audited: 2026-09-19 against main@67117cff, extended 2026-09-23. Evidence for the re-audit is in this comment; this description is the authoritative status list.

Legend: [x] done on main · [~] fix in review (PR open) · [ ] open · [-] de-scoped / not applicable

Status summary

Release packaging parity on Linux and Windows x86-64 is no longer the dominant risk. The remaining risks are: released-library loadability outside Bazel runfiles (in review), compiler-matrix coverage, DPC++ execution, architecture-parameterized release layout for the now-supported non-x86 targets, downstream/ABI validation against Bazel output, and non-hermetic host-tool discovery.

Done on main

Build system and release construction:

• Bazel 9 migration: feat: bazel 9.0 migration #3496
• MKL package extraction fixes: fix: correct MKL strip_prefixes for conda packages in MODULE.bazel #3518
• MKL-free static libraries, hidden MKL objects in dynamic libraries: bazel: fix MKL embedding — static lib must not contain MKL symbols #3532, bazel: align Linux MKL release parity #3662
• Symbol-visibility alignment: bazel: fix symbol visibility parity with Make build #3534
• Release layout, SONAME/versioned links, environment scripts, pkg-config metadata: bazel: add release structure improvements (versioning, SONAME, vars.sh, pkg-config) #3513
• //:release transition to all configured x86 ISA variants and DPC++ artifacts by default: bazel: release target auto-compiles all ISAs and DPC++ by default #3537
• Debug, ASan, TSan, UBSan, MSan, custom --copt/--linkopt: bazel: add debug/sanitizer configs and expand build documentation #3538
• Linux x86-64 release-tree, archive-member/export and binary-metadata parity checks: bazel: align Linux MKL release parity #3662
• Windows x86-64 release package, exported-symbol comparison, DPC++ release construction: Bazel: align Windows release package with make #3670
• Linux and Windows Make/Bazel release comparison in nightly CI; release-comparison symlink handling fixed: ci: fix release comparison failing on every dereferenced Make symlink #3782
• Standalone Windows CMake consumer smoke build from the Bazel release
• Windows PowerShell helper documentation, including 5.1 native-command failure propagation: Bazel: document Windows PowerShell helpers #3701

ISA and compiler behavior (all #3705 unless noted) — closes the former P0 "ISA compilation contract" block except for the verification item:

• GCC/Clang ISA table: dev/bazel/flags.bzl emits -march=nocona / -march=haswell / -march=skylake-avx512, matching gnu.32e.mk:61-64. The "AVX2 and AVX-512 both map to -march=haswell" defect is gone.
• Clang ISA option lists are no longer empty.
• MSVC per-ISA /arch: _MSVC_CPU_FLAGS in dev/bazel/cc/compile.bzl:42-57, matching vc.mkl.32e.mk:47-50. No longer inherited from rules_cc auto-configuration.
• Linux DPC++ archiver discovery: cc_toolchain_lnx.bzl:99-105 derives llvm-ar from dpcc_path and hard-fails when absent, instead of from the host compiler path.

Make options now available in Bazel:

• STDALLOC=yes: bazel: add Linux standard allocator option #3714
• Named LeakSanitizer configuration: bazel: add Linux LeakSanitizer config #3710
• Parameter-library enable/disable: Add Bazel parameter library layout control #3737
• OpenRNG backend, Linux AArch64 and RISC-V64 cross-compile: Bazel: add AArch64/RISC-V64 cross-compile and OpenRNG backend support #3719
• Windows debug/release CRT selection for the custom ICX toolchain (with d suffix): bazel: decouple Windows MSVC runtime from --compilation_mode, add d suffix #3740

Resolved as non-issues:

• [-] "Three-versus-four ISA variants / decide whether mc3/SSE4.2 is supported." Make builds three x86 variants — dev/make/function_definitions/32e.mk:25 is CPUs := sse2 avx2 avx512, and REQCPU is validated against exactly that list (makefile:77-78). mc3_OPT.* survives in the compiler-definition files but is unreferenced, i.e. dead in Make too. Bazel's _ISA_EXTENSIONS_X86 matches Make exactly. Nothing to decide.
• [-] "2× static artifact / MKL bloat" blocker: resolved. bazel: align Linux MKL release parity #3662 measured complete Make and Bazel releases at ~405 MB and ~410 MB. Keep a size threshold, but the residual delta is not embedded MKL.
• Linux pkg-config consumers work in both link modes (verified by compiling and running basic_statistics_dense_batch against a Bazel release via pkg-config --libs dal-dynamic-threading-host / dal-static-threading-host). The capability exists; only a CI lane is missing — see below.

Fixes in review

Blocks cutover

Compiler matrix

• Linux clang host //:release does not build on main. dev/bazel/flags.bzl:150-151 adds -fno-strict-overflow for every non-icx compiler and flags.bzl:18 adds -fwrapv unconditionally; clang then fails under the build's own -Werror with argument unused during compilation: '-fno-strict-overflow'. Make passes neither flag to Linux clang (clang.32e.mk:55); they come from COMPILER.all.gnu (gnu.32e.mk:49). --copt=-Wno-unused-command-line-argument makes the whole clang release build clean, so this single flag is the only blocker. Undetected because there is no clang Bazel lane.
• Linux clang host release lane
• Linux ICX host release lane
• Linux GCC host + ICPX DPC release lane that actually builds (today Azure does --nobuild for release-dpc, ci.yml:378)
• Linux ICX + ICPX DPC release lane
• Windows MSVC cl host release lane (both Windows Bazel lanes use icx; compare_windows_release.ps1:22-27 states the comparison is packaging-only, not compiler equivalence)
• Linux GCC host release lane
• Windows ICX host/DPC release lane

ISA verification

• Verify ISA by compile options, not symbol names. dev/bazel/tests/isa_coverage_test.sh proves that E0/E4/E6-named symbols exist, not that the objects were compiled with distinct instruction sets. bazel aquery 'mnemonic("CppCompile", deps(//cpp/daal:core))' --cpu=all does show the same source compiled three times with the three distinct -march values, and --cpu=auto yields baseline + host ISA — so an aquery-based regression test is straightforward. The repo contains no aquery usage today.
• Add small ISA probe targets that require the intended instructions.
• Add runtime dispatch and representative performance comparison against Make on capable machines.

Architecture-dependent release layout

• Parameterize host/target architecture IDs and release paths. dev/bazel/release.bzl:301,302,620 and dev/bazel/scripts.bzl:95,185,239 hardcode lib/intel64 / redist/intel64; Make parameterizes them (makefile:223,226). After Bazel: add AArch64/RISC-V64 cross-compile and OpenRNG backend support #3719 this is what blocks an AArch64/RISC-V release tree, and it blocks Windows ARM64 in build(bazel): support Windows ARM64 (Make PLAT=winarm parity) #3795.
• Produce a Bazel release tree for AArch64, RISC-V, Windows ARM64 and macOS. The existing non-x86 lanes build only //cpp/daal:core_dynamic and //cpp/oneapi/dal:dynamic and assert on readelf output. The RISC-V Bazel lane also uses GCC where Make uses Clang and executes nothing under QEMU.
• macOS x86-64 Bazel toolchain/build with .dylib install-name/version-link handling.
• Select dependencies by target platform or build them from source; remove x86-specific defaults where inappropriate.
• Remove execution of hardcoded host g++ during target configuration.
• The pkg-config generator is duplicated and the Bazel copy is x86-only. Make preprocesses deploy/pkg-config/pkg-config.cpp, which selects lib/intel64 / lib/arm / lib/riscv64 from target macros; dev/bazel/scripts.bzl:185,239 hardcode lib/intel64. Byte-identical on x86 today (verified by re-running the Make recipe and diffing both .pc files); diverges the moment a non-x86 release tree exists.

DPC++ execution

• Run a packaged SYCL CPU-device example from the standalone release tree on every relevant PR. Bazel currently runs only --config=host tests (ci.yml:678); examples/oneapi/dpc/BUILD exists but no CI command references it.
• Run Windows packaged DPC++ examples/tests, not only build/link plus host examples.
• GPU nightly execution where hardware is available.
• AOT validation, if AOT is part of the supported contract.
• Exercise the repo's own pinned oneMKL SYCL in a real DPC link. The GHA nightly builds release-dpc only after source /opt/intel/oneapi/setvars.sh, which sets MKLROOT and repoints @mkl at system oneAPI; Azure only does --nobuild. The pinned package is therefore never link-tested (this is how bazel: fix DPC++ linking against oneMKL (order and CPU dispatch kernels) #3801's two defects survived).
• Check packaged runtime dependencies for SYCL, MKL SYCL and oneDPL. A Bazel release ships no DPC pkg-config file (BUILD:42-58 emits only host .pc files; deploy/pkg-config/pkg-config.cpp has no dpc/sycl handling) and no oneMKL SYCL runtime, so a DPC consumer must supply a version-matched oneMKL itself.
• Compare representative DPC++ runtime performance against Make.

Downstream and ABI validation

• Build a daal4py/scikit-learn-intelex smoke consumer against the Bazel release. Today abi_check.sh and both sklearnex lanes consume Make artifacts exclusively; no Bazel-produced library is ABI-checked or exercised downstream.
• Cover static/dynamic and host/DPC consumer variants; add a Linux pkg-config consumer lane (the capability is verified, the lane is missing).
• Maintain an explicit public ABI baseline.
• Review and document every ignored symbol prefix in the level-4 comparator. It currently drops all-uppercase symbols (^[A-Z][A-Z0-9_]+(_64)?$), all _ZNSt/_ZSt libstdc++ symbols, the whole _ZN4daal namespace for libonedal_dpc.so, and silently excludes any file whose suffix is in neither TEXT_SUFFIXES nor BINARY_SUFFIXES from the level-3 content comparison. Measured effect on x86/gcc: the ignore lists hide nothing (raw symbol sets are equal except three linker-synthesized __bss_start/_edata/_end), but the uppercase rule is far broader than its stated MKL/Fortran intent.
• Keep separate forbidden-export tests for MKL/internal symbols. (mkl_linkage_test passes today.)
• [~] Compare SONAME/install names, DT_NEEDED, RPATH/RUNPATH and static archive symbols separately — bazel: compare release library linkage, not just exported symbols #3800.

CI wiring and correctness

• dev/bazel/tests/release_structure_test.sh is wired into no CI job. Only mkl_linkage_test, isa_coverage_test and parameters_layout_test.sh run (ci.yml:449,531,536). It is the only check covering the libonedal_core.so → .so.4 → .so.4.0 chain, which compare_release_trees.py:367-376 documents as its own blind spot. It passes 20/20 on a fresh release, so wiring it up is free.
• .github/workflows/ci-aarch64.yml:341 (CI AArch64 status job) is not fork-guarded while both of its dependencies are, so on a fork it reports success with nothing having run.
• Windows Bazel has no release_static example link mode; Make exercises both lib and dll.
• CI time budgeting: the audited nightly passed the Windows Bazel release, level-4 comparison and CMake consumer, then exceeded the 180-minute job budget during later BaseKit compression.

Documentation

• dev/bazel/README.md:213 documents --cpu="avx2,avx512", but dev/bazel/config/config.bzl:164 splits on whitespace, so the comma form is a hard analysis failure in _check_cpu_extensions. Space-separated lists and all/auto/modern work.
• README.md:793 maps PLAT=<isa> to --cpu=<isa>; PLAT is platform selection and REQCPU is the CPU-list knob.
• Remove stale text saying a Make-like release structure is missing; reconcile duplicate statements about whether //:release enables all ISAs and DPC++; document the CPU-list separator and distinguish cpu=auto from the //:release transition; remove Bazelisk version drift between documentation/Azure and the Windows helper.
• Version numbers are hardcoded in dev/bazel/config/config.bzl:308-317 instead of read from makefile.ver — silent drift at the next version bump.

Needed, lower urgency

• REQPROFILE=yes / ITT profiler integration
• Named code-coverage configuration and report CI (bazel: add code coverage build option #3716)
• Threading-layer selection (none/TBB)
• Algorithm-selection profiles, or a dependency-validated custom selection rule
• Complete MemorySanitizer support with an instrumented standard library/sysroot: Sanitized bazel builds with instrumented standard library #3626
• Conda-provided toolchain lane: CI: exercise oneDAL Bazel examples under a conda-provided toolchain #3682
• .nupkg production and documentation-package targets, if literal Make target parity requires them
• Threshold gates for: clean Make vs Bazel build time, one-header and one-source incremental rebuild, warm disk-cache rebuild, release and static archive size, downstream link time, representative algorithm throughput/latency. Measured baseline on one 224-core box, gcc 13.3 both sides, -j96: clean 122.3 s Make vs 66.6 s Bazel; null 12.0 s vs 1.1 s (warm server); touch one .cpp 10.0 s vs 5.1 s; touch one .h with 14 users 11.5 s vs 5.5 s — Bazel wins every case while building strictly more (Make target is libraries, Bazel target is the staged release tree). Both build systems are bit-for-bit reproducible (985/985 identical sha256 across independent builds), so there is no reproducibility regression at cutover.
• Hermeticity: compiler/link tools are discovered via PATH, system include directories and the shell environment are consumed, and Windows uses INCLUDE/LIB. Either package and register pinned compilers, sysroots and link tools, or narrow the goal and document which environment inputs are part of the build contract.
• Make knobs with no Bazel equivalent, listed for completeness rather than as requirements: CHECK_DLL_SIG, BULLSEYEROOT/COVFILE, SYSROOT, WORKDIR, RELEASEDIR, BUILDREV, a named/validated OPTFLAG and LINKER, make help.

Candidates for explicit de-scoping

Maintainer decision required so that cutover is not blocked on them: Bullseye coverage, CHECK_DLL_SIG, WORKDIR/RELEASEDIR relocation, make help, and literal one-to-one target names (daal_c, onedal_c, _release_c, …) where a flag already expresses the same thing.

Note on test targets: Make has no test goals at all (makefile goals are only daal/oneapi/release/clean); all 325 *_test targets are Bazel-only. Unit-test coverage cannot regress at cutover — Bazel is a strict superset.

Definition of done

Close this tracker only when:

1. Every supported Make platform/compiler/backend combination is either supported and continuously tested by Bazel, or explicitly de-scoped with maintainer-approved rationale.
2. Release-tree, linkage and public-ABI comparisons are hard gates on every supported platform.
3. ISA compile options and runtime dispatch are validated, not inferred from symbol names.
4. At least one host and one DPC downstream consumer use the Bazel-produced release.
5. Package size, build time, downstream link time and representative runtime performance stay within agreed thresholds.
6. Every supported Make command-line capability has a documented Bazel equivalent, or a recorded de-scope decision.
7. Documentation matches actual build transitions/defaults and CI behavior.