Add dependency auto download downloader logic

[scaliby]

Description

Add downloader.py script that will download a dependency.

Issue

b/489307581

Testing

Unit testing.

[jamOne-]

Should we fix anything from this list? Maybe 4. Security one?

Gemini review:

Review Report

• Verdict: Request Changes

• Action Items:

  1. [Network & Error Handling] Missing Timeout and OSError Handling:
     • Issue: urllib.request.urlopen(url) lacks a timeout parameter. If a server accepts a connection but stops responding, the tool will hang indefinitely. Also, network drops during stream reading (shutil.copyfileobj) can raise OSError (e.g., ConnectionResetError) which URLError does not catch.
     • Fix: Provide a reasonable timeout (e.g., urlopen(url, timeout=30)) and explicitly catch OSError alongside URLError in _download_file().
  2. [Concurrency] Prevent Text File Busy Errors:
     • Issue: shutil.move in _install_binary() attempts to overwrite the existing binary. If the current binary is currently executing (by the user or another process), Linux will throw OSError: [Errno 26] Text file busy.
     • Fix: Remove the existing file before overwriting. Call target_path.unlink(missing_ok=True) before shutil.move(src_path, target_path).
  3. [Logic] Inexact Archive Matching:
     • Issue: _process_downloaded_file() uses rglob() to find the extracted binary. This can return directories sharing the same name or multiple files in unpredictable orders (e.g., if there's a docs/crane and bin/crane within the archive).
     • Fix: Filter matches to ensure they are strictly files ([m for m in matches if m.is_file()]) and add a sanity check (if len(matches) > 1) to fail explicitly or warn, rather than silently picking a random match.
  4. [Security] Archive Extraction Path Traversal:
     • Issue: shutil.unpack_archive extracts files exactly as they are in the archive, which leaves the tool theoretically vulnerable to path traversal (Zip Slip / Tar Slip) if the checksum were ever compromised. It will also likely be flagged in static analysis tools like Bandit.
     • Fix: If Python >= 3.12 is used, pass shutil.unpack_archive(..., filter='data') to natively enforce safe path extraction.
  5. [Robustness] Case-Insensitive Checksum Match:
     • Issue: hashlib.sha256().hexdigest() always returns lowercase characters. If an uppercase checksum is ever pasted into binary_dependencies.py in the future, validation will incorrectly fail.
     • Fix: In _verify_checksum(), safely compare using sha256.hexdigest() == expected_checksum.lower().

• Overall Impression: The logic is well-structured and properly abstracts dependency fetching into a clean set of modular functions. With a few defensive programming fixes to harden network volatility, file extraction paths, and prevent active execution races, this implementation will be highly robust.