fix: OSV database update consumes excessive resources and process is terminated; GitLab Advisory sync intermittently fails #5771

Description

I am unable to successfully update the vulnerability databases using cve-bin-tool. The issue has been reproducible for at least 5 days across multiple operating systems and installation methods.

The update process consistently reaches the OSV ingestion stage and then either:

• Consumes resources until the operating system terminates the process, or
• Fails to complete after entering the OSV processing phase.

In some runs, GitLab Advisory Database synchronization also fails.

This behavior has been reproduced on multiple systems, which suggests the issue is not specific to a single OS, Python environment, or machine configuration.

Environment

Operating Systems Tested

• Arch Linux
• Kali Linux
• Ubuntu 24.04 LTS (Virtual Machine)
• Windows 10

Hardware

• 12 CPU cores
• Approximately 6.5 GHz aggregate processing capability
• 32 GB RAM

Installation Methods Tested

• System-wide installation
• Python virtual environment installation

The issue occurs regardless of installation method.

Python versions

• Python 3.13
• Python 3.14.5

db-update.log

db-update-2.log

Version

Observed with:

CVE Binary Tool v3.4

Commands Tested

cve-bin-tool --update now

cve-bin-tool --update now --nvd-api-key NVD-API-KEY

cve-bin-tool --update now -d OSV

Expected Behavior

The update process should successfully download and ingest all supported vulnerability sources, including OSV, and complete database initialization/update.

Actual Behavior

Scenario 1: Standard Update

The update progresses through NVD, RedHat, and other sources before reaching OSV:

Getting NVD CVE data...
Getting GitLab Advisory Database CVEs...
Getting RedHat CVEs...
Getting PURL2CPE data...
Getting Open Source Vulnerability Database CVEs...

At this point resource consumption grows significantly and the process is eventually terminated by the operating system:

Killed

Scenario 2: GitLab Advisory Failure

Some runs also report:

Unable to fetch GitLab Advisory Database CVEs, skipping GAD.

before continuing to the OSV phase.

Relevant Log Excerpts

Example run:

INFO     Getting NVD CVE data...
INFO     Getting GitLab Advisory Database CVEs...
INFO     Getting RedHat CVEs...
INFO     Getting PURL2CPE data...
INFO     Getting Open Source Vulnerability Database CVEs...
Killed

Debug logging from Ubuntu 24.04 LTS shows:

DEBUG    Adding 48055 GAD entries
INFO     Adding 48505 RedHat CVE entries
INFO     Getting Open Source Vulnerability Database CVEs...

The update appears to enter OSV processing successfully but never completes.

Additional Observations

Disabling OSV allows the update process to complete:

cve-bin-tool --update now -d OSV

However, the resulting vulnerability dataset appears significantly smaller and insufficient for practical scanning purposes.

Because OSV contributes a substantial portion of the vulnerability coverage, disabling it is not a viable long-term workaround.

Reproducibility

Reproduced on:

• Arch Linux
• Kali Linux
• Ubuntu 24.04 LTS VM
• Windows 10

Reproduced with:

• Fresh installations
• Virtual environments
• Non-virtual-environment installations

The behavior has been consistent across all tested environments.

Questions

1. Is there a known issue with OSV ingestion in v3.4?
2. Has there been a recent upstream OSV schema or dataset change that could cause excessive memory consumption during processing?
3. Is there a recommended way to update OSV data incrementally rather than loading the entire dataset at once?
4. Is the GitLab Advisory Database synchronization failure related to the OSV issue, or are these separate problems?

Attachments

• Debug-level update log from Ubuntu 24.04 LTS VM
• Console output showing process termination during OSV update