Skip to content

chore(analyzer): Allow dry-running without an output directory - #12444

Open
sschuberth wants to merge 8 commits into
mainfrom
dry-run-without-outdir
Open

sschuberth wants to merge 8 commits into
mainfrom
dry-run-without-outdir

Conversation

@sschuberth

Copy link
Copy Markdown
Member

No description provided.

@sschuberth
sschuberth requested a review from a team as a code owner September 13, 2026 10:21
@sschuberth
sschuberth enabled auto-merge (rebase) September 13, 2026 10:21
@sschuberth
sschuberth marked this pull request as draft September 13, 2026 10:29
auto-merge was automatically disabled September 13, 2026 10:29

Pull request was converted to draft

@codecov

codecov Bot commented Sep 13, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 59.21%. Comparing base (fce14c4) to head (d4372f6).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff            @@
##               main   #12444   +/-   ##
=========================================
  Coverage     59.21%   59.21%           
  Complexity     1892     1892           
=========================================
  Files           366      366           
  Lines         13839    13839           
  Branches       1462     1462           
=========================================
  Hits           8195     8195           
  Misses         5114     5114           
  Partials        530      530           
Flag Coverage Δ
funTest-external-tools 16.13% <ø> (ø)
test-ubuntu-26.04 42.61% <ø> (ø)
test-windows-2025 42.59% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
@sschuberth
sschuberth force-pushed the dry-run-without-outdir branch from 4767cf2 to f527559 Compare September 13, 2026 10:55
@sschuberth
sschuberth marked this pull request as ready for review September 13, 2026 10:55
@sschuberth
sschuberth enabled auto-merge (rebase) September 13, 2026 10:56
Comment thread plugins/commands/analyzer/src/main/kotlin/AnalyzeCommand.kt
@sschuberth
sschuberth marked this pull request as draft September 13, 2026 17:44
auto-merge was automatically disabled September 13, 2026 17:44

Pull request was converted to draft

@sschuberth
sschuberth force-pushed the dry-run-without-outdir branch from f527559 to 5255188 Compare September 13, 2026 17:44
Comment thread plugins/commands/downloader/src/main/kotlin/DownloadCommand.kt Fixed
Comment thread plugins/commands/downloader/src/main/kotlin/DownloadCommand.kt Fixed
Comment thread plugins/commands/downloader/src/main/kotlin/DownloadCommand.kt Fixed
@sschuberth
sschuberth force-pushed the dry-run-without-outdir branch 2 times, most recently from a76f3c9 to 5928782 Compare September 13, 2026 19:05
@sschuberth
sschuberth marked this pull request as ready for review September 13, 2026 19:06
@sschuberth
sschuberth requested review from a team and fviernau September 13, 2026 19:06
@sschuberth
sschuberth force-pushed the dry-run-without-outdir branch from 5928782 to 78d888b Compare September 13, 2026 19:09
@fviernau
fviernau dismissed their stale review September 14, 2026 06:21

Addressed.

@sschuberth
sschuberth enabled auto-merge (rebase) September 14, 2026 12:18
downloadPackage(pkg, packageDownloadDirs.getValue(pkg), failureMessages)

// For a dry run the download directory is not used anyway.
val dir = if (dryRun) Os.tempDirectory else packageDownloadDirs.getValue(pkg)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm getting this error for the downloader dry run:

Exception in thread "main" java.lang.IllegalArgumentException: The output directory '/tmp' must not contain any files yet.
	at org.ossreviewtoolkit.downloader.Downloader.verifyOutputDirectory(Downloader.kt:60)
	at org.ossreviewtoolkit.downloader.Downloader.download(Downloader.kt:73)
	at org.ossreviewtoolkit.plugins.commands.downloader.DownloadCommand.downloadPackage(DownloadCommand.kt:431)
	at org.ossreviewtoolkit.plugins.commands.downloader.DownloadCommand.access$downloadPackage(DownloadCommand.kt:109)
	at org.ossreviewtoolkit.plugins.commands.downloader.DownloadCommand$downloadPackages$1$2$1$1.invokeSuspend(DownloadCommand.kt:416)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:34)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
	at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:124)
	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:89)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:798)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704)
	Suppressed: java.lang.IllegalArgumentException: The output directory '/tmp' must not contain any files yet.
		... 13 more

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for checking! I've addressed this in a new commit.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's now some weird behavior, for example when there's a download failure like:

VCS download failed for 'NPM::mime-db:1.33.0': DownloadException: Unhandled VCS URL https://github.com/jshttp/mime-db.git.

that triggers a cleanup for the tmp folder, which deletes all files in the folder, and I guess if I'm reading it correctly it tries to remove the tmp folder itself and for me it fails there because it doesn't have sufficient permissions.

So idk, should the outputDirectory.safeDeleteRecursively(baseDirectory = outputDirectory) calls maybe be ignored for the dry run or something?

Here's the exception:

Exception in thread "main" java.nio.file.FileSystemException: Failed to delete one or more files. See suppressed exceptions for details.
	at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.deleteRecursively(PathRecursiveFunctions.kt:368)
	at org.ossreviewtoolkit.utils.common.FileUtilsKt.safeDeleteRecursively(FileUtils.kt:126)
	at org.ossreviewtoolkit.downloader.Downloader.handleVcsDownload(Downloader.kt:123)
	at org.ossreviewtoolkit.downloader.Downloader.download(Downloader.kt:72)
	at org.ossreviewtoolkit.plugins.commands.downloader.DownloadCommand.downloadPackage(DownloadCommand.kt:432)
	at org.ossreviewtoolkit.plugins.commands.downloader.DownloadCommand.access$downloadPackage(DownloadCommand.kt:109)
	at org.ossreviewtoolkit.plugins.commands.downloader.DownloadCommand$downloadPackages$1$2$1$1.invokeSuspend(DownloadCommand.kt:416)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:34)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
	at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:124)
	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:89)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:798)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704)
	Suppressed: java.nio.file.FileSystemException: /tmp
		at kotlin.io.path.ExceptionsCollector.collect(PathRecursiveFunctions.kt:398)
		at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.handleEntry$PathsKt__PathRecursiveFunctionsKt(PathRecursiveFunctions.kt:596)
		at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.deleteRecursivelyImpl$PathsKt__PathRecursiveFunctionsKt(PathRecursiveFunctions.kt:420)
		at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.deleteRecursively(PathRecursiveFunctions.kt:365)
		... 14 more
	Caused by: java.nio.file.AccessDeniedException: tmp
		at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
		at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
		at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
		at java.base/sun.nio.fs.UnixSecureDirectoryStream.implDelete(UnixSecureDirectoryStream.java:179)
		at java.base/sun.nio.fs.UnixSecureDirectoryStream.deleteDirectory(UnixSecureDirectoryStream.java:193)
		at java.base/sun.nio.fs.UnixSecureDirectoryStream.deleteDirectory(UnixSecureDirectoryStream.java:42)
This is useful in scenarios where only a dry-run is needed, and the user
does not want to be bothered with specifying a (dummy) output directory.

Signed-off-by: Sebastian Schuberth <sebastian@doubleopen.io>
This fits better with the `check...()` Kotlin functions.

Signed-off-by: Sebastian Schuberth <sebastian@doubleopen.io>
As this prevents the command from running at all, the code should be 1.

This aligns all `UsageError`s in the `cli` module to use status code 1.

Signed-off-by: Sebastian Schuberth <sebastian@doubleopen.io>
Do not use the number of download directories but the number of
packages for progress information, which is more straight forward.
Determine the download directories in `downloadAllPackages()` and return
them, which makes a better API.

Signed-off-by: Sebastian Schuberth <sebastian@doubleopen.io>
The "All" part is superfluous, esp. now that the `packages` are passed
directly.

Signed-off-by: Sebastian Schuberth <sebastian@doubleopen.io>
Signed-off-by: Sebastian Schuberth <sebastian@doubleopen.io>
This is useful in scenarios where only a dry-run is needed, and the user
does not want to be bothered with specifying a (dummy) output directory.

Signed-off-by: Sebastian Schuberth <sebastian@doubleopen.io>
@sschuberth
sschuberth force-pushed the dry-run-without-outdir branch from 78d888b to 8d5ebfb Compare September 16, 2026 08:17
Align with `checkOutputFiles()` and move the checking of the output
directory to the CLI callers. This allows CLI user to control the
behavior via the existing `forceOverwrite` option.

It also makes the API less strict in case programmatic users do want to
accumulate downloads in the same directory.

Signed-off-by: Sebastian Schuberth <sebastian@doubleopen.io>
@sschuberth
sschuberth requested a review from lamppu September 16, 2026 09:21

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

4 participants