chore(analyzer): Allow dry-running without an output directory - #12444
sschuberth wants to merge 8 commits into
Conversation
Pull request was converted to draft
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #12444 +/- ##
=========================================
Coverage 59.21% 59.21%
Complexity 1892 1892
=========================================
Files 366 366
Lines 13839 13839
Branches 1462 1462
=========================================
Hits 8195 8195
Misses 5114 5114
Partials 530 530
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
4767cf2 to
f527559
Compare
Pull request was converted to draft
f527559 to
5255188
Compare
a76f3c9 to
5928782
Compare
5928782 to
78d888b
Compare
| downloadPackage(pkg, packageDownloadDirs.getValue(pkg), failureMessages) | ||
|
|
||
| // For a dry run the download directory is not used anyway. | ||
| val dir = if (dryRun) Os.tempDirectory else packageDownloadDirs.getValue(pkg) |
There was a problem hiding this comment.
I'm getting this error for the downloader dry run:
Exception in thread "main" java.lang.IllegalArgumentException: The output directory '/tmp' must not contain any files yet.
at org.ossreviewtoolkit.downloader.Downloader.verifyOutputDirectory(Downloader.kt:60)
at org.ossreviewtoolkit.downloader.Downloader.download(Downloader.kt:73)
at org.ossreviewtoolkit.plugins.commands.downloader.DownloadCommand.downloadPackage(DownloadCommand.kt:431)
at org.ossreviewtoolkit.plugins.commands.downloader.DownloadCommand.access$downloadPackage(DownloadCommand.kt:109)
at org.ossreviewtoolkit.plugins.commands.downloader.DownloadCommand$downloadPackages$1$2$1$1.invokeSuspend(DownloadCommand.kt:416)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:34)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:124)
at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:89)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:798)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704)
Suppressed: java.lang.IllegalArgumentException: The output directory '/tmp' must not contain any files yet.
... 13 more
There was a problem hiding this comment.
Thanks for checking! I've addressed this in a new commit.
There was a problem hiding this comment.
There's now some weird behavior, for example when there's a download failure like:
VCS download failed for 'NPM::mime-db:1.33.0': DownloadException: Unhandled VCS URL https://github.com/jshttp/mime-db.git.
that triggers a cleanup for the tmp folder, which deletes all files in the folder, and I guess if I'm reading it correctly it tries to remove the tmp folder itself and for me it fails there because it doesn't have sufficient permissions.
So idk, should the outputDirectory.safeDeleteRecursively(baseDirectory = outputDirectory) calls maybe be ignored for the dry run or something?
Here's the exception:
Exception in thread "main" java.nio.file.FileSystemException: Failed to delete one or more files. See suppressed exceptions for details.
at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.deleteRecursively(PathRecursiveFunctions.kt:368)
at org.ossreviewtoolkit.utils.common.FileUtilsKt.safeDeleteRecursively(FileUtils.kt:126)
at org.ossreviewtoolkit.downloader.Downloader.handleVcsDownload(Downloader.kt:123)
at org.ossreviewtoolkit.downloader.Downloader.download(Downloader.kt:72)
at org.ossreviewtoolkit.plugins.commands.downloader.DownloadCommand.downloadPackage(DownloadCommand.kt:432)
at org.ossreviewtoolkit.plugins.commands.downloader.DownloadCommand.access$downloadPackage(DownloadCommand.kt:109)
at org.ossreviewtoolkit.plugins.commands.downloader.DownloadCommand$downloadPackages$1$2$1$1.invokeSuspend(DownloadCommand.kt:416)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:34)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:124)
at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:89)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:798)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704)
Suppressed: java.nio.file.FileSystemException: /tmp
at kotlin.io.path.ExceptionsCollector.collect(PathRecursiveFunctions.kt:398)
at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.handleEntry$PathsKt__PathRecursiveFunctionsKt(PathRecursiveFunctions.kt:596)
at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.deleteRecursivelyImpl$PathsKt__PathRecursiveFunctionsKt(PathRecursiveFunctions.kt:420)
at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.deleteRecursively(PathRecursiveFunctions.kt:365)
... 14 more
Caused by: java.nio.file.AccessDeniedException: tmp
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixSecureDirectoryStream.implDelete(UnixSecureDirectoryStream.java:179)
at java.base/sun.nio.fs.UnixSecureDirectoryStream.deleteDirectory(UnixSecureDirectoryStream.java:193)
at java.base/sun.nio.fs.UnixSecureDirectoryStream.deleteDirectory(UnixSecureDirectoryStream.java:42)
This is useful in scenarios where only a dry-run is needed, and the user does not want to be bothered with specifying a (dummy) output directory. Signed-off-by: Sebastian Schuberth <sebastian@doubleopen.io>
This fits better with the `check...()` Kotlin functions. Signed-off-by: Sebastian Schuberth <sebastian@doubleopen.io>
As this prevents the command from running at all, the code should be 1. This aligns all `UsageError`s in the `cli` module to use status code 1. Signed-off-by: Sebastian Schuberth <sebastian@doubleopen.io>
Do not use the number of download directories but the number of packages for progress information, which is more straight forward. Determine the download directories in `downloadAllPackages()` and return them, which makes a better API. Signed-off-by: Sebastian Schuberth <sebastian@doubleopen.io>
The "All" part is superfluous, esp. now that the `packages` are passed directly. Signed-off-by: Sebastian Schuberth <sebastian@doubleopen.io>
Signed-off-by: Sebastian Schuberth <sebastian@doubleopen.io>
This is useful in scenarios where only a dry-run is needed, and the user does not want to be bothered with specifying a (dummy) output directory. Signed-off-by: Sebastian Schuberth <sebastian@doubleopen.io>
78d888b to
8d5ebfb
Compare
Align with `checkOutputFiles()` and move the checking of the output directory to the CLI callers. This allows CLI user to control the behavior via the existing `forceOverwrite` option. It also makes the API less strict in case programmatic users do want to accumulate downloads in the same directory. Signed-off-by: Sebastian Schuberth <sebastian@doubleopen.io>
No description provided.