Skip to content

Retry results download if connection times out#2440

Merged
aeisenberg merged 6 commits intogithub:mainfrom
JarLob:patch-1
May 26, 2023
Merged

Retry results download if connection times out#2440
aeisenberg merged 6 commits intogithub:mainfrom
JarLob:patch-1

Conversation

@JarLob
Copy link
Copy Markdown
Contributor

@JarLob JarLob commented May 23, 2023

Even with a good internet connection I sometimes have seen the error "failed to download results". When doing a MRVA on a 1000 top projects this results in that either you admit your run has incomplete results or have to rerun whole 1000 repositories query run, which is time consuming.
I have noticed that the url in the error message is still valid and I'm able to download the zip manually.
The change makes up to 3 retry attempts in a specific cases like timeout.

Checklist

  • CHANGELOG.md has been updated to incorporate all user visible changes made by this pull request.
  • Issues have been created for any UI or other user-facing changes made by this pull request.
  • [Maintainers only] If this pull request makes user-facing changes that require documentation changes, open a corresponding docs pull request in the github/codeql repo and add the ready-for-doc-review label there.

@JarLob JarLob requested a review from a team as a code owner May 23, 2023 08:59
Copy link
Copy Markdown
Contributor

@aeisenberg aeisenberg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your contribution. This looks reasonable to me. One small suggestion.

Copy link
Copy Markdown
Contributor

@aeisenberg aeisenberg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apologies. I just thought of this. We should log whenever we do a retry.

My code isn't exactly correct since there's no associated import statement.

@aeisenberg
Copy link
Copy Markdown
Contributor

Failing tests are flakes. I'll re-run.

Copy link
Copy Markdown
Member

@koesie10 koesie10 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this! Unfortunately, there is some extra work required to make this work when the connection drops when some parts have already been downloaded. When downloading, we only append to the ZIP file:

If we restart the download without truncating this file, a corrupt ZIP file will be written. Can you add await rm(zipFilePath, { force: true }); (with rm coming from fs-extra) in the download method of the VariantAnalysisResultsManager just before starting the fetch?

@JarLob
Copy link
Copy Markdown
Contributor Author

JarLob commented May 26, 2023

Good to be merged?

Copy link
Copy Markdown
Contributor

@aeisenberg aeisenberg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Thanks for reminding me to take another peak.

@aeisenberg aeisenberg merged commit 3b4f236 into github:main May 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants