Import testproj databases into workspace storage

[aeisenberg]

Commit-by-commit review is recommended.

If a folder that ends with .testproj is encountered, assume it is a
database created by a codeql test. When the user wants to import this
database, copy it into workspace storage. The database can be
re-imported, which first removes the old version before importing it
again.

Ask user if they want to re-import outdated testproj dbs

Before running a query now, do the following:

1. Check if the selected database is imported from a testproj
2. If so, check the last modified time of the codeql-datase.yml file
   of the imported database with that of its origin.
3. If the origin database has a file that is newer, assume that the
   database has been recreated since the last time it was imported.
4. If newer, then ask the user if they want to re-import before running
   the query.

Also, this change appends the (test) label to all test databases in
the database list.

To verify this change, do the following:

1. In the development workspace directory, create a codeql test:

   echo "import javascript
   select 1" | tee  test.ql
   echo "dependencies:
       codeql/javascript-all: \"*\"
   extractor: javascript
   tests: test"
   echo hucairz | tee qlpack.yml
   

2. Start the vscode dev workbench
3. In the vscode editor, go to the test tab (looks like a beaker)
4. Run the test. It should fail
5. Right click on the test.testproj directory and select "(Re-)Import test database"
6. Notice that the database is imported
7. Run the query if you like and you should see 1 as the result.
8. Change test to select 2
9. Re-run the test
10. Re-run the query
11. This time, you should get a popup asking if you want to re-import
12. Select yes
13. See the results are now 2.

Checklist

• CHANGELOG.md has been updated to incorporate all user visible changes made by this pull request.
• Issues have been created for any UI or other user-facing changes made by this pull request.
• [Maintainers only] If this pull request makes user-facing changes that require documentation changes, open a corresponding docs pull request in the github/codeql repo and add the ready-for-doc-review label there.

[koesie10]

I haven't looked at the full PR yet, but is there a reason we need to be using unzipper here? We do already have yauzl as a dependency and some helpers functions in unzip.ts and unzip-concurrently.ts.

[aeisenberg]

Thanks. I wasn't aware of those dependencies. I can switch.

[aeisenberg]

I'm not sure what's happening with the EPERM errors when renaming the directory on windows. It must be that the OS is hanging on to an open file handle somewhere, somehow, but I don't know where this would be happening. I could try to add a 1 second wait before renaming in case there's some sort of cleanup that needs to finish, but that feels very unsatisfactory.

[adityasharad]

Two possibilities come to mind:

• If you've just created the copy, then Defender might be scanning the newly-created files.
• Does an existing CodeQL process have a lock on the DB?

[aeisenberg]

The next thing to try would be to add a few second wait on windows.

[aeisenberg]

OK...that seems to have worked, but it's not a great solution.

[koesie10]

Everything seems to be working well for me, I just have some minor comments.

[koesie10]

LGTM. However, there are some confusing logs when you re-import a database before running a query. I see the following log lines:

Deleting database from filesystem.
Deleted '/Users/koesie10/Library/Application Support/Code/User/workspaceStorage/c3466bf33a6faffab7f413267a6ffe3f/GitHub.vscode-codeql/modeleditor-1'

However, there's nothing to indicate that the database has been imported again, so it'd be good to add that as well.

[koesie10]

I can't leave a comment on the exact line, but there's a message "Database unzipped and imported successfully.", which is somewhat confusing if you're importing a testproj database since nothing is being unzipped. Could this message be changed when you're using a testproj database?

[aeisenberg]

Thanks for such a detailed review, Koen!