(ci): use LFS repo for sight-data
What does this MR do?
This replaces the usage of sight-data with a new repository with LFS support. This speed-ups the download time because git LFS can parallelize batch downloads. This is particularly useful for the CI:
- Previously we used
curl
to retrieve the archive thentar
to decompress the archive, the whole process took around 170s. - The new
git clone
commands, this takes only 45s.
For some ci tasks like deploying issues and merge requests templates, we will use GIT_LFS_SKIP_SMUDGE=1
which allows to skip completely the download of binaries.
Important notice: For now I duplicated the data repository to not break other branches and app repositories. We have three options regarding this:
-
Keep it like this forever, but is sight-data-lfs a good repo name ? I don't think so. We could rename it before merging if we choose this option. -
Delete sight-data and rename sight-data-lfs into sight-data just before the merge. This will break other branches and app repositories. -
Erase sight-data history with sight-data-lfs just before the merge. This will break other branches and app repositories. -
Deprecate sight-data somehow, keep sight-data-lfs for some time and propose a new MR after some weeks/months when everyone has switched.
How to test it?
- Check if the CI pipeline succeeds.
- If you want to test locally, you will need git-lfs support on your computer;
Associated Issues/Merge Requests
- Issues
- Closes #444 (closed)