Skip to content

(ci): use LFS repo for sight-data

Flavien BRIDAULT requested to merge 444-use-lfs-repo-for-sight-data into dev

What does this MR do?

This replaces the usage of sight-data with a new repository with LFS support. This speed-ups the download time because git LFS can parallelize batch downloads. This is particularly useful for the CI:

  • Previously we used curl to retrieve the archive then tar to decompress the archive, the whole process took around 170s.
  • The new git clone commands, this takes only 45s.

For some ci tasks like deploying issues and merge requests templates, we will use GIT_LFS_SKIP_SMUDGE=1 which allows to skip completely the download of binaries.

Important notice: For now I duplicated the data repository to not break other branches and app repositories. We have three options regarding this:

  • Keep it like this forever, but is sight-data-lfs a good repo name ? I don't think so. We could rename it before merging if we choose this option.
  • Delete sight-data and rename sight-data-lfs into sight-data just before the merge. This will break other branches and app repositories.
  • Erase sight-data history with sight-data-lfs just before the merge. This will break other branches and app repositories.
  • Deprecate sight-data somehow, keep sight-data-lfs for some time and propose a new MR after some weeks/months when everyone has switched.

How to test it?

  • Check if the CI pipeline succeeds.
  • If you want to test locally, you will need git-lfs support on your computer;

Associated Issues/Merge Requests

Edited by Flavien BRIDAULT

Merge request reports