(io): Use LZ4 or ZSTD for data serialization instead of zip
Description
We currently use an internal minizip to store and compress our data. This is working quite good, but big application with big image can somewhat take some time to be written and read back. The Idea is to use more advanced data compression algorithm like ZSTD (from Facebook) or LZ4. The sped gain can be huge:
Proposal
Three approaches are possible:
-
- Rework fwZip to use libarchive which can let us choose which algorithm we want to use. Proper solution, but more work, as libarchive is not well prepared to manage archive with random read/write access.
-
- Update the internal minizip to the last version. We currently use a "compatibility" version. Minizip development is still ongoing, and recent release support ZSTD / LZ4 ! Maybe the easiest solution
-
- Same as 2, but with an external minizip conan package. A bit more work, but maybe cleaner.
Outcomes
Going from 36 MB/s to 780 MB/s for writing and from 445 MB/s to 4970 MB/s for reading should be a visible change.