Add ability to unzip artifacts zipped with zstd #527
Conversation
![sentry[bot]](https://avatars.githubusercontent.com/u/1396951?s=60&v=4){kind=small}
| compression_methods = self._detect_compression_methods(zip_path) | ||
| uses_zstd = COMPRESSION_ZSTD in compression_methods | ||
|
|
||
| if uses_zstd: | ||
| logger.debug("Detected Zstandard compression in zip file") | ||
| try: | ||
| import zipfile_zstd # noqa: F401 | ||
| except ImportError: | ||
| raise RuntimeError( | ||
| "Zstandard-compressed zip file detected, but zipfile-zstd package is not installed. " | ||
| "Install it with: pip install zipfile-zstd" |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
NicoHinderling
force-pushed
the
add-zstd-compression-support
branch
from
e1a5384 to
01ee208
Compare
| from typing import List | ||
|
|
||
| import zipfile_zstd # noqa: F401 - Import registers zstd handler with zipfile | ||
|
|
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
NicoHinderling
force-pushed
the
add-zstd-compression-support
branch
from
01ee208 to
7ffa8dc
Compare
|
|
||
| def test_extract_zstd_zip(self) -> None: | ||
| """Test that zstd-compressed zips can be extracted.""" | ||
| with tempfile.NamedTemporaryFile(suffix=".zip", delete=False) as temp_file: |
There was a problem hiding this comment.
nit: why delete=False when we always unlink the file afterwards in the finally block?
|
|
||
| # Create a zstd-compressed zip (compression method 93) | ||
| with zipfile.ZipFile(temp_path, "w") as zf: | ||
| zf.writestr("test.txt", "content", compress_type=93) |
There was a problem hiding this comment.
nit: can this use zipfile.ZIP_ZSTANDARD instead of magic number 93?
There was a problem hiding this comment.
"Unfortunately no - the standard library zipfile module doesn't define ZIP_ZSTANDARD. The zipfile_zstd package only adds the decompression handler, not the constant."
There was a problem hiding this comment.
perhaps this will exist in the python 3.14 world though..
src/launchpad/__init__.py
Outdated
| # Monkey patches - import these first to register handlers globally | ||
| import zipfile_zstd # noqa: F401 - Registers zstd compression support with zipfile module | ||
|
|
||
| __version__ = "0fg.0.1" |
There was a problem hiding this comment.
wasn't intentional, thanks for flagging
| sentry-sdk>=2.36.0 | ||
| sortedcontainers>=2.4.0 | ||
| typing-extensions>=4.15.0 | ||
| zipfile-zstd==0.0.4 |
There was a problem hiding this comment.
It looks like we get zstd for free in Python 3.14, not sure if you looked into that option: https://docs.python.org/3/library/compression.zstd.html#
There was a problem hiding this comment.
i tried upgrading us to 3.14, but that requires upgrading confluent-kafka to 2.12 which then conflicts with sentry-arroyo which has that package pinned to <2.10.0... so not possible right now
hen we're off of arroyo though.. lol
Partner PR to getsentry/sentry-cli#3038
Noticed uploading a certain customer's build took 5+ minutes and upon further inspection the zipping was the lions share of that time. If we take away the actual network transport of the upload:
With level 6 compression: 5 minutes -> ultimately 566MB
with level 1 compression: 2.5 min -> ultimately 658MB
with zstandard: 36s -> ultimately 533MB
This change will allow us to support both types of compression on the Sentry backend