Segfault in NC_check_id #862
Description
Recent changes in HDF4 v4.3.1 seems to trigger a segmentation fault in some packages using HDF4.
Please refer to
- https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1114364
- https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1114329
The issue has been reported also to the rasterio and rioxarray packages:
- Segmentation fault on Debian / sid rasterio/rasterio#3427
- Segmentation fault on Debian / sid corteva/rioxarray#869
A (not so) minimal example to reproduce is to run the following unit tests from the test suite of rioxarray in Debian Sid:
python3 -m pytest -v test/integration/test_integration__io.py -k "test_open_group_load_attrs or test_open_group_filter"The analysis with valgrind reports the following errors:
$ valgrind --trace-children=yes --enable-debuginfod=yes --leak-check=full --show-leak-kinds=all --track-origins=yes python3-dbg -X tracemalloc -X faulthandler -m pytest -v test/integration/test_integration__io.py -k "test_open_group_load_attrs or test_open_group_filter"
==66172== Copyright (C) 2002-2024, and GNU GPL'd, by Julian Seward et al.
==66172== Using Valgrind-3.25.1 and LibVEX; rerun with -h for copyright info
==66172== Command: python3-dbg -X tracemalloc -X faulthandler -m pytest -v test/integration/test_integration__io.py -k test_open_group_load_attrs\ or\ test_open_group_filter
==66172==
/usr/lib/python3/dist-packages/pytest_asyncio/plugin.py:211: PytestDeprecationWarning: The configuration option "asyncio_default_fixture_loop_scope" is unset.
The event loop scope for asynchronous fixtures will default to the fixture caching scope. Future versions of pytest-asyncio will default the loop scope for asynchronous fixtures to function scope. Set the default fixture loop scope explicitly in order to avoid unexpected behavior in the future. Valid fixture loop scopes are: "function", "class", "module", "package", "session"
warnings.warn(PytestDeprecationWarning(_DEFAULT_FIXTURE_LOOP_SCOPE_UNSET))
============================= test session starts ==============================
platform linux -- Python 3.13.9, pytest-8.4.2, pluggy-1.6.0 -- /usr/bin/python3-dbg
cachedir: .pytest_cache
PyQt5 5.15.11 -- Qt runtime 5.15.17 -- Qt compiled 5.15.17
rootdir: /home/antonio/debian/git/python-rioxarray
configfile: pyproject.toml
plugins: anyio-4.11.0, qt-4.4.0, subtests-0.14.2, typeguard-4.4.4, zarr-3.1.3, localserver-0.9.0.post0, instafail-0.5.0, socket-0.7.0, asyncio-1.1.0, timeout-2.4.0, cov-5.0.0, xdist-3.8.0, lazy-fixtures-0.0.0, forked-1.6.0
asyncio: mode=Mode.STRICT, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collecting ... collected 95 items / 89 deselected / 6 selected
test/integration/test_integration__io.py::test_open_group_filter__missing[open_rasterio] PASSED [ 16%]
test/integration/test_integration__io.py::test_open_group_filter__missing[open_rasterio1] PASSED [ 33%]
test/integration/test_integration__io.py::test_open_group_filter[open_rasterio] PASSED [ 50%]
test/integration/test_integration__io.py::test_open_group_filter[open_rasterio1] PASSED [ 66%]
test/integration/test_integration__io.py::test_open_group_load_attrs[open_rasterio] ==66172== Invalid read of size 8
==66172== at 0x6049A60A: sd_NC_check_id (file.c:212)
==66172== by 0x6049EAF2: SDend (mfsd.c:375)
==66172== by 0x5C505D99: ??? (in /usr/lib/x86_64-linux-gnu/libgdal.so.37.3.11.4)
==66172== by 0x5C519934: ??? (in /usr/lib/x86_64-linux-gnu/libgdal.so.37.3.11.4)
==66172== by 0x5C519951: ??? (in /usr/lib/x86_64-linux-gnu/libgdal.so.37.3.11.4)
==66172== by 0x5CBDAA79: GDALClose (in /usr/lib/x86_64-linux-gnu/libgdal.so.37.3.11.4)
==66172== by 0x5B3067C8: ??? (in /usr/lib/python3/dist-packages/rasterio/_base.cpython-313-x86_64-linux-gnu.so)
==66172== by 0x5B30B050: ??? (in /usr/lib/python3/dist-packages/rasterio/_base.cpython-313-x86_64-linux-gnu.so)
==66172== by 0x4B9D50: _PyObject_VectorcallTstate (pycore_call.h:168)
==66172== by 0x4B9E20: PyObject_Vectorcall (call.c:327)
==66172== by 0x5E039D: _PyEval_EvalFrameDefault (generated_cases.c.h:1850)
==66172== by 0x5ECC8D: _PyEval_EvalFrame (pycore_ceval.h:119)
==66172== Address 0x8 is not stack'd, malloc'd or (recently) free'd
==66172==
Fatal Python error: Segmentation fault
Current thread 0x0000000004b9ea40 (most recent call first):
File "/usr/lib/python3/dist-packages/xarray/backends/file_manager.py", line 234 in close
File "/usr/lib/python3/dist-packages/xarray/core/common.py", line 1280 in close
File "/home/antonio/debian/git/python-rioxarray/rioxarray/_io.py", line 791 in _ds_close
File "/usr/lib/python3/dist-packages/xarray/core/common.py", line 1280 in close
File "/usr/lib/python3/dist-packages/xarray/core/common.py", line 1497 in __exit__
File "/home/antonio/debian/git/python-rioxarray/test/integration/test_integration__io.py", line 256 in test_open_group_load_attrs
File "/usr/lib/python3/dist-packages/_pytest/python.py", line 157 in pytest_pyfunc_call
File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 121 in _multicall
File "/usr/lib/python3/dist-packages/pluggy/_manager.py", line 120 in _hookexec
File "/usr/lib/python3/dist-packages/pluggy/_hooks.py", line 512 in __call__
File "/usr/lib/python3/dist-packages/_pytest/python.py", line 1671 in runtest
File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 178 in pytest_runtest_call
File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 121 in _multicall
File "/usr/lib/python3/dist-packages/pluggy/_manager.py", line 120 in _hookexec
File "/usr/lib/python3/dist-packages/pluggy/_hooks.py", line 512 in __call__
File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 246 in <lambda>
File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 344 in from_call
File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 245 in call_and_report
File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 136 in runtestprotocol
File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 117 in pytest_runtest_protocol
File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 121 in _multicall
File "/usr/lib/python3/dist-packages/pluggy/_manager.py", line 120 in _hookexec
File "/usr/lib/python3/dist-packages/pluggy/_hooks.py", line 512 in __call__
File "/usr/lib/python3/dist-packages/_pytest/main.py", line 367 in pytest_runtestloop
File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 121 in _multicall
File "/usr/lib/python3/dist-packages/pluggy/_manager.py", line 120 in _hookexec
File "/usr/lib/python3/dist-packages/pluggy/_hooks.py", line 512 in __call__
File "/usr/lib/python3/dist-packages/_pytest/main.py", line 343 in _main
File "/usr/lib/python3/dist-packages/_pytest/main.py", line 289 in wrap_session
File "/usr/lib/python3/dist-packages/_pytest/main.py", line 336 in pytest_cmdline_main
File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 121 in _multicall
File "/usr/lib/python3/dist-packages/pluggy/_manager.py", line 120 in _hookexec
File "/usr/lib/python3/dist-packages/pluggy/_hooks.py", line 512 in __call__
File "/usr/lib/python3/dist-packages/_pytest/config/__init__.py", line 175 in main
File "/usr/lib/python3/dist-packages/_pytest/config/__init__.py", line 201 in console_main
File "/usr/lib/python3/dist-packages/pytest/__main__.py", line 9 in <module>
File "/usr/lib/python3.13/runpy.py", line 88 in _run_code
File "/usr/lib/python3.13/runpy.py", line 198 in _run_module_as_main
Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, yaml._yaml, numcodecs.compat_ext, numcodecs.blosc, numcodecs.zstd, numcodecs.lz4, numcodecs._shuffle, numcodecs.jenkins, crc32c._crc32c, numcodecs.vlen, numcodecs.fletcher32, msgpack._cmsgpack, lz4._version, lz4.frame._frame, zstandard.backend_c, rasterio._err, rasterio._filepath, rasterio._version, rasterio._env, rasterio.crs, _brotli, charset_normalizer.md, _cffi_backend, rasterio._transform, rasterio._base, rasterio._features, rasterio._warp, rasterio._io, rasterio._vsiopener, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._pcg64, numpy.random._mt19937, numpy.random._generator, numpy.random._philox, numpy.random._sfc64, numpy.random.mtrand, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pandas._libs.ops, numexpr.interpreter, bottleneck.move, bottleneck.nonreduce, bottleneck.nonreduce_axis, bottleneck.reduce, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, cftime._cftime, pyproj._compat, pyproj._context, pyproj._network, pyproj._version, pyproj._geod, pyproj.list, pyproj._crs, pyproj.database, pyproj._transformer, pyproj._sync, psutil._psutil_linux, psutil._psutil_posix, markupsafe._speedups, PyQt5.QtCore, PyQt5.QtGui, PyQt5.QtWidgets, PyQt5.QtTest (total: 102)
==66172==
==66172== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==66172== at 0x4A3B95C: __pthread_kill_implementation (pthread_kill.c:44)
==66172== by 0x49E6CC1: raise (raise.c:26)
==66172== by 0x6F36B3: faulthandler_fatal_error (faulthandler.c:353)
==66172== by 0x49E6DEF: ??? (in /usr/lib/x86_64-linux-gnu/libc.so.6)
==66172== by 0x6049A609: sd_NC_check_id (file.c:212)
==66172==
==66172== HEAP SUMMARY:
==66172== in use at exit: 238,245,619 bytes in 2,619,419 blocks
==66172== total heap usage: 16,533,919 allocs, 13,914,500 frees, 2,456,124,828 bytes allocated
[CUT]
==66172==
==66172== LEAK SUMMARY:
==66172== definitely lost: 2,256 bytes in 19 blocks
==66172== indirectly lost: 0 bytes in 0 blocks
==66172== possibly lost: 178,171,189 bytes in 1,637,590 blocks
==66172== still reachable: 60,024,885 bytes in 981,803 blocks
==66172== of which reachable via heuristic:
==66172== length64 : 2,923,664 bytes in 27,496 blocks
==66172== multipleinheritance: 296 bytes in 3 blocks
==66172== suppressed: 47,289 bytes in 7 blocks
==66172==
==66172== For lists of detected and suppressed errors, rerun with: -s
==66172== ERROR SUMMARY: 3057 errors from 3057 contexts (suppressed: 6 from 6)
Our analysis shows that the issue could be related to the changes introduced in #826.
In particular the attached patch seems to solve the segmentation fault error both in the satpy and rioxarray test suite.
We are not sure, by the way, of the possible side effect of the change and, as minimum, we would expect that the leaks solved by #826 could be re-introduced.