Skip to content

Comments

Add SIMD with dynamic dispatch support for time conversions (mostly AV512 only)#47

Merged
jeandet merged 1 commit intoSciQLop:mainfrom
jeandet:simd_time_conv
Feb 10, 2026
Merged

Add SIMD with dynamic dispatch support for time conversions (mostly AV512 only)#47
jeandet merged 1 commit intoSciQLop:mainfrom
jeandet:simd_time_conv

Conversation

@jeandet
Copy link
Member

@jeandet jeandet commented Jan 2, 2026

This pull request introduces several improvements and additions across the codebase, focusing on enhanced CDF type handling, improved time type representations, benchmarking infrastructure, and utility enhancements. The most significant changes are the addition of CDF type concepts and dispatch utilities, modernization of time type structures, new benchmarking capabilities, and improved debug and helper utilities.

CDF Type Handling and Concepts

  • Added new consteval functions to classify CDF types (is_cdf_time_type, is_cdf_string_type, is_cdf_integer_type, is_cdf_floating_point_type) and a cdf_type_dispatch function for type-based dispatching in cdf-enums.hpp. Also introduced concepts for time types and collections, improving type safety and generic programming. [1] [2] [3]

Time Type Representation

  • Updated tt2000_t, epoch, and epoch16 structures to use more descriptive member names (nseconds, mseconds, seconds, picoseconds), and added literal operator and equality operators for these types. This clarifies their semantics and usage throughout the codebase. [1] [2]

Benchmarking Infrastructure

  • Added a new benchmark suite for chrono operations (benchmarks/chrono/main.cpp) and integrated Google Benchmark support in the build system (benchmarks/meson.build). This enables performance testing of time conversion and leap second calculations. [1] [2]

Debug and Helper Utilities

  • Introduced exception_message utility using std::source_location and fmt for improved error reporting in cdf-debug.hpp. Also expanded helper utilities with new type-checking and string functions in cdf-helpers.hpp. [1] [2]

Miscellaneous Improvements

  • Modernized and cleaned up several utility functions and includes, such as removing unused headers, updating operator overloads, and improving buffer and record utilities for clarity and consistency. [1] [2] [3] [4] [5] [6]

These changes collectively improve the maintainability, performance, and usability of the codebase.

@jeandet jeandet force-pushed the simd_time_conv branch 9 times, most recently from 17dafb5 to 5a5a3a0 Compare January 7, 2026 12:19
@jeandet jeandet added the enhancement New feature or request label Jan 7, 2026
@jeandet jeandet force-pushed the simd_time_conv branch 15 times, most recently from e1d032e to 9fa2779 Compare January 13, 2026 22:53
@jeandet jeandet force-pushed the simd_time_conv branch 5 times, most recently from 0a50d31 to 7f0233e Compare January 18, 2026 16:40
@jeandet jeandet force-pushed the simd_time_conv branch 3 times, most recently from cd48cd8 to f809dc3 Compare January 22, 2026 13:42
@jeandet jeandet force-pushed the simd_time_conv branch 6 times, most recently from b6b5d64 to 3d18a01 Compare February 4, 2026 16:14
@jeandet jeandet requested a review from Copilot February 8, 2026 16:05
@jeandet jeandet marked this pull request as ready for review February 8, 2026 16:10
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR modernizes the chrono/time conversion stack (C++20, refactored time types, SIMD-enabled to_ns_from_1970 with xsimd dynamic dispatch) and updates the Python bindings to infer/validate collection types/shapes in C++ with expanded test/benchmark coverage.

Changes:

  • Add x86 SIMD dispatch path (xsimd) for time conversions and integrate it into the chrono entry points (plus new benchmarks).
  • Refactor Python wrapper value-setting/type inference utilities and add new Python/C++ tests for collection analysis and chrono conversions.
  • Restructure Meson build for tests/benchmarks and update dependency wraps (Catch2, google-benchmark, new xsimd wrap).

Reviewed changes

Copilot reviewed 53 out of 56 changed files in this pull request and generated 14 comments.

Show a summary per file
File Description
tests/zstd_compression/main.cpp Remove Catch2 include fallback logic.
tests/zlib_compression/main.cpp Remove Catch2 include fallback logic.
tests/simple_save/main.cpp Remove Catch2 include fallback logic.
tests/simple_open/main.cpp Remove Catch2 include fallback logic; adjust string literal.
tests/rle_compression/main.cpp Remove Catch2 include fallback logic.
tests/records_saving/main.cpp Remove Catch2 include fallback logic.
tests/records_loading/main.cpp Remove Catch2 include fallback logic.
tests/python_wrapper_cpp/main.cpp New embedded-module C++ tests for analyze_collection.
tests/python_variable_set_values/test.py Adjust datetime64[ms] variable naming in test.
tests/python_saving/test.py Print wrapper version/path during tests.
tests/python_loading/test.py Improve assertions; special-case tt2000 pre-1970 behavior.
tests/python_chrono/test.py New Python chrono conversion tests.
tests/nomap/main.cpp Remove Catch2 include fallback logic.
tests/meson.build New Meson test definitions (C++ + Python + embedded pybind test).
tests/majority/main.cpp Remove Catch2 include fallback logic.
tests/libdeflate_compression/main.cpp Remove Catch2 include fallback logic.
tests/full_corpus/test.py New remote corpus loading tests (single + multithread).
tests/endianness/main.cpp Remove Catch2 include fallback logic.
tests/chrono/test_values.hpp Make test vector inline constexpr.
tests/chrono/main.cpp Add leap-second + vectorized-vs-scalar conversion tests; update renamed fields.
subprojects/xsimd.wrap Add xsimd Meson wrap (git).
subprojects/google-benchmark.wrap Update wrapdb patch version.
subprojects/catch2.wrap Update Catch2 version.
src/arch/x86/chrono_arch.cpp Explicit template instantiations for arch-specific SIMD conversion kernels.
src/arch/x86/chrono.cpp Dynamic dispatch setup and exported vectorized conversion entry points.
simd/meson.build Build per-arch static libs and dispatch list; disable SIMD on non-x86_64.
scripts/chrono_profiling.py New perf-driven profiling script for datetime64 conversions.
scripts/CDF.hexpat Enhance hex pattern types/offset handling and debug output.
pyproject.toml Require Python>=3.9; switch meson-python to C++20.
pycdfpp/variable.hpp Major refactor: C++-side set_values, collection analysis integration, warnings, new helpers.
pycdfpp/meson.build Minor cleanup.
pycdfpp/data_types.hpp New: collection/type inference + compatibility helpers.
pycdfpp/collections.hpp New: collection traversal/transform utilities + fast allocation helpers.
pycdfpp/chrono.hpp Major refactor: faster conversions, list/tuple/array support, datetime64 bridging.
pycdfpp/buffers.hpp Deleted (logic moved into collections.hpp).
pycdfpp/init.py Simplify Python Variable.set_values wrapper (delegates to C++); doc updates.
meson.build Switch project to C++20, integrate simd/, restructure tests/benchmarks, add summaries.
include/cdfpp/vectorized/cdf-chrono.hpp New vectorized chrono conversion API declarations.
include/cdfpp/vectorized/cdf-chrono-impl.hpp New SIMD implementations for epoch/epoch16/tt2000 conversions.
include/cdfpp/variable.hpp Add set_data(pair) overload; richer shape mismatch exception.
include/cdfpp/no_init_vector.hpp Adjust allocator overloads; add MADV_WILLNEED.
include/cdfpp/chrono/cdf-leap-seconds.h Minor formatting change.
include/cdfpp/chrono/cdf-chrono.hpp Integrate vectorized + threaded conversion path; refactor chrono API.
include/cdfpp/chrono/cdf-chrono-impl.hpp New scalar leap-second + conversion impls.
include/cdfpp/cdf-repr.hpp Refactor streaming via cdf_type_dipatch; switch some time formatting to fmt.
include/cdfpp/cdf-io/saving/records-saving.hpp Remove unused includes; adjust constexpr signatures.
include/cdfpp/cdf-io/loading/buffers.hpp Remove redundant inline qualifiers.
include/cdfpp/cdf-helpers.hpp Add is_in, rt_is_in, is_any_of, contains.
include/cdfpp/cdf-enums.hpp Add type classification helpers, concepts, literal op, and cdf_type_dipatch.
include/cdfpp/cdf-debug.hpp Add exception_message(...) helper using std::source_location.
benchmarks/meson.build Add benchmarks meson subdir.
benchmarks/chrono/main.cpp New chrono benchmarks (leap seconds + conversions).
.readthedocs.yaml Install cmake for docs build.
.github/workflows/CI.yml Update corpus test filename; macOS brew cleanup step.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines 396 to 401
if (PyList_Check(obj) or PyTuple_Check(obj))
{
auto inner_min_storage
= _min_storage(py::reinterpret_borrow<py::list>(const_cast<PyObject*>(obj)));
min_storage = min_storage > inner_min_storage ? min_storage : inner_min_storage;
}
Copy link

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_min_storage(const py_list_or_py_tuple auto&) treats both lists and tuples the same but always reinterpret_borrow<py::list> for nested collections. If obj is a tuple this creates a py::list handle to a tuple object (type mismatch/UB). Branch on PyList_Check vs PyTuple_Check and borrow the correct pybind11 type (or accept py::handle and dispatch safely).

Suggested change
if (PyList_Check(obj) or PyTuple_Check(obj))
{
auto inner_min_storage
= _min_storage(py::reinterpret_borrow<py::list>(const_cast<PyObject*>(obj)));
min_storage = min_storage > inner_min_storage ? min_storage : inner_min_storage;
}
if (PyList_Check(obj))
{
auto inner_min_storage = _min_storage(
py::reinterpret_borrow<py::list>(const_cast<PyObject*>(obj)));
min_storage = min_storage > inner_min_storage ? min_storage : inner_min_storage;
}
else if (PyTuple_Check(obj))
{
auto inner_min_storage = _min_storage(
py::reinterpret_borrow<py::tuple>(const_cast<PyObject*>(obj)));
min_storage = min_storage > inner_min_storage ? min_storage : inner_min_storage;
}

Copilot uses AI. Check for mistakes.
Comment on lines 371 to 376
else if (uval > std::numeric_limits<uint32_t>::max())
{
min_storage.size_in_bytes
= min_storage.size_in_bytes > 4 ? min_storage.size_in_bytes : 4;
}
else if (uval > std::numeric_limits<uint16_t>::max())
Copy link

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In _min_storage(PyObject*), the unsigned-width logic has a duplicated condition: else if (uval > uint32_t::max()) repeats the previous if, so the size_in_bytes = 4 path is unreachable. This will mis-infer storage for values that need 32 bits. Adjust the comparisons (e.g., check > uint16_t::max() then > uint8_t::max()/> uint32_t::max() as appropriate).

Suggested change
else if (uval > std::numeric_limits<uint32_t>::max())
{
min_storage.size_in_bytes
= min_storage.size_in_bytes > 4 ? min_storage.size_in_bytes : 4;
}
else if (uval > std::numeric_limits<uint16_t>::max())
else if (uval > std::numeric_limits<uint16_t>::max())
{
min_storage.size_in_bytes
= min_storage.size_in_bytes > 4 ? min_storage.size_in_bytes : 4;
}
else if (uval > std::numeric_limits<uint8_t>::max())

Copilot uses AI. Check for mistakes.
Comment on lines 415 to 416
[[nodiscard]] inline auto cdf_type_dipatch(CDF_Types cdf_type, auto&& f, auto&&... args)
{
Copy link

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Public helper is named cdf_type_dipatch (typo). Since this is part of the API surface and is referenced from multiple headers, consider renaming to cdf_type_dispatch (and optionally providing a deprecated alias) to avoid baking in a misspelling.

Copilot uses AI. Check for mistakes.
Comment on lines 182 to 186
auto [cdf_type, shape] = run_test("['√abc', '√def', '√ghi']");
THEN("The inferred type is CDF_CHAR and shape is [3, 4]")
{
REQUIRE(cdf_type == CDF_Types::CDF_UCHAR);
REQUIRE(shape == std::vector<std::size_t> { 3, 6 });
Copy link

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The THEN(...) description says the shape is [3, 4], but the assertion checks {3, 6}. This makes the test output misleading when it fails; update the description to match the actual expected UTF-8 byte length behavior.

Copilot uses AI. Check for mistakes.
Comment on lines 87 to 91
std::size_t end = chunk_size;
for (std::size_t i = 0; i < threads_count; ++i)
{
threads.emplace_back([input, start, end, output, &function]()
{ function(input.subspan(start, end), output + start); });
Copy link

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the threaded path, std::span::subspan(offset, count) is called as subspan(start, end), but end here is an absolute index, not a count. For iterations after the first this will create a span that overruns the input (and output indexing will also mismatch), leading to out-of-bounds access/UB. Pass end - start as the second argument (or compute a count per chunk) and ensure the last chunk size is correct.

Suggested change
std::size_t end = chunk_size;
for (std::size_t i = 0; i < threads_count; ++i)
{
threads.emplace_back([input, start, end, output, &function]()
{ function(input.subspan(start, end), output + start); });
std::size_t end = std::min(chunk_size, count);
for (std::size_t i = 0; i < threads_count; ++i)
{
threads.emplace_back([input, start, end, output, &function]()
{ function(input.subspan(start, end - start), output + start); });

Copilot uses AI. Check for mistakes.
Comment on lines 177 to 178
GIVEN("A list of strings wuth unicode characters")
{
Copy link

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo in test description: "wuth" -> "with".

Copilot uses AI. Check for mistakes.
Comment on lines 285 to 289
auto [cdf_type, shape] = run_test("[0, 10000000000, 20000000000, 0x10000000000]");
THEN("The inferred type is CDF_UINT8 and shape is [4]")
{
REQUIRE(cdf_type == CDF_Types::CDF_INT8);
REQUIRE(shape == std::vector<std::size_t> { 4 });
Copy link

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The THEN(...) description references CDF_UINT8, but the assertion expects CDF_INT8 (and CDF has no uint64 type). Update the description to avoid confusion.

Copilot uses AI. Check for mistakes.
Comment on lines +569 to +583
.def(
"_set_values",
[](Variable& var, const py::array& values, std::optional<CDF_Types> data_type,
bool force)
{
if (not force and var.type() != CDF_Types::CDF_NONE)
{
py::warnings::warn(
"Overriding existing variable values without force=True is deprecated and "
"will raise an exception in future versions.",
PyExc_DeprecationWarning, 3);
}
set_values(var, ensure_utf8(values), data_type ? data_type : var.type());
},
py::arg("values").noconvert(), py::arg("data_type") = std::nullopt,
Copy link

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The _set_values bindings accept a force argument but do not forward it to set_values(...). As a result, force=True from Python has no effect (and may still raise on type/shape mismatch). Forward force to the underlying set_values overloads.

Copilot uses AI. Check for mistakes.
import sys, os
sys.path.append(".")
try:
import pycdfpp
Copy link

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Import of 'pycdfpp' is not used.

Copilot uses AI. Check for mistakes.
"Please run this script from the build directory where pycdfpp is built."
) from e

import numpy as np
Copy link

Copilot AI Feb 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Import of 'np' is not used.

Suggested change
import numpy as np

Copilot uses AI. Check for mistakes.
@jeandet jeandet force-pushed the simd_time_conv branch 3 times, most recently from ceb090f to f4dbfad Compare February 10, 2026 10:43
@jeandet jeandet requested a review from Copilot February 10, 2026 14:20
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 54 out of 58 changed files in this pull request and generated 7 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

meson.build Outdated

cdfpp_dep = declare_dependency(include_directories: cdfpp_dep_inc,
dependencies: [zlib_dep, hedley_dep, fmt_dep, zstd_dep],
dependencies: [zlib_dep, hedley_dep, fmt_dep, zstd_dep, simd_deps],
Copy link

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cdfpp_dep passes simd_deps as a single list element inside dependencies: [...], which creates a nested array. Meson expects a flat array of dependency objects; this will fail when simd_deps is non-empty. Concatenate instead (e.g., dependencies: [..] + simd_deps).

Suggested change
dependencies: [zlib_dep, hedley_dep, fmt_dep, zstd_dep, simd_deps],
dependencies: [zlib_dep, hedley_dep, fmt_dep, zstd_dep] + simd_deps,

Copilot uses AI. Check for mistakes.
Comment on lines +29 to +32
python_wrapper_cpp = executable('python_wrapper_cpp','python_wrapper_cpp/main.cpp',
dependencies:[pybind11_dep, python3.dependency(embed:true), catch_dep, cdfpp_dep, fmt_dep],
install: false
)
Copy link

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

catch_dep is catch2-with-main, but tests/python_wrapper_cpp/main.cpp defines its own Catch2 runner (CATCH_CONFIG_RUNNER) and main(). Linking against Catch2WithMain will typically cause a duplicate main symbol at link time. Use the Catch2 dependency without main for this executable (or remove the custom runner).

Copilot uses AI. Check for mistakes.
Comment on lines +575 to +620
.def(
"_set_values",
[](Variable& var, const py::array& values, std::optional<CDF_Types> data_type,
bool force)
{
if (not force and var.type() != CDF_Types::CDF_NONE)
{
py::warnings::warn(
"Overriding existing variable values without force=True is deprecated and "
"will raise an exception in future versions.",
PyExc_DeprecationWarning, 3);
}
set_values(var, ensure_utf8(values), data_type ? data_type : var.type());
},
py::arg("values").noconvert(), py::arg("data_type") = std::nullopt,
py::arg("force") = false)
.def(
"_set_values",
[](Variable& var, const py::list& values, std::optional<CDF_Types> data_type,
bool force)
{
if (not force and var.type() != CDF_Types::CDF_NONE)
{
py::warnings::warn(
"Overriding existing variable values without force=True is deprecated and "
"will raise an exception in future versions.",
PyExc_DeprecationWarning, 3);
}
set_values(var, values, data_type ? data_type : var.type());
},
py::arg("values").noconvert(), py::arg("data_type") = std::nullopt,
py::arg("force") = false)
.def(
"_set_values",
[](Variable& var, const py::tuple& values, std::optional<CDF_Types> data_type,
bool force)
{
if (not force and var.type() != CDF_Types::CDF_NONE)
{
py::warnings::warn(
"Overriding existing variable values without force=True is deprecated and "
"will raise an exception in future versions.",
PyExc_DeprecationWarning, 3);
}
set_values(var, values, data_type ? data_type : var.type());
},
Copy link

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The _set_values(..., force) overloads accept a force parameter but do not pass it through to set_values(...) (they always take the default force=false). This makes the Python-level force=True ineffective for list/tuple/ndarray inputs. Pass force into set_values in each overload.

Copilot uses AI. Check for mistakes.
else:
var = self._add_variable(
name=name, is_nrv=is_nrv, compression=compression)
var.set_values(values, data_type)
Copy link

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_add_variable_wrapper no longer handles the case values is None and data_type is not None (it always calls _add_variable(name, ...) without setting the type). Existing tests use cdf.add_variable(..., data_type=...) to create an empty typed variable; this wrapper will leave the variable as CDF_NONE and likely break saving/type expectations. Restore the ability to create an empty variable with an explicit type (e.g., call into the C++ overload that accepts values+data_type, or call var.set_values([], data_type) in a way that preserves the requested type).

Suggested change
var.set_values(values, data_type)
var.set_values(values, data_type)
elif data_type is not None:
# Allow creating an empty variable with an explicit data type
var.set_values([], data_type)

Copilot uses AI. Check for mistakes.
Comment on lines 279 to 281
if var == 'tt2000':
# tt2000 conv is volontarily broken for dates before 1970-01-01
values = values[5:]
Copy link

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo in comment: volontarilyvoluntarily.

Copilot uses AI. Check for mistakes.
Comment on lines +93 to +150
template <typename T>
auto to_numerical(const PyObject* o)
{
if constexpr (std::is_same_v<double, T>)
{
if (PyFloat_Check(const_cast<PyObject*>(o)))
{
return static_cast<T>(PyFloat_AS_DOUBLE(const_cast<PyObject*>(o)));
}
else if (PyLong_Check(const_cast<PyObject*>(o)))
{
return static_cast<T>(PyLong_AsDouble(const_cast<PyObject*>(o)));
}
else
{
throw std::invalid_argument { "Incompatible python and cdf types" };
}
}
if constexpr (helpers::is_any_of_v<T, uint8_t, uint16_t, uint32_t, uint64_t>)
{
if (PyLong_Check(const_cast<PyObject*>(o)))
{
return static_cast<T>(PyLong_AsUnsignedLongLong(const_cast<PyObject*>(o)));
}
else if (PyFloat_Check(const_cast<PyObject*>(o)))
{
return static_cast<T>(PyFloat_AsDouble(const_cast<PyObject*>(o)));
}
else
{
throw std::invalid_argument { "Incompatible python and cdf types" };
}
}
else if constexpr (helpers::is_any_of_v<T, int8_t, int16_t, int32_t, int64_t>)
{
if (PyLong_Check(const_cast<PyObject*>(o)))
{
return static_cast<T>(PyLong_AsLongLong(const_cast<PyObject*>(o)));
}
else if (PyFloat_Check(const_cast<PyObject*>(o)))
{
return static_cast<T>(PyFloat_AS_DOUBLE(const_cast<PyObject*>(o)));
}
else
{
throw std::invalid_argument { "Incompatible python and cdf types" };
}
}
else if constexpr (std::is_same_v<float, T>)
{
if (PyFloat_Check(const_cast<PyObject*>(o)))
{
return static_cast<T>(PyFloat_AS_DOUBLE(const_cast<PyObject*>(o)));
}
else if (PyLong_Check(const_cast<PyObject*>(o)))
{
return static_cast<T>(PyLong_AsDouble(const_cast<PyObject*>(o)));
}
Copy link

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to_numerical uses Python C-API conversion helpers (e.g., PyLong_AsLongLong, PyLong_AsUnsignedLongLong, PyLong_AsDouble) but never checks PyErr_Occurred() for overflow/conversion errors. For out-of-range Python ints this can leave a pending Python exception and produce incorrect values. Check for errors after conversion and translate them into a C++ exception (or propagate the Python error via pybind11::error_already_set).

Copilot uses AI. Check for mistakes.
Comment on lines 54 to 55
[[nodiscard]] constexpr auto exception_message(const auto& msg,
const std::source_location& location = std::source_location::current()) noexcept
Copy link

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exception_message(...) is declared noexcept but calls fmt::format, which can throw (e.g., allocation/format errors). If it throws, the program will std::terminate, and since this helper is used while constructing exceptions it can mask the real error. Drop noexcept (and constexpr, since fmt::format isn’t constexpr) or change implementation to be truly non-throwing.

Suggested change
[[nodiscard]] constexpr auto exception_message(const auto& msg,
const std::source_location& location = std::source_location::current()) noexcept
[[nodiscard]] auto exception_message(const auto& msg,
const std::source_location& location = std::source_location::current())

Copilot uses AI. Check for mistakes.
@jeandet jeandet force-pushed the simd_time_conv branch 4 times, most recently from f27c7d8 to fb72155 Compare February 10, 2026 17:10
@codecov
Copy link

codecov bot commented Feb 10, 2026

Codecov Report

❌ Patch coverage is 87.11832% with 135 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.23%. Comparing base (63129f6) to head (9659380).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
pycdfpp/chrono.hpp 86.93% 29 Missing ⚠️
include/cdfpp/vectorized/cdf-chrono-impl.hpp 61.97% 27 Missing ⚠️
pycdfpp/variable.hpp 78.37% 24 Missing ⚠️
pycdfpp/collections.hpp 88.09% 20 Missing ⚠️
include/cdfpp/cdf-enums.hpp 82.14% 10 Missing ⚠️
pycdfpp/data_types.hpp 91.93% 10 Missing ⚠️
include/cdfpp/chrono/cdf-chrono.hpp 77.14% 8 Missing ⚠️
include/cdfpp/cdf-debug.hpp 0.00% 3 Missing ⚠️
include/cdfpp/chrono/cdf-chrono-impl.hpp 95.12% 2 Missing ⚠️
include/cdfpp/variable.hpp 60.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #47      +/-   ##
==========================================
+ Coverage   88.22%   88.23%   +0.01%     
==========================================
  Files          50       56       +6     
  Lines        2472     3197     +725     
==========================================
+ Hits         2181     2821     +640     
- Misses        291      376      +85     
Flag Coverage Δ
unittests 88.23% <87.11%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jeandet jeandet force-pushed the simd_time_conv branch 2 times, most recently from 06edf3f to b8cd50e Compare February 10, 2026 20:33
This commit introduces significant performance improvements to time processing and completely refactors how data is exchanged between Python and C++.

Key Changes:

1. Vectorized Time Conversions:

    Integrated xsimd to implement SIMD-accelerated conversion for TT2000 and Epoch types.

    Implemented architecture dispatching (AVX512, AVX2, SSE2) in meson.build and source files.

    Refactored leap second handling to use branchless lookups for improved CPU pipelining.

    Added benchmarks/chrono to track throughput improvements.

2. Python Bindings (pycdfpp):

    Implemented data_types.hpp: A C++ engine that recursively inspects Python objects (scalars, nested lists, NumPy arrays) to infer CDF data types and shapes automatically.

    Implemented collections.hpp: Efficient iterators and transformers for bridging Python sequences with C++ spans.

    Enhanced datetime support: Native handling of NumPy datetime64[ns] and Python datetime objects, mapping them to CDF_TIME_TT2000/EPOCH.

    Refactored Variable::set_values: Logic moved from Python to C++ to ensure type safety and shape consistency before memory allocation.

3. Core & Infrastructure:

    Bumped project requirement to C++20.

    Optimized no_init_vector allocation using madvise to tell the kernel that the pages will be used soon

    Updated test suite to use Catch2 v3.

    Updated ImHex pattern file (CDF.hexpat) for V3 format support.
@sonarqubecloud
Copy link

Quality Gate Failed Quality Gate failed

Failed conditions
2 Security Hotspots
D Security Rating on New Code (required ≥ A)
E Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

@jeandet jeandet merged commit 405c150 into SciQLop:main Feb 10, 2026
11 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant