From 639f533d2d314b2483794a706c2b544d7132ae1d Mon Sep 17 00:00:00 2001 From: "randomizedcoder dave.seddon.ca@gmail.com" Date: Mon, 9 Feb 2026 12:00:24 -0800 Subject: [PATCH] nix: Fix segfaults in nix build with ClangTool compatibility patches The nix build was crashing with segfaults due to differences in how ClangTool API behaves in Nix environments vs traditional Linux distros. Add patches to fix two issues: - 01-nix-clang-system-includes.patch: ClangTool bypasses the Nix clang wrapper, missing -isystem flags for headers. Adds include paths from environment variables set by the Nix derivation. - 02-tentative-definition-null-check.patch: C tentative definitions (from XDP2_DECL_PROTO_TABLE macro) return hasInit()=true with void-type InitListExpr on Nix clang 18.1.8+, causing null pointer crash when getAs() returns nullptr. Adds null check to skip these. Refactor the monolithic flake.nix (~1000 lines) into modular files: - nix/packages.nix: Package definitions (nativeBuildInputs, buildInputs, devTools) - nix/llvm.nix: LLVM/Clang configuration with wrapped llvm-config - nix/env-vars.nix: Environment variable exports - nix/devshell.nix: Development shell configuration - nix/derivation.nix: Package derivation for nix build - nix/shell-functions/: Modular shell functions (build, clean, configure, etc.) Also: - Upgrade from pinned LLVM 18 to default LLVM (21.1.8) - Add BPF/XDP dev tools (bpftrace, bcc, perf, pahole, clang-tools) - Update documentation with patch details and toolchain versions --- documentation/nix/nix.md | 87 +- flake.lock | 6 +- flake.nix | 1067 +---------------- nix/derivation.nix | 208 ++++ nix/devshell.nix | 161 +++ nix/env-vars.nix | 67 ++ nix/llvm.nix | 98 ++ nix/packages.nix | 101 ++ .../01-nix-clang-system-includes.patch | 100 ++ .../02-tentative-definition-null-check.patch | 59 + nix/shell-functions/ascii-art.nix | 21 + nix/shell-functions/build.nix | 373 ++++++ nix/shell-functions/clean.nix | 103 ++ nix/shell-functions/configure.nix | 62 + nix/shell-functions/navigation.nix | 84 ++ nix/shell-functions/validation.nix | 179 +++ src/configure.sh | 10 +- 17 files changed, 1757 insertions(+), 1029 deletions(-) create mode 100644 nix/derivation.nix create mode 100644 nix/devshell.nix create mode 100644 nix/env-vars.nix create mode 100644 nix/llvm.nix create mode 100644 nix/packages.nix create mode 100644 nix/patches/01-nix-clang-system-includes.patch create mode 100644 nix/patches/02-tentative-definition-null-check.patch create mode 100644 nix/shell-functions/ascii-art.nix create mode 100644 nix/shell-functions/build.nix create mode 100644 nix/shell-functions/clean.nix create mode 100644 nix/shell-functions/configure.nix create mode 100644 nix/shell-functions/navigation.nix create mode 100644 nix/shell-functions/validation.nix diff --git a/documentation/nix/nix.md b/documentation/nix/nix.md index d5ffeb2..31fc713 100644 --- a/documentation/nix/nix.md +++ b/documentation/nix/nix.md @@ -11,7 +11,7 @@ The goals of using Nix in this repository are to: > ⚠️ **Linux only:** This flake currently supports **Linux only** because [`libbpf`](https://github.com/NixOS/nixpkgs/blob/nixos-unstable/pkgs/os-specific/linux/libbpf/default.nix) is Linux-specific. -Feedback and merge requests are welcome. If we're missing a tool, please open an issue or PR. See the `corePackages` section in `flake.nix`. +Feedback and merge requests are welcome. If we're missing a tool, please open an issue or PR. See `nix/packages.nix` for package definitions. --- @@ -63,12 +63,76 @@ Feedback and merge requests are welcome. If we're missing a tool, please open an ### What This Repository Provides -This repository includes `flake.nix` and `flake.lock`: -- **`flake.nix`** defines the development environment (compilers, libraries, tools, helper functions) -- **`flake.lock`** pins exact versions so all developers use **identical** inputs +This repository includes `flake.nix`, `flake.lock`, and modular Nix files in `nix/`: + +- **`flake.nix`** - Main entry point (~90 lines), imports modules from `nix/` +- **`flake.lock`** - Pins exact versions so all developers use **identical** inputs +- **`nix/packages.nix`** - Package definitions (nativeBuildInputs, buildInputs, devTools) +- **`nix/llvm.nix`** - LLVM/Clang configuration with wrapped llvm-config +- **`nix/env-vars.nix`** - Environment variable exports +- **`nix/devshell.nix`** - Development shell configuration +- **`nix/derivation.nix`** - Package derivation for `nix build` +- **`nix/patches/`** - Source code patches for Nix compatibility (see [Nix-Specific Patches](#nix-specific-patches)) +- **`nix/shell-functions/`** - Modular shell functions (build, clean, configure, etc.) Running `nix develop` spawns a shell with the correct toolchains, libraries, and environment variables configured for you. +### Current Toolchain Versions + +The Nix development environment provides the following tool and library versions (as of February 2026): + +| Package | Version | Purpose | +|---------|---------|---------| +| GCC | 15.2.0 | Primary C/C++ compiler for final libraries | +| LLVM/Clang | 21.1.8 | Host compiler for xdp2-compiler, AST parsing | +| Boost | 1.87.0 | C++ libraries (graph, wave, program_options) | +| libbpf | 1.5.0 | BPF library for eBPF/XDP programs | +| libelf | 0.192 | ELF file handling | +| libpcap | 1.10.5 | Packet capture library | +| Python | 3.13.3 | Scripting and packet generation (with scapy) | + +### Nix-Specific Patches + +The Nix build uses patches to handle differences between traditional Linux distributions and the Nix build environment. These patches are located in `nix/patches/` and are automatically applied during `nix build`. + +**Why patches?** When using libclang's `ClangTool` API directly (as xdp2-compiler does), it bypasses the Nix clang wrapper that normally sets up include paths. Additionally, different clang versions handle certain C constructs differently. The patches address these Nix-specific issues without modifying the original source code. + +#### Patch 1: System Include Paths (`01-nix-clang-system-includes.patch`) + +**Problem:** ClangTool bypasses the Nix clang wrapper script which normally adds `-isystem` flags for system headers. Without these flags, header resolution fails and the AST contains error nodes. + +**Solution:** Reads include paths from environment variables set by the Nix derivation and adds them as `-isystem` arguments to ClangTool. These environment variables are only set during `nix build`, so this is a no-op on Ubuntu/Fedora. + +Environment variables used: +- `XDP2_C_INCLUDE_PATH`: Clang builtins (stddef.h, stdint.h, etc.) +- `XDP2_GLIBC_INCLUDE_PATH`: glibc headers (stdlib.h, stdio.h, etc.) +- `XDP2_LINUX_HEADERS_PATH`: Linux kernel headers (, etc.) + +#### Patch 2: Tentative Definition Null Check (`02-tentative-definition-null-check.patch`) + +**Problem:** C tentative definitions like `static const struct T name;` (created by `XDP2_DECL_PROTO_TABLE` macro) behave differently across clang versions: +- Ubuntu clang 18.1.3: `hasInit()` returns false, these are skipped +- Nix clang 18.1.8+: `hasInit()` returns true with void-type InitListExpr + +When `getAs()` is called on void type, it returns nullptr, causing a segfault. + +**Solution:** Adds a null check and skips tentative definitions gracefully. The actual definition is processed when encountered later in the AST. + +For detailed investigation notes, see [phase6_segfault_defect.md](phase6_segfault_defect.md). + +### BPF/XDP Development Tools + +The development shell includes additional tools for BPF/XDP development and debugging: + +| Tool | Purpose | +|------|---------| +| `bpftools` | BPF program inspection and manipulation | +| `bpftrace` | High-level tracing language for eBPF | +| `bcc` | BPF Compiler Collection with Python bindings | +| `perf` | Linux performance analysis tool | +| `pahole` | DWARF debugging info analyzer (useful for BTF) | +| `clang-tools` | clang-tidy, clang-format, and other code quality tools | + --- ## Quick Start @@ -161,13 +225,26 @@ After running `build-all` once, the necessary changes will have been applied to ### Debugging nix develop -The `flake.nix` and embedded bash code make use of the environment variable `XDP2_NIX_DEBUG`. This variable uses syslog levels between 0 (default) and 7. +The shell functions use the environment variable `XDP2_NIX_DEBUG` at runtime. This variable uses syslog-style levels between 0 (default) and 7. + +Debug levels: +- **0** - No debug output (default) +- **3** - Basic debug info +- **5** - Show compiler selection and config.mk details +- **7** - Maximum verbosity (all debug info) For maximum debugging: ```bash XDP2_NIX_DEBUG=7 nix develop --verbose --print-build-logs ``` +You can also set the debug level after entering the shell: +```bash +nix develop +export XDP2_NIX_DEBUG=5 +build-all # Will show debug output +``` + ### Shellcheck The `flake.nix` checks all the bash code within the flake to ensure there are no issues. diff --git a/flake.lock b/flake.lock index 4d1227e..923e9f1 100644 --- a/flake.lock +++ b/flake.lock @@ -20,11 +20,11 @@ }, "nixpkgs": { "locked": { - "lastModified": 1758427187, - "narHash": "sha256-pHpxZ/IyCwoTQPtFIAG2QaxuSm8jWzrzBGjwQZIttJc=", + "lastModified": 1770562336, + "narHash": "sha256-ub1gpAONMFsT/GU2hV6ZWJjur8rJ6kKxdm9IlCT0j84=", "owner": "NixOS", "repo": "nixpkgs", - "rev": "554be6495561ff07b6c724047bdd7e0716aa7b46", + "rev": "d6c71932130818840fc8fe9509cf50be8c64634f", "type": "github" }, "original": { diff --git a/flake.nix b/flake.nix index ef57bc6..a922ab3 100644 --- a/flake.nix +++ b/flake.nix @@ -1,15 +1,19 @@ # -# flake.nix for XDP2 - Development Shell Only +# flake.nix for XDP2 # -# WARNING - THIS FLAKE IS CURRENTLY BROKEN (2025/11/06) FIXES COMING SOON -# -# This flake.nix provides a fast development environment for the XDP2 project +# This flake provides: +# - Development environment: nix develop +# - Package build: nix build # # To enter the development environment: # nix develop - -# If flakes are not enabled, use the following command to enter the development environment: +# +# To build the package: +# nix build +# +# If flakes are not enabled, use the following command: # nix --extra-experimental-features 'nix-command flakes' develop . +# nix --extra-experimental-features 'nix-command flakes' build . # # To enable flakes, you may need to enable them in your system configuration: # test -d /etc/nix || sudo mkdir /etc/nix @@ -18,17 +22,16 @@ # Debugging: # XDP2_NIX_DEBUG=7 nix develop --verbose --print-build-logs # -# Not really sure what the difference between the two is, but the second one is faster +# Alternative commands: # nix --extra-experimental-features 'nix-command flakes' --option eval-cache false develop # nix --extra-experimental-features 'nix-command flakes' develop --no-write-lock-file -# # nix --extra-experimental-features 'nix-command flakes' print-dev-env --json # -# Recommended term +# Recommended term: # export TERM=xterm-256color # { - description = "XDP2 development environment"; + description = "XDP2 packet processing framework"; inputs = { nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable"; @@ -40,1025 +43,51 @@ let pkgs = nixpkgs.legacyPackages.${system}; lib = nixpkgs.lib; - llvmP = pkgs.llvmPackages_20; - - # Create a Python environment with scapy - pythonWithScapy = pkgs.python3.withPackages (ps: [ ps.scapy ]); - - - sharedConfig = { - - # Debug configuration - nixDebug = let - envDebug = builtins.getEnv "XDP2_NIX_DEBUG"; - in - if envDebug == "" then 0 else builtins.fromJSON envDebug; - - # GCC-only configuration. These variables could be used to select clang - useGCC = true; - selectedCCPkgs = pkgs.gcc; - selectedCXXPkgs = pkgs.gcc; - selectedCCBin = "gcc"; - selectedCXXBin = "g++"; - compilerInfo = "GCC"; - - configAgeWarningDays = 14; # Configurable threshold for stale config warnings + # Import LLVM configuration module + # Use default LLVM version from nixpkgs (no pinning required) + llvmConfig = import ./nix/llvm.nix { inherit pkgs lib; }; + llvmPackages = llvmConfig.llvmPackages; - # https://nixos.wiki/wiki/C#Hardening_flags - # hardeningDisable = [ "fortify" "fortify3" "stackprotector" "strictoverflow" ]; - # Disable all hardening flags for now, but might restore some later - hardeningDisable = [ "all" ]; + # Import packages module + packagesModule = import ./nix/packages.nix { inherit pkgs llvmPackages; }; - # Library packages - corePackages = with pkgs; [ - # Build tools - gnumake pkg-config bison flex - # Core utilities - bash coreutils gnused gawk gnutar xz git - # Libraries - boost - libpcap - libelf - libbpf - pythonWithScapy - # Development tools - graphviz - bpftools - # Compilers - gcc - llvmP.clang - llvmP.llvm.dev - llvmP.clang-unwrapped - llvmP.libclang - llvmP.lld - # Debugging tools - glibc_multi.bin - gdb - valgrind - strace - ltrace - # Code quality - shellcheck - # ASCII art generator for logo display - jp2a - # Locale support for cross-distribution compatibility - glibcLocales - ]; - - buildInputs = with pkgs; [ - boost - libpcap - libelf - libbpf - pythonWithScapy - llvmP.llvm - llvmP.llvm.dev - llvmP.clang-unwrapped - llvmP.libclang - llvmP.lld - ]; - - nativeBuildInputs = [ - pkgs.pkg-config - llvmP.clang - llvmP.llvm.dev - ]; + # Compiler configuration + compilerConfig = { + cc = pkgs.gcc; + cxx = pkgs.gcc; + ccBin = "gcc"; + cxxBin = "g++"; }; - # Create a wrapper for llvm-config to include clang paths (for libclang) - llvm-config-wrapped = pkgs.runCommand "llvm-config-wrapped" { } '' - mkdir -p $out/bin - cat > $out/bin/llvm-config <\\n#include \\n' include/cpp2util.h\n" - fi - sed -i '1i#include \n#include \n' include/cpp2util.h - - # Level 3: Build step details - if [ "$debug_level" -ge 3 ]; then - echo "[DEBUG] Building cppfront-compiler with make" - fi - - # Build cppfront with error checking - if HOST_CXX="$CXX" HOST_CC="$CC" make -j"$NIX_BUILD_CORES"; then - echo "✓ cppfront make completed successfully" - else - echo "✗ ERROR: cppfront make failed" - return 1 - fi - - # Return to repository root - navigate-to-repo-root || return 1 - - # Add to the PATH - add-to-path "$PWD/thirdparty/cppfront" - - # Level 2: Validation step - if [ "$debug_level" -ge 2 ]; then - echo "[DEBUG] Validating cppfront-compiler binary" - fi - - # Validate binary was created - if [ -x "./thirdparty/cppfront/cppfront-compiler" ]; then - echo "✓ cppfront-compiler binary created and executable" - - # Test the binary runs correctly - echo "Testing cppfront-compiler..." - set +e # Temporarily disable exit on error - - # Debug output for validation command - if [ "$debug_level" -gt 3 ]; then - echo "[DEBUG] About to run: ./thirdparty/cppfront/cppfront-compiler -version" - fi - ./thirdparty/cppfront/cppfront-compiler -version - test_exit_code=$? - set -e # Re-enable exit on error - - if [ "$test_exit_code" -eq 0 ] || [ "$test_exit_code" -eq 1 ]; then - echo "✓ cppfront-compiler runs correctly (exit code: $test_exit_code)" - else - echo "⚠ WARNING: cppfront-compiler returned unexpected exit code: $test_exit_code" - echo "But binary exists and is executable, continuing..." - fi - else - echo "✗ ERROR: cppfront-compiler binary not found or not executable" - return 1 - fi - - # End timing for debug levels > 3 - if [ "$debug_level" -gt 3 ]; then - end_time=$(date +%s) - local duration=$((end_time - start_time)) - echo "[DEBUG] build-cppfront completed in $duration seconds" - fi - - echo "cppfront-compiler built and validated successfully ( ./thirdparty/cppfront/cppfront-compiler )" - } - ''; - - check-cppfront-age-fn = '' - check-cppfront-age() { - if [ -n "$XDP2_NIX_DEBUG" ]; then - local debug_level=$XDP2_NIX_DEBUG - else - local debug_level=0 - fi - local start_time="" - local end_time="" - - # Start timing for debug levels > 3 - if [ "$debug_level" -gt 3 ]; then - start_time=$(date +%s) - echo "[DEBUG] check-cppfront-age started at $(date)" - fi - - # Level 1: Function start - if [ "$debug_level" -ge 1 ]; then - echo "[DEBUG] Starting check-cppfront-age function" - fi - - local cppfront_binary="thirdparty/cppfront/cppfront-compiler" - - # Level 2: File check - if [ "$debug_level" -ge 2 ]; then - echo "[DEBUG] Checking cppfront binary: $cppfront_binary" - fi - - if [ -f "$cppfront_binary" ]; then - local file_time - file_time=$(stat -c %Y "$cppfront_binary") - local current_time - current_time=$(date +%s) - local age_days=$(( (current_time - file_time) / 86400 )) - - # Level 3: Age calculation details - if [ "$debug_level" -ge 3 ]; then - echo "[DEBUG] File modification time: $file_time" - echo "[DEBUG] Current time: $current_time" - echo "[DEBUG] Calculated age: $age_days days" - fi - - if [ "$age_days" -gt 7 ]; then - echo "cppfront is $age_days days old, rebuilding..." - build-cppfront - else - echo "cppfront is up to date ($age_days days old)" - fi - else - echo "cppfront not found, building..." - build-cppfront - fi - - # End timing for debug levels > 3 - if [ "$debug_level" -gt 3 ]; then - end_time=$(date +%s) - local duration=$((end_time - start_time)) - echo "[DEBUG] check-cppfront-age completed in $duration seconds" - fi - } - ''; - - build-xdp2-compiler-fn = '' - build-xdp2-compiler() { - if [ -n "$XDP2_NIX_DEBUG" ]; then - local debug_level=$XDP2_NIX_DEBUG - else - local debug_level=0 - fi - local start_time="" - local end_time="" - - # Start timing for debug levels > 3 - if [ "$debug_level" -gt 3 ]; then - start_time=$(date +%s) - echo "[DEBUG] build-xdp2-compiler started at $(date)" - fi - - # Level 1: Function start - if [ "$debug_level" -ge 1 ]; then - echo "[DEBUG] Starting build-xdp2-compiler function" - fi - - # Level 2: Clean step - if [ "$debug_level" -ge 2 ]; then - echo "[DEBUG] Cleaning xdp2-compiler build directory" - fi - echo "Cleaning and building xdp2-compiler..." - - # Navigate to repository root first - navigate-to-repo-root || return 1 - - # Clean previous build artifacts (before navigating to component) - clean-xdp2-compiler - - # Navigate to xdp2-compiler directory - navigate-to-component "src/tools/compiler" || return 1 - - # Level 3: Build step details - if [ "$debug_level" -ge 3 ]; then - echo "[DEBUG] Building xdp2-compiler with make" - fi - - # Build xdp2-compiler with error checking - if CFLAGS_PYTHON="$CFLAGS_PYTHON" LDFLAGS_PYTHON="$LDFLAGS_PYTHON" make -j"$NIX_BUILD_CORES"; then - echo "✓ xdp2-compiler make completed successfully" - else - echo "✗ ERROR: xdp2-compiler make failed" - return 1 - fi - - # Level 2: Validation step - if [ "$debug_level" -ge 2 ]; then - echo "[DEBUG] Validating xdp2-compiler binary" - fi - - # Validate binary was created - if [ -x "./xdp2-compiler" ]; then - echo "✓ xdp2-compiler binary created and executable" - - # Test the binary runs correctly - echo "Testing xdp2-compiler..." - set +e # Temporarily disable exit on error - - # Debug output for validation command - if [ "$debug_level" -gt 3 ]; then - echo "[DEBUG] About to run: ./xdp2-compiler --help" - fi - ./xdp2-compiler --help - test_exit_code=$? - set -e # Re-enable exit on error - - if [ "$test_exit_code" -eq 0 ] || [ "$test_exit_code" -eq 1 ]; then - echo "✓ xdp2-compiler runs correctly (exit code: $test_exit_code)" - else - echo "⚠ WARNING: xdp2-compiler returned unexpected exit code: $test_exit_code" - echo "But binary exists and is executable, continuing..." - fi - else - echo "✗ ERROR: xdp2-compiler binary not found or not executable" - return 1 - fi - - # Return to repository root - navigate-to-repo-root || return 1 - - # End timing for debug levels > 3 - if [ "$debug_level" -gt 3 ]; then - end_time=$(date +%s) - local duration=$((end_time - start_time)) - echo "[DEBUG] build-xdp2-compiler completed in $duration seconds" - fi - - echo "xdp2-compiler built and validated successfully ( ./src/tools/compiler/xdp2-compiler )" - } - ''; - - build-xdp2-fn = '' - build-xdp2() { - if [ -n "$XDP2_NIX_DEBUG" ]; then - local debug_level=$XDP2_NIX_DEBUG - else - local debug_level=0 - fi - local start_time="" - local end_time="" - - # Start timing for debug levels > 3 - if [ "$debug_level" -gt 3 ]; then - start_time=$(date +%s) - echo "[DEBUG] build-xdp2 started at $(date)" - fi - - # Level 1: Function start - if [ "$debug_level" -ge 1 ]; then - echo "[DEBUG] Starting build-xdp2 function" - fi - - # Level 2: Clean step - if [ "$debug_level" -ge 2 ]; then - echo "[DEBUG] Cleaning xdp2 project build directory" - fi - echo "Cleaning and building xdp2 project..." - - # Navigate to repository root first - navigate-to-repo-root || return 1 - - # Clean previous build artifacts (before navigating to component) - clean-xdp2 - - # Navigate to src directory - navigate-to-component "src" || return 1 - - # Ensure xdp2-compiler is available in PATH - add-to-path "$PWD/tools/compiler" - echo "Added tools/compiler to PATH" - - # Level 3: Build step details - if [ "$debug_level" -ge 3 ]; then - echo "[DEBUG] Building xdp2 project with make" - fi - - # Build the main xdp2 project - if make -j"$NIX_BUILD_CORES"; then - echo "✓ xdp2 project make completed successfully" - else - echo "✗ ERROR: xdp2 project make failed" - echo " Check the error messages above for details" - return 1 - fi - - # Return to repository root - navigate-to-repo-root || return 1 - - # End timing for debug levels > 3 - if [ "$debug_level" -gt 3 ]; then - end_time=$(date +%s) - local duration=$((end_time - start_time)) - echo "[DEBUG] build-xdp2 completed in $duration seconds" - fi - - echo "xdp2 project built successfully" - } - ''; - - build-all-fn = '' - build-all() { - if [ -n "$XDP2_NIX_DEBUG" ]; then - local debug_level=$XDP2_NIX_DEBUG - else - local debug_level=0 - fi - - echo "Building all XDP2 components..." - - if [ "$debug_level" -ge 3 ]; then - echo "[DEBUG] Building cppfront: build-cppfront" - fi - build-cppfront - - if [ "$debug_level" -ge 3 ]; then - echo "[DEBUG] Building xdp2-compiler: build-xdp2-compiler" - fi - build-xdp2-compiler - - if [ "$debug_level" -ge 3 ]; then - echo "[DEBUG] Building xdp2: build-xdp2" - fi - build-xdp2 - - echo "✓ All components built successfully" - } - ''; - - clean-all-fn = '' - clean-all() { - echo "Cleaning all build artifacts..." - - # Clean each component using centralized clean functions - clean-cppfront - clean-xdp2-compiler - clean-xdp2 - - echo "✓ All build artifacts cleaned" - } - ''; - - # Shellcheck function registry - list of all bash functions that should be validated by shellcheck - # IMPORTANT: When adding or removing bash functions, update this list accordingly - shellcheckFunctionRegistry = [ - "smart-configure" - "build-cppfront" - "check-cppfront-age" - "build-xdp2-compiler" - "build-xdp2" - "build-all" - "clean-all" - "check-platform-compatibility" - "detect-repository-root" - "setup-locale-support" - "xdp2-help" - "navigate-to-repo-root" - "navigate-to-component" - "add-to-path" - "clean-cppfront" - "clean-xdp2-compiler" - "clean-xdp2" - ]; - - # Generate complete shellcheck validation function in Nix - generate-shellcheck-validation = let - functionNames = shellcheckFunctionRegistry; - totalFunctions = builtins.length functionNames; - - # Generate individual function checks - functionChecks = lib.concatStringsSep "\n" (map (name: '' - echo "Checking ${name}..." - if declare -f "${name}" >/dev/null 2>&1; then - # Create temporary script with function definition - # TODO use mktemp and trap 'rm -f "$temp_script"' EXIT - local temp_script="/tmp/validate_${name}.sh" - declare -f "${name}" > "$temp_script" - echo "#!/bin/bash" > "$temp_script.tmp" - cat "$temp_script" >> "$temp_script.tmp" - mv "$temp_script.tmp" "$temp_script" - - # Run shellcheck on the function - if shellcheck -s bash "$temp_script" 2>/dev/null; then - echo "✓ ${name} passed shellcheck validation" - passed_functions=$((passed_functions + 1)) - else - echo "✗ ${name} failed shellcheck validation:" - shellcheck -s bash "$temp_script" - failed_functions+=("${name}") - fi - rm -f "$temp_script" - else - echo "✗ ${name} not found" - failed_functions+=("${name}") - fi - echo "" - '') functionNames); - - # Generate failed functions reporting - failedFunctionsReporting = lib.concatStringsSep "\n" (map (name: '' - if [[ "$${failed_functions[*]}" == *"${name}"* ]]; then - echo " - ${name}" - fi - '') functionNames); - - in '' - run-shellcheck() { - if [ -n "$XDP2_NIX_DEBUG" ]; then - local debug_level=$XDP2_NIX_DEBUG - else - local debug_level=0 - fi - - echo "Running shellcheck validation on shell functions..." - - local failed_functions=() - local total_functions=${toString totalFunctions} - local passed_functions=0 - - # Pre-generated function checks from Nix - ${functionChecks} - - # Report results - echo "=== Shellcheck Validation Complete ===" - echo "Total functions: $total_functions" - echo "Passed: $passed_functions" - echo "Failed: $((total_functions - passed_functions))" - - if [ $((total_functions - passed_functions)) -eq 0 ]; then - echo "✓ All functions passed shellcheck validation" - return 0 - else - echo "✗ Some functions failed validation:" - # Pre-generated failed functions reporting from Nix - ${failedFunctionsReporting} - return 1 - fi - } - ''; - - run-shellcheck-fn = generate-shellcheck-validation; - - disable-exit-fn = '' - disable-exit() { - set +e - } - ''; - - platform-compatibility-check-fn = '' - check-platform-compatibility() { - if [ "$(uname)" != "Linux" ]; then - echo "⚠️ PLATFORM COMPATIBILITY NOTICE -================================== - -🍎 You are running on $(uname) (not Linux) - -The XDP2 development environment includes Linux-specific packages -like libbpf that are not available on $(uname) systems. - -📋 Available platforms: - ✅ Linux (x86_64-linux, aarch64-linux, etc.) - ❌ macOS (x86_64-darwin, aarch64-darwin) - ❌ Other Unix systems - -Exiting development shell..." - exit 1 - fi - } - ''; - - detect-repository-root-fn = '' - detect-repository-root() { - XDP2_REPO_ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd) - export XDP2_REPO_ROOT - - if [ ! -d "$XDP2_REPO_ROOT" ]; then - echo "⚠ WARNING: Could not detect valid repository root" - XDP2_REPO_ROOT="$PWD" - else - echo "📁 Repository root: $XDP2_REPO_ROOT" - fi - } - ''; - - setup-locale-support-fn = let - bashVarExpansion = "$"; - bashDefaultSyntax = "{LANG:-C.UTF-8}"; - bashDefaultSyntaxLC = "{LC_ALL:-C.UTF-8}"; - in '' - setup-locale-support() { - # Only set locale if user hasn't already configured it - if [ -z "$LANG" ] || [ -z "$LC_ALL" ]; then - # Try to use system default, fallback to C.UTF-8 - export LANG=${bashVarExpansion}${bashDefaultSyntax} - export LC_ALL=${bashVarExpansion}${bashDefaultSyntaxLC} - fi - - # Verify locale is available (only if locale command exists) - if command -v locale >/dev/null 2>&1; then - if ! locale -a 2>/dev/null | grep -q "$LANG"; then - # Fallback to C.UTF-8 if user's locale is not available - export LANG=C.UTF-8 - export LC_ALL=C.UTF-8 - fi - fi - } - ''; - - xdp2-help-fn = '' - xdp2-help() { - echo "🚀 === XDP2 Development Shell Help === - -📦 Compiler: GCC -🔧 GCC and Clang are available in the environment. -🐛 Debugging tools: gdb, valgrind, strace, ltrace - -🔍 DEBUGGING: - XDP2_NIX_DEBUG=0 - No extra debug. Default - XDP2_NIX_DEBUG=3 - Basic debug - XDP2_NIX_DEBUG=5 - Show compiler selection and config.mk - XDP2_NIX_DEBUG=7 - Show all debug info - -🔧 BUILD COMMANDS: - build-cppfront - Build cppfront compiler - build-xdp2-compiler - Build xdp2 compiler - build-xdp2 - Build main XDP2 project - build-all - Build all components - -🧹 CLEAN COMMANDS: - clean-cppfront - Clean cppfront build artifacts - clean-xdp2-compiler - Clean xdp2-compiler build artifacts - clean-xdp2 - Clean xdp2 build artifacts - clean-all - Clean all build artifacts - -🔍 VALIDATION: - run-shellcheck - Validate all shell functions - -📁 PROJECT STRUCTURE: - • src/ - Main source code - • tools/ - Build tools and utilities - • thirdparty/ - Third-party dependencies - • samples/ - Example code and parsers - • documentation/ - Project documentation - -🎯 Ready to develop! 'xdp2-help' for help" - } - ''; - - shell-aliases = '' - alias xdp2-src='cd src' - alias xdp2-samples='cd samples' - alias xdp2-docs='cd documentation' - alias xdp2-cppfront='cd thirdparty/cppfront' - ''; - - colored-prompt = '' - export PS1="\[\033[0;32m\][XDP2-${sharedConfig.compilerInfo}] \[\033[01;34m\][\u@\h:\w]\$ \[\033[0m\]" - ''; - - ascii-art-logo = '' - if command -v jp2a >/dev/null 2>&1 && [ -f "./documentation/images/xdp2-big.png" ]; then - echo "$(jp2a --colors ./documentation/images/xdp2-big.png)" - echo "" - else - echo "🚀 === XDP2 Development Shell ===" - fi - ''; - - minimal-shell-entry = '' - echo "🚀 === XDP2 Development Shell ===" - echo "📦 Compiler: ${sharedConfig.compilerInfo}" - echo "🔧 GCC and Clang are available in the environment" - echo "🐛 Debugging tools: gdb, valgrind, strace, ltrace" - echo "🎯 Ready to develop! 'xdp2-help' for help" - ''; - - debug-compiler-selection = '' - if [ ${toString sharedConfig.nixDebug} -gt 4 ]; then - echo "=== COMPILER SELECTION ===" - echo "Using compiler: ${sharedConfig.compilerInfo}" - echo "HOST_CC: $HOST_CC" - echo "HOST_CXX: $HOST_CXX" - $HOST_CC --version - $HOST_CXX --version - echo "=== End compiler selection ===" - fi - ''; - - debug-environment-vars = '' - if [ ${toString sharedConfig.nixDebug} -gt 5 ]; then - echo "=== Environment Variables ===" - env - echo "=== End Environment Variables ===" - fi - ''; - - - navigate-to-repo-root-fn = '' - navigate-to-repo-root() { - if [ -n "$XDP2_REPO_ROOT" ]; then - cd "$XDP2_REPO_ROOT" || return 1 - else - echo "✗ ERROR: XDP2_REPO_ROOT not set" - return 1 - fi - } - ''; - - navigate-to-component-fn = '' - navigate-to-component() { - local component="$1" - local target_dir="$XDP2_REPO_ROOT/$component" - - if [ ! -d "$target_dir" ]; then - echo "✗ ERROR: Component directory not found: $target_dir" - return 1 - fi - - cd "$target_dir" || return 1 - } - ''; - - add-to-path-fn = '' - # Add path to PATH environment variable if not already present - add-to-path() { - local path_to_add="$1" - - if [ -n "$XDP2_NIX_DEBUG" ]; then - local debug_level=$XDP2_NIX_DEBUG - else - local debug_level=0 - fi - - # Check if path is already in PATH - if [[ ":$PATH:" == *":$path_to_add:"* ]]; then - if [ "$debug_level" -gt 3 ]; then - echo "[DEBUG] Path already in PATH: $path_to_add" - fi - return 0 - fi - - # Add path to beginning of PATH - if [ "$debug_level" -gt 3 ]; then - echo "[DEBUG] Adding to PATH: $path_to_add" - echo "[DEBUG] PATH before: $PATH" - fi - - export PATH="$path_to_add:$PATH" - - if [ "$debug_level" -gt 3 ]; then - echo "[DEBUG] PATH after: $PATH" - fi - } - ''; - - clean-cppfront-fn = '' - # Clean cppfront build artifacts - clean-cppfront() { - if [ -n "$XDP2_NIX_DEBUG" ]; then - local debug_level=$XDP2_NIX_DEBUG - else - local debug_level=0 - fi - - # Navigate to repository root first - navigate-to-repo-root || return 1 - - # Navigate to cppfront directory - navigate-to-component "thirdparty/cppfront" || return 1 - - # Debug output for clean command - if [ "$debug_level" -gt 3 ]; then - echo "[DEBUG] About to run: rm -f cppfront-compiler" - fi - rm -f cppfront-compiler # Remove the binary directly since Makefile has no clean target - - # Return to repository root - navigate-to-repo-root || return 1 - } - ''; - - clean-xdp2-compiler-fn = '' - # Clean xdp2-compiler build artifacts - clean-xdp2-compiler() { - if [ -n "$XDP2_NIX_DEBUG" ]; then - local debug_level=$XDP2_NIX_DEBUG - else - local debug_level=0 - fi - - # Navigate to repository root first - navigate-to-repo-root || return 1 - - # Navigate to xdp2-compiler directory - navigate-to-component "src/tools/compiler" || return 1 - - # Debug output for clean command - if [ "$debug_level" -gt 3 ]; then - echo "[DEBUG] About to run: make clean" - fi - make clean || true # Don't fail if clean fails - - # Return to repository root - navigate-to-repo-root || return 1 - } - ''; - - clean-xdp2-fn = '' - # Clean xdp2 build artifacts - clean-xdp2() { - if [ -n "$XDP2_NIX_DEBUG" ]; then - local debug_level=$XDP2_NIX_DEBUG - else - local debug_level=0 - fi - - # Navigate to repository root first - navigate-to-repo-root || return 1 - - # Navigate to src directory - navigate-to-component "src" || return 1 - - # Debug output for clean command - if [ "$debug_level" -gt 3 ]; then - echo "[DEBUG] About to run: make clean" - fi - make clean || true # Don't fail if clean fails - - # Return to repository root - navigate-to-repo-root || return 1 - } - ''; - - # Combined build functions (ordered to avoid SC2218 - functions called before definition) - build-functions = '' - # Navigation functions (called by all other functions) - ${navigate-to-repo-root-fn} - ${navigate-to-component-fn} + # Import environment variables module + envVars = import ./nix/env-vars.nix { + inherit pkgs llvmConfig compilerConfig; + packages = packagesModule; + configAgeWarningDays = 14; + }; - # Utility functions - ${add-to-path-fn} - ${check-cppfront-age-fn} + # Import package derivation + xdp2 = import ./nix/derivation.nix { + inherit pkgs lib llvmConfig; + inherit (packagesModule) nativeBuildInputs buildInputs; + }; - # Individual clean functions (called by build functions and clean-build) - ${clean-cppfront-fn} - ${clean-xdp2-compiler-fn} - ${clean-xdp2-fn} - # Individual build functions (called by build-all) - ${build-cppfront-fn} - ${build-xdp2-compiler-fn} - ${build-xdp2-fn} - # Composite functions (call the individual functions above) - ${build-all-fn} - ${clean-all-fn} - # Validation and help functions - ${platform-compatibility-check-fn} - ${detect-repository-root-fn} - ${setup-locale-support-fn} - ${run-shellcheck-fn} - ${xdp2-help-fn} - ''; + # Import development shell module + devshell = import ./nix/devshell.nix { + inherit pkgs lib llvmConfig compilerConfig envVars; + packages = packagesModule; + }; in { - devShells.default = pkgs.mkShell { - packages = sharedConfig.corePackages; - - shellHook = '' - ${sharedEnvVars} - - ${build-functions} - - check-platform-compatibility - detect-repository-root - setup-locale-support - - ${debug-compiler-selection} - ${debug-environment-vars} - - ${ascii-art-logo} - - ${smart-configure} - smart-configure - - ${shell-aliases} - - ${colored-prompt} - - ${disable-exit-fn} - - ${minimal-shell-entry} - ''; + # Package outputs + packages = { + default = xdp2; + xdp2 = xdp2; }; + + # Development shell + devShells.default = devshell; }); } diff --git a/nix/derivation.nix b/nix/derivation.nix new file mode 100644 index 0000000..3f4b9ba --- /dev/null +++ b/nix/derivation.nix @@ -0,0 +1,208 @@ +# nix/derivation.nix +# +# Package derivation for XDP2 +# +# This module defines the actual XDP2 package using stdenv.mkDerivation. +# It enables `nix build` support and follows nixpkgs conventions. +# +# Build order: +# 1. Patch source files (postPatch) +# 2. Run configure script (configurePhase) +# 3. Build cppfront, xdp2-compiler, then xdp2 (buildPhase) +# 4. Install binaries and libraries (installPhase) +# +# Usage in flake.nix: +# packages.default = import ./nix/derivation.nix { +# inherit pkgs lib llvmConfig; +# inherit (import ./nix/packages.nix { inherit pkgs llvmPackages; }) nativeBuildInputs buildInputs; +# }; +# + +{ pkgs +, lib +, llvmConfig +, nativeBuildInputs +, buildInputs +}: + +let + llvmPackages = llvmConfig.llvmPackages; +in +pkgs.stdenv.mkDerivation rec { + pname = "xdp2"; + version = "0.1.0"; + + src = ./..; + + # Nix-specific patches for xdp2-compiler + # + # These patches are required because xdp2-compiler uses libclang's ClangTool API + # directly, which bypasses the Nix clang wrapper. The wrapper normally handles + # include path setup, but ClangTool doesn't benefit from it. + # + # See documentation/nix/phase6_segfault_defect.md for full investigation details. + patches = [ + # Patch 1: System include paths for Nix environment + # + # Problem: ClangTool bypasses the Nix clang wrapper which normally adds + # -isystem flags for system headers. Without these, header resolution fails + # and the AST contains RecoveryExpr/contains-errors nodes. + # + # Solution: Read include paths from environment variables set by the Nix + # derivation (XDP2_C_INCLUDE_PATH, XDP2_GLIBC_INCLUDE_PATH, XDP2_LINUX_HEADERS_PATH) + # and add them as -isystem arguments to ClangTool. + # + # Also adds optional debug diagnostic output (enable with -DXDP2_COMPILER_DEBUG) + # using LLVM 21 compatible API. + ./patches/01-nix-clang-system-includes.patch + + # Patch 2: Null check for tentative definitions + # + # Problem: C tentative definitions like "static const struct T name;" create + # VarDecls where hasInit() behavior differs between clang versions: + # - Ubuntu clang 18.1.3: hasInit() returns false + # - Nix clang 18.1.8+: hasInit() returns true with void-type InitListExpr + # + # When getAs() is called on a void type, it returns nullptr, + # causing a segfault when ->getDecl() is called. + # + # Solution: Check if getAs() returns null and skip tentative + # definitions. The actual definition will be processed when encountered later. + ./patches/02-tentative-definition-null-check.patch + ]; + + inherit nativeBuildInputs buildInputs; + + # Disable hardening flags that interfere with XDP/BPF code + hardeningDisable = [ "all" ]; + + # Set up environment variables for the build + HOST_CC = "${pkgs.gcc}/bin/gcc"; + HOST_CXX = "${pkgs.gcc}/bin/g++"; + HOST_LLVM_CONFIG = "${llvmConfig.llvm-config-wrapped}/bin/llvm-config"; + XDP2_CLANG_VERSION = llvmConfig.version; + XDP2_CLANG_RESOURCE_PATH = llvmConfig.paths.clangResourceDir; + + # Add LLVM/Clang libs to library path + LD_LIBRARY_PATH = lib.makeLibraryPath [ + llvmPackages.llvm + llvmPackages.libclang.lib + pkgs.boost + ]; + + # Post-patch phase: Fix paths and apply Nix-specific patches + postPatch = '' + # Fix cppfront Makefile to use source directory path + substituteInPlace thirdparty/cppfront/Makefile \ + --replace-fail 'include ../../src/config.mk' '# config.mk not needed for standalone build' + + # Add functional header to cppfront (required for newer GCC) + sed -i '1i#include \n#include \n' thirdparty/cppfront/include/cpp2util.h + ''; + + # Configure phase: Generate config.mk + configurePhase = '' + runHook preConfigure + + cd src + + # Set up environment for configure + export CC="${pkgs.gcc}/bin/gcc" + export CXX="${pkgs.gcc}/bin/g++" + export HOST_CC="$CC" + export HOST_CXX="$CXX" + export HOST_LLVM_CONFIG="${llvmConfig.llvm-config-wrapped}/bin/llvm-config" + + # Run configure script + bash configure.sh --build-opt-parser + + # Fix PATH_ARG for Nix environment (remove hardcoded paths) + if grep -q 'PATH_ARG="--with-path=' config.mk; then + sed -i 's|PATH_ARG="--with-path=.*"|PATH_ARG=""|' config.mk + fi + + cd .. + + runHook postConfigure + ''; + + # Build phase: Build all components in order + buildPhase = '' + runHook preBuild + + # Set up environment + export HOST_CC="${pkgs.gcc}/bin/gcc" + export HOST_CXX="${pkgs.gcc}/bin/g++" + export HOST_LLVM_CONFIG="${llvmConfig.llvm-config-wrapped}/bin/llvm-config" + export NIX_BUILD_CORES=$NIX_BUILD_CORES + export XDP2_CLANG_VERSION="${llvmConfig.version}" + export XDP2_CLANG_RESOURCE_PATH="${llvmConfig.paths.clangResourceDir}" + + # Include paths for xdp2-compiler's libclang usage + # These are needed because ClangTool bypasses the Nix clang wrapper + export XDP2_C_INCLUDE_PATH="${llvmConfig.paths.clangResourceDir}/include" + export XDP2_GLIBC_INCLUDE_PATH="${pkgs.stdenv.cc.libc.dev}/include" + export XDP2_LINUX_HEADERS_PATH="${pkgs.linuxHeaders}/include" + + # 1. Build cppfront compiler + echo "Building cppfront..." + cd thirdparty/cppfront + $HOST_CXX -std=c++20 source/cppfront.cpp -o cppfront-compiler + cd ../.. + + # 2. Build xdp2-compiler + echo "Building xdp2-compiler..." + cd src/tools/compiler + make -j$NIX_BUILD_CORES + cd ../../.. + + # 3. Build main xdp2 project + echo "Building xdp2..." + cd src + make -j$NIX_BUILD_CORES + cd .. + + runHook postBuild + ''; + + # Install phase: Install binaries and libraries + installPhase = '' + runHook preInstall + + # Create output directories + mkdir -p $out/bin + mkdir -p $out/lib + mkdir -p $out/include + mkdir -p $out/share/xdp2 + + # Install xdp2-compiler + install -m 755 src/tools/compiler/xdp2-compiler $out/bin/ + + # Install cppfront-compiler (useful for development) + install -m 755 thirdparty/cppfront/cppfront-compiler $out/bin/ + + # Install libraries (if any are built as shared) + find src/lib -name "*.so" -exec install -m 755 {} $out/lib/ \; 2>/dev/null || true + find src/lib -name "*.a" -exec install -m 644 {} $out/lib/ \; 2>/dev/null || true + + # Install headers (use -L to dereference symlinks like arch -> platform/...) + cp -rL src/include/* $out/include/ 2>/dev/null || true + + # Install templates + cp -r src/templates $out/share/xdp2/ 2>/dev/null || true + + runHook postInstall + ''; + + meta = with lib; { + description = "XDP2 packet processing framework"; + longDescription = '' + XDP2 is a high-performance packet processing framework that uses + eBPF/XDP for fast packet handling in the Linux kernel. + ''; + homepage = "https://github.com/xdp2/xdp2"; + license = licenses.mit; # Update if different + platforms = platforms.linux; + maintainers = [ ]; + }; +} diff --git a/nix/devshell.nix b/nix/devshell.nix new file mode 100644 index 0000000..ec16c61 --- /dev/null +++ b/nix/devshell.nix @@ -0,0 +1,161 @@ +# nix/devshell.nix +# +# Development shell configuration for XDP2 +# +# This module creates the development shell with all necessary +# packages, environment variables, and shell functions. +# +# Usage in flake.nix: +# devshell = import ./nix/devshell.nix { +# inherit pkgs lib llvmConfig packages compilerConfig envVars; +# }; +# devShells.default = devshell; +# + +{ pkgs +, lib +, llvmConfig +, packages +, compilerConfig +, envVars +}: + +let + llvmPackages = llvmConfig.llvmPackages; + + # Shared configuration + sharedConfig = { + # Compiler info for display + compilerInfo = "GCC"; + + # Configurable threshold for stale config warnings + configAgeWarningDays = 14; + }; + + # Shellcheck function registry - list of all bash functions to validate + # IMPORTANT: When adding or removing bash functions, update this list + shellcheckFunctionRegistry = [ + "smart-configure" + "build-cppfront" + "check-cppfront-age" + "build-xdp2-compiler" + "build-xdp2" + "build-all" + "clean-all" + "check-platform-compatibility" + "detect-repository-root" + "setup-locale-support" + "xdp2-help" + "navigate-to-repo-root" + "navigate-to-component" + "add-to-path" + "clean-cppfront" + "clean-xdp2-compiler" + "clean-xdp2" + ]; + + # Import shell function modules + navigationFns = import ./shell-functions/navigation.nix { }; + cleanFns = import ./shell-functions/clean.nix { }; + buildFns = import ./shell-functions/build.nix { }; + configureFns = import ./shell-functions/configure.nix { + configAgeWarningDays = sharedConfig.configAgeWarningDays; + }; + validationFns = import ./shell-functions/validation.nix { + inherit lib shellcheckFunctionRegistry; + }; + asciiArtFn = import ./shell-functions/ascii-art.nix { }; + + # Shell snippets + disable-exit-fn = '' + disable-exit() { + set +e + } + ''; + + shell-aliases = '' + alias xdp2-src='cd src' + alias xdp2-samples='cd samples' + alias xdp2-docs='cd documentation' + alias xdp2-cppfront='cd thirdparty/cppfront' + ''; + + colored-prompt = '' + export PS1="\[\033[0;32m\][XDP2-${sharedConfig.compilerInfo}] \[\033[01;34m\][\u@\h:\w]\$ \[\033[0m\]" + ''; + + minimal-shell-entry = '' + echo "🚀 === XDP2 Development Shell ===" + echo "📦 Compiler: ${sharedConfig.compilerInfo}" + echo "🔧 GCC and Clang are available in the environment" + echo "🐛 Debugging tools: gdb, valgrind, strace, ltrace" + echo "🎯 Ready to develop! 'xdp2-help' for help" + ''; + + # Debug snippets - check XDP2_NIX_DEBUG shell variable at runtime + # Usage: XDP2_NIX_DEBUG=7 nix develop + debug-compiler-selection = '' + if [ "''${XDP2_NIX_DEBUG:-0}" -gt 4 ]; then + echo "=== COMPILER SELECTION ===" + echo "Using compiler: ${sharedConfig.compilerInfo}" + echo "HOST_CC: $HOST_CC" + echo "HOST_CXX: $HOST_CXX" + $HOST_CC --version + $HOST_CXX --version + echo "=== End compiler selection ===" + fi + ''; + + debug-environment-vars = '' + if [ "''${XDP2_NIX_DEBUG:-0}" -gt 5 ]; then + echo "=== Environment Variables ===" + env + echo "=== End Environment Variables ===" + fi + ''; + + # Combined build functions (ordered to avoid SC2218 - functions called before definition) + build-functions = '' + # Navigation functions (from nix/shell-functions/navigation.nix) + ${navigationFns} + + # Clean functions (from nix/shell-functions/clean.nix) + ${cleanFns} + + # Build functions (from nix/shell-functions/build.nix) + ${buildFns} + + # Validation and help functions (from nix/shell-functions/validation.nix) + ${validationFns} + ''; + +in +pkgs.mkShell { + packages = packages.allPackages; + + shellHook = '' + ${envVars} + + ${build-functions} + + check-platform-compatibility + detect-repository-root + setup-locale-support + + ${debug-compiler-selection} + ${debug-environment-vars} + + ${asciiArtFn} + + ${configureFns} + smart-configure + + ${shell-aliases} + + ${colored-prompt} + + ${disable-exit-fn} + + ${minimal-shell-entry} + ''; +} diff --git a/nix/env-vars.nix b/nix/env-vars.nix new file mode 100644 index 0000000..96f5593 --- /dev/null +++ b/nix/env-vars.nix @@ -0,0 +1,67 @@ +# nix/env-vars.nix +# +# Environment variable definitions for XDP2 +# +# This module defines all environment variables needed for: +# - Compiler configuration (CC, CXX, HOST_CC, HOST_CXX) +# - LLVM/Clang paths (via llvmConfig) +# - Python configuration +# - Library paths +# - Build configuration +# +# Usage in flake.nix: +# envVars = import ./nix/env-vars.nix { +# inherit pkgs llvmConfig packages; +# compilerConfig = { cc = pkgs.gcc; cxx = pkgs.gcc; ccBin = "gcc"; cxxBin = "g++"; }; +# configAgeWarningDays = 14; +# }; +# + +{ pkgs +, llvmConfig +, packages +, compilerConfig +, configAgeWarningDays ? 14 +}: + +'' + # Compiler settings + export CC=${compilerConfig.cc}/bin/${compilerConfig.ccBin} + export CXX=${compilerConfig.cxx}/bin/${compilerConfig.cxxBin} + export HOST_CC=${compilerConfig.cc}/bin/${compilerConfig.ccBin} + export HOST_CXX=${compilerConfig.cxx}/bin/${compilerConfig.cxxBin} + + # LLVM/Clang environment variables (from llvmConfig module) + # Sets: XDP2_CLANG_VERSION, XDP2_CLANG_RESOURCE_PATH, XDP2_C_INCLUDE_PATH, + # HOST_LLVM_CONFIG, LLVM_LIBS, CLANG_LIBS, LIBCLANG_PATH + ${llvmConfig.envVars} + + # Glibc include path for xdp2-compiler (needed because libclang bypasses clang wrapper) + export XDP2_GLIBC_INCLUDE_PATH="${pkgs.stdenv.cc.libc.dev}/include" + + # Linux kernel headers (provides etc.) + export XDP2_LINUX_HEADERS_PATH="${pkgs.linuxHeaders}/include" + + # LD_LIBRARY_PATH for libclang + export LD_LIBRARY_PATH=${llvmConfig.ldLibraryPath}:$LD_LIBRARY_PATH + + # Python environment + export CFLAGS_PYTHON="$(pkg-config --cflags python3-embed)" + export LDFLAGS_PYTHON="$(pkg-config --libs python3-embed)" + export PYTHON_VER=3 + export PYTHONPATH="${pkgs.python3}/lib/python3.13/site-packages:$PYTHONPATH" + + # Boost libraries + export BOOST_LIBS="-lboost_wave -lboost_thread -lboost_filesystem -lboost_system -lboost_program_options" + + # Other libraries + export LIBS="-lpthread -ldl -lutil" + export PATH_ARG="" + + # Build configuration + export PKG_CONFIG_PATH=${pkgs.lib.makeSearchPath "lib/pkgconfig" packages.allPackages} + export XDP2_COMPILER_DEBUG=1 + + # Configuration management + export CONFIG_AGE_WARNING_DAYS=${toString configAgeWarningDays} +'' diff --git a/nix/llvm.nix b/nix/llvm.nix new file mode 100644 index 0000000..ba319b1 --- /dev/null +++ b/nix/llvm.nix @@ -0,0 +1,98 @@ +# nix/llvm.nix +# +# LLVM/Clang configuration for XDP2 +# +# This module centralizes all LLVM and Clang configuration: +# - llvmPackages selection (configurable, defaults to pkgs.llvmPackages) +# - llvm-config wrapper for correct include/lib paths +# - Environment variables for LLVM tools +# - Paths for use with substituteInPlace +# +# Usage in flake.nix: +# llvmConfig = import ./nix/llvm.nix { inherit pkgs lib; }; +# # Or with custom version: +# llvmConfig = import ./nix/llvm.nix { inherit pkgs lib; llvmVersion = 19; }; +# + +{ pkgs +, lib +, llvmVersion ? null # null means use default llvmPackages +}: + +let + # Select llvmPackages based on version parameter + # If no version specified, use the default llvmPackages + llvmPackages = + if llvmVersion == null then + pkgs.llvmPackages + else + pkgs."llvmPackages_${toString llvmVersion}"; + + # Extract major version from LLVM version (e.g., "18.1.8" -> "18") + llvmMajorVersion = lib.versions.major llvmPackages.llvm.version; + + # Create a wrapper for llvm-config to include clang paths (for libclang) + # This is needed because the xdp2-compiler uses libclang and needs correct paths + llvm-config-wrapped = pkgs.runCommand "llvm-config-wrapped" { } '' + mkdir -p $out/bin + cat > $out/bin/llvm-config </lib/clang/ + # The libclang.lib path is incomplete (only has include, missing lib and share) + clangResourceDir = "${llvmPackages.clang}/resource-root"; + llvmLib = "${llvmPackages.llvm}/lib"; + libclangLib = "${llvmPackages.libclang.lib}/lib"; + }; + + # Environment variable exports (as shell script fragment) + envVars = '' + # LLVM/Clang version + export XDP2_CLANG_VERSION="$(${llvmPackages.llvm.dev}/bin/llvm-config --version)" + # Clang resource directory - use the wrapper's resource-root which has complete structure + # matching Ubuntu's /usr/lib/llvm-/lib/clang/ + export XDP2_CLANG_RESOURCE_PATH="${llvmPackages.clang}/resource-root" + # C include path for clang headers + export XDP2_C_INCLUDE_PATH="${llvmPackages.clang}/resource-root/include" + + # LLVM/Clang settings + export HOST_LLVM_CONFIG="${llvm-config-wrapped}/bin/llvm-config" + export LLVM_LIBS="-L${llvmPackages.llvm}/lib" + export CLANG_LIBS="-lclang -lLLVM -lclang-cpp" + + # libclang configuration + export LIBCLANG_PATH="${llvmPackages.libclang.lib}/lib" + ''; + + # LD_LIBRARY_PATH addition (separate to allow conditional use) + ldLibraryPath = "${llvmPackages.libclang.lib}/lib"; +} diff --git a/nix/packages.nix b/nix/packages.nix new file mode 100644 index 0000000..ae98c5a --- /dev/null +++ b/nix/packages.nix @@ -0,0 +1,101 @@ +# nix/packages.nix +# +# Package definitions for XDP2 +# +# This module defines all package dependencies, properly separated into: +# - nativeBuildInputs: Build-time tools (compilers, generators, etc.) +# - buildInputs: Libraries needed at build and runtime +# - devTools: Additional tools for development only +# +# Usage in flake.nix: +# packages = import ./nix/packages.nix { inherit pkgs llvmPackages; }; +# + +{ pkgs, llvmPackages }: + +let + # Create a Python environment with scapy for packet generation + pythonWithScapy = pkgs.python3.withPackages (ps: [ ps.scapy ]); +in +{ + # Build-time tools only - these run on the build machine + nativeBuildInputs = [ + # Build system + pkgs.gnumake + pkgs.pkg-config + pkgs.bison + pkgs.flex + + # Core utilities needed during build + pkgs.bash + pkgs.coreutils + pkgs.gnused + pkgs.gawk + pkgs.gnutar + pkgs.xz + pkgs.git + + # Compilers + pkgs.gcc + llvmPackages.clang + llvmPackages.llvm.dev # Provides llvm-config + llvmPackages.lld + ]; + + # Libraries needed at build and runtime + buildInputs = [ + # Core libraries + pkgs.boost + pkgs.libpcap + pkgs.libelf + pkgs.libbpf + + # Linux kernel headers (provides etc.) + pkgs.linuxHeaders + + # Python environment + pythonWithScapy + + # LLVM/Clang libraries (needed for xdp2-compiler) + llvmPackages.llvm + llvmPackages.libclang + llvmPackages.clang-unwrapped + ]; + + # Development-only tools (not needed for building, only for dev workflow) + devTools = [ + # Debugging + pkgs.gdb + pkgs.valgrind + pkgs.strace + pkgs.ltrace + pkgs.glibc_multi.bin + + # BPF/XDP development tools + pkgs.bpftools + pkgs.bpftrace # High-level tracing language for eBPF + pkgs.bcc # BPF Compiler Collection with Python bindings + pkgs.linuxPackages.perf # Linux performance analysis tool + pkgs.pahole # DWARF debugging info analyzer (useful for BTF) + + # Visualization + pkgs.graphviz + + # Code quality + pkgs.shellcheck + llvmPackages.clang-tools # clang-tidy, clang-format, etc. + + # Utilities + pkgs.jp2a # ASCII art for logo + pkgs.glibcLocales # Locale support + ]; + + # Combined list for dev shell (all packages) + # This replaces the old corePackages + allPackages = + let self = import ./packages.nix { inherit pkgs llvmPackages; }; + in self.nativeBuildInputs ++ self.buildInputs ++ self.devTools; + + # Export pythonWithScapy for use elsewhere + inherit pythonWithScapy; +} diff --git a/nix/patches/01-nix-clang-system-includes.patch b/nix/patches/01-nix-clang-system-includes.patch new file mode 100644 index 0000000..ebecdd9 --- /dev/null +++ b/nix/patches/01-nix-clang-system-includes.patch @@ -0,0 +1,100 @@ +# nix/patches/01-nix-clang-system-includes.patch +# +# Purpose: Add system include paths for Nix build environments +# +# When xdp2-compiler uses libclang's ClangTool API directly, it bypasses the +# Nix clang wrapper that normally sets up include paths. This causes header +# resolution failures and AST parse errors. +# +# This patch reads include paths from environment variables set by the Nix +# derivation and adds them as -isystem arguments to ClangTool. The environment +# variables are only set during `nix build`, so this is a no-op on Ubuntu/Fedora. +# +# Also adds optional debug diagnostic output (LLVM 21 compatible). +# +# See: documentation/nix/phase6_segfault_defect.md +# +diff --git a/src/tools/compiler/src/main.cpp b/src/tools/compiler/src/main.cpp +index ec547c4..426356a 100644 +--- a/src/tools/compiler/src/main.cpp ++++ b/src/tools/compiler/src/main.cpp +@@ -67,6 +67,7 @@ + // Clang + #include + #include ++#include // [nix-patch] For optional debug diagnostics + + // LLVM + #include +@@ -203,8 +204,6 @@ clang::tooling::ClangTool create_clang_tool( + // The actual resource directory is set via -resource-dir flag below. + plog::log(std::cout) + << "/usr/lib/clang/" << version << "/include" << std::endl; +- if (getenv("XDP2_C_INCLUDE_PATH")) +- setenv("C_INCLUDE_PATH", getenv("XDP2_C_INCLUDE_PATH"), 1); + + plog::log(std::cout) << "OptionsParser->getSourcePathList()" << std::endl; + for (auto &&item : OptionsParser->getSourcePathList()) +@@ -240,6 +239,43 @@ clang::tooling::ClangTool create_clang_tool( + } + #endif + ++ // [nix-patch] Add system include paths for Nix environments. ++ // ++ // PROBLEM: When using libclang/ClangTool directly (as xdp2-compiler does), ++ // we bypass the Nix clang wrapper script which normally adds -isystem flags ++ // for system headers. Without these flags, header resolution fails and the ++ // AST contains RecoveryExpr/contains-errors nodes, causing hasInit() to ++ // return unexpected values and ultimately leading to null pointer crashes. ++ // ++ // SOLUTION: Read include paths from environment variables set by the Nix ++ // derivation and add them as -isystem arguments. These env vars are only ++ // set during `nix build`, so this code is a no-op on traditional systems. ++ // ++ // Environment variables (set in nix/derivation.nix buildPhase): ++ // XDP2_C_INCLUDE_PATH: Clang builtins (stddef.h, stdint.h, etc.) ++ // XDP2_GLIBC_INCLUDE_PATH: glibc headers (stdlib.h, stdio.h, etc.) ++ // XDP2_LINUX_HEADERS_PATH: Linux kernel headers (, etc.) ++ // ++ // See: documentation/nix/phase6_segfault_defect.md for full details. ++ const char* clang_include = getenv("XDP2_C_INCLUDE_PATH"); ++ const char* glibc_include = getenv("XDP2_GLIBC_INCLUDE_PATH"); ++ const char* linux_headers = getenv("XDP2_LINUX_HEADERS_PATH"); ++ if (linux_headers) { ++ Tool.appendArgumentsAdjuster(clang::tooling::getInsertArgumentAdjuster( ++ {"-isystem", linux_headers}, ++ clang::tooling::ArgumentInsertPosition::BEGIN)); ++ } ++ if (glibc_include) { ++ Tool.appendArgumentsAdjuster(clang::tooling::getInsertArgumentAdjuster( ++ {"-isystem", glibc_include}, ++ clang::tooling::ArgumentInsertPosition::BEGIN)); ++ } ++ if (clang_include) { ++ Tool.appendArgumentsAdjuster(clang::tooling::getInsertArgumentAdjuster( ++ {"-isystem", clang_include}, ++ clang::tooling::ArgumentInsertPosition::BEGIN)); ++ } ++ + return Tool; + }; + +@@ -271,8 +307,18 @@ void parse_file(G &g, std::vector> &roots, + // Extract basic graph information + graph_info graph_consumed_data{ &g, &roots }; + ++ // [nix-patch] Optional diagnostic output for debugging AST parse errors. ++ // Enable by adding -DXDP2_COMPILER_DEBUG to CXXFLAGS in compiler Makefile. ++ // Uses LLVM 21+ compatible API (DiagnosticOptions by reference, not pointer). ++#ifdef XDP2_COMPILER_DEBUG ++ clang::DiagnosticOptions diagOpts; ++ diagOpts.ShowColors = true; ++ clang::TextDiagnosticPrinter diagPrinter(llvm::errs(), diagOpts); ++ Tool.setDiagnosticConsumer(&diagPrinter); ++#else + clang::IgnoringDiagConsumer diagConsumer; + Tool.setDiagnosticConsumer(&diagConsumer); ++#endif + + int action1 = + Tool.run(extract_graph_constants_factory(graph_consumed_data).get()); diff --git a/nix/patches/02-tentative-definition-null-check.patch b/nix/patches/02-tentative-definition-null-check.patch new file mode 100644 index 0000000..ec9dd0b --- /dev/null +++ b/nix/patches/02-tentative-definition-null-check.patch @@ -0,0 +1,59 @@ +# nix/patches/02-tentative-definition-null-check.patch +# +# Purpose: Fix null pointer crash with C tentative definitions +# +# C tentative definitions like "static const struct T name;" (created by +# XDP2_DECL_PROTO_TABLE macro) behave differently across clang versions: +# +# - Ubuntu clang 18.1.3: hasInit() returns false, these are skipped +# - Nix clang 18.1.8+: hasInit() returns true with void-type InitListExpr +# +# When getAs() is called on void type, it returns nullptr. +# The original code calls ->getDecl() on this nullptr, causing a segfault. +# +# This patch adds a null check and skips tentative definitions gracefully. +# The actual definition is processed when encountered later in the AST. +# +# See: documentation/nix/phase6_segfault_defect.md +# +diff --git a/src/tools/compiler/include/xdp2gen/ast-consumer/proto-tables.h b/src/tools/compiler/include/xdp2gen/ast-consumer/proto-tables.h +index 6616cf7..3aaa7c5 100644 +--- a/src/tools/compiler/include/xdp2gen/ast-consumer/proto-tables.h ++++ b/src/tools/compiler/include/xdp2gen/ast-consumer/proto-tables.h +@@ -89,11 +89,32 @@ public: + clang::dyn_cast( + initializer_expr); + ++ // [nix-patch] Handle tentative definitions to prevent null pointer crash. ++ // ++ // PROBLEM: C tentative definitions like: ++ // static const struct xdp2_proto_table ip_table; ++ // are created by XDP2_DECL_PROTO_TABLE macro before the actual definition. ++ // ++ // Different clang versions handle hasInit() differently for these: ++ // - Ubuntu clang 18.1.3: hasInit() returns false (skipped entirely) ++ // - Nix clang 18.1.8+: hasInit() returns true with void-type InitListExpr ++ // ++ // When getAs() is called on void type, it returns nullptr. ++ // The original code then calls ->getDecl() on nullptr, causing segfault. ++ // ++ // SOLUTION: Check if RecordType is null and skip tentative definitions. ++ // The actual definition will be processed when encountered later in the AST. ++ // ++ // See: documentation/nix/phase6_segfault_defect.md for full investigation. ++ clang::QualType initType = initializer_list_expr->getType(); ++ auto *recordType = initType->getAs(); ++ if (!recordType) { ++ // Skip tentative definitions - actual definition processed later ++ return true; ++ } ++ + // Extracts current analyzed InitListDecl +- clang::RecordDecl *initializer_list_decl = +- initializer_list_expr->getType() +- ->getAs() +- ->getDecl(); ++ clang::RecordDecl *initializer_list_decl = recordType->getDecl(); + + // Proto table consumed infos + xdp2_proto_table_extract_data table_data; diff --git a/nix/shell-functions/ascii-art.nix b/nix/shell-functions/ascii-art.nix new file mode 100644 index 0000000..34f4fc9 --- /dev/null +++ b/nix/shell-functions/ascii-art.nix @@ -0,0 +1,21 @@ +# nix/shell-functions/ascii-art.nix +# +# ASCII art logo display for XDP2 development shell +# +# Uses jp2a to convert the XDP2 logo to colored ASCII art. +# Falls back to a simple text banner if jp2a is not available. +# +# Usage in devshell.nix: +# asciiArt = import ./shell-functions/ascii-art.nix { }; +# + +{ }: + +'' + if command -v jp2a >/dev/null 2>&1 && [ -f "./documentation/images/xdp2-big.png" ]; then + echo "$(jp2a --colors ./documentation/images/xdp2-big.png)" + echo "" + else + echo "🚀 === XDP2 Development Shell ===" + fi +'' diff --git a/nix/shell-functions/build.nix b/nix/shell-functions/build.nix new file mode 100644 index 0000000..c7c93ab --- /dev/null +++ b/nix/shell-functions/build.nix @@ -0,0 +1,373 @@ +# nix/shell-functions/build.nix +# +# Build shell functions for XDP2 +# +# Functions: +# - build-cppfront: Build the cppfront compiler +# - check-cppfront-age: Check if cppfront needs rebuilding +# - build-xdp2-compiler: Build the xdp2 compiler +# - build-xdp2: Build the main xdp2 project +# - build-all: Build all components in order +# +# Note: These functions depend on navigation and clean functions +# +# Usage in flake.nix: +# buildFns = import ./nix/shell-functions/build.nix { }; +# + +{ }: + +'' + # Build the cppfront compiler + build-cppfront() { + if [ -n "$XDP2_NIX_DEBUG" ]; then + local debug_level=$XDP2_NIX_DEBUG + else + local debug_level=0 + fi + local start_time="" + local end_time="" + + if [ "$debug_level" -gt 3 ]; then + start_time=$(date +%s) + echo "[DEBUG] build-cppfront started at $(date)" + fi + + # Level 1: Function start + if [ "$debug_level" -ge 1 ]; then + echo "[DEBUG] Starting build-cppfront function" + fi + + # Clean + if [ "$debug_level" -ge 2 ]; then + echo "[DEBUG] Cleaning cppfront build directory" + fi + echo "Cleaning and building cppfront-compiler..." + + # Navigate to repository root first + navigate-to-repo-root || return 1 + + # Clean previous build artifacts (before navigating to component) + clean-cppfront + + # Navigate to cppfront directory + navigate-to-component "thirdparty/cppfront" || return 1 + + # Apply essential header fix for cppfront + if [ "$debug_level" -ge 3 ]; then + echo "[DEBUG] Applying cppfront header fix" + printf "sed -i '1i#include \\n#include \\n' include/cpp2util.h\n" + fi + sed -i '1i#include \n#include \n' include/cpp2util.h + + # Level 3: Build step details + if [ "$debug_level" -ge 3 ]; then + echo "[DEBUG] Building cppfront-compiler with make" + fi + + # Build cppfront with error checking + if HOST_CXX="$CXX" HOST_CC="$CC" make -j"$NIX_BUILD_CORES"; then + echo "✓ cppfront make completed successfully" + else + echo "✗ ERROR: cppfront make failed" + return 1 + fi + + # Return to repository root + navigate-to-repo-root || return 1 + + # Add to the PATH + add-to-path "$PWD/thirdparty/cppfront" + + # Level 2: Validation step + if [ "$debug_level" -ge 2 ]; then + echo "[DEBUG] Validating cppfront-compiler binary" + fi + + # Validate binary was created + if [ -x "./thirdparty/cppfront/cppfront-compiler" ]; then + echo "✓ cppfront-compiler binary created and executable" + + # Test the binary runs correctly + echo "Testing cppfront-compiler..." + set +e # Temporarily disable exit on error + + # Debug output for validation command + if [ "$debug_level" -gt 3 ]; then + echo "[DEBUG] About to run: ./thirdparty/cppfront/cppfront-compiler -version" + fi + ./thirdparty/cppfront/cppfront-compiler -version + test_exit_code=$? + set -e # Re-enable exit on error + + if [ "$test_exit_code" -eq 0 ] || [ "$test_exit_code" -eq 1 ]; then + echo "✓ cppfront-compiler runs correctly (exit code: $test_exit_code)" + else + echo "⚠ WARNING: cppfront-compiler returned unexpected exit code: $test_exit_code" + echo "But binary exists and is executable, continuing..." + fi + else + echo "✗ ERROR: cppfront-compiler binary not found or not executable" + return 1 + fi + + # End timing for debug levels > 3 + if [ "$debug_level" -gt 3 ]; then + end_time=$(date +%s) + local duration=$((end_time - start_time)) + echo "[DEBUG] build-cppfront completed in $duration seconds" + fi + + echo "cppfront-compiler built and validated successfully ( ./thirdparty/cppfront/cppfront-compiler )" + } + + # Check if cppfront needs rebuilding based on age + check-cppfront-age() { + if [ -n "$XDP2_NIX_DEBUG" ]; then + local debug_level=$XDP2_NIX_DEBUG + else + local debug_level=0 + fi + local start_time="" + local end_time="" + + # Start timing for debug levels > 3 + if [ "$debug_level" -gt 3 ]; then + start_time=$(date +%s) + echo "[DEBUG] check-cppfront-age started at $(date)" + fi + + # Level 1: Function start + if [ "$debug_level" -ge 1 ]; then + echo "[DEBUG] Starting check-cppfront-age function" + fi + + local cppfront_binary="thirdparty/cppfront/cppfront-compiler" + + # Level 2: File check + if [ "$debug_level" -ge 2 ]; then + echo "[DEBUG] Checking cppfront binary: $cppfront_binary" + fi + + if [ -f "$cppfront_binary" ]; then + local file_time + file_time=$(stat -c %Y "$cppfront_binary") + local current_time + current_time=$(date +%s) + local age_days=$(( (current_time - file_time) / 86400 )) + + # Level 3: Age calculation details + if [ "$debug_level" -ge 3 ]; then + echo "[DEBUG] File modification time: $file_time" + echo "[DEBUG] Current time: $current_time" + echo "[DEBUG] Calculated age: $age_days days" + fi + + if [ "$age_days" -gt 7 ]; then + echo "cppfront is $age_days days old, rebuilding..." + build-cppfront + else + echo "cppfront is up to date ($age_days days old)" + fi + else + echo "cppfront not found, building..." + build-cppfront + fi + + # End timing for debug levels > 3 + if [ "$debug_level" -gt 3 ]; then + end_time=$(date +%s) + local duration=$((end_time - start_time)) + echo "[DEBUG] check-cppfront-age completed in $duration seconds" + fi + } + + # Build the xdp2 compiler + build-xdp2-compiler() { + if [ -n "$XDP2_NIX_DEBUG" ]; then + local debug_level=$XDP2_NIX_DEBUG + else + local debug_level=0 + fi + local start_time="" + local end_time="" + + # Start timing for debug levels > 3 + if [ "$debug_level" -gt 3 ]; then + start_time=$(date +%s) + echo "[DEBUG] build-xdp2-compiler started at $(date)" + fi + + # Level 1: Function start + if [ "$debug_level" -ge 1 ]; then + echo "[DEBUG] Starting build-xdp2-compiler function" + fi + + # Level 2: Clean step + if [ "$debug_level" -ge 2 ]; then + echo "[DEBUG] Cleaning xdp2-compiler build directory" + fi + echo "Cleaning and building xdp2-compiler..." + + # Navigate to repository root first + navigate-to-repo-root || return 1 + + # Clean previous build artifacts (before navigating to component) + clean-xdp2-compiler + + # Navigate to xdp2-compiler directory + navigate-to-component "src/tools/compiler" || return 1 + + # Level 3: Build step details + if [ "$debug_level" -ge 3 ]; then + echo "[DEBUG] Building xdp2-compiler with make" + fi + + # Build xdp2-compiler with error checking + if CFLAGS_PYTHON="$CFLAGS_PYTHON" LDFLAGS_PYTHON="$LDFLAGS_PYTHON" make -j"$NIX_BUILD_CORES"; then + echo "✓ xdp2-compiler make completed successfully" + else + echo "✗ ERROR: xdp2-compiler make failed" + return 1 + fi + + # Level 2: Validation step + if [ "$debug_level" -ge 2 ]; then + echo "[DEBUG] Validating xdp2-compiler binary" + fi + + # Validate binary was created + if [ -x "./xdp2-compiler" ]; then + echo "✓ xdp2-compiler binary created and executable" + + # Test the binary runs correctly + echo "Testing xdp2-compiler..." + set +e # Temporarily disable exit on error + + # Debug output for validation command + if [ "$debug_level" -gt 3 ]; then + echo "[DEBUG] About to run: ./xdp2-compiler --help" + fi + ./xdp2-compiler --help + test_exit_code=$? + set -e # Re-enable exit on error + + if [ "$test_exit_code" -eq 0 ] || [ "$test_exit_code" -eq 1 ]; then + echo "✓ xdp2-compiler runs correctly (exit code: $test_exit_code)" + else + echo "⚠ WARNING: xdp2-compiler returned unexpected exit code: $test_exit_code" + echo "But binary exists and is executable, continuing..." + fi + else + echo "✗ ERROR: xdp2-compiler binary not found or not executable" + return 1 + fi + + # Return to repository root + navigate-to-repo-root || return 1 + + # End timing for debug levels > 3 + if [ "$debug_level" -gt 3 ]; then + end_time=$(date +%s) + local duration=$((end_time - start_time)) + echo "[DEBUG] build-xdp2-compiler completed in $duration seconds" + fi + + echo "xdp2-compiler built and validated successfully ( ./src/tools/compiler/xdp2-compiler )" + } + + # Build the main xdp2 project + build-xdp2() { + if [ -n "$XDP2_NIX_DEBUG" ]; then + local debug_level=$XDP2_NIX_DEBUG + else + local debug_level=0 + fi + local start_time="" + local end_time="" + + # Start timing for debug levels > 3 + if [ "$debug_level" -gt 3 ]; then + start_time=$(date +%s) + echo "[DEBUG] build-xdp2 started at $(date)" + fi + + # Level 1: Function start + if [ "$debug_level" -ge 1 ]; then + echo "[DEBUG] Starting build-xdp2 function" + fi + + # Level 2: Clean step + if [ "$debug_level" -ge 2 ]; then + echo "[DEBUG] Cleaning xdp2 project build directory" + fi + echo "Cleaning and building xdp2 project..." + + # Navigate to repository root first + navigate-to-repo-root || return 1 + + # Clean previous build artifacts (before navigating to component) + clean-xdp2 + + # Navigate to src directory + navigate-to-component "src" || return 1 + + # Ensure xdp2-compiler is available in PATH + add-to-path "$PWD/tools/compiler" + echo "Added tools/compiler to PATH" + + # Level 3: Build step details + if [ "$debug_level" -ge 3 ]; then + echo "[DEBUG] Building xdp2 project with make" + fi + + # Build the main xdp2 project + if make -j"$NIX_BUILD_CORES"; then + echo "✓ xdp2 project make completed successfully" + else + echo "✗ ERROR: xdp2 project make failed" + echo " Check the error messages above for details" + return 1 + fi + + # Return to repository root + navigate-to-repo-root || return 1 + + # End timing for debug levels > 3 + if [ "$debug_level" -gt 3 ]; then + end_time=$(date +%s) + local duration=$((end_time - start_time)) + echo "[DEBUG] build-xdp2 completed in $duration seconds" + fi + + echo "xdp2 project built successfully" + } + + # Build all XDP2 components + build-all() { + if [ -n "$XDP2_NIX_DEBUG" ]; then + local debug_level=$XDP2_NIX_DEBUG + else + local debug_level=0 + fi + + echo "Building all XDP2 components..." + + if [ "$debug_level" -ge 3 ]; then + echo "[DEBUG] Building cppfront: build-cppfront" + fi + build-cppfront + + if [ "$debug_level" -ge 3 ]; then + echo "[DEBUG] Building xdp2-compiler: build-xdp2-compiler" + fi + build-xdp2-compiler + + if [ "$debug_level" -ge 3 ]; then + echo "[DEBUG] Building xdp2: build-xdp2" + fi + build-xdp2 + + echo "✓ All components built successfully" + } +'' diff --git a/nix/shell-functions/clean.nix b/nix/shell-functions/clean.nix new file mode 100644 index 0000000..9c8e9dd --- /dev/null +++ b/nix/shell-functions/clean.nix @@ -0,0 +1,103 @@ +# nix/shell-functions/clean.nix +# +# Clean shell functions for XDP2 +# +# Functions: +# - clean-cppfront: Clean cppfront build artifacts +# - clean-xdp2-compiler: Clean xdp2-compiler build artifacts +# - clean-xdp2: Clean xdp2 build artifacts +# - clean-all: Clean all build artifacts +# +# Note: These functions depend on navigation functions from navigation.nix +# +# Usage in flake.nix: +# cleanFns = import ./nix/shell-functions/clean.nix { }; +# + +{ }: + +'' + # Clean cppfront build artifacts + clean-cppfront() { + if [ -n "$XDP2_NIX_DEBUG" ]; then + local debug_level=$XDP2_NIX_DEBUG + else + local debug_level=0 + fi + + # Navigate to repository root first + navigate-to-repo-root || return 1 + + # Navigate to cppfront directory + navigate-to-component "thirdparty/cppfront" || return 1 + + # Debug output for clean command + if [ "$debug_level" -gt 3 ]; then + echo "[DEBUG] About to run: rm -f cppfront-compiler" + fi + rm -f cppfront-compiler # Remove the binary directly since Makefile has no clean target + + # Return to repository root + navigate-to-repo-root || return 1 + } + + # Clean xdp2-compiler build artifacts + clean-xdp2-compiler() { + if [ -n "$XDP2_NIX_DEBUG" ]; then + local debug_level=$XDP2_NIX_DEBUG + else + local debug_level=0 + fi + + # Navigate to repository root first + navigate-to-repo-root || return 1 + + # Navigate to xdp2-compiler directory + navigate-to-component "src/tools/compiler" || return 1 + + # Debug output for clean command + if [ "$debug_level" -gt 3 ]; then + echo "[DEBUG] About to run: make clean" + fi + make clean || true # Don't fail if clean fails + + # Return to repository root + navigate-to-repo-root || return 1 + } + + # Clean xdp2 build artifacts + clean-xdp2() { + if [ -n "$XDP2_NIX_DEBUG" ]; then + local debug_level=$XDP2_NIX_DEBUG + else + local debug_level=0 + fi + + # Navigate to repository root first + navigate-to-repo-root || return 1 + + # Navigate to src directory + navigate-to-component "src" || return 1 + + # Debug output for clean command + if [ "$debug_level" -gt 3 ]; then + echo "[DEBUG] About to run: make clean" + fi + make clean || true # Don't fail if clean fails + + # Return to repository root + navigate-to-repo-root || return 1 + } + + # Clean all build artifacts + clean-all() { + echo "Cleaning all build artifacts..." + + # Clean each component using centralized clean functions + clean-cppfront + clean-xdp2-compiler + clean-xdp2 + + echo "✓ All build artifacts cleaned" + } +'' diff --git a/nix/shell-functions/configure.nix b/nix/shell-functions/configure.nix new file mode 100644 index 0000000..01c6c3b --- /dev/null +++ b/nix/shell-functions/configure.nix @@ -0,0 +1,62 @@ +# nix/shell-functions/configure.nix +# +# Configure shell functions for XDP2 +# +# Functions: +# - smart-configure: Smart configure script with age checking +# +# Note: These functions depend on navigation functions from navigation.nix +# +# Usage in flake.nix: +# configureFns = import ./nix/shell-functions/configure.nix { configAgeWarningDays = 14; }; +# + +{ configAgeWarningDays ? 14 }: + +'' + # Smart configure script execution with age checking + # This simply includes a check to see if the config.mk file exists, and + # it generates a warning if the file is older than the threshold + smart-configure() { + local config_file="./src/config.mk" + local warning_days=${toString configAgeWarningDays} + + if [ -f "$config_file" ]; then + echo "✓ config.mk found, skipping configure step" + + # Check age of config.mk + local file_time + file_time=$(stat -c %Y "$config_file") + local current_time + current_time=$(date +%s) + local age_days=$(( (current_time - file_time) / 86400 )) + + if [ "$age_days" -gt "$warning_days" ]; then + echo "⚠️ WARNING: config.mk is $age_days days old (threshold: $warning_days days)" + echo " Consider running 'configure' manually if you've made changes to:" + echo " • Build configuration" + echo " • Compiler settings" + echo " • Library paths" + echo " • Platform-specific settings" + echo "" + else + echo "✓ config.mk is up to date ($age_days days old)" + fi + else + echo "config.mk not found, running configure script..." + cd src || return 1 + rm -f config.mk + ./configure.sh --build-opt-parser + + # Apply PATH_ARG fix for Nix environment + if grep -q 'PATH_ARG="--with-path=' config.mk; then + echo "Applying PATH_ARG fix for Nix environment..." + sed -i 's|PATH_ARG="--with-path=.*"|PATH_ARG=""|' config.mk + fi + echo "PATH_ARG in config.mk: $(grep '^PATH_ARG=' config.mk)" + + cd .. || return 1 + echo "✓ config.mk generated successfully" + fi + } +'' diff --git a/nix/shell-functions/navigation.nix b/nix/shell-functions/navigation.nix new file mode 100644 index 0000000..7e325a6 --- /dev/null +++ b/nix/shell-functions/navigation.nix @@ -0,0 +1,84 @@ +# nix/shell-functions/navigation.nix +# +# Navigation shell functions for XDP2 +# +# Functions: +# - detect-repository-root: Detect and export XDP2_REPO_ROOT +# - navigate-to-repo-root: Change to repository root directory +# - navigate-to-component: Change to a component subdirectory +# - add-to-path: Add a directory to PATH if not already present +# +# Usage in flake.nix: +# navigationFns = import ./nix/shell-functions/navigation.nix { }; +# + +{ }: + +'' + # Detect and export the repository root directory + detect-repository-root() { + XDP2_REPO_ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd) + export XDP2_REPO_ROOT + + if [ ! -d "$XDP2_REPO_ROOT" ]; then + echo "⚠ WARNING: Could not detect valid repository root" + XDP2_REPO_ROOT="$PWD" + else + echo "📁 Repository root: $XDP2_REPO_ROOT" + fi + } + + # Navigate to the repository root directory + navigate-to-repo-root() { + if [ -n "$XDP2_REPO_ROOT" ]; then + cd "$XDP2_REPO_ROOT" || return 1 + else + echo "✗ ERROR: XDP2_REPO_ROOT not set" + return 1 + fi + } + + # Navigate to a component subdirectory + navigate-to-component() { + local component="$1" + local target_dir="$XDP2_REPO_ROOT/$component" + + if [ ! -d "$target_dir" ]; then + echo "✗ ERROR: Component directory not found: $target_dir" + return 1 + fi + + cd "$target_dir" || return 1 + } + + # Add path to PATH environment variable if not already present + add-to-path() { + local path_to_add="$1" + + if [ -n "$XDP2_NIX_DEBUG" ]; then + local debug_level=$XDP2_NIX_DEBUG + else + local debug_level=0 + fi + + # Check if path is already in PATH + if [[ ":$PATH:" == *":$path_to_add:"* ]]; then + if [ "$debug_level" -gt 3 ]; then + echo "[DEBUG] Path already in PATH: $path_to_add" + fi + return 0 + fi + + # Add path to beginning of PATH + if [ "$debug_level" -gt 3 ]; then + echo "[DEBUG] Adding to PATH: $path_to_add" + echo "[DEBUG] PATH before: $PATH" + fi + + export PATH="$path_to_add:$PATH" + + if [ "$debug_level" -gt 3 ]; then + echo "[DEBUG] PATH after: $PATH" + fi + } +'' diff --git a/nix/shell-functions/validation.nix b/nix/shell-functions/validation.nix new file mode 100644 index 0000000..04b1f94 --- /dev/null +++ b/nix/shell-functions/validation.nix @@ -0,0 +1,179 @@ +# nix/shell-functions/validation.nix +# +# Validation and help shell functions for XDP2 +# +# Functions: +# - check-platform-compatibility: Check if running on Linux +# - setup-locale-support: Configure locale settings +# - run-shellcheck: Validate all shell functions with shellcheck +# - xdp2-help: Display help information +# +# Usage in flake.nix: +# validationFns = import ./nix/shell-functions/validation.nix { +# inherit lib; +# shellcheckFunctionRegistry = [ "func1" "func2" ... ]; +# }; +# + +{ lib, shellcheckFunctionRegistry ? [] }: + +let + # Generate shellcheck validation function + totalFunctions = builtins.length shellcheckFunctionRegistry; + + # Generate individual function checks + functionChecks = lib.concatStringsSep "\n" (map (name: '' + echo "Checking ${name}..." + if declare -f "${name}" >/dev/null 2>&1; then + # Create temporary script with function definition + # TODO use mktemp and trap 'rm -f "$temp_script"' EXIT + local temp_script="/tmp/validate_${name}.sh" + declare -f "${name}" > "$temp_script" + echo "#!/bin/bash" > "$temp_script.tmp" + cat "$temp_script" >> "$temp_script.tmp" + mv "$temp_script.tmp" "$temp_script" + + # Run shellcheck on the function + if shellcheck -s bash "$temp_script" 2>/dev/null; then + echo "✓ ${name} passed shellcheck validation" + passed_functions=$((passed_functions + 1)) + else + echo "✗ ${name} failed shellcheck validation:" + shellcheck -s bash "$temp_script" + failed_functions+=("${name}") + fi + rm -f "$temp_script" + else + echo "✗ ${name} not found" + failed_functions+=("${name}") + fi + echo "" + '') shellcheckFunctionRegistry); + + # Generate failed functions reporting + failedFunctionsReporting = lib.concatStringsSep "\n" (map (name: '' + if [[ "$${failed_functions[*]}" == *"${name}"* ]]; then + echo " - ${name}" + fi + '') shellcheckFunctionRegistry); + + # Bash variable expansion helpers for setup-locale-support + bashVarExpansion = "$"; + bashDefaultSyntax = "{LANG:-C.UTF-8}"; + bashDefaultSyntaxLC = "{LC_ALL:-C.UTF-8}"; + +in +'' + # Check platform compatibility (Linux only) + check-platform-compatibility() { + if [ "$(uname)" != "Linux" ]; then + echo "⚠️ PLATFORM COMPATIBILITY NOTICE +================================== + +🍎 You are running on $(uname) (not Linux) + +The XDP2 development environment includes Linux-specific packages +like libbpf that are not available on $(uname) systems. + +📋 Available platforms: + ✅ Linux (x86_64-linux, aarch64-linux, etc.) + ❌ macOS (x86_64-darwin, aarch64-darwin) + ❌ Other Unix systems + +Exiting development shell..." + exit 1 + fi + } + + # Setup locale support for cross-distribution compatibility + setup-locale-support() { + # Only set locale if user hasn't already configured it + if [ -z "$LANG" ] || [ -z "$LC_ALL" ]; then + # Try to use system default, fallback to C.UTF-8 + export LANG=${bashVarExpansion}${bashDefaultSyntax} + export LC_ALL=${bashVarExpansion}${bashDefaultSyntaxLC} + fi + + # Verify locale is available (only if locale command exists) + if command -v locale >/dev/null 2>&1; then + if ! locale -a 2>/dev/null | grep -q "$LANG"; then + # Fallback to C.UTF-8 if user's locale is not available + export LANG=C.UTF-8 + export LC_ALL=C.UTF-8 + fi + fi + } + + # Run shellcheck validation on all registered shell functions + run-shellcheck() { + if [ -n "$XDP2_NIX_DEBUG" ]; then + local debug_level=$XDP2_NIX_DEBUG + else + local debug_level=0 + fi + + echo "Running shellcheck validation on shell functions..." + + local failed_functions=() + local total_functions=${toString totalFunctions} + local passed_functions=0 + + # Pre-generated function checks from Nix + ${functionChecks} + + # Report results + echo "=== Shellcheck Validation Complete ===" + echo "Total functions: $total_functions" + echo "Passed: $passed_functions" + echo "Failed: $((total_functions - passed_functions))" + + if [ $((total_functions - passed_functions)) -eq 0 ]; then + echo "✓ All functions passed shellcheck validation" + return 0 + else + echo "✗ Some functions failed validation:" + # Pre-generated failed functions reporting from Nix + ${failedFunctionsReporting} + return 1 + fi + } + + # Display help information for XDP2 development shell + xdp2-help() { + echo "🚀 === XDP2 Development Shell Help === + +📦 Compiler: GCC +🔧 GCC and Clang are available in the environment. +🐛 Debugging tools: gdb, valgrind, strace, ltrace + +🔍 DEBUGGING: + XDP2_NIX_DEBUG=0 - No extra debug. Default + XDP2_NIX_DEBUG=3 - Basic debug + XDP2_NIX_DEBUG=5 - Show compiler selection and config.mk + XDP2_NIX_DEBUG=7 - Show all debug info + +🔧 BUILD COMMANDS: + build-cppfront - Build cppfront compiler + build-xdp2-compiler - Build xdp2 compiler + build-xdp2 - Build main XDP2 project + build-all - Build all components + +🧹 CLEAN COMMANDS: + clean-cppfront - Clean cppfront build artifacts + clean-xdp2-compiler - Clean xdp2-compiler build artifacts + clean-xdp2 - Clean xdp2 build artifacts + clean-all - Clean all build artifacts + +🔍 VALIDATION: + run-shellcheck - Validate all shell functions + +📁 PROJECT STRUCTURE: + • src/ - Main source code + • tools/ - Build tools and utilities + • thirdparty/ - Third-party dependencies + • samples/ - Example code and parsers + • documentation/ - Project documentation + +🎯 Ready to develop! 'xdp2-help' for help" + } +'' diff --git a/src/configure.sh b/src/configure.sh index 9caf6b5..298f36a 100755 --- a/src/configure.sh +++ b/src/configure.sh @@ -769,8 +769,14 @@ fi echo "XDP2_CLANG_VERSION=${XDP2_CLANG_VERSION}" echo "XDP2_CLANG_VERSION=${XDP2_CLANG_VERSION}" >> $CONFIG -XDP2_CLANG_RESOURCE_PATH="`${HOST_LLVM_CONFIG} --libdir`/clang/`echo ${XDP2_CLANG_VERSION} | cut -d'.' -f1`" -XDP2_C_INCLUDE_PATH="`${HOST_LLVM_CONFIG} --libdir`/clang/`echo ${XDP2_CLANG_VERSION} | cut -d'.' -f1`/include" + +# Check for environment variable overrides (useful for Nix which has different path structure) +if [ -z "${XDP2_CLANG_RESOURCE_PATH}" ]; then + XDP2_CLANG_RESOURCE_PATH="`${HOST_LLVM_CONFIG} --libdir`/clang/`echo ${XDP2_CLANG_VERSION} | cut -d'.' -f1`" +fi +if [ -z "${XDP2_C_INCLUDE_PATH}" ]; then + XDP2_C_INCLUDE_PATH="${XDP2_CLANG_RESOURCE_PATH}/include" +fi echo "XDP2_C_INCLUDE_PATH=${XDP2_C_INCLUDE_PATH}" echo "XDP2_C_INCLUDE_PATH=${XDP2_C_INCLUDE_PATH}" >> $CONFIG echo "XDP2_CLANG_RESOURCE_PATH=${XDP2_CLANG_RESOURCE_PATH}"