diff --git a/documentation/nix/nix.md b/documentation/nix/nix.md index d5ffeb2..31fc713 100644 --- a/documentation/nix/nix.md +++ b/documentation/nix/nix.md @@ -11,7 +11,7 @@ The goals of using Nix in this repository are to: > ⚠️ **Linux only:** This flake currently supports **Linux only** because [`libbpf`](https://github.com/NixOS/nixpkgs/blob/nixos-unstable/pkgs/os-specific/linux/libbpf/default.nix) is Linux-specific. -Feedback and merge requests are welcome. If we're missing a tool, please open an issue or PR. See the `corePackages` section in `flake.nix`. +Feedback and merge requests are welcome. If we're missing a tool, please open an issue or PR. See `nix/packages.nix` for package definitions. --- @@ -63,12 +63,76 @@ Feedback and merge requests are welcome. If we're missing a tool, please open an ### What This Repository Provides -This repository includes `flake.nix` and `flake.lock`: -- **`flake.nix`** defines the development environment (compilers, libraries, tools, helper functions) -- **`flake.lock`** pins exact versions so all developers use **identical** inputs +This repository includes `flake.nix`, `flake.lock`, and modular Nix files in `nix/`: + +- **`flake.nix`** - Main entry point (~90 lines), imports modules from `nix/` +- **`flake.lock`** - Pins exact versions so all developers use **identical** inputs +- **`nix/packages.nix`** - Package definitions (nativeBuildInputs, buildInputs, devTools) +- **`nix/llvm.nix`** - LLVM/Clang configuration with wrapped llvm-config +- **`nix/env-vars.nix`** - Environment variable exports +- **`nix/devshell.nix`** - Development shell configuration +- **`nix/derivation.nix`** - Package derivation for `nix build` +- **`nix/patches/`** - Source code patches for Nix compatibility (see [Nix-Specific Patches](#nix-specific-patches)) +- **`nix/shell-functions/`** - Modular shell functions (build, clean, configure, etc.) Running `nix develop` spawns a shell with the correct toolchains, libraries, and environment variables configured for you. +### Current Toolchain Versions + +The Nix development environment provides the following tool and library versions (as of February 2026): + +| Package | Version | Purpose | +|---------|---------|---------| +| GCC | 15.2.0 | Primary C/C++ compiler for final libraries | +| LLVM/Clang | 21.1.8 | Host compiler for xdp2-compiler, AST parsing | +| Boost | 1.87.0 | C++ libraries (graph, wave, program_options) | +| libbpf | 1.5.0 | BPF library for eBPF/XDP programs | +| libelf | 0.192 | ELF file handling | +| libpcap | 1.10.5 | Packet capture library | +| Python | 3.13.3 | Scripting and packet generation (with scapy) | + +### Nix-Specific Patches + +The Nix build uses patches to handle differences between traditional Linux distributions and the Nix build environment. These patches are located in `nix/patches/` and are automatically applied during `nix build`. + +**Why patches?** When using libclang's `ClangTool` API directly (as xdp2-compiler does), it bypasses the Nix clang wrapper that normally sets up include paths. Additionally, different clang versions handle certain C constructs differently. The patches address these Nix-specific issues without modifying the original source code. + +#### Patch 1: System Include Paths (`01-nix-clang-system-includes.patch`) + +**Problem:** ClangTool bypasses the Nix clang wrapper script which normally adds `-isystem` flags for system headers. Without these flags, header resolution fails and the AST contains error nodes. + +**Solution:** Reads include paths from environment variables set by the Nix derivation and adds them as `-isystem` arguments to ClangTool. These environment variables are only set during `nix build`, so this is a no-op on Ubuntu/Fedora. + +Environment variables used: +- `XDP2_C_INCLUDE_PATH`: Clang builtins (stddef.h, stdint.h, etc.) +- `XDP2_GLIBC_INCLUDE_PATH`: glibc headers (stdlib.h, stdio.h, etc.) +- `XDP2_LINUX_HEADERS_PATH`: Linux kernel headers (, etc.) + +#### Patch 2: Tentative Definition Null Check (`02-tentative-definition-null-check.patch`) + +**Problem:** C tentative definitions like `static const struct T name;` (created by `XDP2_DECL_PROTO_TABLE` macro) behave differently across clang versions: +- Ubuntu clang 18.1.3: `hasInit()` returns false, these are skipped +- Nix clang 18.1.8+: `hasInit()` returns true with void-type InitListExpr + +When `getAs()` is called on void type, it returns nullptr, causing a segfault. + +**Solution:** Adds a null check and skips tentative definitions gracefully. The actual definition is processed when encountered later in the AST. + +For detailed investigation notes, see [phase6_segfault_defect.md](phase6_segfault_defect.md). + +### BPF/XDP Development Tools + +The development shell includes additional tools for BPF/XDP development and debugging: + +| Tool | Purpose | +|------|---------| +| `bpftools` | BPF program inspection and manipulation | +| `bpftrace` | High-level tracing language for eBPF | +| `bcc` | BPF Compiler Collection with Python bindings | +| `perf` | Linux performance analysis tool | +| `pahole` | DWARF debugging info analyzer (useful for BTF) | +| `clang-tools` | clang-tidy, clang-format, and other code quality tools | + --- ## Quick Start @@ -161,13 +225,26 @@ After running `build-all` once, the necessary changes will have been applied to ### Debugging nix develop -The `flake.nix` and embedded bash code make use of the environment variable `XDP2_NIX_DEBUG`. This variable uses syslog levels between 0 (default) and 7. +The shell functions use the environment variable `XDP2_NIX_DEBUG` at runtime. This variable uses syslog-style levels between 0 (default) and 7. + +Debug levels: +- **0** - No debug output (default) +- **3** - Basic debug info +- **5** - Show compiler selection and config.mk details +- **7** - Maximum verbosity (all debug info) For maximum debugging: ```bash XDP2_NIX_DEBUG=7 nix develop --verbose --print-build-logs ``` +You can also set the debug level after entering the shell: +```bash +nix develop +export XDP2_NIX_DEBUG=5 +build-all # Will show debug output +``` + ### Shellcheck The `flake.nix` checks all the bash code within the flake to ensure there are no issues. diff --git a/flake.lock b/flake.lock index 4d1227e..923e9f1 100644 --- a/flake.lock +++ b/flake.lock @@ -20,11 +20,11 @@ }, "nixpkgs": { "locked": { - "lastModified": 1758427187, - "narHash": "sha256-pHpxZ/IyCwoTQPtFIAG2QaxuSm8jWzrzBGjwQZIttJc=", + "lastModified": 1770562336, + "narHash": "sha256-ub1gpAONMFsT/GU2hV6ZWJjur8rJ6kKxdm9IlCT0j84=", "owner": "NixOS", "repo": "nixpkgs", - "rev": "554be6495561ff07b6c724047bdd7e0716aa7b46", + "rev": "d6c71932130818840fc8fe9509cf50be8c64634f", "type": "github" }, "original": { diff --git a/flake.nix b/flake.nix index ef57bc6..a922ab3 100644 --- a/flake.nix +++ b/flake.nix @@ -1,15 +1,19 @@ # -# flake.nix for XDP2 - Development Shell Only +# flake.nix for XDP2 # -# WARNING - THIS FLAKE IS CURRENTLY BROKEN (2025/11/06) FIXES COMING SOON -# -# This flake.nix provides a fast development environment for the XDP2 project +# This flake provides: +# - Development environment: nix develop +# - Package build: nix build # # To enter the development environment: # nix develop - -# If flakes are not enabled, use the following command to enter the development environment: +# +# To build the package: +# nix build +# +# If flakes are not enabled, use the following command: # nix --extra-experimental-features 'nix-command flakes' develop . +# nix --extra-experimental-features 'nix-command flakes' build . # # To enable flakes, you may need to enable them in your system configuration: # test -d /etc/nix || sudo mkdir /etc/nix @@ -18,17 +22,16 @@ # Debugging: # XDP2_NIX_DEBUG=7 nix develop --verbose --print-build-logs # -# Not really sure what the difference between the two is, but the second one is faster +# Alternative commands: # nix --extra-experimental-features 'nix-command flakes' --option eval-cache false develop # nix --extra-experimental-features 'nix-command flakes' develop --no-write-lock-file -# # nix --extra-experimental-features 'nix-command flakes' print-dev-env --json # -# Recommended term +# Recommended term: # export TERM=xterm-256color # { - description = "XDP2 development environment"; + description = "XDP2 packet processing framework"; inputs = { nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable"; @@ -40,1025 +43,51 @@ let pkgs = nixpkgs.legacyPackages.${system}; lib = nixpkgs.lib; - llvmP = pkgs.llvmPackages_20; - - # Create a Python environment with scapy - pythonWithScapy = pkgs.python3.withPackages (ps: [ ps.scapy ]); - - - sharedConfig = { - - # Debug configuration - nixDebug = let - envDebug = builtins.getEnv "XDP2_NIX_DEBUG"; - in - if envDebug == "" then 0 else builtins.fromJSON envDebug; - - # GCC-only configuration. These variables could be used to select clang - useGCC = true; - selectedCCPkgs = pkgs.gcc; - selectedCXXPkgs = pkgs.gcc; - selectedCCBin = "gcc"; - selectedCXXBin = "g++"; - compilerInfo = "GCC"; - - configAgeWarningDays = 14; # Configurable threshold for stale config warnings + # Import LLVM configuration module + # Use default LLVM version from nixpkgs (no pinning required) + llvmConfig = import ./nix/llvm.nix { inherit pkgs lib; }; + llvmPackages = llvmConfig.llvmPackages; - # https://nixos.wiki/wiki/C#Hardening_flags - # hardeningDisable = [ "fortify" "fortify3" "stackprotector" "strictoverflow" ]; - # Disable all hardening flags for now, but might restore some later - hardeningDisable = [ "all" ]; + # Import packages module + packagesModule = import ./nix/packages.nix { inherit pkgs llvmPackages; }; - # Library packages - corePackages = with pkgs; [ - # Build tools - gnumake pkg-config bison flex - # Core utilities - bash coreutils gnused gawk gnutar xz git - # Libraries - boost - libpcap - libelf - libbpf - pythonWithScapy - # Development tools - graphviz - bpftools - # Compilers - gcc - llvmP.clang - llvmP.llvm.dev - llvmP.clang-unwrapped - llvmP.libclang - llvmP.lld - # Debugging tools - glibc_multi.bin - gdb - valgrind - strace - ltrace - # Code quality - shellcheck - # ASCII art generator for logo display - jp2a - # Locale support for cross-distribution compatibility - glibcLocales - ]; - - buildInputs = with pkgs; [ - boost - libpcap - libelf - libbpf - pythonWithScapy - llvmP.llvm - llvmP.llvm.dev - llvmP.clang-unwrapped - llvmP.libclang - llvmP.lld - ]; - - nativeBuildInputs = [ - pkgs.pkg-config - llvmP.clang - llvmP.llvm.dev - ]; + # Compiler configuration + compilerConfig = { + cc = pkgs.gcc; + cxx = pkgs.gcc; + ccBin = "gcc"; + cxxBin = "g++"; }; - # Create a wrapper for llvm-config to include clang paths (for libclang) - llvm-config-wrapped = pkgs.runCommand "llvm-config-wrapped" { } '' - mkdir -p $out/bin - cat > $out/bin/llvm-config <\\n#include \\n' include/cpp2util.h\n" - fi - sed -i '1i#include \n#include \n' include/cpp2util.h - - # Level 3: Build step details - if [ "$debug_level" -ge 3 ]; then - echo "[DEBUG] Building cppfront-compiler with make" - fi - - # Build cppfront with error checking - if HOST_CXX="$CXX" HOST_CC="$CC" make -j"$NIX_BUILD_CORES"; then - echo "✓ cppfront make completed successfully" - else - echo "✗ ERROR: cppfront make failed" - return 1 - fi - - # Return to repository root - navigate-to-repo-root || return 1 - - # Add to the PATH - add-to-path "$PWD/thirdparty/cppfront" - - # Level 2: Validation step - if [ "$debug_level" -ge 2 ]; then - echo "[DEBUG] Validating cppfront-compiler binary" - fi - - # Validate binary was created - if [ -x "./thirdparty/cppfront/cppfront-compiler" ]; then - echo "✓ cppfront-compiler binary created and executable" - - # Test the binary runs correctly - echo "Testing cppfront-compiler..." - set +e # Temporarily disable exit on error - - # Debug output for validation command - if [ "$debug_level" -gt 3 ]; then - echo "[DEBUG] About to run: ./thirdparty/cppfront/cppfront-compiler -version" - fi - ./thirdparty/cppfront/cppfront-compiler -version - test_exit_code=$? - set -e # Re-enable exit on error - - if [ "$test_exit_code" -eq 0 ] || [ "$test_exit_code" -eq 1 ]; then - echo "✓ cppfront-compiler runs correctly (exit code: $test_exit_code)" - else - echo "⚠ WARNING: cppfront-compiler returned unexpected exit code: $test_exit_code" - echo "But binary exists and is executable, continuing..." - fi - else - echo "✗ ERROR: cppfront-compiler binary not found or not executable" - return 1 - fi - - # End timing for debug levels > 3 - if [ "$debug_level" -gt 3 ]; then - end_time=$(date +%s) - local duration=$((end_time - start_time)) - echo "[DEBUG] build-cppfront completed in $duration seconds" - fi - - echo "cppfront-compiler built and validated successfully ( ./thirdparty/cppfront/cppfront-compiler )" - } - ''; - - check-cppfront-age-fn = '' - check-cppfront-age() { - if [ -n "$XDP2_NIX_DEBUG" ]; then - local debug_level=$XDP2_NIX_DEBUG - else - local debug_level=0 - fi - local start_time="" - local end_time="" - - # Start timing for debug levels > 3 - if [ "$debug_level" -gt 3 ]; then - start_time=$(date +%s) - echo "[DEBUG] check-cppfront-age started at $(date)" - fi - - # Level 1: Function start - if [ "$debug_level" -ge 1 ]; then - echo "[DEBUG] Starting check-cppfront-age function" - fi - - local cppfront_binary="thirdparty/cppfront/cppfront-compiler" - - # Level 2: File check - if [ "$debug_level" -ge 2 ]; then - echo "[DEBUG] Checking cppfront binary: $cppfront_binary" - fi - - if [ -f "$cppfront_binary" ]; then - local file_time - file_time=$(stat -c %Y "$cppfront_binary") - local current_time - current_time=$(date +%s) - local age_days=$(( (current_time - file_time) / 86400 )) - - # Level 3: Age calculation details - if [ "$debug_level" -ge 3 ]; then - echo "[DEBUG] File modification time: $file_time" - echo "[DEBUG] Current time: $current_time" - echo "[DEBUG] Calculated age: $age_days days" - fi - - if [ "$age_days" -gt 7 ]; then - echo "cppfront is $age_days days old, rebuilding..." - build-cppfront - else - echo "cppfront is up to date ($age_days days old)" - fi - else - echo "cppfront not found, building..." - build-cppfront - fi - - # End timing for debug levels > 3 - if [ "$debug_level" -gt 3 ]; then - end_time=$(date +%s) - local duration=$((end_time - start_time)) - echo "[DEBUG] check-cppfront-age completed in $duration seconds" - fi - } - ''; - - build-xdp2-compiler-fn = '' - build-xdp2-compiler() { - if [ -n "$XDP2_NIX_DEBUG" ]; then - local debug_level=$XDP2_NIX_DEBUG - else - local debug_level=0 - fi - local start_time="" - local end_time="" - - # Start timing for debug levels > 3 - if [ "$debug_level" -gt 3 ]; then - start_time=$(date +%s) - echo "[DEBUG] build-xdp2-compiler started at $(date)" - fi - - # Level 1: Function start - if [ "$debug_level" -ge 1 ]; then - echo "[DEBUG] Starting build-xdp2-compiler function" - fi - - # Level 2: Clean step - if [ "$debug_level" -ge 2 ]; then - echo "[DEBUG] Cleaning xdp2-compiler build directory" - fi - echo "Cleaning and building xdp2-compiler..." - - # Navigate to repository root first - navigate-to-repo-root || return 1 - - # Clean previous build artifacts (before navigating to component) - clean-xdp2-compiler - - # Navigate to xdp2-compiler directory - navigate-to-component "src/tools/compiler" || return 1 - - # Level 3: Build step details - if [ "$debug_level" -ge 3 ]; then - echo "[DEBUG] Building xdp2-compiler with make" - fi - - # Build xdp2-compiler with error checking - if CFLAGS_PYTHON="$CFLAGS_PYTHON" LDFLAGS_PYTHON="$LDFLAGS_PYTHON" make -j"$NIX_BUILD_CORES"; then - echo "✓ xdp2-compiler make completed successfully" - else - echo "✗ ERROR: xdp2-compiler make failed" - return 1 - fi - - # Level 2: Validation step - if [ "$debug_level" -ge 2 ]; then - echo "[DEBUG] Validating xdp2-compiler binary" - fi - - # Validate binary was created - if [ -x "./xdp2-compiler" ]; then - echo "✓ xdp2-compiler binary created and executable" - - # Test the binary runs correctly - echo "Testing xdp2-compiler..." - set +e # Temporarily disable exit on error - - # Debug output for validation command - if [ "$debug_level" -gt 3 ]; then - echo "[DEBUG] About to run: ./xdp2-compiler --help" - fi - ./xdp2-compiler --help - test_exit_code=$? - set -e # Re-enable exit on error - - if [ "$test_exit_code" -eq 0 ] || [ "$test_exit_code" -eq 1 ]; then - echo "✓ xdp2-compiler runs correctly (exit code: $test_exit_code)" - else - echo "⚠ WARNING: xdp2-compiler returned unexpected exit code: $test_exit_code" - echo "But binary exists and is executable, continuing..." - fi - else - echo "✗ ERROR: xdp2-compiler binary not found or not executable" - return 1 - fi - - # Return to repository root - navigate-to-repo-root || return 1 - - # End timing for debug levels > 3 - if [ "$debug_level" -gt 3 ]; then - end_time=$(date +%s) - local duration=$((end_time - start_time)) - echo "[DEBUG] build-xdp2-compiler completed in $duration seconds" - fi - - echo "xdp2-compiler built and validated successfully ( ./src/tools/compiler/xdp2-compiler )" - } - ''; - - build-xdp2-fn = '' - build-xdp2() { - if [ -n "$XDP2_NIX_DEBUG" ]; then - local debug_level=$XDP2_NIX_DEBUG - else - local debug_level=0 - fi - local start_time="" - local end_time="" - - # Start timing for debug levels > 3 - if [ "$debug_level" -gt 3 ]; then - start_time=$(date +%s) - echo "[DEBUG] build-xdp2 started at $(date)" - fi - - # Level 1: Function start - if [ "$debug_level" -ge 1 ]; then - echo "[DEBUG] Starting build-xdp2 function" - fi - - # Level 2: Clean step - if [ "$debug_level" -ge 2 ]; then - echo "[DEBUG] Cleaning xdp2 project build directory" - fi - echo "Cleaning and building xdp2 project..." - - # Navigate to repository root first - navigate-to-repo-root || return 1 - - # Clean previous build artifacts (before navigating to component) - clean-xdp2 - - # Navigate to src directory - navigate-to-component "src" || return 1 - - # Ensure xdp2-compiler is available in PATH - add-to-path "$PWD/tools/compiler" - echo "Added tools/compiler to PATH" - - # Level 3: Build step details - if [ "$debug_level" -ge 3 ]; then - echo "[DEBUG] Building xdp2 project with make" - fi - - # Build the main xdp2 project - if make -j"$NIX_BUILD_CORES"; then - echo "✓ xdp2 project make completed successfully" - else - echo "✗ ERROR: xdp2 project make failed" - echo " Check the error messages above for details" - return 1 - fi - - # Return to repository root - navigate-to-repo-root || return 1 - - # End timing for debug levels > 3 - if [ "$debug_level" -gt 3 ]; then - end_time=$(date +%s) - local duration=$((end_time - start_time)) - echo "[DEBUG] build-xdp2 completed in $duration seconds" - fi - - echo "xdp2 project built successfully" - } - ''; - - build-all-fn = '' - build-all() { - if [ -n "$XDP2_NIX_DEBUG" ]; then - local debug_level=$XDP2_NIX_DEBUG - else - local debug_level=0 - fi - - echo "Building all XDP2 components..." - - if [ "$debug_level" -ge 3 ]; then - echo "[DEBUG] Building cppfront: build-cppfront" - fi - build-cppfront - - if [ "$debug_level" -ge 3 ]; then - echo "[DEBUG] Building xdp2-compiler: build-xdp2-compiler" - fi - build-xdp2-compiler - - if [ "$debug_level" -ge 3 ]; then - echo "[DEBUG] Building xdp2: build-xdp2" - fi - build-xdp2 - - echo "✓ All components built successfully" - } - ''; - - clean-all-fn = '' - clean-all() { - echo "Cleaning all build artifacts..." - - # Clean each component using centralized clean functions - clean-cppfront - clean-xdp2-compiler - clean-xdp2 - - echo "✓ All build artifacts cleaned" - } - ''; - - # Shellcheck function registry - list of all bash functions that should be validated by shellcheck - # IMPORTANT: When adding or removing bash functions, update this list accordingly - shellcheckFunctionRegistry = [ - "smart-configure" - "build-cppfront" - "check-cppfront-age" - "build-xdp2-compiler" - "build-xdp2" - "build-all" - "clean-all" - "check-platform-compatibility" - "detect-repository-root" - "setup-locale-support" - "xdp2-help" - "navigate-to-repo-root" - "navigate-to-component" - "add-to-path" - "clean-cppfront" - "clean-xdp2-compiler" - "clean-xdp2" - ]; - - # Generate complete shellcheck validation function in Nix - generate-shellcheck-validation = let - functionNames = shellcheckFunctionRegistry; - totalFunctions = builtins.length functionNames; - - # Generate individual function checks - functionChecks = lib.concatStringsSep "\n" (map (name: '' - echo "Checking ${name}..." - if declare -f "${name}" >/dev/null 2>&1; then - # Create temporary script with function definition - # TODO use mktemp and trap 'rm -f "$temp_script"' EXIT - local temp_script="/tmp/validate_${name}.sh" - declare -f "${name}" > "$temp_script" - echo "#!/bin/bash" > "$temp_script.tmp" - cat "$temp_script" >> "$temp_script.tmp" - mv "$temp_script.tmp" "$temp_script" - - # Run shellcheck on the function - if shellcheck -s bash "$temp_script" 2>/dev/null; then - echo "✓ ${name} passed shellcheck validation" - passed_functions=$((passed_functions + 1)) - else - echo "✗ ${name} failed shellcheck validation:" - shellcheck -s bash "$temp_script" - failed_functions+=("${name}") - fi - rm -f "$temp_script" - else - echo "✗ ${name} not found" - failed_functions+=("${name}") - fi - echo "" - '') functionNames); - - # Generate failed functions reporting - failedFunctionsReporting = lib.concatStringsSep "\n" (map (name: '' - if [[ "$${failed_functions[*]}" == *"${name}"* ]]; then - echo " - ${name}" - fi - '') functionNames); - - in '' - run-shellcheck() { - if [ -n "$XDP2_NIX_DEBUG" ]; then - local debug_level=$XDP2_NIX_DEBUG - else - local debug_level=0 - fi - - echo "Running shellcheck validation on shell functions..." - - local failed_functions=() - local total_functions=${toString totalFunctions} - local passed_functions=0 - - # Pre-generated function checks from Nix - ${functionChecks} - - # Report results - echo "=== Shellcheck Validation Complete ===" - echo "Total functions: $total_functions" - echo "Passed: $passed_functions" - echo "Failed: $((total_functions - passed_functions))" - - if [ $((total_functions - passed_functions)) -eq 0 ]; then - echo "✓ All functions passed shellcheck validation" - return 0 - else - echo "✗ Some functions failed validation:" - # Pre-generated failed functions reporting from Nix - ${failedFunctionsReporting} - return 1 - fi - } - ''; - - run-shellcheck-fn = generate-shellcheck-validation; - - disable-exit-fn = '' - disable-exit() { - set +e - } - ''; - - platform-compatibility-check-fn = '' - check-platform-compatibility() { - if [ "$(uname)" != "Linux" ]; then - echo "⚠️ PLATFORM COMPATIBILITY NOTICE -================================== - -🍎 You are running on $(uname) (not Linux) - -The XDP2 development environment includes Linux-specific packages -like libbpf that are not available on $(uname) systems. - -📋 Available platforms: - ✅ Linux (x86_64-linux, aarch64-linux, etc.) - ❌ macOS (x86_64-darwin, aarch64-darwin) - ❌ Other Unix systems - -Exiting development shell..." - exit 1 - fi - } - ''; - - detect-repository-root-fn = '' - detect-repository-root() { - XDP2_REPO_ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd) - export XDP2_REPO_ROOT - - if [ ! -d "$XDP2_REPO_ROOT" ]; then - echo "⚠ WARNING: Could not detect valid repository root" - XDP2_REPO_ROOT="$PWD" - else - echo "📁 Repository root: $XDP2_REPO_ROOT" - fi - } - ''; - - setup-locale-support-fn = let - bashVarExpansion = "$"; - bashDefaultSyntax = "{LANG:-C.UTF-8}"; - bashDefaultSyntaxLC = "{LC_ALL:-C.UTF-8}"; - in '' - setup-locale-support() { - # Only set locale if user hasn't already configured it - if [ -z "$LANG" ] || [ -z "$LC_ALL" ]; then - # Try to use system default, fallback to C.UTF-8 - export LANG=${bashVarExpansion}${bashDefaultSyntax} - export LC_ALL=${bashVarExpansion}${bashDefaultSyntaxLC} - fi - - # Verify locale is available (only if locale command exists) - if command -v locale >/dev/null 2>&1; then - if ! locale -a 2>/dev/null | grep -q "$LANG"; then - # Fallback to C.UTF-8 if user's locale is not available - export LANG=C.UTF-8 - export LC_ALL=C.UTF-8 - fi - fi - } - ''; - - xdp2-help-fn = '' - xdp2-help() { - echo "🚀 === XDP2 Development Shell Help === - -📦 Compiler: GCC -🔧 GCC and Clang are available in the environment. -🐛 Debugging tools: gdb, valgrind, strace, ltrace - -🔍 DEBUGGING: - XDP2_NIX_DEBUG=0 - No extra debug. Default - XDP2_NIX_DEBUG=3 - Basic debug - XDP2_NIX_DEBUG=5 - Show compiler selection and config.mk - XDP2_NIX_DEBUG=7 - Show all debug info - -🔧 BUILD COMMANDS: - build-cppfront - Build cppfront compiler - build-xdp2-compiler - Build xdp2 compiler - build-xdp2 - Build main XDP2 project - build-all - Build all components - -🧹 CLEAN COMMANDS: - clean-cppfront - Clean cppfront build artifacts - clean-xdp2-compiler - Clean xdp2-compiler build artifacts - clean-xdp2 - Clean xdp2 build artifacts - clean-all - Clean all build artifacts - -🔍 VALIDATION: - run-shellcheck - Validate all shell functions - -📁 PROJECT STRUCTURE: - • src/ - Main source code - • tools/ - Build tools and utilities - • thirdparty/ - Third-party dependencies - • samples/ - Example code and parsers - • documentation/ - Project documentation - -🎯 Ready to develop! 'xdp2-help' for help" - } - ''; - - shell-aliases = '' - alias xdp2-src='cd src' - alias xdp2-samples='cd samples' - alias xdp2-docs='cd documentation' - alias xdp2-cppfront='cd thirdparty/cppfront' - ''; - - colored-prompt = '' - export PS1="\[\033[0;32m\][XDP2-${sharedConfig.compilerInfo}] \[\033[01;34m\][\u@\h:\w]\$ \[\033[0m\]" - ''; - - ascii-art-logo = '' - if command -v jp2a >/dev/null 2>&1 && [ -f "./documentation/images/xdp2-big.png" ]; then - echo "$(jp2a --colors ./documentation/images/xdp2-big.png)" - echo "" - else - echo "🚀 === XDP2 Development Shell ===" - fi - ''; - - minimal-shell-entry = '' - echo "🚀 === XDP2 Development Shell ===" - echo "📦 Compiler: ${sharedConfig.compilerInfo}" - echo "🔧 GCC and Clang are available in the environment" - echo "🐛 Debugging tools: gdb, valgrind, strace, ltrace" - echo "🎯 Ready to develop! 'xdp2-help' for help" - ''; - - debug-compiler-selection = '' - if [ ${toString sharedConfig.nixDebug} -gt 4 ]; then - echo "=== COMPILER SELECTION ===" - echo "Using compiler: ${sharedConfig.compilerInfo}" - echo "HOST_CC: $HOST_CC" - echo "HOST_CXX: $HOST_CXX" - $HOST_CC --version - $HOST_CXX --version - echo "=== End compiler selection ===" - fi - ''; - - debug-environment-vars = '' - if [ ${toString sharedConfig.nixDebug} -gt 5 ]; then - echo "=== Environment Variables ===" - env - echo "=== End Environment Variables ===" - fi - ''; - - - navigate-to-repo-root-fn = '' - navigate-to-repo-root() { - if [ -n "$XDP2_REPO_ROOT" ]; then - cd "$XDP2_REPO_ROOT" || return 1 - else - echo "✗ ERROR: XDP2_REPO_ROOT not set" - return 1 - fi - } - ''; - - navigate-to-component-fn = '' - navigate-to-component() { - local component="$1" - local target_dir="$XDP2_REPO_ROOT/$component" - - if [ ! -d "$target_dir" ]; then - echo "✗ ERROR: Component directory not found: $target_dir" - return 1 - fi - - cd "$target_dir" || return 1 - } - ''; - - add-to-path-fn = '' - # Add path to PATH environment variable if not already present - add-to-path() { - local path_to_add="$1" - - if [ -n "$XDP2_NIX_DEBUG" ]; then - local debug_level=$XDP2_NIX_DEBUG - else - local debug_level=0 - fi - - # Check if path is already in PATH - if [[ ":$PATH:" == *":$path_to_add:"* ]]; then - if [ "$debug_level" -gt 3 ]; then - echo "[DEBUG] Path already in PATH: $path_to_add" - fi - return 0 - fi - - # Add path to beginning of PATH - if [ "$debug_level" -gt 3 ]; then - echo "[DEBUG] Adding to PATH: $path_to_add" - echo "[DEBUG] PATH before: $PATH" - fi - - export PATH="$path_to_add:$PATH" - - if [ "$debug_level" -gt 3 ]; then - echo "[DEBUG] PATH after: $PATH" - fi - } - ''; - - clean-cppfront-fn = '' - # Clean cppfront build artifacts - clean-cppfront() { - if [ -n "$XDP2_NIX_DEBUG" ]; then - local debug_level=$XDP2_NIX_DEBUG - else - local debug_level=0 - fi - - # Navigate to repository root first - navigate-to-repo-root || return 1 - - # Navigate to cppfront directory - navigate-to-component "thirdparty/cppfront" || return 1 - - # Debug output for clean command - if [ "$debug_level" -gt 3 ]; then - echo "[DEBUG] About to run: rm -f cppfront-compiler" - fi - rm -f cppfront-compiler # Remove the binary directly since Makefile has no clean target - - # Return to repository root - navigate-to-repo-root || return 1 - } - ''; - - clean-xdp2-compiler-fn = '' - # Clean xdp2-compiler build artifacts - clean-xdp2-compiler() { - if [ -n "$XDP2_NIX_DEBUG" ]; then - local debug_level=$XDP2_NIX_DEBUG - else - local debug_level=0 - fi - - # Navigate to repository root first - navigate-to-repo-root || return 1 - - # Navigate to xdp2-compiler directory - navigate-to-component "src/tools/compiler" || return 1 - - # Debug output for clean command - if [ "$debug_level" -gt 3 ]; then - echo "[DEBUG] About to run: make clean" - fi - make clean || true # Don't fail if clean fails - - # Return to repository root - navigate-to-repo-root || return 1 - } - ''; - - clean-xdp2-fn = '' - # Clean xdp2 build artifacts - clean-xdp2() { - if [ -n "$XDP2_NIX_DEBUG" ]; then - local debug_level=$XDP2_NIX_DEBUG - else - local debug_level=0 - fi - - # Navigate to repository root first - navigate-to-repo-root || return 1 - - # Navigate to src directory - navigate-to-component "src" || return 1 - - # Debug output for clean command - if [ "$debug_level" -gt 3 ]; then - echo "[DEBUG] About to run: make clean" - fi - make clean || true # Don't fail if clean fails - - # Return to repository root - navigate-to-repo-root || return 1 - } - ''; - - # Combined build functions (ordered to avoid SC2218 - functions called before definition) - build-functions = '' - # Navigation functions (called by all other functions) - ${navigate-to-repo-root-fn} - ${navigate-to-component-fn} + # Import environment variables module + envVars = import ./nix/env-vars.nix { + inherit pkgs llvmConfig compilerConfig; + packages = packagesModule; + configAgeWarningDays = 14; + }; - # Utility functions - ${add-to-path-fn} - ${check-cppfront-age-fn} + # Import package derivation + xdp2 = import ./nix/derivation.nix { + inherit pkgs lib llvmConfig; + inherit (packagesModule) nativeBuildInputs buildInputs; + }; - # Individual clean functions (called by build functions and clean-build) - ${clean-cppfront-fn} - ${clean-xdp2-compiler-fn} - ${clean-xdp2-fn} - # Individual build functions (called by build-all) - ${build-cppfront-fn} - ${build-xdp2-compiler-fn} - ${build-xdp2-fn} - # Composite functions (call the individual functions above) - ${build-all-fn} - ${clean-all-fn} - # Validation and help functions - ${platform-compatibility-check-fn} - ${detect-repository-root-fn} - ${setup-locale-support-fn} - ${run-shellcheck-fn} - ${xdp2-help-fn} - ''; + # Import development shell module + devshell = import ./nix/devshell.nix { + inherit pkgs lib llvmConfig compilerConfig envVars; + packages = packagesModule; + }; in { - devShells.default = pkgs.mkShell { - packages = sharedConfig.corePackages; - - shellHook = '' - ${sharedEnvVars} - - ${build-functions} - - check-platform-compatibility - detect-repository-root - setup-locale-support - - ${debug-compiler-selection} - ${debug-environment-vars} - - ${ascii-art-logo} - - ${smart-configure} - smart-configure - - ${shell-aliases} - - ${colored-prompt} - - ${disable-exit-fn} - - ${minimal-shell-entry} - ''; + # Package outputs + packages = { + default = xdp2; + xdp2 = xdp2; }; + + # Development shell + devShells.default = devshell; }); } diff --git a/nix/derivation.nix b/nix/derivation.nix new file mode 100644 index 0000000..3f4b9ba --- /dev/null +++ b/nix/derivation.nix @@ -0,0 +1,208 @@ +# nix/derivation.nix +# +# Package derivation for XDP2 +# +# This module defines the actual XDP2 package using stdenv.mkDerivation. +# It enables `nix build` support and follows nixpkgs conventions. +# +# Build order: +# 1. Patch source files (postPatch) +# 2. Run configure script (configurePhase) +# 3. Build cppfront, xdp2-compiler, then xdp2 (buildPhase) +# 4. Install binaries and libraries (installPhase) +# +# Usage in flake.nix: +# packages.default = import ./nix/derivation.nix { +# inherit pkgs lib llvmConfig; +# inherit (import ./nix/packages.nix { inherit pkgs llvmPackages; }) nativeBuildInputs buildInputs; +# }; +# + +{ pkgs +, lib +, llvmConfig +, nativeBuildInputs +, buildInputs +}: + +let + llvmPackages = llvmConfig.llvmPackages; +in +pkgs.stdenv.mkDerivation rec { + pname = "xdp2"; + version = "0.1.0"; + + src = ./..; + + # Nix-specific patches for xdp2-compiler + # + # These patches are required because xdp2-compiler uses libclang's ClangTool API + # directly, which bypasses the Nix clang wrapper. The wrapper normally handles + # include path setup, but ClangTool doesn't benefit from it. + # + # See documentation/nix/phase6_segfault_defect.md for full investigation details. + patches = [ + # Patch 1: System include paths for Nix environment + # + # Problem: ClangTool bypasses the Nix clang wrapper which normally adds + # -isystem flags for system headers. Without these, header resolution fails + # and the AST contains RecoveryExpr/contains-errors nodes. + # + # Solution: Read include paths from environment variables set by the Nix + # derivation (XDP2_C_INCLUDE_PATH, XDP2_GLIBC_INCLUDE_PATH, XDP2_LINUX_HEADERS_PATH) + # and add them as -isystem arguments to ClangTool. + # + # Also adds optional debug diagnostic output (enable with -DXDP2_COMPILER_DEBUG) + # using LLVM 21 compatible API. + ./patches/01-nix-clang-system-includes.patch + + # Patch 2: Null check for tentative definitions + # + # Problem: C tentative definitions like "static const struct T name;" create + # VarDecls where hasInit() behavior differs between clang versions: + # - Ubuntu clang 18.1.3: hasInit() returns false + # - Nix clang 18.1.8+: hasInit() returns true with void-type InitListExpr + # + # When getAs() is called on a void type, it returns nullptr, + # causing a segfault when ->getDecl() is called. + # + # Solution: Check if getAs() returns null and skip tentative + # definitions. The actual definition will be processed when encountered later. + ./patches/02-tentative-definition-null-check.patch + ]; + + inherit nativeBuildInputs buildInputs; + + # Disable hardening flags that interfere with XDP/BPF code + hardeningDisable = [ "all" ]; + + # Set up environment variables for the build + HOST_CC = "${pkgs.gcc}/bin/gcc"; + HOST_CXX = "${pkgs.gcc}/bin/g++"; + HOST_LLVM_CONFIG = "${llvmConfig.llvm-config-wrapped}/bin/llvm-config"; + XDP2_CLANG_VERSION = llvmConfig.version; + XDP2_CLANG_RESOURCE_PATH = llvmConfig.paths.clangResourceDir; + + # Add LLVM/Clang libs to library path + LD_LIBRARY_PATH = lib.makeLibraryPath [ + llvmPackages.llvm + llvmPackages.libclang.lib + pkgs.boost + ]; + + # Post-patch phase: Fix paths and apply Nix-specific patches + postPatch = '' + # Fix cppfront Makefile to use source directory path + substituteInPlace thirdparty/cppfront/Makefile \ + --replace-fail 'include ../../src/config.mk' '# config.mk not needed for standalone build' + + # Add functional header to cppfront (required for newer GCC) + sed -i '1i#include \n#include \n' thirdparty/cppfront/include/cpp2util.h + ''; + + # Configure phase: Generate config.mk + configurePhase = '' + runHook preConfigure + + cd src + + # Set up environment for configure + export CC="${pkgs.gcc}/bin/gcc" + export CXX="${pkgs.gcc}/bin/g++" + export HOST_CC="$CC" + export HOST_CXX="$CXX" + export HOST_LLVM_CONFIG="${llvmConfig.llvm-config-wrapped}/bin/llvm-config" + + # Run configure script + bash configure.sh --build-opt-parser + + # Fix PATH_ARG for Nix environment (remove hardcoded paths) + if grep -q 'PATH_ARG="--with-path=' config.mk; then + sed -i 's|PATH_ARG="--with-path=.*"|PATH_ARG=""|' config.mk + fi + + cd .. + + runHook postConfigure + ''; + + # Build phase: Build all components in order + buildPhase = '' + runHook preBuild + + # Set up environment + export HOST_CC="${pkgs.gcc}/bin/gcc" + export HOST_CXX="${pkgs.gcc}/bin/g++" + export HOST_LLVM_CONFIG="${llvmConfig.llvm-config-wrapped}/bin/llvm-config" + export NIX_BUILD_CORES=$NIX_BUILD_CORES + export XDP2_CLANG_VERSION="${llvmConfig.version}" + export XDP2_CLANG_RESOURCE_PATH="${llvmConfig.paths.clangResourceDir}" + + # Include paths for xdp2-compiler's libclang usage + # These are needed because ClangTool bypasses the Nix clang wrapper + export XDP2_C_INCLUDE_PATH="${llvmConfig.paths.clangResourceDir}/include" + export XDP2_GLIBC_INCLUDE_PATH="${pkgs.stdenv.cc.libc.dev}/include" + export XDP2_LINUX_HEADERS_PATH="${pkgs.linuxHeaders}/include" + + # 1. Build cppfront compiler + echo "Building cppfront..." + cd thirdparty/cppfront + $HOST_CXX -std=c++20 source/cppfront.cpp -o cppfront-compiler + cd ../.. + + # 2. Build xdp2-compiler + echo "Building xdp2-compiler..." + cd src/tools/compiler + make -j$NIX_BUILD_CORES + cd ../../.. + + # 3. Build main xdp2 project + echo "Building xdp2..." + cd src + make -j$NIX_BUILD_CORES + cd .. + + runHook postBuild + ''; + + # Install phase: Install binaries and libraries + installPhase = '' + runHook preInstall + + # Create output directories + mkdir -p $out/bin + mkdir -p $out/lib + mkdir -p $out/include + mkdir -p $out/share/xdp2 + + # Install xdp2-compiler + install -m 755 src/tools/compiler/xdp2-compiler $out/bin/ + + # Install cppfront-compiler (useful for development) + install -m 755 thirdparty/cppfront/cppfront-compiler $out/bin/ + + # Install libraries (if any are built as shared) + find src/lib -name "*.so" -exec install -m 755 {} $out/lib/ \; 2>/dev/null || true + find src/lib -name "*.a" -exec install -m 644 {} $out/lib/ \; 2>/dev/null || true + + # Install headers (use -L to dereference symlinks like arch -> platform/...) + cp -rL src/include/* $out/include/ 2>/dev/null || true + + # Install templates + cp -r src/templates $out/share/xdp2/ 2>/dev/null || true + + runHook postInstall + ''; + + meta = with lib; { + description = "XDP2 packet processing framework"; + longDescription = '' + XDP2 is a high-performance packet processing framework that uses + eBPF/XDP for fast packet handling in the Linux kernel. + ''; + homepage = "https://github.com/xdp2/xdp2"; + license = licenses.mit; # Update if different + platforms = platforms.linux; + maintainers = [ ]; + }; +} diff --git a/nix/devshell.nix b/nix/devshell.nix new file mode 100644 index 0000000..ec16c61 --- /dev/null +++ b/nix/devshell.nix @@ -0,0 +1,161 @@ +# nix/devshell.nix +# +# Development shell configuration for XDP2 +# +# This module creates the development shell with all necessary +# packages, environment variables, and shell functions. +# +# Usage in flake.nix: +# devshell = import ./nix/devshell.nix { +# inherit pkgs lib llvmConfig packages compilerConfig envVars; +# }; +# devShells.default = devshell; +# + +{ pkgs +, lib +, llvmConfig +, packages +, compilerConfig +, envVars +}: + +let + llvmPackages = llvmConfig.llvmPackages; + + # Shared configuration + sharedConfig = { + # Compiler info for display + compilerInfo = "GCC"; + + # Configurable threshold for stale config warnings + configAgeWarningDays = 14; + }; + + # Shellcheck function registry - list of all bash functions to validate + # IMPORTANT: When adding or removing bash functions, update this list + shellcheckFunctionRegistry = [ + "smart-configure" + "build-cppfront" + "check-cppfront-age" + "build-xdp2-compiler" + "build-xdp2" + "build-all" + "clean-all" + "check-platform-compatibility" + "detect-repository-root" + "setup-locale-support" + "xdp2-help" + "navigate-to-repo-root" + "navigate-to-component" + "add-to-path" + "clean-cppfront" + "clean-xdp2-compiler" + "clean-xdp2" + ]; + + # Import shell function modules + navigationFns = import ./shell-functions/navigation.nix { }; + cleanFns = import ./shell-functions/clean.nix { }; + buildFns = import ./shell-functions/build.nix { }; + configureFns = import ./shell-functions/configure.nix { + configAgeWarningDays = sharedConfig.configAgeWarningDays; + }; + validationFns = import ./shell-functions/validation.nix { + inherit lib shellcheckFunctionRegistry; + }; + asciiArtFn = import ./shell-functions/ascii-art.nix { }; + + # Shell snippets + disable-exit-fn = '' + disable-exit() { + set +e + } + ''; + + shell-aliases = '' + alias xdp2-src='cd src' + alias xdp2-samples='cd samples' + alias xdp2-docs='cd documentation' + alias xdp2-cppfront='cd thirdparty/cppfront' + ''; + + colored-prompt = '' + export PS1="\[\033[0;32m\][XDP2-${sharedConfig.compilerInfo}] \[\033[01;34m\][\u@\h:\w]\$ \[\033[0m\]" + ''; + + minimal-shell-entry = '' + echo "🚀 === XDP2 Development Shell ===" + echo "📦 Compiler: ${sharedConfig.compilerInfo}" + echo "🔧 GCC and Clang are available in the environment" + echo "🐛 Debugging tools: gdb, valgrind, strace, ltrace" + echo "🎯 Ready to develop! 'xdp2-help' for help" + ''; + + # Debug snippets - check XDP2_NIX_DEBUG shell variable at runtime + # Usage: XDP2_NIX_DEBUG=7 nix develop + debug-compiler-selection = '' + if [ "''${XDP2_NIX_DEBUG:-0}" -gt 4 ]; then + echo "=== COMPILER SELECTION ===" + echo "Using compiler: ${sharedConfig.compilerInfo}" + echo "HOST_CC: $HOST_CC" + echo "HOST_CXX: $HOST_CXX" + $HOST_CC --version + $HOST_CXX --version + echo "=== End compiler selection ===" + fi + ''; + + debug-environment-vars = '' + if [ "''${XDP2_NIX_DEBUG:-0}" -gt 5 ]; then + echo "=== Environment Variables ===" + env + echo "=== End Environment Variables ===" + fi + ''; + + # Combined build functions (ordered to avoid SC2218 - functions called before definition) + build-functions = '' + # Navigation functions (from nix/shell-functions/navigation.nix) + ${navigationFns} + + # Clean functions (from nix/shell-functions/clean.nix) + ${cleanFns} + + # Build functions (from nix/shell-functions/build.nix) + ${buildFns} + + # Validation and help functions (from nix/shell-functions/validation.nix) + ${validationFns} + ''; + +in +pkgs.mkShell { + packages = packages.allPackages; + + shellHook = '' + ${envVars} + + ${build-functions} + + check-platform-compatibility + detect-repository-root + setup-locale-support + + ${debug-compiler-selection} + ${debug-environment-vars} + + ${asciiArtFn} + + ${configureFns} + smart-configure + + ${shell-aliases} + + ${colored-prompt} + + ${disable-exit-fn} + + ${minimal-shell-entry} + ''; +} diff --git a/nix/env-vars.nix b/nix/env-vars.nix new file mode 100644 index 0000000..96f5593 --- /dev/null +++ b/nix/env-vars.nix @@ -0,0 +1,67 @@ +# nix/env-vars.nix +# +# Environment variable definitions for XDP2 +# +# This module defines all environment variables needed for: +# - Compiler configuration (CC, CXX, HOST_CC, HOST_CXX) +# - LLVM/Clang paths (via llvmConfig) +# - Python configuration +# - Library paths +# - Build configuration +# +# Usage in flake.nix: +# envVars = import ./nix/env-vars.nix { +# inherit pkgs llvmConfig packages; +# compilerConfig = { cc = pkgs.gcc; cxx = pkgs.gcc; ccBin = "gcc"; cxxBin = "g++"; }; +# configAgeWarningDays = 14; +# }; +# + +{ pkgs +, llvmConfig +, packages +, compilerConfig +, configAgeWarningDays ? 14 +}: + +'' + # Compiler settings + export CC=${compilerConfig.cc}/bin/${compilerConfig.ccBin} + export CXX=${compilerConfig.cxx}/bin/${compilerConfig.cxxBin} + export HOST_CC=${compilerConfig.cc}/bin/${compilerConfig.ccBin} + export HOST_CXX=${compilerConfig.cxx}/bin/${compilerConfig.cxxBin} + + # LLVM/Clang environment variables (from llvmConfig module) + # Sets: XDP2_CLANG_VERSION, XDP2_CLANG_RESOURCE_PATH, XDP2_C_INCLUDE_PATH, + # HOST_LLVM_CONFIG, LLVM_LIBS, CLANG_LIBS, LIBCLANG_PATH + ${llvmConfig.envVars} + + # Glibc include path for xdp2-compiler (needed because libclang bypasses clang wrapper) + export XDP2_GLIBC_INCLUDE_PATH="${pkgs.stdenv.cc.libc.dev}/include" + + # Linux kernel headers (provides etc.) + export XDP2_LINUX_HEADERS_PATH="${pkgs.linuxHeaders}/include" + + # LD_LIBRARY_PATH for libclang + export LD_LIBRARY_PATH=${llvmConfig.ldLibraryPath}:$LD_LIBRARY_PATH + + # Python environment + export CFLAGS_PYTHON="$(pkg-config --cflags python3-embed)" + export LDFLAGS_PYTHON="$(pkg-config --libs python3-embed)" + export PYTHON_VER=3 + export PYTHONPATH="${pkgs.python3}/lib/python3.13/site-packages:$PYTHONPATH" + + # Boost libraries + export BOOST_LIBS="-lboost_wave -lboost_thread -lboost_filesystem -lboost_system -lboost_program_options" + + # Other libraries + export LIBS="-lpthread -ldl -lutil" + export PATH_ARG="" + + # Build configuration + export PKG_CONFIG_PATH=${pkgs.lib.makeSearchPath "lib/pkgconfig" packages.allPackages} + export XDP2_COMPILER_DEBUG=1 + + # Configuration management + export CONFIG_AGE_WARNING_DAYS=${toString configAgeWarningDays} +'' diff --git a/nix/llvm.nix b/nix/llvm.nix new file mode 100644 index 0000000..ba319b1 --- /dev/null +++ b/nix/llvm.nix @@ -0,0 +1,98 @@ +# nix/llvm.nix +# +# LLVM/Clang configuration for XDP2 +# +# This module centralizes all LLVM and Clang configuration: +# - llvmPackages selection (configurable, defaults to pkgs.llvmPackages) +# - llvm-config wrapper for correct include/lib paths +# - Environment variables for LLVM tools +# - Paths for use with substituteInPlace +# +# Usage in flake.nix: +# llvmConfig = import ./nix/llvm.nix { inherit pkgs lib; }; +# # Or with custom version: +# llvmConfig = import ./nix/llvm.nix { inherit pkgs lib; llvmVersion = 19; }; +# + +{ pkgs +, lib +, llvmVersion ? null # null means use default llvmPackages +}: + +let + # Select llvmPackages based on version parameter + # If no version specified, use the default llvmPackages + llvmPackages = + if llvmVersion == null then + pkgs.llvmPackages + else + pkgs."llvmPackages_${toString llvmVersion}"; + + # Extract major version from LLVM version (e.g., "18.1.8" -> "18") + llvmMajorVersion = lib.versions.major llvmPackages.llvm.version; + + # Create a wrapper for llvm-config to include clang paths (for libclang) + # This is needed because the xdp2-compiler uses libclang and needs correct paths + llvm-config-wrapped = pkgs.runCommand "llvm-config-wrapped" { } '' + mkdir -p $out/bin + cat > $out/bin/llvm-config </lib/clang/ + # The libclang.lib path is incomplete (only has include, missing lib and share) + clangResourceDir = "${llvmPackages.clang}/resource-root"; + llvmLib = "${llvmPackages.llvm}/lib"; + libclangLib = "${llvmPackages.libclang.lib}/lib"; + }; + + # Environment variable exports (as shell script fragment) + envVars = '' + # LLVM/Clang version + export XDP2_CLANG_VERSION="$(${llvmPackages.llvm.dev}/bin/llvm-config --version)" + # Clang resource directory - use the wrapper's resource-root which has complete structure + # matching Ubuntu's /usr/lib/llvm-/lib/clang/ + export XDP2_CLANG_RESOURCE_PATH="${llvmPackages.clang}/resource-root" + # C include path for clang headers + export XDP2_C_INCLUDE_PATH="${llvmPackages.clang}/resource-root/include" + + # LLVM/Clang settings + export HOST_LLVM_CONFIG="${llvm-config-wrapped}/bin/llvm-config" + export LLVM_LIBS="-L${llvmPackages.llvm}/lib" + export CLANG_LIBS="-lclang -lLLVM -lclang-cpp" + + # libclang configuration + export LIBCLANG_PATH="${llvmPackages.libclang.lib}/lib" + ''; + + # LD_LIBRARY_PATH addition (separate to allow conditional use) + ldLibraryPath = "${llvmPackages.libclang.lib}/lib"; +} diff --git a/nix/packages.nix b/nix/packages.nix new file mode 100644 index 0000000..ae98c5a --- /dev/null +++ b/nix/packages.nix @@ -0,0 +1,101 @@ +# nix/packages.nix +# +# Package definitions for XDP2 +# +# This module defines all package dependencies, properly separated into: +# - nativeBuildInputs: Build-time tools (compilers, generators, etc.) +# - buildInputs: Libraries needed at build and runtime +# - devTools: Additional tools for development only +# +# Usage in flake.nix: +# packages = import ./nix/packages.nix { inherit pkgs llvmPackages; }; +# + +{ pkgs, llvmPackages }: + +let + # Create a Python environment with scapy for packet generation + pythonWithScapy = pkgs.python3.withPackages (ps: [ ps.scapy ]); +in +{ + # Build-time tools only - these run on the build machine + nativeBuildInputs = [ + # Build system + pkgs.gnumake + pkgs.pkg-config + pkgs.bison + pkgs.flex + + # Core utilities needed during build + pkgs.bash + pkgs.coreutils + pkgs.gnused + pkgs.gawk + pkgs.gnutar + pkgs.xz + pkgs.git + + # Compilers + pkgs.gcc + llvmPackages.clang + llvmPackages.llvm.dev # Provides llvm-config + llvmPackages.lld + ]; + + # Libraries needed at build and runtime + buildInputs = [ + # Core libraries + pkgs.boost + pkgs.libpcap + pkgs.libelf + pkgs.libbpf + + # Linux kernel headers (provides etc.) + pkgs.linuxHeaders + + # Python environment + pythonWithScapy + + # LLVM/Clang libraries (needed for xdp2-compiler) + llvmPackages.llvm + llvmPackages.libclang + llvmPackages.clang-unwrapped + ]; + + # Development-only tools (not needed for building, only for dev workflow) + devTools = [ + # Debugging + pkgs.gdb + pkgs.valgrind + pkgs.strace + pkgs.ltrace + pkgs.glibc_multi.bin + + # BPF/XDP development tools + pkgs.bpftools + pkgs.bpftrace # High-level tracing language for eBPF + pkgs.bcc # BPF Compiler Collection with Python bindings + pkgs.linuxPackages.perf # Linux performance analysis tool + pkgs.pahole # DWARF debugging info analyzer (useful for BTF) + + # Visualization + pkgs.graphviz + + # Code quality + pkgs.shellcheck + llvmPackages.clang-tools # clang-tidy, clang-format, etc. + + # Utilities + pkgs.jp2a # ASCII art for logo + pkgs.glibcLocales # Locale support + ]; + + # Combined list for dev shell (all packages) + # This replaces the old corePackages + allPackages = + let self = import ./packages.nix { inherit pkgs llvmPackages; }; + in self.nativeBuildInputs ++ self.buildInputs ++ self.devTools; + + # Export pythonWithScapy for use elsewhere + inherit pythonWithScapy; +} diff --git a/nix/patches/01-nix-clang-system-includes.patch b/nix/patches/01-nix-clang-system-includes.patch new file mode 100644 index 0000000..ebecdd9 --- /dev/null +++ b/nix/patches/01-nix-clang-system-includes.patch @@ -0,0 +1,100 @@ +# nix/patches/01-nix-clang-system-includes.patch +# +# Purpose: Add system include paths for Nix build environments +# +# When xdp2-compiler uses libclang's ClangTool API directly, it bypasses the +# Nix clang wrapper that normally sets up include paths. This causes header +# resolution failures and AST parse errors. +# +# This patch reads include paths from environment variables set by the Nix +# derivation and adds them as -isystem arguments to ClangTool. The environment +# variables are only set during `nix build`, so this is a no-op on Ubuntu/Fedora. +# +# Also adds optional debug diagnostic output (LLVM 21 compatible). +# +# See: documentation/nix/phase6_segfault_defect.md +# +diff --git a/src/tools/compiler/src/main.cpp b/src/tools/compiler/src/main.cpp +index ec547c4..426356a 100644 +--- a/src/tools/compiler/src/main.cpp ++++ b/src/tools/compiler/src/main.cpp +@@ -67,6 +67,7 @@ + // Clang + #include + #include ++#include // [nix-patch] For optional debug diagnostics + + // LLVM + #include +@@ -203,8 +204,6 @@ clang::tooling::ClangTool create_clang_tool( + // The actual resource directory is set via -resource-dir flag below. + plog::log(std::cout) + << "/usr/lib/clang/" << version << "/include" << std::endl; +- if (getenv("XDP2_C_INCLUDE_PATH")) +- setenv("C_INCLUDE_PATH", getenv("XDP2_C_INCLUDE_PATH"), 1); + + plog::log(std::cout) << "OptionsParser->getSourcePathList()" << std::endl; + for (auto &&item : OptionsParser->getSourcePathList()) +@@ -240,6 +239,43 @@ clang::tooling::ClangTool create_clang_tool( + } + #endif + ++ // [nix-patch] Add system include paths for Nix environments. ++ // ++ // PROBLEM: When using libclang/ClangTool directly (as xdp2-compiler does), ++ // we bypass the Nix clang wrapper script which normally adds -isystem flags ++ // for system headers. Without these flags, header resolution fails and the ++ // AST contains RecoveryExpr/contains-errors nodes, causing hasInit() to ++ // return unexpected values and ultimately leading to null pointer crashes. ++ // ++ // SOLUTION: Read include paths from environment variables set by the Nix ++ // derivation and add them as -isystem arguments. These env vars are only ++ // set during `nix build`, so this code is a no-op on traditional systems. ++ // ++ // Environment variables (set in nix/derivation.nix buildPhase): ++ // XDP2_C_INCLUDE_PATH: Clang builtins (stddef.h, stdint.h, etc.) ++ // XDP2_GLIBC_INCLUDE_PATH: glibc headers (stdlib.h, stdio.h, etc.) ++ // XDP2_LINUX_HEADERS_PATH: Linux kernel headers (, etc.) ++ // ++ // See: documentation/nix/phase6_segfault_defect.md for full details. ++ const char* clang_include = getenv("XDP2_C_INCLUDE_PATH"); ++ const char* glibc_include = getenv("XDP2_GLIBC_INCLUDE_PATH"); ++ const char* linux_headers = getenv("XDP2_LINUX_HEADERS_PATH"); ++ if (linux_headers) { ++ Tool.appendArgumentsAdjuster(clang::tooling::getInsertArgumentAdjuster( ++ {"-isystem", linux_headers}, ++ clang::tooling::ArgumentInsertPosition::BEGIN)); ++ } ++ if (glibc_include) { ++ Tool.appendArgumentsAdjuster(clang::tooling::getInsertArgumentAdjuster( ++ {"-isystem", glibc_include}, ++ clang::tooling::ArgumentInsertPosition::BEGIN)); ++ } ++ if (clang_include) { ++ Tool.appendArgumentsAdjuster(clang::tooling::getInsertArgumentAdjuster( ++ {"-isystem", clang_include}, ++ clang::tooling::ArgumentInsertPosition::BEGIN)); ++ } ++ + return Tool; + }; + +@@ -271,8 +307,18 @@ void parse_file(G &g, std::vector> &roots, + // Extract basic graph information + graph_info graph_consumed_data{ &g, &roots }; + ++ // [nix-patch] Optional diagnostic output for debugging AST parse errors. ++ // Enable by adding -DXDP2_COMPILER_DEBUG to CXXFLAGS in compiler Makefile. ++ // Uses LLVM 21+ compatible API (DiagnosticOptions by reference, not pointer). ++#ifdef XDP2_COMPILER_DEBUG ++ clang::DiagnosticOptions diagOpts; ++ diagOpts.ShowColors = true; ++ clang::TextDiagnosticPrinter diagPrinter(llvm::errs(), diagOpts); ++ Tool.setDiagnosticConsumer(&diagPrinter); ++#else + clang::IgnoringDiagConsumer diagConsumer; + Tool.setDiagnosticConsumer(&diagConsumer); ++#endif + + int action1 = + Tool.run(extract_graph_constants_factory(graph_consumed_data).get()); diff --git a/nix/patches/02-tentative-definition-null-check.patch b/nix/patches/02-tentative-definition-null-check.patch new file mode 100644 index 0000000..ec9dd0b --- /dev/null +++ b/nix/patches/02-tentative-definition-null-check.patch @@ -0,0 +1,59 @@ +# nix/patches/02-tentative-definition-null-check.patch +# +# Purpose: Fix null pointer crash with C tentative definitions +# +# C tentative definitions like "static const struct T name;" (created by +# XDP2_DECL_PROTO_TABLE macro) behave differently across clang versions: +# +# - Ubuntu clang 18.1.3: hasInit() returns false, these are skipped +# - Nix clang 18.1.8+: hasInit() returns true with void-type InitListExpr +# +# When getAs() is called on void type, it returns nullptr. +# The original code calls ->getDecl() on this nullptr, causing a segfault. +# +# This patch adds a null check and skips tentative definitions gracefully. +# The actual definition is processed when encountered later in the AST. +# +# See: documentation/nix/phase6_segfault_defect.md +# +diff --git a/src/tools/compiler/include/xdp2gen/ast-consumer/proto-tables.h b/src/tools/compiler/include/xdp2gen/ast-consumer/proto-tables.h +index 6616cf7..3aaa7c5 100644 +--- a/src/tools/compiler/include/xdp2gen/ast-consumer/proto-tables.h ++++ b/src/tools/compiler/include/xdp2gen/ast-consumer/proto-tables.h +@@ -89,11 +89,32 @@ public: + clang::dyn_cast( + initializer_expr); + ++ // [nix-patch] Handle tentative definitions to prevent null pointer crash. ++ // ++ // PROBLEM: C tentative definitions like: ++ // static const struct xdp2_proto_table ip_table; ++ // are created by XDP2_DECL_PROTO_TABLE macro before the actual definition. ++ // ++ // Different clang versions handle hasInit() differently for these: ++ // - Ubuntu clang 18.1.3: hasInit() returns false (skipped entirely) ++ // - Nix clang 18.1.8+: hasInit() returns true with void-type InitListExpr ++ // ++ // When getAs() is called on void type, it returns nullptr. ++ // The original code then calls ->getDecl() on nullptr, causing segfault. ++ // ++ // SOLUTION: Check if RecordType is null and skip tentative definitions. ++ // The actual definition will be processed when encountered later in the AST. ++ // ++ // See: documentation/nix/phase6_segfault_defect.md for full investigation. ++ clang::QualType initType = initializer_list_expr->getType(); ++ auto *recordType = initType->getAs(); ++ if (!recordType) { ++ // Skip tentative definitions - actual definition processed later ++ return true; ++ } ++ + // Extracts current analyzed InitListDecl +- clang::RecordDecl *initializer_list_decl = +- initializer_list_expr->getType() +- ->getAs() +- ->getDecl(); ++ clang::RecordDecl *initializer_list_decl = recordType->getDecl(); + + // Proto table consumed infos + xdp2_proto_table_extract_data table_data; diff --git a/nix/shell-functions/ascii-art.nix b/nix/shell-functions/ascii-art.nix new file mode 100644 index 0000000..34f4fc9 --- /dev/null +++ b/nix/shell-functions/ascii-art.nix @@ -0,0 +1,21 @@ +# nix/shell-functions/ascii-art.nix +# +# ASCII art logo display for XDP2 development shell +# +# Uses jp2a to convert the XDP2 logo to colored ASCII art. +# Falls back to a simple text banner if jp2a is not available. +# +# Usage in devshell.nix: +# asciiArt = import ./shell-functions/ascii-art.nix { }; +# + +{ }: + +'' + if command -v jp2a >/dev/null 2>&1 && [ -f "./documentation/images/xdp2-big.png" ]; then + echo "$(jp2a --colors ./documentation/images/xdp2-big.png)" + echo "" + else + echo "🚀 === XDP2 Development Shell ===" + fi +'' diff --git a/nix/shell-functions/build.nix b/nix/shell-functions/build.nix new file mode 100644 index 0000000..c7c93ab --- /dev/null +++ b/nix/shell-functions/build.nix @@ -0,0 +1,373 @@ +# nix/shell-functions/build.nix +# +# Build shell functions for XDP2 +# +# Functions: +# - build-cppfront: Build the cppfront compiler +# - check-cppfront-age: Check if cppfront needs rebuilding +# - build-xdp2-compiler: Build the xdp2 compiler +# - build-xdp2: Build the main xdp2 project +# - build-all: Build all components in order +# +# Note: These functions depend on navigation and clean functions +# +# Usage in flake.nix: +# buildFns = import ./nix/shell-functions/build.nix { }; +# + +{ }: + +'' + # Build the cppfront compiler + build-cppfront() { + if [ -n "$XDP2_NIX_DEBUG" ]; then + local debug_level=$XDP2_NIX_DEBUG + else + local debug_level=0 + fi + local start_time="" + local end_time="" + + if [ "$debug_level" -gt 3 ]; then + start_time=$(date +%s) + echo "[DEBUG] build-cppfront started at $(date)" + fi + + # Level 1: Function start + if [ "$debug_level" -ge 1 ]; then + echo "[DEBUG] Starting build-cppfront function" + fi + + # Clean + if [ "$debug_level" -ge 2 ]; then + echo "[DEBUG] Cleaning cppfront build directory" + fi + echo "Cleaning and building cppfront-compiler..." + + # Navigate to repository root first + navigate-to-repo-root || return 1 + + # Clean previous build artifacts (before navigating to component) + clean-cppfront + + # Navigate to cppfront directory + navigate-to-component "thirdparty/cppfront" || return 1 + + # Apply essential header fix for cppfront + if [ "$debug_level" -ge 3 ]; then + echo "[DEBUG] Applying cppfront header fix" + printf "sed -i '1i#include \\n#include \\n' include/cpp2util.h\n" + fi + sed -i '1i#include \n#include \n' include/cpp2util.h + + # Level 3: Build step details + if [ "$debug_level" -ge 3 ]; then + echo "[DEBUG] Building cppfront-compiler with make" + fi + + # Build cppfront with error checking + if HOST_CXX="$CXX" HOST_CC="$CC" make -j"$NIX_BUILD_CORES"; then + echo "✓ cppfront make completed successfully" + else + echo "✗ ERROR: cppfront make failed" + return 1 + fi + + # Return to repository root + navigate-to-repo-root || return 1 + + # Add to the PATH + add-to-path "$PWD/thirdparty/cppfront" + + # Level 2: Validation step + if [ "$debug_level" -ge 2 ]; then + echo "[DEBUG] Validating cppfront-compiler binary" + fi + + # Validate binary was created + if [ -x "./thirdparty/cppfront/cppfront-compiler" ]; then + echo "✓ cppfront-compiler binary created and executable" + + # Test the binary runs correctly + echo "Testing cppfront-compiler..." + set +e # Temporarily disable exit on error + + # Debug output for validation command + if [ "$debug_level" -gt 3 ]; then + echo "[DEBUG] About to run: ./thirdparty/cppfront/cppfront-compiler -version" + fi + ./thirdparty/cppfront/cppfront-compiler -version + test_exit_code=$? + set -e # Re-enable exit on error + + if [ "$test_exit_code" -eq 0 ] || [ "$test_exit_code" -eq 1 ]; then + echo "✓ cppfront-compiler runs correctly (exit code: $test_exit_code)" + else + echo "⚠ WARNING: cppfront-compiler returned unexpected exit code: $test_exit_code" + echo "But binary exists and is executable, continuing..." + fi + else + echo "✗ ERROR: cppfront-compiler binary not found or not executable" + return 1 + fi + + # End timing for debug levels > 3 + if [ "$debug_level" -gt 3 ]; then + end_time=$(date +%s) + local duration=$((end_time - start_time)) + echo "[DEBUG] build-cppfront completed in $duration seconds" + fi + + echo "cppfront-compiler built and validated successfully ( ./thirdparty/cppfront/cppfront-compiler )" + } + + # Check if cppfront needs rebuilding based on age + check-cppfront-age() { + if [ -n "$XDP2_NIX_DEBUG" ]; then + local debug_level=$XDP2_NIX_DEBUG + else + local debug_level=0 + fi + local start_time="" + local end_time="" + + # Start timing for debug levels > 3 + if [ "$debug_level" -gt 3 ]; then + start_time=$(date +%s) + echo "[DEBUG] check-cppfront-age started at $(date)" + fi + + # Level 1: Function start + if [ "$debug_level" -ge 1 ]; then + echo "[DEBUG] Starting check-cppfront-age function" + fi + + local cppfront_binary="thirdparty/cppfront/cppfront-compiler" + + # Level 2: File check + if [ "$debug_level" -ge 2 ]; then + echo "[DEBUG] Checking cppfront binary: $cppfront_binary" + fi + + if [ -f "$cppfront_binary" ]; then + local file_time + file_time=$(stat -c %Y "$cppfront_binary") + local current_time + current_time=$(date +%s) + local age_days=$(( (current_time - file_time) / 86400 )) + + # Level 3: Age calculation details + if [ "$debug_level" -ge 3 ]; then + echo "[DEBUG] File modification time: $file_time" + echo "[DEBUG] Current time: $current_time" + echo "[DEBUG] Calculated age: $age_days days" + fi + + if [ "$age_days" -gt 7 ]; then + echo "cppfront is $age_days days old, rebuilding..." + build-cppfront + else + echo "cppfront is up to date ($age_days days old)" + fi + else + echo "cppfront not found, building..." + build-cppfront + fi + + # End timing for debug levels > 3 + if [ "$debug_level" -gt 3 ]; then + end_time=$(date +%s) + local duration=$((end_time - start_time)) + echo "[DEBUG] check-cppfront-age completed in $duration seconds" + fi + } + + # Build the xdp2 compiler + build-xdp2-compiler() { + if [ -n "$XDP2_NIX_DEBUG" ]; then + local debug_level=$XDP2_NIX_DEBUG + else + local debug_level=0 + fi + local start_time="" + local end_time="" + + # Start timing for debug levels > 3 + if [ "$debug_level" -gt 3 ]; then + start_time=$(date +%s) + echo "[DEBUG] build-xdp2-compiler started at $(date)" + fi + + # Level 1: Function start + if [ "$debug_level" -ge 1 ]; then + echo "[DEBUG] Starting build-xdp2-compiler function" + fi + + # Level 2: Clean step + if [ "$debug_level" -ge 2 ]; then + echo "[DEBUG] Cleaning xdp2-compiler build directory" + fi + echo "Cleaning and building xdp2-compiler..." + + # Navigate to repository root first + navigate-to-repo-root || return 1 + + # Clean previous build artifacts (before navigating to component) + clean-xdp2-compiler + + # Navigate to xdp2-compiler directory + navigate-to-component "src/tools/compiler" || return 1 + + # Level 3: Build step details + if [ "$debug_level" -ge 3 ]; then + echo "[DEBUG] Building xdp2-compiler with make" + fi + + # Build xdp2-compiler with error checking + if CFLAGS_PYTHON="$CFLAGS_PYTHON" LDFLAGS_PYTHON="$LDFLAGS_PYTHON" make -j"$NIX_BUILD_CORES"; then + echo "✓ xdp2-compiler make completed successfully" + else + echo "✗ ERROR: xdp2-compiler make failed" + return 1 + fi + + # Level 2: Validation step + if [ "$debug_level" -ge 2 ]; then + echo "[DEBUG] Validating xdp2-compiler binary" + fi + + # Validate binary was created + if [ -x "./xdp2-compiler" ]; then + echo "✓ xdp2-compiler binary created and executable" + + # Test the binary runs correctly + echo "Testing xdp2-compiler..." + set +e # Temporarily disable exit on error + + # Debug output for validation command + if [ "$debug_level" -gt 3 ]; then + echo "[DEBUG] About to run: ./xdp2-compiler --help" + fi + ./xdp2-compiler --help + test_exit_code=$? + set -e # Re-enable exit on error + + if [ "$test_exit_code" -eq 0 ] || [ "$test_exit_code" -eq 1 ]; then + echo "✓ xdp2-compiler runs correctly (exit code: $test_exit_code)" + else + echo "⚠ WARNING: xdp2-compiler returned unexpected exit code: $test_exit_code" + echo "But binary exists and is executable, continuing..." + fi + else + echo "✗ ERROR: xdp2-compiler binary not found or not executable" + return 1 + fi + + # Return to repository root + navigate-to-repo-root || return 1 + + # End timing for debug levels > 3 + if [ "$debug_level" -gt 3 ]; then + end_time=$(date +%s) + local duration=$((end_time - start_time)) + echo "[DEBUG] build-xdp2-compiler completed in $duration seconds" + fi + + echo "xdp2-compiler built and validated successfully ( ./src/tools/compiler/xdp2-compiler )" + } + + # Build the main xdp2 project + build-xdp2() { + if [ -n "$XDP2_NIX_DEBUG" ]; then + local debug_level=$XDP2_NIX_DEBUG + else + local debug_level=0 + fi + local start_time="" + local end_time="" + + # Start timing for debug levels > 3 + if [ "$debug_level" -gt 3 ]; then + start_time=$(date +%s) + echo "[DEBUG] build-xdp2 started at $(date)" + fi + + # Level 1: Function start + if [ "$debug_level" -ge 1 ]; then + echo "[DEBUG] Starting build-xdp2 function" + fi + + # Level 2: Clean step + if [ "$debug_level" -ge 2 ]; then + echo "[DEBUG] Cleaning xdp2 project build directory" + fi + echo "Cleaning and building xdp2 project..." + + # Navigate to repository root first + navigate-to-repo-root || return 1 + + # Clean previous build artifacts (before navigating to component) + clean-xdp2 + + # Navigate to src directory + navigate-to-component "src" || return 1 + + # Ensure xdp2-compiler is available in PATH + add-to-path "$PWD/tools/compiler" + echo "Added tools/compiler to PATH" + + # Level 3: Build step details + if [ "$debug_level" -ge 3 ]; then + echo "[DEBUG] Building xdp2 project with make" + fi + + # Build the main xdp2 project + if make -j"$NIX_BUILD_CORES"; then + echo "✓ xdp2 project make completed successfully" + else + echo "✗ ERROR: xdp2 project make failed" + echo " Check the error messages above for details" + return 1 + fi + + # Return to repository root + navigate-to-repo-root || return 1 + + # End timing for debug levels > 3 + if [ "$debug_level" -gt 3 ]; then + end_time=$(date +%s) + local duration=$((end_time - start_time)) + echo "[DEBUG] build-xdp2 completed in $duration seconds" + fi + + echo "xdp2 project built successfully" + } + + # Build all XDP2 components + build-all() { + if [ -n "$XDP2_NIX_DEBUG" ]; then + local debug_level=$XDP2_NIX_DEBUG + else + local debug_level=0 + fi + + echo "Building all XDP2 components..." + + if [ "$debug_level" -ge 3 ]; then + echo "[DEBUG] Building cppfront: build-cppfront" + fi + build-cppfront + + if [ "$debug_level" -ge 3 ]; then + echo "[DEBUG] Building xdp2-compiler: build-xdp2-compiler" + fi + build-xdp2-compiler + + if [ "$debug_level" -ge 3 ]; then + echo "[DEBUG] Building xdp2: build-xdp2" + fi + build-xdp2 + + echo "✓ All components built successfully" + } +'' diff --git a/nix/shell-functions/clean.nix b/nix/shell-functions/clean.nix new file mode 100644 index 0000000..9c8e9dd --- /dev/null +++ b/nix/shell-functions/clean.nix @@ -0,0 +1,103 @@ +# nix/shell-functions/clean.nix +# +# Clean shell functions for XDP2 +# +# Functions: +# - clean-cppfront: Clean cppfront build artifacts +# - clean-xdp2-compiler: Clean xdp2-compiler build artifacts +# - clean-xdp2: Clean xdp2 build artifacts +# - clean-all: Clean all build artifacts +# +# Note: These functions depend on navigation functions from navigation.nix +# +# Usage in flake.nix: +# cleanFns = import ./nix/shell-functions/clean.nix { }; +# + +{ }: + +'' + # Clean cppfront build artifacts + clean-cppfront() { + if [ -n "$XDP2_NIX_DEBUG" ]; then + local debug_level=$XDP2_NIX_DEBUG + else + local debug_level=0 + fi + + # Navigate to repository root first + navigate-to-repo-root || return 1 + + # Navigate to cppfront directory + navigate-to-component "thirdparty/cppfront" || return 1 + + # Debug output for clean command + if [ "$debug_level" -gt 3 ]; then + echo "[DEBUG] About to run: rm -f cppfront-compiler" + fi + rm -f cppfront-compiler # Remove the binary directly since Makefile has no clean target + + # Return to repository root + navigate-to-repo-root || return 1 + } + + # Clean xdp2-compiler build artifacts + clean-xdp2-compiler() { + if [ -n "$XDP2_NIX_DEBUG" ]; then + local debug_level=$XDP2_NIX_DEBUG + else + local debug_level=0 + fi + + # Navigate to repository root first + navigate-to-repo-root || return 1 + + # Navigate to xdp2-compiler directory + navigate-to-component "src/tools/compiler" || return 1 + + # Debug output for clean command + if [ "$debug_level" -gt 3 ]; then + echo "[DEBUG] About to run: make clean" + fi + make clean || true # Don't fail if clean fails + + # Return to repository root + navigate-to-repo-root || return 1 + } + + # Clean xdp2 build artifacts + clean-xdp2() { + if [ -n "$XDP2_NIX_DEBUG" ]; then + local debug_level=$XDP2_NIX_DEBUG + else + local debug_level=0 + fi + + # Navigate to repository root first + navigate-to-repo-root || return 1 + + # Navigate to src directory + navigate-to-component "src" || return 1 + + # Debug output for clean command + if [ "$debug_level" -gt 3 ]; then + echo "[DEBUG] About to run: make clean" + fi + make clean || true # Don't fail if clean fails + + # Return to repository root + navigate-to-repo-root || return 1 + } + + # Clean all build artifacts + clean-all() { + echo "Cleaning all build artifacts..." + + # Clean each component using centralized clean functions + clean-cppfront + clean-xdp2-compiler + clean-xdp2 + + echo "✓ All build artifacts cleaned" + } +'' diff --git a/nix/shell-functions/configure.nix b/nix/shell-functions/configure.nix new file mode 100644 index 0000000..01c6c3b --- /dev/null +++ b/nix/shell-functions/configure.nix @@ -0,0 +1,62 @@ +# nix/shell-functions/configure.nix +# +# Configure shell functions for XDP2 +# +# Functions: +# - smart-configure: Smart configure script with age checking +# +# Note: These functions depend on navigation functions from navigation.nix +# +# Usage in flake.nix: +# configureFns = import ./nix/shell-functions/configure.nix { configAgeWarningDays = 14; }; +# + +{ configAgeWarningDays ? 14 }: + +'' + # Smart configure script execution with age checking + # This simply includes a check to see if the config.mk file exists, and + # it generates a warning if the file is older than the threshold + smart-configure() { + local config_file="./src/config.mk" + local warning_days=${toString configAgeWarningDays} + + if [ -f "$config_file" ]; then + echo "✓ config.mk found, skipping configure step" + + # Check age of config.mk + local file_time + file_time=$(stat -c %Y "$config_file") + local current_time + current_time=$(date +%s) + local age_days=$(( (current_time - file_time) / 86400 )) + + if [ "$age_days" -gt "$warning_days" ]; then + echo "⚠️ WARNING: config.mk is $age_days days old (threshold: $warning_days days)" + echo " Consider running 'configure' manually if you've made changes to:" + echo " • Build configuration" + echo " • Compiler settings" + echo " • Library paths" + echo " • Platform-specific settings" + echo "" + else + echo "✓ config.mk is up to date ($age_days days old)" + fi + else + echo "config.mk not found, running configure script..." + cd src || return 1 + rm -f config.mk + ./configure.sh --build-opt-parser + + # Apply PATH_ARG fix for Nix environment + if grep -q 'PATH_ARG="--with-path=' config.mk; then + echo "Applying PATH_ARG fix for Nix environment..." + sed -i 's|PATH_ARG="--with-path=.*"|PATH_ARG=""|' config.mk + fi + echo "PATH_ARG in config.mk: $(grep '^PATH_ARG=' config.mk)" + + cd .. || return 1 + echo "✓ config.mk generated successfully" + fi + } +'' diff --git a/nix/shell-functions/navigation.nix b/nix/shell-functions/navigation.nix new file mode 100644 index 0000000..7e325a6 --- /dev/null +++ b/nix/shell-functions/navigation.nix @@ -0,0 +1,84 @@ +# nix/shell-functions/navigation.nix +# +# Navigation shell functions for XDP2 +# +# Functions: +# - detect-repository-root: Detect and export XDP2_REPO_ROOT +# - navigate-to-repo-root: Change to repository root directory +# - navigate-to-component: Change to a component subdirectory +# - add-to-path: Add a directory to PATH if not already present +# +# Usage in flake.nix: +# navigationFns = import ./nix/shell-functions/navigation.nix { }; +# + +{ }: + +'' + # Detect and export the repository root directory + detect-repository-root() { + XDP2_REPO_ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd) + export XDP2_REPO_ROOT + + if [ ! -d "$XDP2_REPO_ROOT" ]; then + echo "⚠ WARNING: Could not detect valid repository root" + XDP2_REPO_ROOT="$PWD" + else + echo "📁 Repository root: $XDP2_REPO_ROOT" + fi + } + + # Navigate to the repository root directory + navigate-to-repo-root() { + if [ -n "$XDP2_REPO_ROOT" ]; then + cd "$XDP2_REPO_ROOT" || return 1 + else + echo "✗ ERROR: XDP2_REPO_ROOT not set" + return 1 + fi + } + + # Navigate to a component subdirectory + navigate-to-component() { + local component="$1" + local target_dir="$XDP2_REPO_ROOT/$component" + + if [ ! -d "$target_dir" ]; then + echo "✗ ERROR: Component directory not found: $target_dir" + return 1 + fi + + cd "$target_dir" || return 1 + } + + # Add path to PATH environment variable if not already present + add-to-path() { + local path_to_add="$1" + + if [ -n "$XDP2_NIX_DEBUG" ]; then + local debug_level=$XDP2_NIX_DEBUG + else + local debug_level=0 + fi + + # Check if path is already in PATH + if [[ ":$PATH:" == *":$path_to_add:"* ]]; then + if [ "$debug_level" -gt 3 ]; then + echo "[DEBUG] Path already in PATH: $path_to_add" + fi + return 0 + fi + + # Add path to beginning of PATH + if [ "$debug_level" -gt 3 ]; then + echo "[DEBUG] Adding to PATH: $path_to_add" + echo "[DEBUG] PATH before: $PATH" + fi + + export PATH="$path_to_add:$PATH" + + if [ "$debug_level" -gt 3 ]; then + echo "[DEBUG] PATH after: $PATH" + fi + } +'' diff --git a/nix/shell-functions/validation.nix b/nix/shell-functions/validation.nix new file mode 100644 index 0000000..04b1f94 --- /dev/null +++ b/nix/shell-functions/validation.nix @@ -0,0 +1,179 @@ +# nix/shell-functions/validation.nix +# +# Validation and help shell functions for XDP2 +# +# Functions: +# - check-platform-compatibility: Check if running on Linux +# - setup-locale-support: Configure locale settings +# - run-shellcheck: Validate all shell functions with shellcheck +# - xdp2-help: Display help information +# +# Usage in flake.nix: +# validationFns = import ./nix/shell-functions/validation.nix { +# inherit lib; +# shellcheckFunctionRegistry = [ "func1" "func2" ... ]; +# }; +# + +{ lib, shellcheckFunctionRegistry ? [] }: + +let + # Generate shellcheck validation function + totalFunctions = builtins.length shellcheckFunctionRegistry; + + # Generate individual function checks + functionChecks = lib.concatStringsSep "\n" (map (name: '' + echo "Checking ${name}..." + if declare -f "${name}" >/dev/null 2>&1; then + # Create temporary script with function definition + # TODO use mktemp and trap 'rm -f "$temp_script"' EXIT + local temp_script="/tmp/validate_${name}.sh" + declare -f "${name}" > "$temp_script" + echo "#!/bin/bash" > "$temp_script.tmp" + cat "$temp_script" >> "$temp_script.tmp" + mv "$temp_script.tmp" "$temp_script" + + # Run shellcheck on the function + if shellcheck -s bash "$temp_script" 2>/dev/null; then + echo "✓ ${name} passed shellcheck validation" + passed_functions=$((passed_functions + 1)) + else + echo "✗ ${name} failed shellcheck validation:" + shellcheck -s bash "$temp_script" + failed_functions+=("${name}") + fi + rm -f "$temp_script" + else + echo "✗ ${name} not found" + failed_functions+=("${name}") + fi + echo "" + '') shellcheckFunctionRegistry); + + # Generate failed functions reporting + failedFunctionsReporting = lib.concatStringsSep "\n" (map (name: '' + if [[ "$${failed_functions[*]}" == *"${name}"* ]]; then + echo " - ${name}" + fi + '') shellcheckFunctionRegistry); + + # Bash variable expansion helpers for setup-locale-support + bashVarExpansion = "$"; + bashDefaultSyntax = "{LANG:-C.UTF-8}"; + bashDefaultSyntaxLC = "{LC_ALL:-C.UTF-8}"; + +in +'' + # Check platform compatibility (Linux only) + check-platform-compatibility() { + if [ "$(uname)" != "Linux" ]; then + echo "⚠️ PLATFORM COMPATIBILITY NOTICE +================================== + +🍎 You are running on $(uname) (not Linux) + +The XDP2 development environment includes Linux-specific packages +like libbpf that are not available on $(uname) systems. + +📋 Available platforms: + ✅ Linux (x86_64-linux, aarch64-linux, etc.) + ❌ macOS (x86_64-darwin, aarch64-darwin) + ❌ Other Unix systems + +Exiting development shell..." + exit 1 + fi + } + + # Setup locale support for cross-distribution compatibility + setup-locale-support() { + # Only set locale if user hasn't already configured it + if [ -z "$LANG" ] || [ -z "$LC_ALL" ]; then + # Try to use system default, fallback to C.UTF-8 + export LANG=${bashVarExpansion}${bashDefaultSyntax} + export LC_ALL=${bashVarExpansion}${bashDefaultSyntaxLC} + fi + + # Verify locale is available (only if locale command exists) + if command -v locale >/dev/null 2>&1; then + if ! locale -a 2>/dev/null | grep -q "$LANG"; then + # Fallback to C.UTF-8 if user's locale is not available + export LANG=C.UTF-8 + export LC_ALL=C.UTF-8 + fi + fi + } + + # Run shellcheck validation on all registered shell functions + run-shellcheck() { + if [ -n "$XDP2_NIX_DEBUG" ]; then + local debug_level=$XDP2_NIX_DEBUG + else + local debug_level=0 + fi + + echo "Running shellcheck validation on shell functions..." + + local failed_functions=() + local total_functions=${toString totalFunctions} + local passed_functions=0 + + # Pre-generated function checks from Nix + ${functionChecks} + + # Report results + echo "=== Shellcheck Validation Complete ===" + echo "Total functions: $total_functions" + echo "Passed: $passed_functions" + echo "Failed: $((total_functions - passed_functions))" + + if [ $((total_functions - passed_functions)) -eq 0 ]; then + echo "✓ All functions passed shellcheck validation" + return 0 + else + echo "✗ Some functions failed validation:" + # Pre-generated failed functions reporting from Nix + ${failedFunctionsReporting} + return 1 + fi + } + + # Display help information for XDP2 development shell + xdp2-help() { + echo "🚀 === XDP2 Development Shell Help === + +📦 Compiler: GCC +🔧 GCC and Clang are available in the environment. +🐛 Debugging tools: gdb, valgrind, strace, ltrace + +🔍 DEBUGGING: + XDP2_NIX_DEBUG=0 - No extra debug. Default + XDP2_NIX_DEBUG=3 - Basic debug + XDP2_NIX_DEBUG=5 - Show compiler selection and config.mk + XDP2_NIX_DEBUG=7 - Show all debug info + +🔧 BUILD COMMANDS: + build-cppfront - Build cppfront compiler + build-xdp2-compiler - Build xdp2 compiler + build-xdp2 - Build main XDP2 project + build-all - Build all components + +🧹 CLEAN COMMANDS: + clean-cppfront - Clean cppfront build artifacts + clean-xdp2-compiler - Clean xdp2-compiler build artifacts + clean-xdp2 - Clean xdp2 build artifacts + clean-all - Clean all build artifacts + +🔍 VALIDATION: + run-shellcheck - Validate all shell functions + +📁 PROJECT STRUCTURE: + • src/ - Main source code + • tools/ - Build tools and utilities + • thirdparty/ - Third-party dependencies + • samples/ - Example code and parsers + • documentation/ - Project documentation + +🎯 Ready to develop! 'xdp2-help' for help" + } +'' diff --git a/src/configure.sh b/src/configure.sh index 9caf6b5..298f36a 100755 --- a/src/configure.sh +++ b/src/configure.sh @@ -769,8 +769,14 @@ fi echo "XDP2_CLANG_VERSION=${XDP2_CLANG_VERSION}" echo "XDP2_CLANG_VERSION=${XDP2_CLANG_VERSION}" >> $CONFIG -XDP2_CLANG_RESOURCE_PATH="`${HOST_LLVM_CONFIG} --libdir`/clang/`echo ${XDP2_CLANG_VERSION} | cut -d'.' -f1`" -XDP2_C_INCLUDE_PATH="`${HOST_LLVM_CONFIG} --libdir`/clang/`echo ${XDP2_CLANG_VERSION} | cut -d'.' -f1`/include" + +# Check for environment variable overrides (useful for Nix which has different path structure) +if [ -z "${XDP2_CLANG_RESOURCE_PATH}" ]; then + XDP2_CLANG_RESOURCE_PATH="`${HOST_LLVM_CONFIG} --libdir`/clang/`echo ${XDP2_CLANG_VERSION} | cut -d'.' -f1`" +fi +if [ -z "${XDP2_C_INCLUDE_PATH}" ]; then + XDP2_C_INCLUDE_PATH="${XDP2_CLANG_RESOURCE_PATH}/include" +fi echo "XDP2_C_INCLUDE_PATH=${XDP2_C_INCLUDE_PATH}" echo "XDP2_C_INCLUDE_PATH=${XDP2_C_INCLUDE_PATH}" >> $CONFIG echo "XDP2_CLANG_RESOURCE_PATH=${XDP2_CLANG_RESOURCE_PATH}"