Skip to content

Add asynchronous archive creation with concurrent file processing#3

Draft
Copilot wants to merge 4 commits intomainfrom
copilot/fix-2
Draft

Add asynchronous archive creation with concurrent file processing#3
Copilot wants to merge 4 commits intomainfrom
copilot/fix-2

Conversation

Copy link

Copilot AI commented Jun 15, 2025

This PR implements optional asynchronous archive creation for the go-pxar library, allowing concurrent file processing while maintaining full compatibility with the existing synchronous operations and PXAR format requirements.

Features Added

Async Configuration Options

pa := PBSArchive{
    Filename:     "backup.pxar",
    AsyncMode:    true,        // Enable async processing
    AsyncWorkers: 4,           // Use 4 concurrent workers (optional, defaults to CPU count)
}

New Async Methods

  • ToBufferAsync() - Asynchronous archive creation to buffer
  • ToChannelAsync() - Asynchronous archive creation to channel
  • Extended NodeRef interface with async methods for all node types

Usage Example

// Synchronous (existing behavior)
err := pa.ToBuffer(buf)

// Asynchronous (new functionality)  
err := pa.ToBufferAsync(buf)

// Both produce identical results

Technical Implementation

The async implementation uses a worker pool pattern where:

  1. Concurrent Processing: Multiple files are read and processed simultaneously using configurable worker goroutines
  2. Ordered Assembly: Results are collected and buffered in the correct order to maintain PXAR format requirements
  3. Sequential Writing: Final archive writing maintains proper positioning and goodbye record calculations
  4. Recursive Async: Nested directories use async methods recursively for maximum performance benefit

Key Benefits

  • Performance: Concurrent file I/O processing for improved throughput on multi-file archives
  • Format Compliance: Maintains exact PXAR format requirements with correct ordering
  • Identical Results: Async and sync methods produce byte-for-byte identical archives (verified by comprehensive tests)
  • Backward Compatibility: All existing code continues to work unchanged
  • Configurable: Worker count can be tuned based on system capabilities

Testing

Comprehensive test suite added including:

  • Sync vs async result validation (identical output verification)
  • Multiple worker configuration testing (1, 2, 4, 8 workers)
  • Consistency testing across multiple async runs
  • Performance benchmarking
  • Large file handling validation

All existing tests pass with no regressions.

Documentation

Updated README.md with:

  • Usage examples for async functionality
  • Configuration options explanation
  • Performance characteristics description

Fixes #2.


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Copilot AI and others added 3 commits June 15, 2025 22:50
Co-authored-by: scheibling <24367830+scheibling@users.noreply.github.com>
Co-authored-by: scheibling <24367830+scheibling@users.noreply.github.com>
Co-authored-by: scheibling <24367830+scheibling@users.noreply.github.com>
Copilot AI changed the title [WIP] Add asynchronous archive creation Add asynchronous archive creation with concurrent file processing Jun 15, 2025
Copilot AI requested a review from scheibling June 15, 2025 23:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add asynchronous archive creation

2 participants