Skip to content

Add SQL dump optimizer for INSERT batching#32

Merged
w3spi5 merged 6 commits intomainfrom
develop
Dec 31, 2025
Merged

Add SQL dump optimizer for INSERT batching#32
w3spi5 merged 6 commits intomainfrom
develop

Conversation

@w3spi5
Copy link
Owner

@w3spi5 w3spi5 commented Dec 31, 2025

Add standalone CLI tool (cli.php) that rewrites SQL dump files with
INSERT statement batching for optimized imports.

Features:

  • Reads .sql, .sql.gz, .sql.bz2 input files
  • Outputs optimized .sql (uncompressed)
  • Two profiles: conservative (2000) and aggressive (5000)
  • Custom --batch-size override
  • Time-based progress reporting (every 2s)
  • Standalone mode (no config.php or database required)

Usage:
php cli.php dump.sql -o optimized.sql
php cli.php dump.sql.gz -o optimized.sql --profile=aggressive
php cli.php dump.sql -o optimized.sql --batch-size=3000

New files:

  • cli.php (entry point)
  • src/Services/CliFileReader.php
  • src/Services/CliSqlParser.php
  • src/Services/CliOptimizerService.php
  • 7 test files (38 tests)

  SSE Reliability Fixes:
  - Fix PHP session locking blocking SSE connections
  - Add session_write_close() before rendering import page
  - Add SSE retry mechanism with exponential backoff (100ms-1600ms)
  - Fix output buffering order in SseService (ini_set before ob_end_clean)
  - Add ~8KB padding to force Apache/mod_fcgid buffer flush
  - Change return to exit after SSE error to prevent Response corruption

  Documentation:
  - Add "How It Works" section explaining staggered import behavior
  - Enhance Troubleshooting with Laragon-specific config paths
  - Add quick diagnostic command (php -S localhost:8000)
  - Document server configurations for SSE (Apache, nginx, Laragon)

  Git Cleanup:
  - Fix .gitignore to use **/CLAUDE.md pattern for all subdirectories
  - Remove CLAUDE.md files from git tracking
Add support for .bz2 and .sql.bz2 compressed SQL dump imports alongside
  existing .gz support, with full resume functionality via seek workaround.

  Key features:
  - BZ2 file detection with case-insensitive extension matching
  - bzopen/bzread/bzclose integration in FileHandler
  - ADR-001: Seek workaround using re-read strategy (O(n) resume time)
  - Conditional support based on PHP ext-bz2 availability
  - Purple "BZ2" badge in file listing
  - Frontend validation with custom error messages
  - Comprehensive test suite (24 tests across 4 files)

  Files modified:
  - src/Models/FileHandler.php - Core BZ2 support
  - src/Config/Config.php - Extension config + isBz2Supported()
  - templates/home.php - UI badge + data-bz2-supported
  - assets/js/fileupload.js - Conditional validation

  Test files created:
  - tests/FileHandlerBz2Test.php
  - tests/ConfigBz2Test.php
  - tests/FrontendBz2Test.php
  - tests/Bz2IntegrationTest.php
  - tests/fixtures/test_bz2_import.sql.bz2
@w3spi5 w3spi5 merged commit d922333 into main Dec 31, 2025
1 check passed
w3spi5 added a commit that referenced this pull request Jan 13, 2026
* feat(sse): fix session locking and improve reliability

  SSE Reliability Fixes:
  - Fix PHP session locking blocking SSE connections
  - Add session_write_close() before rendering import page
  - Add SSE retry mechanism with exponential backoff (100ms-1600ms)
  - Fix output buffering order in SseService (ini_set before ob_end_clean)
  - Add ~8KB padding to force Apache/mod_fcgid buffer flush
  - Change return to exit after SSE error to prevent Response corruption

  Documentation:
  - Add "How It Works" section explaining staggered import behavior
  - Enhance Troubleshooting with Laragon-specific config paths
  - Add quick diagnostic command (php -S localhost:8000)
  - Document server configurations for SSE (Apache, nginx, Laragon)

  Git Cleanup:
  - Fix .gitignore to use **/CLAUDE.md pattern for all subdirectories
  - Remove CLAUDE.md files from git tracking

* feat(compression): add BZ2 (bzip2) compressed file support

Add support for .bz2 and .sql.bz2 compressed SQL dump imports alongside
  existing .gz support, with full resume functionality via seek workaround.

  Key features:
  - BZ2 file detection with case-insensitive extension matching
  - bzopen/bzread/bzclose integration in FileHandler
  - ADR-001: Seek workaround using re-read strategy (O(n) resume time)
  - Conditional support based on PHP ext-bz2 availability
  - Purple "BZ2" badge in file listing
  - Frontend validation with custom error messages
  - Comprehensive test suite (24 tests across 4 files)

  Files modified:
  - src/Models/FileHandler.php - Core BZ2 support
  - src/Config/Config.php - Extension config + isBz2Supported()
  - templates/home.php - UI badge + data-bz2-supported
  - assets/js/fileupload.js - Conditional validation

  Test files created:
  - tests/FileHandlerBz2Test.php
  - tests/ConfigBz2Test.php
  - tests/FrontendBz2Test.php
  - tests/Bz2IntegrationTest.php
  - tests/fixtures/test_bz2_import.sql.bz2

* build: Auto-compiled assets [skip ci]

* remove legacy XML response code (closes #29)

* add SQL dump optimizer for INSERT batching
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

Comments