Skip to content

Add benchmark check #268

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
superdav42 wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Add benchmark check #268

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
98c4bd8
Add benchmark check
superdav42 Nov 15, 2025
c218952
Add some more tests
superdav42 Nov 18, 2025
b2b7d4f
add doc
superdav42 Nov 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
191 changes: 191 additions & 0 deletions .github/workflows/performance.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,191 @@
name: Performance Tests

on:
pull_request:
branches: [main]
paths-ignore:
- '**.md'
- 'docs/**'
push:
branches: [main]
paths-ignore:
- '**.md'
- 'docs/**'

jobs:
performance:
name: Performance Benchmark
runs-on: ubuntu-latest

services:
mysql:
image: mysql:5.7
env:
MYSQL_ROOT_PASSWORD: root
ports: [3306]
options: >-
--health-cmd="mysqladmin ping --silent"
--health-interval=10s
--health-timeout=5s
--health-retries=5

container:
image: cimg/php:8.1
options: --user root

steps:
- name: Checkout PR Branch
uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Checkout Base Branch
uses: actions/checkout@v4
with:
ref: ${{ github.base_ref }}
path: baseline
fetch-depth: 0

- name: Set up Composer global bin path
run: |
mkdir -p "$HOME/.config/composer/vendor/bin"
echo "PATH=$HOME/.config/composer/vendor/bin:$PATH" >> $GITHUB_ENV

- name: Install System Dependencies
run: |
apt-get update && apt-get install -y subversion mariadb-client

- name: Install PHP Extensions
run: |
install-php-extensions mysqli gd bcmath || echo "Extensions may already be installed."

- name: Self-update Composer
run: |
composer self-update || echo "Composer update skipped due to permission issue."

- name: Install PHP Dependencies (PR Branch)
run: |
composer install --no-interaction --prefer-dist

- name: Install PHP Dependencies (Baseline)
run: |
cd baseline && composer install --no-interaction --prefer-dist

- name: Wait for MySQL to be ready
run: |
for i in {1..30}; do
if mysqladmin ping -h mysql --silent; then
echo "MySQL is up"
break
fi
echo "Waiting for MySQL..."
sleep 2
done

- name: Prepare WordPress Tests
run: |
rm -rf /tmp/wordpress-tests-lib /tmp/wordpress/
bash bin/install-wp-tests.sh wordpress_test root root mysql latest

- name: Run Baseline Performance Tests
run: |
cd baseline
php tests/performance-benchmark.php > ../baseline-results.json
echo "Baseline results:"
cat ../baseline-results.json | jq '.'

- name: Run Current Branch Performance Tests
run: |
php tests/performance-benchmark.php > current-results.json
echo "Current results:"
cat current-results.json | jq '.'
Comment on lines +90 to +101
Copy link
Contributor

@coderabbitai coderabbitai bot Nov 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Critical: Redirecting stdout creates malformed JSON files.

Lines 93 and 99 redirect the entire script output to JSON files using >, which includes console echo statements alongside the JSON output. This creates the malformed files seen in current-test.json and baseline-test.json (status messages + JSON + "Results saved" text).

The benchmark scripts already save JSON via their save_results() methods. Either use those saved files, or capture only the JSON output.

Option 1 - Use the saved JSON files instead of stdout redirect:

       - name: Run Baseline Performance Tests
         run: |
           cd baseline
-          php tests/performance-benchmark.php > ../baseline-results.json
+          php tests/performance-benchmark.php
+          cp tests/performance-results-*.json ../baseline-results.json
           echo "Baseline results:"
           cat ../baseline-results.json | jq '.'
       
       - name: Run Current Branch Performance Tests
         run: |
-          php tests/performance-benchmark.php > current-results.json
+          php tests/performance-benchmark.php
+          cp tests/performance-results-*.json current-results.json
           echo "Current results:"
           cat current-results.json | jq '.'

Option 2 - Suppress console output and only output JSON:

Modify the benchmark scripts to only echo JSON when stdout is redirected, or add a --quiet flag.

Committable suggestion skipped: line range outside the PR's diff.


- name: Compare Performance Results
run: |
php tests/performance-comparator.php baseline-results.json current-results.json > performance-report.md
echo "Performance comparison:"
cat performance-report.md

- name: Upload Performance Results
uses: actions/upload-artifact@v4
if: always()
with:
name: performance-results-${{ github.run_number }}
path: |
baseline-results.json
current-results.json
performance-report.md
retention-days: 30

- name: Comment PR with Performance Results
uses: actions/github-script@v7
if: github.event_name == 'pull_request'
with:
script: |
const fs = require('fs');

try {
const report = fs.readFileSync('performance-report.md', 'utf8');

// Find previous performance comment
const { data: comments } = await github.rest.issues.listComments({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
});

const existingComment = comments.find(comment =>
comment.user.type === 'Bot' &&
comment.body.includes('Performance Test Results')
);

if (existingComment) {
// Update existing comment
await github.rest.issues.updateComment({
comment_id: existingComment.id,
owner: context.repo.owner,
repo: context.repo.repo,
body: report
});
} else {
// Create new comment
await github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: report
});
}
} catch (error) {
console.log('Error posting performance comment:', error);
}

- name: Check for Performance Regressions
run: |
# Check if performance report contains critical regressions
if grep -q "Critical Regressions.*[1-9]" performance-report.md; then
echo "🚨 Critical performance regressions detected!"
echo "Please review the performance report and optimize before merging."
exit 1
fi

# Check if performance report contains any regressions
if grep -q "Regressions.*[1-9]" performance-report.md; then
echo "⚠️ Performance regressions detected."
echo "Consider optimizing before merging, but not blocking."
fi

echo "✅ Performance tests passed!"

- name: Performance Summary
if: always()
run: |
echo "## Performance Test Summary" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY

if [ -f "performance-report.md" ]; then
# Extract summary from performance report
sed -n '/## Summary/,/## Detailed Results/p' performance-report.md >> $GITHUB_STEP_SUMMARY
else
echo "Performance report not available." >> $GITHUB_STEP_SUMMARY
fi
49 changes: 49 additions & 0 deletions baseline-test.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
Starting simplified performance benchmarks...
✓ Dashboard loading benchmark completed
✓ Checkout process benchmark completed
✓ Site creation validation benchmark completed
✓ Membership operations benchmark completed
✓ API endpoints benchmark completed
✓ Database queries benchmark completed
{
"test_run": "2025-12.55 20:22:06",
Copy link
Contributor

@coderabbitai coderabbitai bot Nov 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Invalid timestamp format in baseline data.

The timestamp "2025-12.55 20:22:06" is malformed—month cannot be "12.55". This should follow the format "Y-m-d H:i:s" (e.g., "2025-11-15 20:22:06"). While this won't break JSON parsing if the file format is fixed, it will cause confusion in reports and may break date parsing logic downstream.

Apply this diff:

-    "test_run": "2025-12.55 20:22:06",
+    "test_run": "2025-11-15 20:22:06",
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
"test_run": "2025-12.55 20:22:06",
"test_run": "2025-11-15 20:22:06",
🧰 Tools
🪛 Biome (2.1.2)

[error] 8-9: String values must be double quoted.

(parse)

[error] 9-49: End of file expected

Use an array for a sequence of values: [1, 2]

(parse)

🤖 Prompt for AI Agents 
In baseline-test.json around line 9, the "test_run" timestamp "2025-12.55
20:22:06" is malformed (month contains decimal); replace it with a valid "Y-m-d
H:i:s" timestamp (for example "2025-11-15 20:22:06" or the correct actual test
run datetime) so downstream date parsing uses a proper format.

"php_version": "8.4.12",
"wordpress_version": "6.4.3",
"plugin_version": "2.4.7",
"dashboard_loading": {
"execution_time_ms": 2.52,
"memory_usage_mb": 0,
"peak_memory_mb": 2,
"database_queries": 0
},
"checkout_process": {
"execution_time_ms": 0.59,
"memory_usage_mb": 0,
"peak_memory_mb": 2,
"database_queries": 0
},
"site_creation_validation": {
"execution_time_ms": 0.58,
"memory_usage_mb": 0,
"peak_memory_mb": 2,
"database_queries": 0
},
"membership_operations": {
"execution_time_ms": 0.01,
"memory_usage_mb": 0,
"peak_memory_mb": 2,
"database_queries": 0
},
"api_endpoints": {
"execution_time_ms": 2.1,
"memory_usage_mb": 0,
"peak_memory_mb": 2,
"database_queries": 0
},
"database_queries": {
"execution_time_ms": 0.27,
"memory_usage_mb": 0,
"peak_memory_mb": 2,
"database_queries": 5
}
}Results saved to: /home/dave/multisite/ultimate-multisite/tests/simple-performance-results-2025-12.55-20-22-06.json
49 changes: 49 additions & 0 deletions current-test.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
Starting simplified performance benchmarks...
✓ Dashboard loading benchmark completed
✓ Checkout process benchmark completed
✓ Site creation validation benchmark completed
✓ Membership operations benchmark completed
✓ API endpoints benchmark completed
✓ Database queries benchmark completed
{
"test_run": "2025-11-15 20:22:14",
"php_version": "8.4.12",
"wordpress_version": "6.4.3",
"plugin_version": "2.4.7",
"dashboard_loading": {
"execution_time_ms": 1.09,
"memory_usage_mb": 0,
"peak_memory_mb": 2,
"database_queries": 0
},
"checkout_process": {
"execution_time_ms": 0.6,
"memory_usage_mb": 0,
"peak_memory_mb": 2,
"database_queries": 0
},
"site_creation_validation": {
"execution_time_ms": 0.59,
"memory_usage_mb": 0,
"peak_memory_mb": 2,
"database_queries": 0
},
"membership_operations": {
"execution_time_ms": 0.01,
"memory_usage_mb": 0,
"peak_memory_mb": 2,
"database_queries": 0
},
"api_endpoints": {
"execution_time_ms": 0.89,
"memory_usage_mb": 0,
"peak_memory_mb": 2,
"database_queries": 0
},
"database_queries": {
"execution_time_ms": 0.26,
"memory_usage_mb": 0,
"peak_memory_mb": 2,
"database_queries": 5
}
}Results saved to: /home/dave/multisite/ultimate-multisite/tests/simple-performance-results-2025-11-15-20-22-14.json
Comment on lines +1 to +49
Copy link
Contributor

@coderabbitai coderabbitai bot Nov 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Critical: Malformed JSON structure will break the comparator.

This file contains mixed console output and JSON, making it unparseable as JSON. Lines 1-7 contain status messages, and line 49 concatenates the JSON closing brace with a file path message. The Performance_Comparator::load_results() method (lines 31-44 in tests/performance-comparator.php) uses json_decode() which will fail on this format.

These files should contain only the pure JSON object (lines 8-48), without the console output prefix or the "Results saved to:" suffix.

Apply this approach to fix:

  1. Strip lines 1-7 (console output)
  2. Remove the "Results saved to:" text from line 49, keeping only the closing }

Or ensure the benchmark scripts write pure JSON to these files, separate from console output.

🧰 Tools
🪛 Biome (2.1.2)

[error] 1-1: String values must be double quoted.

(parse)

[error] 1-1: String values must be double quoted.

(parse)

[error] 1-1: String values must be double quoted.

(parse)

[error] 1-1: String values must be double quoted.

(parse)

[error] 1-1: unexpected character .

(parse)

[error] 1-1: unexpected character .

(parse)

[error] 1-1: unexpected character .

(parse)

[error] 1-2: unexpected character ✓

(parse)

[error] 2-2: String values must be double quoted.

(parse)

[error] 2-2: String values must be double quoted.

(parse)

[error] 2-2: String values must be double quoted.

(parse)

[error] 2-3: String values must be double quoted.

(parse)

[error] 3-3: unexpected character ✓

(parse)

[error] 3-3: String values must be double quoted.

(parse)

[error] 3-3: String values must be double quoted.

(parse)

[error] 3-3: String values must be double quoted.

(parse)

[error] 3-4: String values must be double quoted.

(parse)

[error] 4-4: unexpected character ✓

(parse)

[error] 4-4: String values must be double quoted.

(parse)

[error] 4-4: String values must be double quoted.

(parse)

[error] 4-4: String values must be double quoted.

(parse)

[error] 4-4: String values must be double quoted.

(parse)

[error] 4-5: String values must be double quoted.

(parse)

[error] 5-5: unexpected character ✓

(parse)

[error] 5-5: String values must be double quoted.

(parse)

[error] 5-5: String values must be double quoted.

(parse)

[error] 5-5: String values must be double quoted.

(parse)

[error] 5-6: String values must be double quoted.

(parse)

[error] 6-6: unexpected character ✓

(parse)

[error] 6-6: String values must be double quoted.

(parse)

[error] 6-6: String values must be double quoted.

(parse)

[error] 6-6: String values must be double quoted.

(parse)

[error] 6-7: String values must be double quoted.

(parse)

[error] 7-7: unexpected character ✓

(parse)

[error] 7-7: String values must be double quoted.

(parse)

[error] 7-7: String values must be double quoted.

(parse)

[error] 7-8: String values must be double quoted.

(parse)

[error] 8-9: String values must be double quoted.

(parse)

[error] 9-49: End of file expected

Use an array for a sequence of values: [1, 2]

(parse)

[error] 49-49: String values must be double quoted.

(parse)

[error] 49-49: String values must be double quoted.

(parse)

[error] 49-49: String values must be double quoted.

(parse)

[error] 49-49: End of file expected

Use an array for a sequence of values: [1, 2]

(parse)

[error] 49-49: unexpected character /

(parse)

[error] 49-49: unexpected character /

(parse)

[error] 49-49: unexpected character /

(parse)

[error] 49-49: unexpected character /

(parse)

[error] 49-49: Minus must be followed by a digit

(parse)

[error] 49-49: unexpected character /

(parse)

[error] 49-49: unexpected character /

(parse)

[error] 49-49: Minus must be followed by a digit

(parse)

[error] 49-49: Minus must be followed by a digit

(parse)

[error] 49-49: End of file expected

Use an array for a sequence of values: [1, 2]

(parse)

[error] 49-49: End of file expected

Use an array for a sequence of values: [1, 2]

(parse)

[error] 49-49: End of file expected

Use an array for a sequence of values: [1, 2]

(parse)

[error] 49-49: End of file expected

Use an array for a sequence of values: [1, 2]

(parse)

🤖 Prompt for AI Agents 
In current-test.json lines 1-49: the file mingles console output with JSON which
breaks json_decode; remove the leading status lines (lines 1-7) so the file
starts with the JSON object and strip the trailing "Results saved to: ..." text
on line 49 so the file ends exactly with the JSON closing brace; alternatively
modify the benchmark writer to emit only the pure JSON payload to the results
file and write console logs to stdout/stderr or a separate log file.

Loading
Loading