Register webhook and script in create PR command by richardhjtan · Pull Request #3998 · cardstack/boxel

richardhjtan · 2026-02-14T04:41:50Z

GitHub Webhook Setup for Local Testing

Quick guide for testing webhooks locally with ngrok.

Quick Start

1. Start ngrok

ngrok http 4201

Copy the URL from output: https://abc123.ngrok.io

⚠️ Free ngrok URL changes each restart - you'll need to update GitHub webhook URL

2. Register webhook

cd packages/matrix
PUBLIC_URL=https://abc123.ngrok.io \
MATRIX_USERNAME=@admin:localhost \
MATRIX_PASSWORD=password \
pnpm exec ts-node scripts/register-github-webhook.ts

Copy the webhook URL and signing secret from output.

3. Add to GitHub

Go to your repo → Settings → Webhooks → Add webhook
URL: (paste from script output)
Secret: (paste from script output)
Content type: application/json
Events: Select "Pull requests" and "Pull request reviews"
Click "Add webhook"

4. Test

Create a PR or use GitHub's webhook test feature to verify it works.

When ngrok Restarts (URL Changed)

# 1. Get new ngrok URL
# 2. Re-run script from packages/matrix
cd packages/matrix
PUBLIC_URL=https://NEW_URL.ngrok.io pnpm exec ts-node scripts/register-github-webhook.ts

# 3. Update GitHub webhook:
#    Settings → Webhooks → Edit → Change URL → Update
#    (Keep the same secret - script reuses it)

Troubleshooting

Webhooks not working after ngrok restart?

→ ngrok URL changed. Re-run script and update URL in GitHub.

Getting 401 errors?

→ Verify signing secret in GitHub matches the script output.

Webhooks not received?

# Check ngrok is running
curl http://localhost:4040/api/tunnels

# Check realm server is running
curl http://localhost:4201/_health

Monitor webhook deliveries

ngrok dashboard: http://localhost:4040
GitHub: Repo → Settings → Webhooks → Recent Deliveries

For more details, see GitHub Webhooks Documentation.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e1b34cd1f4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-02-14T04:46:49Z

packages/host/app/commands/create-listing-pr.ts

+      log.debug('Webhook registered for PR:', {
+        webhookUrl: `${this.realmServer.url.href}_webhooks/${webhookData.path}`,
+        signingSecret: webhookData.signingSecret,
+        prNumber: prResult.prNumber,


Stop logging webhook signing secrets

The debug log currently includes signingSecret, which is the credential used to authenticate webhook payloads. In any environment where debug logging is enabled (staging, incident debugging, or local logs shipped to a shared sink), this leaks enough information to forge valid X-Hub-Signature-256 headers and spoof GitHub webhook events for the registered path.

Useful? React with 👍 / 👎.

github-actions · 2026-02-14T04:49:38Z

Preview deployments

github-actions · 2026-02-14T05:01:43Z

Host Test Results

1 files ± 0 1 suites ±0 1h 33m 3s ⏱️ - 11m 10s
1 633 tests - 223 1 619 ✅ - 223 14 💤 ±0 0 ❌ ±0
1 647 runs - 224 1 633 ✅ - 224 14 💤 ±0 0 ❌ ±0

Results for commit feef6f6. ± Comparison against base commit 82e0e25.

This pull request removes 223 tests.

Chrome ‑ Acceptance | code-submode | field playground > multiple realms: can autogenerate new Spec and field instance (no preexisting Spec)
Chrome ‑ Acceptance | code-submode | field playground > multiple realms: can autogenerate new field instance when spec exists but has no examples
Chrome ‑ Acceptance | code-submode | field playground > multiple realms: can create new field instance (has preexisting Spec)
Chrome ‑ Acceptance | code-submode | field playground > single realm: can autogenerate new Spec and field instance (no preexisting Spec)
Chrome ‑ Acceptance | code-submode | field playground > single realm: can autogenerate new field instance when spec exists but has no examples
Chrome ‑ Acceptance | code-submode | field playground > single realm: can create new field instance (has preexisting Spec)
Chrome ‑ Acceptance | code-submode | field playground > single realm: can not preview non-exports or primitives
Chrome ‑ Acceptance | code-submode | field playground > single realm: can populate instance chooser dropdown options with containedExamples from Spec
Chrome ‑ Acceptance | code-submode | field playground > single realm: can preview compound field instance
Chrome ‑ Acceptance | code-submode | field playground > single realm: can request AI assistant to bulk generate samples
…

♻️ This comment has been updated with latest results.

Copilot

Pull request overview

This pull request adds webhook registration functionality to enable GitHub webhook notifications for pull requests. The implementation includes infrastructure for creating and managing incoming webhooks, filtering webhook commands by event type and PR number, and a script for easy webhook setup during local development with ngrok.

Changes:

Added webhook registration capability to the create-listing-pr command to automatically set up GitHub webhooks for PR status updates
Implemented command filtering logic in the webhook receiver to filter by event type and PR number
Created a standalone script for manual webhook registration with idempotent behavior

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
packages/runtime-common/pr-manifest.ts	Adds webhook metadata (id, path, signingSecret) to PrManifest interface
packages/realm-server/handlers/handle-webhook-receiver.ts	Implements webhook command lookup and filtering by event type and PR number
packages/realm-server/tests/server-endpoints/webhook-receiver-test.ts	Adds comprehensive tests for command execution and filtering scenarios
packages/matrix/scripts/register-github-webhook.ts	New script for registering GitHub webhooks with idempotent behavior
packages/host/app/services/realm-server.ts	Adds service methods for creating webhooks and webhook commands
packages/host/app/commands/create-listing-pr.ts	Integrates webhook registration into PR creation flow

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-16T15:01:40Z

packages/host/app/commands/create-listing-pr.ts

+  private async registerPRWebhook(
+    prNumber: number,
+  ): Promise<{ id: string; path: string; signingSecret: string } | null> {
+    try {
+      const webhook = await this.realmServer.createIncomingWebhook({
+        verificationType: 'HMAC_SHA256_HEADER',
+        verificationConfig: {
+          header: 'x-hub-signature-256',
+          encoding: 'hex',
+        },
+      });
+
+      log.debug('Created incoming webhook:', {
+        id: webhook.id,
+        path: webhook.webhookPath,
+      });
+
+      await this.realmServer.createWebhookCommand({
+        incomingWebhookId: webhook.id,
+        command: `${this.realmServer.url.href}catalog-realm/commands/process-github-webhook`,
+        filter: {
+          eventType: 'pull_request',
+          prNumber: prNumber,
+        },
+      });
+
+      log.debug('Registered webhook command for PR', { prNumber });
+
+      return {
+        id: webhook.id,
+        path: webhook.webhookPath,
+        signingSecret: webhook.signingSecret,
+      };
+    } catch (error: any) {
+      log.error('Failed to register PR webhook:', error);
+      // Don't fail the entire PR creation if webhook registration fails
+      // Just log and continue
+      return null;
+    }
+  }


The registerPRWebhook method always creates a new webhook and webhook command without checking if one already exists for this PR. This could lead to duplicate webhooks being created if the PR creation is retried or if the command is called multiple times. Consider implementing idempotent webhook registration similar to the approach in register-github-webhook.ts (see ensureIncomingWebhook and ensureWebhookCommand functions) to prevent duplicate webhooks.

Copilot · 2026-02-16T15:01:41Z

packages/runtime-common/pr-manifest.ts

+  webhook?: {
+    id: string;              // UUID from incoming_webhooks table
+    path: string;            // webhook_path (e.g., "whk_abc123...")
+    signingSecret: string;   // For GitHub webhook configuration


The signing secret is sensitive information used to verify GitHub webhook signatures. Storing it in the PrManifest may pose a security risk if this manifest is persisted or exposed through any API endpoints. Consider whether the signing secret needs to be stored in the manifest at all - it may be sufficient to only store the webhook ID and path, and retrieve the signing secret from the database when needed for configuration purposes. If the manifest is only used transiently and never persisted or exposed, this is acceptable, but the comment "Store minimal manifest in room state" suggests it may be stored.

Suggested change

signingSecret: string; // For GitHub webhook configuration

signingSecretId: string; // Reference/ID for secret used to verify GitHub webhooks (not the raw secret)

Copilot · 2026-02-16T15:01:41Z

packages/matrix/scripts/register-github-webhook.ts

+const webhookConfig = {
+  verificationType: 'HMAC_SHA256_HEADER' as const,
+  verificationConfig: {
+    header: 'x-hub-signature-256',


The header name should be 'X-Hub-Signature-256' with capital letters instead of 'x-hub-signature-256'. While HTTP headers are case-insensitive, the verification logic in handle-webhook-receiver.ts normalizes headers to lowercase before comparison (line 178), but the GitHub API documentation and all existing tests in the codebase consistently use 'X-Hub-Signature-256' with capital letters. This inconsistency could cause confusion and should be corrected for consistency with the rest of the codebase.

Suggested change

header: 'x-hub-signature-256',

header: 'X-Hub-Signature-256',

Copilot · 2026-02-16T15:01:41Z

packages/host/app/commands/create-listing-pr.ts

+      const webhook = await this.realmServer.createIncomingWebhook({
+        verificationType: 'HMAC_SHA256_HEADER',
+        verificationConfig: {
+          header: 'x-hub-signature-256',


The header name should be 'X-Hub-Signature-256' with capital letters instead of 'x-hub-signature-256'. While HTTP headers are case-insensitive, the GitHub API documentation and all existing tests in the codebase consistently use 'X-Hub-Signature-256' with capital letters. This inconsistency could cause confusion and should be corrected for consistency with the rest of the codebase.

Suggested change

header: 'x-hub-signature-256',

header: 'X-Hub-Signature-256',

richardhjtan requested review from a team and tintinthong February 14, 2026 04:42

chatgpt-codex-connector bot reviewed Feb 14, 2026

View reviewed changes

habdelra requested a review from Copilot February 16, 2026 14:56

Copilot started reviewing on behalf of habdelra February 16, 2026 14:57 View session

Copilot AI reviewed Feb 16, 2026

View reviewed changes

Register webhook and script in create PR command

feef6f6

richardhjtan force-pushed the CS-10190-register-webhook-in-create-pr-command branch from e1b34cd to feef6f6 Compare February 20, 2026 03:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

Register webhook and script in create PR command#3998

Register webhook and script in create PR command#3998
richardhjtan wants to merge 1 commit intomainfrom
CS-10190-register-webhook-in-create-pr-command

richardhjtan commented Feb 14, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Uh oh!

chatgpt-codex-connector bot Feb 14, 2026

Uh oh!

github-actions bot commented Feb 14, 2026

Uh oh!

github-actions bot commented Feb 14, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Feb 16, 2026

Uh oh!

Copilot AI Feb 16, 2026

Uh oh!

Copilot AI Feb 16, 2026

Uh oh!

Copilot AI Feb 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

	signingSecret: string; // For GitHub webhook configuration
	signingSecretId: string; // Reference/ID for secret used to verify GitHub webhooks (not the raw secret)

	header: 'x-hub-signature-256',
	header: 'X-Hub-Signature-256',

Comments

Conversation

richardhjtan commented Feb 14, 2026

GitHub Webhook Setup for Local Testing

Quick Start

1. Start ngrok

2. Register webhook

3. Add to GitHub

4. Test

When ngrok Restarts (URL Changed)

Troubleshooting

Webhooks not working after ngrok restart?

Getting 401 errors?

Webhooks not received?

Monitor webhook deliveries

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector bot Feb 14, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 14, 2026

Preview deployments

Uh oh!

github-actions bot commented Feb 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Host Test Results

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 16, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

github-actions bot commented Feb 14, 2026 •

edited

Loading