Skip to content

chore(ci): integration test schedule #199

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
hntrl wants to merge 1 commit into
base: main
Choose a base branch
from
+70 −32
Open

chore(ci): integration test schedule #199

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 0 additions & 32 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -134,35 +134,3 @@ jobs:
run: tar -xzf build-artifacts.tar.gz
- name: Run Unit Tests
run: pnpm test

test-int:
name: Integration Tests
needs: [build]
if: github.event_name != 'pull_request' || (github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]')
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
- name: Install pnpm
uses: pnpm/action-setup@41ff72655975bd51cab0327fa583b6e92b6d3061 # v4.2.0
- name: Use Node.js 24.x
uses: actions/setup-node@395ad3262231945c25e8478fd5baf05154b1d79f # v6.1.0
with:
node-version-file: .nvmrc
cache: "pnpm"
- name: Install dependencies
run: pnpm install --frozen-lockfile
- name: Download build artifacts
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v4
with:
name: build-artifacts
- name: Extract build artifacts
run: tar -xzf build-artifacts.tar.gz
- name: Run Integration Tests
run: pnpm test:int
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
DENO_DEPLOY_TOKEN: ${{ secrets.DENO_DEPLOY_TOKEN }}
DAYTONA_API_KEY: ${{ secrets.DAYTONA_API_KEY }}
MODAL_TOKEN_ID: ${{ secrets.MODAL_TOKEN_ID }}
MODAL_TOKEN_SECRET: ${{ secrets.MODAL_TOKEN_SECRET }}
70 changes: 70 additions & 0 deletions .github/workflows/integration-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# Run integration tests weekly and on-demand

name: Integration Tests

permissions:
contents: read
Copy link

@everywheredennise-oss everywheredennise-oss Feb 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Overly-permissive workflow permission: actions: write

The workflow requests actions: write at the workflow-level permissions. actions: write grants the workflow the ability to create or modify workflow runs and is broader than typically required for a scheduled integration job. Scheduled workflows that run with repository secrets should use least privilege to reduce risk.

🔧 Suggested fix 
# Reduce permissions to the minimum required. If you only need to read contents and run actions, prefer read-only.
permissions:
  contents: read
  actions: read
# Or remove the `actions` key entirely if not required by any step.

If a specific step requires higher privileges, escalate permissions only for that job and document why.

</details>

<details>
<summary>🤖 Prompt for AI Agents</summary>

In `.github/workflows/integration-tests.yml` at line ~6, change workflow-level permissions to least-privilege. If any steps truly require `actions: write`, add a comment explaining why and scope that permission to the smallest job possible.
</details>

actions: write

on:
schedule:
# Run every Monday at 14:00 UTC
Copy link

@everywheredennise-oss everywheredennise-oss Feb 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Major

Potential mismatch between pinned action SHA and input support (setup-node)

The workflow pins actions/setup-node to a specific commit SHA but uses the node-version-file input. Not all historical commit SHAs accept this input; if the pinned commit doesn't support it, the step may ignore the input or fail unexpectedly.

🔧 Suggested fix

Prefer using a released tag that supports the input, e.g. actions/setup-node@v4, or explicitly specify node-version:

- uses: actions/setup-node@v4
  with:
    node-version-file: .nvmrc
    cache: pnpm
# or
- uses: actions/setup-node@v4
  with:
    node-version: '24'
    cache: pnpm

Pinning to a stable release tag reduces the risk of using a commit SHA that lacks expected features.

🤖 Prompt for AI Agents

In .github/workflows/integration-tests.yml at line ~11, replace the commit SHA pin for actions/setup-node with a maintained release tag (e.g., v4) or change the input to node-version if you prefer explicit version pinning.

- cron: "0 14 * * 1"
workflow_dispatch: # Allows triggering the workflow manually in GitHub UI

jobs:
build:
name: Build
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
- name: Install pnpm
uses: pnpm/action-setup@41ff72655975bd51cab0327fa583b6e92b6d3061 # v4.2.0
Copy link

@everywheredennise-oss everywheredennise-oss Feb 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Robustness: archive creation suppresses errors and may produce an empty/corrupt artifact

The tar invocation suppresses stderr and falls back, hiding errors. If none of the referenced paths exist, or tar fails, the command will either produce an empty archive or the fallback may also fail — but errors are hidden. This can cause actions/upload-artifact to upload a missing or invalid artifact.

🔧 Suggested fix 
- name: Create build archive
  run: |
    set -euo pipefail
    FILES=()
    for p in libs/*/dist libs/*/.turbo .turbo; do
      if [ -e "$p" ]; then
        FILES+=("$p")
      fi
    done
    if [ ${#FILES[@]} -eq 0 ]; then
      echo "No build artifacts found" >&2
      exit 1
    fi
    tar -czf build-artifacts.tar.gz "${FILES[@]}"

This avoids hiding errors and ensures a valid artifact is produced. Also remove redundant paths (e.g., .turbo listed twice).

🤖 Prompt for AI Agents

In .github/workflows/integration-tests.yml at line ~22 replace the single-line tar command with a robust multi-line script that checks for artifact presence and exits non-zero if none are found.

Copy link

@everywheredennise-oss everywheredennise-oss Feb 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Silent / suppressed archive creation failures (tar)

The tar command in the workflow suppresses stderr and provides a fallback, which can hide failures and result in empty or corrupted artifacts being uploaded. This is a critical reliability issue for the integration workflow — if artifacts are missing or invalid, downstream test steps will fail unpredictably or run against stale data.

🔧 Suggested fix

Replace the single-line tar invocation with a robust script that: (1) uses strict shell flags, (2) checks which paths exist before archiving, and (3) exits non-zero with a clear error if nothing to archive. Example:

- name: Create build archive
  run: |
    set -euo pipefail
    trap 'echo "ERROR: archive creation failed at line $LINENO" >&2' ERR
    FILES=()
    for p in libs/*/dist libs/*/.turbo .turbo; do
      [ -e "$p" ] && FILES+=("$p")
    done
    if [ ${#FILES[@]} -eq 0 ]; then
      echo "No build artifacts found" >&2
      exit 1
    fi
    tar -czf build-artifacts.tar.gz "${FILES[@]}"

This ensures failures are visible and the artifact is valid before upload.

🤖 Prompt for AI Agents

In .github/workflows/integration-tests.yml at line ~22, replace the existing tar command with the robust multi-line script above and remove any stderr suppressions (e.g., 2>/dev/null or || fallbacks).

- name: Use Node.js
uses: actions/setup-node@395ad3262231945c25e8478fd5baf05154b1d79f # v6.1.0
with:
node-version-file: .nvmrc
cache: "pnpm"
- name: Install dependencies
run: pnpm install --frozen-lockfile
- name: Build
run: pnpm build
- name: Create build archive
run: tar -czf build-artifacts.tar.gz libs/*/dist libs/*/.turbo .turbo 2>/dev/null || tar -czf build-artifacts.tar.gz libs/*/dist
- name: Upload build artifacts
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v4
with:
name: build-artifacts
path: build-artifacts.tar.gz
retention-days: 1

test-int:
name: Integration Tests
needs: [build]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
- name: Install pnpm
uses: pnpm/action-setup@41ff72655975bd51cab0327fa583b6e92b6d3061 # v4.2.0
- name: Use Node.js 24.x
uses: actions/setup-node@395ad3262231945c25e8478fd5baf05154b1d79f # v6.1.0
with:
node-version-file: .nvmrc
cache: "pnpm"
- name: Install dependencies
run: pnpm install --frozen-lockfile
- name: Download build artifacts
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v4
with:
name: build-artifacts
- name: Extract build artifacts
run: tar -xzf build-artifacts.tar.gz
- name: Run Integration Tests
run: pnpm test:int
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
DENO_DEPLOY_TOKEN: ${{ secrets.DENO_DEPLOY_TOKEN }}
DAYTONA_API_KEY: ${{ secrets.DAYTONA_API_KEY }}
MODAL_TOKEN_ID: ${{ secrets.MODAL_TOKEN_ID }}
MODAL_TOKEN_SECRET: ${{ secrets.MODAL_TOKEN_SECRET }}
Loading