Add GitHub Pages deployment with automated notebook execution

[Copilot]

Automates deployment of course materials to GitHub Pages, executing Jupyter notebooks and converting all content to HTML while preserving the README-centric navigation structure.

Implementation

GitHub Actions Workflow (.github/workflows/deploy-pages.yml)

• Triggers on master branch pushes
• Python 3.11 with dependencies: jupyter, nbconvert, numpy, pandas, matplotlib, seaborn, scikit-learn, scipy, markdown
• Deploys to Pages using actions/deploy-pages@v4

Build Script (scripts/build_site.py)

• Converts README.md → index.html, all .md → .html, all .ipynb → .html with execution
• Rewrites internal links: .md → .html, .ipynb → .html (preserves anchors, query params, external links)
• Copies static assets maintaining directory structure to _site/
• Fallback conversion without execution when notebooks fail
• GitHub-like responsive CSS styling

Key Technical Decisions

Link Rewriting Pattern

# Handles anchors and query parameters correctly
r'href="(?!http://|https://|#)([^"#?]*?)\.md([#?][^"]*)?(")'
r'href="\1.html\2\3'

Error Handling

• 600s timeout per notebook with 60s subprocess buffer
• Continues build on notebook execution failures
• Fallback to static conversion preserves repository content even with missing dependencies

Path Matching

# Checks path parts to avoid false positives (e.g., "git_tutorial.md")
path_parts = path.parts
for pattern in ignore_patterns:
    if pattern in path_parts:
        return True

Result

After merge, enable GitHub Pages in Settings → Pages → Source: GitHub Actions. Site available at https://avidaldo.github.io/ia25/.

Screenshots

Landing Page

Notebook Page

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• api.openml.org
  • Triggering command: /usr/bin/python /usr/bin/python -m ipykernel_launcher -f /tmp/tmp14apqpnt.json --HistoryManager.hist_file=:memory: (dns block)
  • Triggering command: /usr/bin/python /usr/bin/python -m ipykernel_launcher -f /tmp/tmpphay0tw1.json --HistoryManager.hist_file=:memory: (dns block)
  • Triggering command: /usr/bin/python /usr/bin/python -m ipykernel_launcher -f /tmp/tmp24ql593h.json --HistoryManager.hist_file=:memory: (dns block)
• ndownloader.figshare.com
  • Triggering command: /usr/bin/python /usr/bin/python -m ipykernel_launcher -f /tmp/tmppa5evzvc.json --HistoryManager.hist_file=:memory: (dns block)
  • Triggering command: /usr/bin/python /usr/bin/python -m ipykernel_launcher -f /tmp/tmpb2msq422.json --HistoryManager.hist_file=:memory: (dns block)
  • Triggering command: /usr/bin/python /usr/bin/python -m ipykernel_launcher -f /tmp/tmpj5na5nfy.json --HistoryManager.hist_file=:memory: origin uname (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Original prompt

Add GitHub Pages Deployment for Course Materials

Objective

Create a GitHub Actions workflow that automatically deploys the course materials to GitHub Pages, maintaining the current README-centric navigation structure while executing and rendering Jupyter notebooks with outputs.

Requirements

1. Create GitHub Actions Workflow (.github/workflows/deploy-pages.yml)

The workflow should:

• Trigger on pushes to master branch (note: this repo uses master, not main)
• Set up Python 3.11
• Install necessary dependencies for executing notebooks:
  • jupyter
  • nbconvert
  • numpy
  • pandas
  • matplotlib
  • seaborn
  • scikit-learn
  • scipy
• Convert all content while preserving directory structure:
  • Convert README.md → index.html (as the landing page)
  • Convert all .md files to .html files in their respective directories
  • Execute and convert all .ipynb files to .html files with outputs
  • Copy all static assets (images, PDFs, etc.)
• Deploy to GitHub Pages using the actions/deploy-pages@v4 action

2. Create Build Script (scripts/build_site.py)

Create a Python script that:

• Finds all markdown files and converts them to HTML using a simple converter or nbconvert
• Finds all Jupyter notebooks and executes them with jupyter nbconvert --execute --to html
• Updates links in the generated HTML files:
  • .md → .html
  • .ipynb → .html
  • Preserves external links (starting with http:// or https://)
• Copies the entire directory structure to _site/ output directory
• Applies consistent styling that resembles GitHub's rendering (simple, clean, readable)
• Handles relative links correctly so navigation works as expected

3. Styling

Create a simple CSS template that:

• Uses GitHub-like styling (clean, readable, familiar to students)
• Has responsive design for mobile viewing
• Preserves code block styling
• Makes notebook outputs clear and readable
• Includes proper spacing and typography

4. Error Handling

The workflow should:

• Continue on error (some notebooks might fail to execute)
• Log which notebooks failed but still deploy the rest
• Use a reasonable timeout for notebook execution (e.g., 600 seconds per notebook)

Technical Specifications

Workflow file: .github/workflows/deploy-pages.yml
Build script: scripts/build_site.py
Output directory: _site/ (will be ignored in git)

Required permissions in workflow:

permissions:
  contents: read
  pages: write
  id-token: write

Key steps:

1. Checkout repository
2. Setup Python
3. Install dependencies
4. Run build script
5. Upload Pages artifact
6. Deploy to Pages

Expected Result

After merging and enabling GitHub Pages (Settings → Pages → Source: GitHub Actions):

• Students can visit https://avidaldo.github.io/ia25/
• The landing page shows the README content with working links
• All markdown files are rendered as HTML
• All notebooks are executed and show outputs
• Directory structure is preserved
• Navigation works exactly like browsing the repo, but with rendered outputs

Notes

• Keep the solution simple and maintainable
• Don't require manual steps for each new file added
• The README is the main navigation - preserve its role
• Some notebooks might take time to execute - use appropriate timeouts
• This repo uses master as the default branch, not main

This pull request was created from Copilot chat.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.