GitHub Downloads Prometheus Exporter

A lightweight, bash-based Prometheus exporter for GitHub repository release download statistics. This exporter uses only bash, curl, jq, and socat to provide comprehensive download metrics for monitoring with Prometheus and Grafana.

Features

• Pure Bash Implementation: No external dependencies except socat, curl, and jq
• Repository-Level Metrics: Exports GitHub download statistics including:
  • Total download counts per repository
  • Repository release and asset counts
  • GitHub API rate limit status
• Regex-Based Asset Filtering: Filter assets using regular expressions
• Multi-Repository Monitoring: Monitor multiple specified repositories simultaneously
• HTTP Server: Built-in HTTP server using socat for serving metrics
• Rate Limiting: Configurable delays between API calls to avoid rate limits
• Systemd Integration: Ready-to-use systemd service file
• GitHub Enterprise Support: Works with GitHub Enterprise instances

Quick Start

Prerequisites

• GitHub account or organization to monitor
• socat package installed
• curl and jq packages installed
• GitHub API token (optional but recommended for higher rate limits)
• Prometheus server for scraping metrics

Basic Installation

1. Clone the repository:

git clone https://github.com/itefixnet/prometheus-github-downloads-exporter.git
cd prometheus-github-downloads-exporter

2. Configure the exporter:

# Edit config.sh or set environment variables
export GITHUB_REPOS="account/repo1,account/repo2"  # Comma-separated list
export GITHUB_TOKEN="your-github-token"  # Optional but recommended

3. Test the exporter:

./github-downloads-exporter.sh test

4. Start the HTTP server:

./http-server.sh start

5. Access metrics at http://localhost:9168/metrics

System Installation

For production deployment, install as a system service:

# Create user and directories
sudo useradd -r -s /bin/false github-downloads-exporter
sudo mkdir -p /opt/github-downloads-exporter

# Copy files
sudo cp *.sh /opt/github-downloads-exporter/
sudo cp config.sh /opt/github-downloads-exporter/
sudo cp github-downloads-exporter.conf /opt/github-downloads-exporter/
sudo cp github-downloads-exporter.service /etc/systemd/system/

# Set permissions
sudo chown -R github-downloads-exporter:github-downloads-exporter /opt/github-downloads-exporter
sudo chmod +x /opt/github-downloads-exporter/*.sh

# Enable and start service
sudo systemctl daemon-reload
sudo systemctl enable github-downloads-exporter
sudo systemctl start github-downloads-exporter

Configuration

Environment Variables

The exporter can be configured using environment variables or configuration files:

Variable	Default	Description
GITHUB_TOKEN	(none)	GitHub API token (strongly recommended)
GITHUB_REPOS	(required)	Comma-separated list of repositories (format: account/repo)
GITHUB_API_URL	https://api.github.com	GitHub API URL (for Enterprise)
RELEASE_REGEX	.*	Regex to filter releases
ASSET_REGEX	.*	Regex to filter assets
LISTEN_PORT	9168	HTTP server port
LISTEN_ADDRESS	0.0.0.0	HTTP server bind address
METRICS_PREFIX	github_downloads	Prometheus metrics prefix
RATE_LIMIT_DELAY	0.2	Delay between API calls in seconds
MAX_PROCESSING_TIME	60	Maximum processing time in seconds

Configuration Files

1. config.sh: Shell configuration file (sourced by scripts)
2. github-downloads-exporter.conf: Systemd environment file

GitHub Token Setup

While not required, using a GitHub token is strongly recommended to increase API rate limits:

1. Go to GitHub Settings → Developer settings → Personal access tokens
2. Generate a new token with public_repo scope (or repo for private repositories)
3. Set the token: export GITHUB_TOKEN="your-token-here"

Metrics

The exporter provides the following Prometheus metrics:

Repository-Level Metrics

• github_downloads_repository_downloads_total{repository="account/repo"} - Total downloads per repository (counter)
• github_downloads_repository_releases_count{repository="account/repo"} - Number of releases per repository (gauge)
• github_downloads_repository_assets_count{repository="account/repo"} - Number of assets per repository (gauge)

Global Metrics

• github_downloads_total_downloads - Total downloads across all monitored repositories (counter)
• github_downloads_total_releases - Total number of releases across all repositories (gauge)
• github_downloads_total_assets - Total number of assets across all repositories (gauge)

API Rate Limit Metrics

• github_downloads_api_rate_limit - GitHub API rate limit (gauge)
• github_downloads_api_rate_remaining - Remaining API calls (gauge)
• github_downloads_api_rate_reset_timestamp - Rate limit reset timestamp (gauge)

Exporter Metrics

• github_downloads_scrape_duration_seconds - Time spent scraping GitHub API (gauge)
• github_downloads_scrape_timestamp - Timestamp of last successful scrape (gauge)

Usage Examples

Monitor Specific Repository

export GITHUB_REPOS="prometheus/prometheus"
./github-downloads-exporter.sh collect

Monitor Multiple Repositories

export GITHUB_REPOS="prometheus/prometheus,grafana/grafana,kubernetes/kubernetes"
./github-downloads-exporter.sh collect

Filter with Regex Patterns

# Monitor specific repositories with release filtering
export GITHUB_REPOS="prometheus/prometheus,grafana/grafana"

# Monitor only stable releases (no pre-releases)
export RELEASE_REGEX="^v?[0-9]+\.[0-9]+\.[0-9]+$"

# Monitor only binary assets
export ASSET_REGEX="\.(tar\.gz|zip|deb|rpm|exe)$"

./github-downloads-exporter.sh collect

Manual Testing

# Test connection
./github-downloads-exporter.sh test

# Collect metrics once
./github-downloads-exporter.sh collect

# Start HTTP server manually
./http-server.sh start

# Test HTTP endpoints
curl http://localhost:9168/metrics
curl http://localhost:9168/health
curl http://localhost:9168/

Prometheus Configuration

Add to your prometheus.yml:

scrape_configs:
  - job_name: 'github-downloads-exporter'
    static_configs:
      - targets: ['localhost:9168']
    scrape_interval: 5m  # GitHub data doesn't change frequently
    scrape_timeout: 60s   # API calls can take time
    metrics_path: /metrics
    
  # Multiple instances
  - job_name: 'github-downloads'
    static_configs:
      - targets: ['server1:9168', 'server2:9168']
        labels:
          environment: 'production'
    scrape_interval: 5m

Grafana Dashboard

The repository includes a Grafana dashboard (grafana-dashboard.json) with the following features:

Core Visualizations:

• Total Downloads: Overall download trends across all repositories
• Top Repositories: Bar chart showing most popular repositories by downloads
• Repository Statistics: Complete overview table with download counts, releases, and assets

Monitoring & Health:

• GitHub API Rate Limits: Current usage and remaining quota
• Scrape Performance: Exporter performance metrics (duration and timestamp)
• Download Trends: Time-series graphs showing download growth

Interactive Features:

• Instance Filter: Monitor multiple exporter instances
• Repository Filter: Focus on specific repositories
• Time Range Selection: Analyze trends over different periods

Installation:

1. Import grafana-dashboard.json into your Grafana instance
2. Configure the Prometheus datasource
3. Use the template variables to filter data as needed

Troubleshooting

Common Issues

1. Rate Limit Exceeded:

   • Add a GitHub token: export GITHUB_TOKEN="your-token"
   • Increase RATE_LIMIT_DELAY or CACHE_TTL
   • Monitor rate limit metrics

2. API Connection Failed:

   • Check internet connectivity
   • Verify GitHub account exists
   • Test manually: curl https://api.github.com/users/ACCOUNT

3. No Data Returned:

   • Verify repositories exist and have releases
   • Check that GITHUB_REPOS is correctly formatted (account/repo)
   • Check regex patterns aren't too restrictive
   • Ensure repositories are public (or token has access to private repos)

4. Missing Dependencies:

   # Install on Ubuntu/Debian
   sudo apt-get install socat curl jq
   
   # Install on CentOS/RHEL
   sudo yum install socat curl jq

Logging

• Service logs: journalctl -u github-downloads-exporter -f
• Manual logs: Scripts output to stderr
• Enable debug: LOG_LEVEL=debug

Performance Tuning

For high-frequency monitoring:

• Adjust RATE_LIMIT_DELAY based on your rate limits (default: 0.2 seconds)
• Set MAX_PROCESSING_TIME appropriately for your number of repositories
• Monitor fewer repositories per instance if experiencing timeouts
• Consider running multiple exporter instances for different repository sets
• Use longer Prometheus scrape intervals (5-15 minutes) since GitHub data doesn't change frequently

Development

Testing

# Run basic tests
./github-downloads-exporter.sh test
./http-server.sh test

# Test with different configurations
GITHUB_ACCOUNT=prometheus ./github-downloads-exporter.sh test

Contributing

1. Fork the repository
2. Create a feature branch
3. Test thoroughly with different GitHub accounts/repositories
4. Submit a pull request

License

This project is licensed under the BSD 2-Clause License - see the LICENSE file for details.

Support

• GitHub Issues: https://github.com/itefixnet/prometheus-github-downloads-exporter/issues
• Documentation: This README and inline script comments

Use Cases

This exporter is particularly useful for:

• Open Source Projects: Track adoption through download metrics
• Release Management: Monitor which versions are most popular
• Asset Performance: Understand which package formats are preferred
• Growth Tracking: Measure project growth over time
• Release Strategy: Optimize release cadence based on download patterns

Alternatives

For more advanced features or different languages, consider:

• GitHub API directly with custom scripts
• Grafana GitHub datasource plugin
• Custom Prometheus collectors in Go/Python