🗺️ CadenceMaps

Open, Reproducible OpenStreetMap Data Pipeline
Transforming OSM data into analysis-ready GeoParquet format

🚀 Quick Start

Access ready-to-use OSM data with a simple duckdb query:

INSTALL httpfs;
LOAD httpfs;
INSTALL spatial;
LOAD spatial;
SET s3_endpoint='api.cadencemaps.infs.ch';
SET s3_url_style='path';


SELECT names.primary
FROM read_parquet(
  's3://cadencemaps/release/2025-05-13/theme=places/type=place/country=CH/*',
  filename=true,
  hive_partitioning=1
)
WHERE categories.primary = 'restaurant';

Scan to access the CadenceMaps website

📖 About

A modular pipeline for converting OpenStreetMap data into cloud-native GeoParquet files using DuckDB, PostGIS, and Docker. Built as part of a thesis project, it addresses the challenge of making OSM's rich dataset (5+ million POIs for the D-A-CH-LI) accessible for modern cloud-native analytics.

🎯 Key Features

• 🔄 Automated Pipeline: Weekly or Daily updates possible using CI/CD DataOps principles
• 📊 Analysis-Ready: Direct SQL querying without server infrastructure
• 🌍 Standardized Schema: Overture Maps-compatible format for interoperability
• ⚡ High Performance: Optimized spatial partitioning
• 🔍 Quality Assured: Automated validation and vandalism detection
• 🏗️ Scalable Architecture: Country-scale processing in <24 hours

🏛️ Architecture

Data Processing Pipeline

CadenceMaps employs a fully automated CI/CD pipeline that orchestrates the conversion of OpenStreetMap data into analysis-ready GeoParquet files. The pipeline is triggered weekly and processes the entire D-A-CH-LI region in parallel across multiple countries.

Pipeline Stages

The GitLab CI/CD pipeline consists of seven distinct stages that ensure reliable, reproducible data processing:

1. 🔧 Publish Stage: Builds and publishes Docker images for all services (converter, importer, database, validator, web, reverse-proxy, cert-extractor) to the GitLab container registry.

2. ⚙️ Setup Stage: Initializes the production environment by deploying core infrastructure services including MinIO S3 storage, PostgreSQL database, reverse proxy, and the web interface.

3. 🚀 Deploy Stage: Executes parallel data processing for each country in the D-A-CH-LI region:

   • Downloads latest OSM PBF files from Geofabrik (with fallback to OSM France)
   • Imports data into PostgreSQL using osm2pgsql with custom Lua transformations
   • Converts PostgreSQL data to GeoParquet format using DuckDB
   • Uploads results to staging area in MinIO

4. 🧹 Cleanup Stage: Performs system maintenance by removing unused Docker images and volumes to free up server resources.

5. ✅ Validate Stage: Runs vandalism detection algorithms.

6. 👍 Approve Stage: Manual approval gate where validated data is promoted from staging to production release bucket, making it publicly accessible.

7. 🛑 Stop Proxy Stage: Optional manual stage to stop the reverse proxy for maintenance purposes.

Detailed pipeline Execution Flow

Automation Features

• ⏰ Scheduled Execution: Pipeline runs weekly or daily automatically, ensuring fresh data availability
• 🔄 Fault Tolerance: Automatic fallback to alternative OSM data sources (OSM France) if Geofabrik fails
• 📊 Parallel Processing: All four countries (DE, AT, CH, LI) process simultaneously for optimal performance
• 💾 Resource Management: Aggressive cleanup of Docker images and volumes to maintain server health
• 📈 Monitoring: Artifact collection of validation reports
• 🚦 Quality Gates: Manual approval step prevents bad data from reaching production

The entire pipeline typically completes in under 24 hours for the full D-A-CH-LI region, processing 5+ million points of interest.

Technology Stack

Component	Technology	Purpose
Data Ingestion	osm2pgsql	OSM to PostgreSQL conversion
Spatial Processing	PostgreSQL/PostGIS	Geographic data preprocessing
Format Conversion	Python + DuckDB	Tabular data transformation
Storage Format	GeoParquet	Cloud-optimized geospatial data
Hosting	MinIO (S3-compatible)	Scalable object storage
Orchestration	GitLab CI/CD + Docker	Automated pipeline execution

📁 Repository Structure

cadencemaps/
├── 🐳 .docker/            # Docker container configurations for production
│   ├── cert-extractor/    # SSL certificate extraction service for minio
│   ├── converter/         # Converter service Docker setup
│   ├── importer/          # Importer service Docker setup
│   ├── reverse-proxy/     # Caddy reverse proxy configuration
│   ├── spatial-database/  # PostGIS database Docker setup
│   ├── validator/         # Validator service Docker setup
│   ├── web/               # Web service Docker setup
│   └── docker-compose.yml # Production Docker compose configuration
├── 🔄 converter/          # Data conversion logic (PostgreSQL to GeoParquet)
├── 📥 importer/           # osm2pgsql data import tools and mapping scripts
│   └── mappers/           # Lua scripts for data transformation
│       ├── divisions/     # Administrative boundary mapping
│       ├── places/        # Points of interest mapping
│       └── utils/         # Shared utility functions
├── ✅ validator/          # Data quality validation and testing
├── 🌐 web/                # Public website and documentation
│   └── release-page/      # Next.js frontend application
├── 🐳 docker-compose.yml  # Local development environment setup

🗺️ Coverage

Current Region: D-A-CH-LI

• 🇩🇪 Germany (Deutschland)
• 🇦🇹 Austria (Österreich)
• 🇨🇭 Switzerland (Schweiz)
• 🇱🇮 Liechtenstein

Data Themes

• places:
  • place (POIs like restaurants, parks, etc. as points)
• divisions:
  • division (logical definition of administrative boundaries as points)
  • division_area (definition of administrative boundaries as polygons or multipolygons)
  • division_boundary (definition of administrative boundaries as linestrings or multilinestrings)

🛠️ Usage

Local Development

1. Clone the repository

git clone <repository-url>
cd cadencemaps

2. Start Docker services

docker-compose up -d

Accessing Data

Access examples can be found on our Website in the Usage section.

📊 Performance

Benchmarks (D-A-CH-LI Region)

• Processing Time: <24 hours for full update
• Data Volume: ~5M POIs processed

A typical client query like "all restaurants in D-A-CH-LI region" takes around 20 seconds to execute as per our testing:

SELECT * FROM read_parquet('s3://cadencemaps/release/2025-06-04/theme=places/type=place/*/*', hive_partitioning=1)
    WHERE
      country IN ('DE', 'CH', 'LI', 'AT')
      AND (categories.primary = 'restaurant'
      OR 'restaurant' IN categories.alternate);

Query Performance

Run	Execution Time
Run 1	20.17 seconds
Run 2	19.17 seconds
Run 3	19.40 seconds
Run 4	19.68 seconds
Run 5	19.05 seconds

🤝 Contributing

We welcome contributions! Please see our contributing guidelines:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

Development Areas

• 🌍 Geographic Expansion: Add new countries/regions
• ⚡ Performance: Optimize processing algorithms
• 🧮 Analytics: New schema mappings and transformations
• 📖 Documentation: Improve guides and examples

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• OpenStreetMap Community - For the incredible open geospatial data
• Geofabrik - For reliable OSM extracts
• Overture Maps Foundation - For schema standardization efforts

---

Built by Matthias Hersche, Fadil Smajilbasic and Nils Robin-Grob as part of the FS25 Bachelor Thesis at the OST