Skip to content

Conversation

@cbb330
Copy link
Collaborator

@cbb330 cbb330 commented Jan 23, 2026

Summary

Adds new Gradle tasks to simplify the Docker-based local development workflow, replacing the manual multi-step process with a single command.

the OpenHouse build currently depends on shadowJars, which significantly slows builds (in CI and ELR so this time compounds) This dependency existed to simplify the local testing UX.As a first step toward removing shadowJars(which cuts the build time in half from ~5 minutes to ~2 minutes) I've added a one-line command to start the OpenHouse local server, effectively replacing the old shadowJars-based workflow.

Before (Manual Process)

# Step 1: Build all JARs, this task explicitly depends on shadowJars publish step
./gradlew build

# Step 2: Navigate to recipe directory
cd infra/recipes/docker-compose/oh-hadoop-spark

# Step 3: Build Docker images using the shadowjars in step1
docker compose build

# Step 4: Start containers
docker compose up -d

After (Single Command)

./gradlew dockerUp -Precipe=oh-hadoop-spark

New Gradle Tasks

Task Description
dockerPrereqs Builds all JAR files required by Docker images
dockerBuild Builds Docker images after ensuring prerequisites
dockerUp One-command build and start (JARs → images → containers)
dockerDown Stop and remove containers

Usage

# Quick start with lightweight recipe
./gradlew dockerUp -Precipe=oh-only

# Full stack with Spark (default)
./gradlew dockerUp -Precipe=oh-hadoop-spark

# Stop containers
./gradlew dockerDown -Precipe=oh-only

Implementation Details

Explicit JAR Dependencies: The dockerPrereqs task declares all JAR dependencies explicitly, enabling proper Gradle parallelism:

  • Service bootJars: tables, housetables, jobs
  • Spark runtime uber JARs: spark-3.1, spark-3.5
  • Spark apps uber JAR
  • Utility JAR: dummytokens

Recipe Selection: Use -Precipe=<name> to select docker-compose recipe:

  • oh-only - Lightweight, local filesystem (fastest startup)
  • oh-hadoop - With HDFS
  • oh-hadoop-spark - Full stack with Spark (default)

Design Decisions:

  • Tasks are in separate docker group, NOT integrated into ./gradlew build
  • Helpful output messages show service URLs and next steps
  • Error handling for invalid recipe names

Documentation Updates

  • README.md: Added quick start commands in "Running OpenHouse with Docker Compose"
  • SETUP.md:
    • New "Quick Start (Recommended)" section at top
    • Task reference table
    • Restructured with "Manual Docker Compose (Advanced)" section for users who need fine-grained control

Test Plan

  • Verified build works from clean state (no build/ directory)
  • Verified docker tasks are NOT part of ./gradlew build (independent)
  • Verified proper Gradle dependency resolution and parallelism (85 tasks, 66 executed in parallel)
  • Verified services start and respond correctly:
    • Tables Service (8000): 200 OK - Create/Read/Delete table API tested
    • HouseTables Service (8001): 200 OK
    • Prometheus (9090): 200 OK
  • Verified dockerDown properly stops and removes containers

Add new Gradle tasks to simplify the Docker-based local development workflow:

- dockerPrereqs: Builds all JAR files required by Docker images
- dockerBuild: Builds Docker images after ensuring prerequisites
- dockerUp: One-command build and start (JARs → images → containers)
- dockerDown: Stop and remove containers

Usage:
  ./gradlew dockerUp -Precipe=oh-only      # Lightweight local testing
  ./gradlew dockerUp -Precipe=oh-hadoop-spark  # Full stack (default)
  ./gradlew dockerDown -Precipe=oh-only

Benefits:
- Single command replaces manual multi-step process
- Explicit JAR dependencies enable proper Gradle parallelism
- Recipe selection via -Precipe parameter
- Not integrated into standard build (docker tasks are opt-in)
- Helpful output messages with service URLs and next steps

Updated documentation:
- README.md: Added quick start commands
- SETUP.md: New "Quick Start" section, task reference table
@cbb330 cbb330 force-pushed the enable-parallel-build-default branch from 9e65ab0 to d50661e Compare January 23, 2026 01:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant