Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 22 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

**A ClickHouse migrations CLI with templated SQL and environment-aware config.**

[![NPM Version](https://img.shields.io/npm/v/clisma?labelColor=EE4B2B&color=FFBF00&style=badge)](https://www.npmjs.com/package/clisma) [![Visual Studio Marketplace](https://img.shields.io/badge/Visual_Studio_Marketplace-Extension-blue?labelColor=black&color=white&style=badge)](https://marketplace.visualstudio.com/items?itemName=StopMakingThatBigFace.vscode-clisma)

### 💅 [See full Docs →](https://clisma.poorquality.tech/)

## What is for?
Expand Down Expand Up @@ -68,10 +70,22 @@ variable "ttl_days" {

env "production" {
url = env("CLICKHOUSE_PROD_URL")
cluster_name = "prod-cluster"

migrations {
dir = "migrations"

table {
name = "schema_migrations"

is_replicated = true

# Optional: force a specific cluster for ON CLUSTER.
cluster_name = "prod-cluster"

# If replication_path is set, is_replicated can be omitted.
replication_path = "/clickhouse/tables/cluster-{cluster}/shard-{shard}/{database}/schema_migrations"
}

vars = {
is_replicated = true
create_table_options = "ON CLUSTER prod-cluster"
Expand All @@ -81,17 +95,20 @@ env "production" {
}
```

**`cluster_name`** affects how the migrations tracking table is created (replicated or not). And the CLI will warn if the actual cluster does not match the config.
**`migrations.table`** controls the tracking table:

#### If your ClickHouse server has clusters configured, `cluster_name` is required
- `name` sets a custom table name.
- `is_replicated = true` enables replicated tracking.
- `cluster_name` optionally selects cluster for `ON CLUSTER`.
- `replication_path` overrides the default replication path (and also enables replicated mode if `is_replicated` is omitted).

### TLS certificates (custom CA and mTLS)

If your ClickHouse endpoint uses a self-signed certificate, add a `tls` block so clisma can trust your CA.

```hcl
env "production" {
url = env("CLICKHOUSE_URL") # e.g. https://user:pass@host:8443/db?secure=true
url = env("CLICKHOUSE_URL") # e.g. https://user:pass@host:8443/db

tls {
ca_file = env("CLICKHOUSE_CA_FILE")
Expand All @@ -108,12 +125,10 @@ Notes:
- `ca_file` is required when `tls` is set.
- `cert_file` and `key_file` must be provided together.
- Relative paths are resolved from the directory where `clisma.hcl` lives.
- URL query params (like `?secure=true`) are preserved.

## 🧪 Templates

Templates are [Handlebars](https://handlebarsjs.com/guide/expressions.html). Variables come from `migrations.vars` (and
`cluster_name` is available as `{{cluster_name}}`).
Templates are [Handlebars](https://handlebarsjs.com/guide/expressions.html). Variables come from `migrations.vars`.

```sql
CREATE TABLE IF NOT EXISTS events {{create_table_options}} (
Expand Down
18 changes: 1 addition & 17 deletions docs/configuration/env-and-vars.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -38,20 +38,4 @@ env "local" {
}
}
}
```

## Cluster name

If you want the migrations table to be replicated and use cluster-aware templates, set
`cluster_name` in the environment:

```hcl
env "production" {
url = env("CLICKHOUSE_PROD_URL")
cluster_name = "prod-cluster"

migrations {
dir = "migrations"
}
}
```
```
9 changes: 7 additions & 2 deletions docs/configuration/examples.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ env "staging" {

migrations {
dir = "migrations"

vars = {
replication_factor = 2
ttl_days = 30
Expand All @@ -37,7 +38,6 @@ env "staging" {
```hcl
env "production" {
url = env("CLICKHOUSE_PROD_URL")
cluster_name = "prod-cluster"
exclude = ["system.*", "_tmp_*"]

tls {
Expand All @@ -48,7 +48,12 @@ env "production" {

migrations {
dir = "migrations"
table_name = "schema_migrations"

table {
name = "schema_migrations"
is_replicated = true
}

vars = {
replication_factor = var.replication_factor
ttl_days = 90
Expand Down
23 changes: 17 additions & 6 deletions docs/configuration/overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -29,16 +29,25 @@ env "local" {

env "production" {
url = "http://default:password@localhost:8123/mydb"
cluster_name = "dwh"

tls {
ca_file = "certs/ca.pem"
}

migrations {
dir = "migrations"
table_name = "schema_migrations"
replication_path = "/clickhouse/tables/{cluster}/schema_migrations"

table {
name = "schema_migrations"

is_replicated = true

# Optional: force a specific cluster for ON CLUSTER.
cluster_name = "dwh"

# If replication_path is set, is_replicated can be omitted.
replication_path = "/clickhouse/tables/{cluster}/schema_migrations"
}

vars = {
ttl_days = 30
Expand All @@ -50,7 +59,6 @@ env "production" {
Fields:

- `url` (string) Connection string. Supports `env("NAME")`.
- `cluster_name` (string, optional) Cluster name used for template context and replicated tracking. Required when your server has clusters configured.
- `exclude` (list\<string\>, optional) Patterns to ignore.
- `migrations` (block) Migration settings.
- `tls` (block, optional) TLS certificate settings for custom CA and mTLS:
Expand All @@ -61,8 +69,11 @@ Fields:
## migrations

- `dir` (string) Path to migrations directory.
- `table_name` (string, optional) Custom tracking table.
- `replication_path` (string, optional) Replication path for the tracking table.
- `table` (block, optional) Tracking table settings.
- `table.name` (string, optional) Custom tracking table.
- `table.is_replicated` (bool, optional, default `false`) Use replicated tracking table with `ON CLUSTER`.
- `table.cluster_name` (string, optional) Cluster name override for `ON CLUSTER`. Useful when multiple clusters exist.
- `table.replication_path` (string, optional) Replication path for the tracking table. If set, replicated mode is enabled even when `table.is_replicated` is omitted.
- `vars` (object, optional) Variables for Handlebars templates.

## variable "name"
Expand Down
105 changes: 97 additions & 8 deletions docs/migrations/faq.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,10 @@ env "local" {

migrations {
dir = "migrations"
table_name = "custom_schema_migrations"

table {
name = "custom_schema_migrations"
}
}
}
```
Expand All @@ -26,27 +29,113 @@ In clustered setups you can override the migrations table replication path used
```hcl
env "production" {
url = env("CLICKHOUSE_PROD_URL")
cluster_name = "prod-cluster"

migrations {
dir = "migrations"
table_name = "custom_schema_migrations"
replication_path = "/clickhouse/foo/bar/custom_schema_migrations"

table {
name = "custom_schema_migrations"

is_replicated = true

# Optional: force a specific cluster for ON CLUSTER.
cluster_name = "prod-cluster"

# If replication_path is set, is_replicated can be omitted.
replication_path = "/clickhouse/foo/bar/custom_schema_migrations"
}
}
}
```

Default replication path:

```text
/clickhouse/tables/cluster-{cluster}/shard-{shard}/{database}/${migrations.table_name}
/clickhouse/tables/cluster-{cluster}/shard-{shard}/{database}/${migrations.table.name}
```

### Summary

If `env.cluster_name` is set, clisma treats the environment as clustered and creates the migrations table using `ReplicatedReplacingMergeTree(...)` with `ON CLUSTER "<cluster_name>"`.
If `migrations.table.is_replicated` is `true`, clisma treats the environment as clustered and creates the migrations table using `ReplicatedReplacingMergeTree(...)` with `ON CLUSTER "<detected_cluster>"`.

If `migrations.table.is_replicated` is not set (or `false`), it uses `ReplacingMergeTree()` in standalone mode.

If `migrations.table.replication_path` is set, clisma also enables replicated mode even when `migrations.table.is_replicated` is omitted.

If your ClickHouse has multiple non-default clusters, set `migrations.table.cluster_name` to choose which cluster should be used in `ON CLUSTER`.

## How config affects migrations table DDL

### 1) Standalone (default)

Config:

```hcl
migrations {
dir = "migrations"
}
```

DDL shape:

```sql
CREATE TABLE IF NOT EXISTS schema_migrations (
...
) ENGINE = ReplacingMergeTree()
ORDER BY version;
```

### 2) Replicated (auto cluster detection)

If `cluster_name` is not set in config and ClickHouse reports configured clusters, clisma stops and asks you to set `cluster_name`. If there are no clusters, it uses `ReplacingMergeTree()` in standalone mode.
Config:

```hcl
migrations {
dir = "migrations"
table {
is_replicated = true
}
}
```

DDL shape:

```sql
CREATE TABLE IF NOT EXISTS schema_migrations ON CLUSTER "<detected_cluster>" (
...
) ENGINE = ReplicatedReplacingMergeTree(
'/clickhouse/tables/cluster-{cluster}/shard-{shard}/{database}/schema_migrations',
'{replica}'
)
ORDER BY version;
```

### 3) Replicated with explicit cluster and path

Config:

```hcl
migrations {
dir = "migrations"
table {
name = "custom_schema_migrations"
cluster_name = "prod-cluster"
replication_path = "/clickhouse/foo/bar/custom_schema_migrations"
}
}
```

DDL shape:

```sql
CREATE TABLE IF NOT EXISTS custom_schema_migrations ON CLUSTER "prod-cluster" (
...
) ENGINE = ReplicatedReplacingMergeTree(
'/clickhouse/foo/bar/custom_schema_migrations',
'{replica}'
)
ORDER BY version;
```

## Does clisma support down migrations?

Expand All @@ -61,4 +150,4 @@ Yes. You can include multiple SQL statements in one file. Separate statements wi
If a migration file changes after being applied, clisma will fail with a checksum mismatch. You have two options:

- Revert the migration file back to the applied version and create a new forward migration.
- If you really need to override the checksum, update the stored checksum in the migrations table (use `clisma checksum <migration path>`). This is risky and should only be done if you fully understand the consequences.
- If you really need to override the checksum, update the stored checksum in the migrations table (use `clisma checksum <migration path>`). This is risky and should only be done if you fully understand the consequences.
10 changes: 7 additions & 3 deletions docs/migrations/templates.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,7 @@ description: "Use Handlebars variables in migrations to adapt across environment
icon: "notes-medical"
---

Template variables come from `migrations.vars` in your config. If you set
`cluster_name` on the environment, it is also available as `{{cluster_name}}`.
Template variables come only from `migrations.vars` in your config.

> If you have used Atlas, this will feel familiar. The idea is the same as
> [Atlas templated migrations](https://atlasgo.io/concepts/migrations#template),
Expand All @@ -31,10 +30,15 @@ env "local" {

env "production" {
url = env("CLICKHOUSE_PROD_URL")
cluster_name = "prod-cluster"

migrations {
dir = "migrations"

table {
is_replicated = true
cluster_name = "prod-cluster"
}

vars = {
is_replicated = true
create_table_options = "ON CLUSTER prod-cluster"
Expand Down
14 changes: 13 additions & 1 deletion example/clisma.hcl
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@ env "local" {

env "production" {
url = "http://default:password@localhost:8123/default"
cluster_name = "prod-cluster"

tls {
ca_file = "certs/ca.pem"
Expand All @@ -18,6 +17,19 @@ env "production" {

migrations {
dir = "migrations"

table {
name = "schema_migrations"

is_replicated = true

# Optional: force a specific cluster for ON CLUSTER.
cluster_name = "prod-cluster"

# If replication_path is set, is_replicated can be omitted.
replication_path = "/clickhouse/tables/cluster-{cluster}/shard-{shard}/{database}/schema_migrations"
}

vars = {
is_replicated = true
create_table_options = "ON CLUSTER prod-cluster"
Expand Down
Loading