feat(zero-cache): valitas-based flag/env configuration

[darkgnotic]

All your config are belong to valitas

---

valitas-based library for automatically deriving command line flags + environment variables from a valitas-style configuration object schema.

The Options type is structured similarly to the familiar valitas v.object() specification.

e.g.

const options = {
  port: {type: v.number().default(4848), desc: ['blah blah blah']},
  replicaDBFile: v.string(),
  litestream: v.boolean().optional(),
  log: {
    format: v.union(v.literal('text'), v.literal('json')).default('text'),
  },
  shard: {
    id: {type: v.string().default('0'), desc: ['blah blah blah']},
    publications: {type: v.array(v.string()).optional(() => [])},
  },
};

with the two important differences being:

1. The top-level object and non-value groups are plain Records, rather than v.object() instances
2. A value can be a v.Type or a {type: v.Type, desc: string[]} wrapper that allows documenting the configuration value in the same place that the value is defined.

The resulting configuration object type is derived from the Options instance:

And, most importantly, flag and environment variable parsing is dynamically configured from the Options object:

const config: TestConfig = parseOptions(options, argv);

The library includes support for:

• primitive types number, string, boolean
• v.array() types thereof
• v.union()
• .optional() semantics
• .default() semantics

The precedence for determining the values of the final configuration object are, from low to high:

• Defaults defined by .default(...) in the Options object.
• Values defined in the environment variables.
• Values supplied on the command line

Implementation Notes

The general approach leverages the command-line-args library for command line parsing and valitas for validation. (command-line-args explicitly focuses on command line parsing and delegates validation).

valitas is similarly laser-focused on JSON messages and doesn't always provide convenient APIs for using it in other contexts. However, we already use valitas everywhere, it is good at what it does, and it packages pretty much all of the semantic information that we need for flags parsing (with the exception being description / documentation).

On balance, the convenience of being able to stick to valitas, and automatically derive our flags and environment variables from it, seems to justify the sometimes obtuse implementation necessary to make it work (e.g. you have to figure out what Type.func() and Type.toTerminals() do).

Luckily, the valitas package is small and simple enough that it's not too bad.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
replicache-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Nov 1, 2024 9:37pm
zbugs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Nov 1, 2024 9:37pm

[arv]

before digging into this... See comments regarding pitfalls with default at https://github.com/badrap/valita/releases/tag/v0.3.11

[arv]

LGTM

NIH :-)

But I understand the need.

A few things.

• How do we deal with things like fooJSON vs fooJson?
• It is common to support short flags like -p for --port. In git one can do git commit -am "Hello world" which is the same as git commit --add --message "Hello world".

[arv]

What happens with shardID for example?

[darkgnotic]

before digging into this... See comments regarding pitfalls with default at https://github.com/badrap/valita/releases/tag/v0.3.11

Very nice, thanks for the tip. I added an instance of that use case in the test (particularly, for the array).

[darkgnotic]

LGTM

NIH :-)

But I understand the need.

I am of the same mind. My first attempt was actually completely based on the https://github.com/Roaders/ts-command-line-args library, which is a similar type-safe wrapper around command-line-args (but doesn't use valitas).

I even implemented a feature request and added env vars to it: Roaders/ts-command-line-args#48

But as I went down that path, I found a few things lacking:

• Grouped options to mirror the groups that we already have in our config. Not a huge deal; we could have flattened everything, but given that (1) both Matt and I intuitively structured config options in groups, and (2) command-line-args supports grouped parsing, it seemed like a good thing to have.
• Unions and other validation parsing are not in either flags packages. So I started adding custom validation functions for our unions like the LOG_LEVEL and LOG_FORMAT. And then I realized I could use valitas directly ...
• And it became clear that declaring valitas validators for both the config object and the flags parsing was redundant and felt a bit silly.
• So I started over with a different approach of using the same valitas schema for the config object and flags.

I like how it turned out. But just so you know, I explored quite a few options before deciding they were all NIH. 🤣

A few things.

• How do we deal with things like fooJSON vs fooJson?

It looks like lodash.camelcase handles things that are already like that, like replicaDBFile. But yes, things like id are camelcased to shardId. Two options that come to mind are:

1. Just live with it.
2. Add a name override field to the {type: ... , desc: ...} Option specification to force it to be caps, i.e.

shard: {
  id: {type: v.string().default('0'),
  name: 'ID',
  desc: ['blah blah blah'],
}

wdyt?

• It is common to support short flags like -p for --port. In git one can do git commit -am "Hello world" which is the same as git commit --add --message "Hello world".

command-line-args supports an alias field, so I can add that to the wrapper too. I know that handles the short versions, but I'm not sure about the combination "-am" thing.

[darkgnotic]

Thank you for the thoughtful review!

[darkgnotic]

2. Add a name override field to the {type: ... , desc: ...} Option specification to force it to be caps

On second thought, the name override idea is a bit tricky because we later have to convert the parsed flag back into the field defined in the schema. I guess we could dig back into the schema to find out what to rename to.

Or, we could capitalize the filed itself, but that's kind of weird too.

shard: {
  ID: ...
}

[darkgnotic]

command-line-args supports an alias field, so I can add that to the wrapper too. I know that handles the short versions, but I'm not sure about the combination "-am" thing.

Looks like the combining of short aliases is supported by command-line-args:

https://github.com/75lb/command-line-args/blob/master/doc/option-definition.md#module_option-definition--OptionDefinition+alias

[darkgnotic]

2. Add a name override field to the {type: ... , desc: ...} Option specification to force it to be caps

On second thought, the name override idea is a bit tricky because we later have to convert the parsed flag back into the field defined in the schema. I guess we could dig back into the schema to find out what to rename to.

Or, we could capitalize the filed itself, but that's kind of weird too.

shard: {
  ID: ...
}

I went for an allCaps option. This turned out to be the simplest.

Accidentally committed directly into main: 79c6aca