Make flist format OCI compliant

[zaibon]

This is the first step to be able to use flist format run any OCI runtime (runc, ...)
The definition of an OCI bundle: https://github.com/opencontainers/runtime-spec/blob/master/bundle.md

So it really only requires 2 things:

• config.json
• the root filesystem of the application.

Flist format already provide a way to have a full filesystem mounted using 0-fs

So the only things missing is the config.json.

[muhamadazmy]

As per this comment here threefoldtech/zos#5 (comment) we can have the flist (which is basically a tar) have (next to the sql database) a config.json file which has a subset of the OCI config.json file. The one included can define exec command, env vars, ports, etc ... where the system config.json will define permissions. we can have different variants of the system config.json file (profiles) which defines different privileged levels.

[zaibon]

yep that's pretty much how I was seeing things too.
@muhamadazmy can you take this one over and try to define the exact subset of the spec we want to allow to be defined by the user ? full spec is there: https://github.com/opencontainers/runtime-spec/blob/master/spec.md

[zaibon]

After a bit more thinking about this, I don't think we should make the flist OCI compilant directly.
But instead define a "metadata" file format in which we can let the user creating the flist define some how the flist should be used. This format would be understood by 0-OS and it could convert it to a proper config.json file OCI compliant.

example of configurable :

• entry point
• volumes
• environment variables
• exposed ports

[maxux]

Implemented on the latest zflist-edit:

• Backend address
• Entrypoint
• Environment variables
• Exposed ports
• Volumes

Data structure

Backend

The backend is stored in a json object with theses members:

• host: hostname (address) of the server
• port: the port
• socket: optional unix socket (port is 0 and host should not be set then)
• namespace: a specific zdb namespace to use
• password: a password to provide when switching namespace

Entrypoint

It's just a json array of the arguments, eg:

["/sbin/sshd", "-D", "-v", "-N"]

Environment variables

It's a json dictionary, key is the variable name, content is a string with the value contents, eg:

{"HOME": "/root", "PATH": "/sbin:/bin", "EDITOR": "/bin/vi"}

Exposed ports

It's a json dictionary, key is protocol/in-port and value is an integer with destination port, eg:

{"tcp/80": 80, "tcp/22": 2222, "udp/53": 53}

Volumes

It's a json dictionary, key is the host path, value is an object: key target contains container path, eg:

{"/dev": {"target": "/dev"}, "/mnt/storage": {"target": "/storage"}}

Implementation

All the output above are raw output from latest implementation on zflist-edit code, everything was generated by the code, here are the following commands used:

./zflist-edit metadata entrypoint -- /sbin/sshd -D -v -N

./zflist-edit metadata environ --name HELLO --value world
./zflist-edit metadata environ --name HOME --value /root
./zflist-edit metadata environ --name PATH --value /sbin:/bin
./zflist-edit metadata environ --name EDITOR --value /bin/vi
./zflist-edit metadata environ --name HELLO --unset

./zflist-edit metadata port --in 80
./zflist-edit metadata port --in 22 --out 2222
./zflist-edit metadata port --protocol udp --in 53

./zflist-edit metadata volume --host /dev
./zflist-edit metadata volume --host /mnt/storage --target /storage

Command line

Each metadata have a --reset flag which remove the metadata from the database. For the backend and entrypoint, you overwrite everything on any changes (theses are unique values). For all the others, you always have --unset argument to remove the specified key.

When not providing any values (eg: ./zflist-edit metadata port), the current value is printed.

Each metadata subcommands have help message, eg:

./zflist-edit metadata port --help
[...]
[+] action: metadata: arguments:
[+]   -p --protocol    protocol name (should be tcp or udp, default: tcp)
[+]   -i --in          input port number
[+]   -o --out         destination port (default: same as in)
[+]   -u --unset       remove the specified input port
[+]   -r --reset       remove metadata (all exported ports)
[+]   -h --help        show this message

[maxux]

I'm adding a metadata: Readme
This metadata will be used to allow user to set arbitrary text to describe the flist and include an optional licence name

[maxux]

I made a modification on volumes metadata: the dictionary key is the target mountpoint and not the host mountpoint. Like this, you can specify which mountpoint your flist wants, without specifying the host mountpoint.

Eg: you can set on your flist I need to get /data mounted, but not specifying where this is on the host, this can be specified on runtime by environment. You can optionally set the host point. This is way more useful to describe what does the flist needs.