feat: add ?meta=eof for trailling metadata dag-json

[rvagg]

Prototype for an option for implementing ipfs/specs#431

Defines a new meta query parameter that, if it has a value of eof, will do the following:

• Terminate the CAR body with a NUL byte, such that the CARv1 is only now parseable with the ZeroLengthSectionAsEOF mode (I don't think we have this in js-car yet but it's not hard as an opt-in), but it cleanly terminates.
• Write a trailing metadata message in dag-json format with the schema defined in metadata/metadata.ipldsch. This message is versioned, so we can upgrade it fairly easily just by adding a new version to the parent enum. It contains the requested parameters, and properties of the CAR returned - bytes, data bytes (block content only), block count, and a checksum which is a multihash of the entire CARv1 payload (blake3 here). If there was an error producing the CAR, then the properties are omitted and replaced with an error field which has a message.

Lassie should be able to pass in ?meta=eof to HTTP retrievals, strip this message out (where it exists - remotes may not have it), and present the information separately; either via an API return property, or to stdout for the CLI.

Examples (which you can run with curl):

curl 'http://ipfs.va.gg:3747/ipfs/bafybeiaysi4s6lnjev27ln5icwm6tueaw2vdykrtjkwiphwekaywqhcjze/wiki/Oregon_Cartoon_Institute?dag-scope=entity&meta=eof' -o foo.car

Pretty printed dag-json trailing bytes:

{
    "car-metadata/v1": {
        "properties": {
            "block_count": 5,
            "car_bytes": 94183,
            "checksum": { "/": { "bytes": "HiAmqA2NmdW37sogTFF6FOHQmQ8UOHJVHjmEUQevbUg/TQ" }
            },
            "data_bytes": 93933
        },
        "request": {
            "dups": true,
            "path": "/wiki/Oregon_Cartoon_Institute",
            "root": { "/": "bafybeiaysi4s6lnjev27ln5icwm6tueaw2vdykrtjkwiphwekaywqhcjze" },
            "scope": "entity"
        }
    }
}

/cc @bajtos

[rvagg]

• Doesn't do any of the signing proposed in the IPIP, that needs to be added in some form. Either replace or add to the checksum field with a signed blob as suggested?
• Needs consideration of caching. Etag should either include a bit for meta or even the peerid if we're doing signing and it matters for uniqueness in caches (maybe not, but I'm not clear on the flow here).

[lidel]

nit: this is a CAR content type parameter, and should go to Accept and Content-Type headers, like ones from IPIP-412. (mentioning early to avoid costly renames later 😅)

[rvagg]

Ah ok, so the reasoning here is that anything that changes the format of the output should go in Content-Type or Accept? The content itself is dictated by the query parameters, but the way it's presented is dictated by these headers? Is that the right way to think about it?

So something like:

Accept: application/vnd.ipld.car; version=1; order=dfs; dups=y; meta=eof;

Shouldn't be hard to adapt this. We have a ParseAccept in lassie that I'm using here which I'll just extend if that's where it goes.