Setup steel thread version of METjson2db

[ian-noaa]

We want to be able to demonstrate that METjson2db can be used to process MET stat files in the cloud, and send them to our Capella database.

To that end, we want to:

• Create an S3 bucket, and upload sample MET stat output to it
• Create an EC2 instance, and install METjson2db on it
• Use the aws CLI to copy data from S3 to the EC2 instance
• Use METjson2db to parse a subset of the data (i.e. a single file) into our JSON format
• Use METjson2db to send the JSON formatted data to our Capella instance

[ian-noaa]

I've gone ahead and done this, this morning. To do so, I:

1. Created an s3 bucket (vx-metjson-test)
   2. Uploaded tc_data.zip and G2G_v12.zip to the bucket
2. Created an EC2 instance (vx-metjson-test)
3. Installed Go & METjson2db on the machine
4. Used the aws cli to copy data from the bucket to the instance
5. unzipped the tc_data.zip into a $/data directory
6. Created a new "ec2-test" user in Capella
7. Updated the credentials file with the connection information for Capella
8. Updated the load_spec.json file to have a unique dataset name (ian-test), and local output directory.
9. used METjson2db in archive and then in upload mode to first create a JSON archive so I could verify content was correct, and then to upload the stat file contents directly to Couchbase.Resulting DocIDs are:

• MET:DD:MET:ian-test:V11.1.0:UKX:BEST:AL972023:AL:97:INVEST:1698710400:TCMPR
• MET:DD:MET:ian-test:V11.1.0:UKX:BEST:AL972023:AL:97:INVEST:1698753600:TCMPR

Notes I took during the process:

# Notes

Data at: `../data/tc_data/UKX/2023110700`

## Running

Did: `./metjson2db -f ../data/tc_data/UKX/2023110700/tc_pairs_al97.dat.tcst -m CREATE_JSON_DOC_ARCHIVE` to create a gzip'd JSON document.

I also updated the `load_spec.json` file with a new `datasetName` field so we could tell that these documents were from this experiment.

## Running direct to Couchbase

Did: `./metjson2db -f ../data/tc_data/UKX/2023110700/tc_pairs_al97.dat.tcst -c ./credentials`

Relevant docID's now resident in Capella:

* MET:DD:MET:ian-test:V11.1.0:UKX:BEST:AL972023:AL:97:INVEST:1698710400:TCMPR
* MET:DD:MET:ian-test:V11.1.0:UKX:BEST:AL972023:AL:97:INVEST:1698753600:TCMPR

Takeaways:

• I found the load_spec.json file was confusing.
  • large portions of the example load_spec.json file in the repo appear to be unneeded.
• METjson2db could have more CLI help output
  • It was unclear how the flags interacted with the load_spec.json file. (E.g. - does-f mode need a load_spec.json?)
• The README could be updated with a focus on end users as the audience, and moving some of the developer documentation to a separate README.
• It would have been nice to have a precompiled METjson2db on the Github releases page.