wip: SDAP-TODO: feat: S3 obj store

[wphyojpl]

• add class to store tile blob in S3 in addition to Cassandra
• update configuration to accept either S3 or Cassandra
• refactor main method to avoid code duplication
• TODO: add test case
• TODO: AWS credentials

How SDAP handles S3 credentials

• Query repo already has S3 class to extract data from S3: https://github.com/apache/incubator-sdap-nexus/blob/master/data-access/nexustiles/dao/S3Proxy.py
• Collection Manager also has a class to observe files in S3 bucket to be ingested: https://github.com/apache/incubator-sdap-ingester/blob/dev/collection_manager/collection_manager/services/S3Observer.py
• Both classes assume AWS credentials are handled at a different place since they don't have code to do so.
• How are they being handled as we will need that info to write a test case.

[frankinspace]

This was the "old" ingest code that was able to write to S3 instead of cassandra: https://github.com/apache/incubator-sdap-ningester/blob/master/src/main/java/org/apache/sdap/ningester/writer/S3Store.java

The analysis side as you see is able to read data from S3 instead of cassandra. I think @jjacob7734 might have worked on an instance of SDAP where that was configured.

As far as credentials go, it all uses the boto3 library so you can provide credentials in a number of different ways:
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html
I don't recall off the top of my head but typically they would be provided either through environment variables or a ~/.aws/credentials file on the node running the code.

[jjacob7734]

Adding S3 as a datastore option was work done by David Silvan, a summer intern at JPL, in Summer of 2017. I think he worked primarily with @ngachung on that. @ngachung , Thomas Huang, @fgreg and I received an email from David on 8/9/2017 in which he attached some benchmarks comparing the S3 datastore to Cassandra. It looks like at that time his S3 implementation was 2x to 5x (maybe 3x on average) slower than Cassandra.