Decide on a storage medium.

[hearnderek]

It would be nice to be able to save work directly to an output file.
It would also be nice to use said output files when working with multiprocess/machine workloads.
It would also be important when dealing with high memory workloads.

Ideas:

1. results_to_dataframe() -> dataframe_to_csv()
   • limiting on what can be stored as a string
2. results_to_json()
3. results_to_sqlite()
4. result_to_zipdb()

[hearnderek]

in commit 82744bc
I am using sqlite as the storage medium, and also using that as the communication method from child to parent processes.

[hearnderek]

I saw some TEXT columns which should have been some numeric type. Should create some tests to view how numbers can be misinterpreted when converting from DataFrame -> python -> DataFrame -> sqlite3

[hearnderek]

Storing the data is likely to be a heavier than the calculation itself.

One option is to use zipfiles as a document store.

[hearnderek]

Spent a good chunk of time researching and testing the validity of the idea. Decided that while possible it would be a terrible option as the filestore grows. This is because of the need to rebuild the file from scratch every time we want to update or delete something. At that point we're better off creating a hidden folder and using that instead of a datastore.