-
Notifications
You must be signed in to change notification settings - Fork 3
Open
Labels
Description
This is probably out of scope for milestone 3, but given we have a use case for it, I'd rate it more important than S3 buckets (which was already labelled for a yet to negotiate milestone 4).
We would like to be able to do the following
- Run a query on some DB1 and mirror it into a temporary table/storage
- Run another query on some DB2 and mirror it into a temporary table/storage
- Combine the two results and upload the final table to Dune
Currently this is not possible. Potentially using csv files (which I assume would be part of #59) a strict "append" combination would be possible.
However, to be maximally flexible I wonder if the container couldn't ship with a built in local sqlite or postgres DB that could be used as a data source and sink?
Then, we could order jobs accordingly to make sure the local table is populated before it gets synced to Dune.
bh2smith