-
Notifications
You must be signed in to change notification settings - Fork 0
Closed
Description
Data upload from a simulated CML data source is now working. Next step is to parse the data to the DB.
Main questions:
- We already have a container for parsing but it is not doing something. Should we implement the parser there or is it better to integrate it into one of the other services (the flask webserver)? Probably not because CPU load could spike for the parser. But this needs to be decided first.
- Parsing and handling of metadata and raw data has to be done separately. That should be easy to do, but has to be take into account, in particular the aspect that metadata and raw data need to be linked via a CML ID and a sublink_id. This linkage needs to be documented somewhere but it might also be enough to only parse data to the DB that has this clear linkage via CML IDs.
- Raw data and metadata files need to be moved to an archive directory once they have been parsed succesfully and if not there needs to be an intermediate (quarantine?) directory to keep them.
- There must be a way to test the parsers and the result in the DB which must be separate from a running production system.
Implementation plan:
- ...to be added... (iterated while implementing things in Add data parser service #14)
Metadata
Metadata
Assignees
Labels
No labels