While working the test adjustments to #129 - I noticed that we are only catching and logging errors. Not actually failing. Furthermore, we return the number of data points as if that were the number of affected rows. This can lead to incomplete jobs passing silently.
Some ideas:
- Allow the errors to be raised
- return boolean from job.run based on success or raised error
- collect failures at the end
- Report on failed jobs.
I am currently working on this task directly involving DuneDestination.save. Not sure if this should be fixed independently or just repaired in the overhaul.