Add consolidated metadata manifest for full pipeline auditability

[AhmedAli58]

Adds a single metadata.json manifest generated on each successful publish. Consolidates download, normalization, and publish metadata into one auditable source of truth, improving reproducibility, provenance, and trust.

[stiles]

Hi @AhmedAli58. Thanks for this contribution! The consolidated metadata manifest is a good idea for improving pipeline auditability. Your implementation looks solid and the code is well-documented.

Before merging, could you please address these items:

Required:

1. Add unit tests for the new _create_consolidated_metadata() function
2. Update README.md to document the new metadata.json file and its fields
3. Add a CHANGELOG.md entry for this feature

Recommended:
4. Add a metadata_version field to the consolidated metadata for future schema evolution
5. Consider warning/failing if critical fields are missing rather than silently using empty defaults

Let me know if you have questions about any of these! Looking forward to the updates.

[AhmedAli58]

Hi, I’ve addressed all the requested items: added unit tests, updated the README with metadata.json documentation, and added a CHANGELOG entry. The PR now includes the requested changes. Thanks!