Skip to content

Add chunking to serializing models into Store #192

@bjester

Description

@bjester

Overview

Serializing huge amounts of data into the Store could cause large increases in memory usage. Currently _serialize_into_store processes batches by model, but loads all of the model's data into memory before saving the bulk writes to the Store model.

Deliverables

  • Relocate and split up _serialize_into_store into smaller functions
  • Have the serialization process the model objects in chunks, or via a queryset iterator() (ref)

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions