Skip to content

Ovewriting previous states with new states to save memory and sampling progress #59

@EL-MZ

Description

@EL-MZ

Dear Karamanis,

I would like to thank you for this amazing implementation of PMC and a very useful sampler.

I have been using your sampler recently on challenging posterior sampling.
And, I have experienced two issues on my side since saving every state.
When running a long HPC job for a high-dimensional problem.
Every single progress made by the sampler is very valuable.
Saving one each in multiple iterations would mean losing important progress, especially when every iteration takes hours.
Progress would be lost if the job were terminated due to the time limit.
The second issue is that saving multiple states for high-dimensional and challenging problems is expensive.
For example, I am testing the sampler on ndim~140, and every single state takes 90mb at least.

I think a good solution would be adding the ability to overwrite previous states so one can save each iteration, which will save memory and progress.

Thank you.

Regards,
EL Mehdi

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions