Serialization

[stevemarin]

Have you thought about adding serialization/saving to disk? At the very least, would you point out what to store for, say, the ngram vectorizer?

Thank you very much for the examples on training word/document vectors using this and comparing to USE! Giving it a try now on my data and it looks pretty good! Thank you!

[stevemarin]

Does this portion capture what's needed for the ngram vectorizer?

https://github.com/TutteInstitute/vectorizers/blob/master/vectorizers/ngram_vectorizer.py#L269-L276

[stevemarin]

Sorry for all the comments! Pickling the ngram vectorizer seems to work as long as all the numba attributes in the class are converted back to regular Python objects. Does this agree with your understanding?

[lmcinnes]

Yes -- to properly handle things we really need to set up the __getstate__ and __setstate__ methods to properly convert the numba based attributes back and forth. Usually this isn't too hard, but it is something that we haven't gotten to yet. With that done pickling should just work. I'll add an issue for that so that we hopefully don't forget it.