Compact Index Format

[mandykoh]

Now that the serialisation has been separated from IndexNode, we can begin addressing its various inadequacies, which include:

• Inefficient use of space (2.4GB of actual data took up 38GB of space in one use case).
• Creating large numbers of filesystem objects.
• Difficulty to extend with caching/thread-safety/etc.
• Non-handling of hash collisions.
• Requiring very long filesystem paths.

This work is partly being done as the Keva key-value store library.

[mandykoh]

Now that Keva is ready, we can begin to replace the various bespoke disk operations in DiskIndexStore with Keva operations instead. It probably makes sense to have one Keva store for nodes (and their entries).

[gonzalo-bulnes]

Hi! I've seen Keva quickly and I think I get the idea (the concept of bucket is mainly a performance concern, am I right?).

I'm now reading through the simian code to understand what the DiskIndexStore is about, I'm taking notes so I can ask you questions, but for now I'm very much in discovery mode. (It's entertaining so far! I'm trying to remember your presentation as I progress.)

[mandykoh]

Yup, the buckets are similar to the idea of paging: reduce overheads by storing a bunch of things together instead of individually. The main point is that you can store lots and lots of objects by unique key and not have to worry about very long paths or hash collisions or such. To do this, you normally define a bucket/page size, and then either have a set number of buckets up front, or use some sort of allocator to manage what goes in each bucket/page. Keva’s solution is to split buckets once they get full, using the filesystem and a special key format to recursively find buckets in a way that is invariant of the splitting.

Simian was (and is still currently) using one directory per node, one file per fingerprint, etc. However, filesystems become inefficient if you have lots and lots of very sparse directories and small files (eg at the very least, a file takes up a whole block regardless of how much data is in it). Because we want to support indexing very large numbers of images, this poses a problem. Keva solves that and abstracts it away.

The DiskIndexStore is about to change a lot because we’ll remove the custom storage operations from it and use Keva instead.

The one exception is thumbnails—Keva doesn’t have an efficient way to handle large-ish binary objects, so for now the best we can do is store them separately to the nodes. (This was done in the last commit.) This is still an improvement because then we never have to move them if the nodes change (eg during a split operation), but in future the idea is that we don’t store thumbnails in the index at all (and instead the user just stores a reference to the original image as metadata when indexing an image).

We currently rely on the thumbnails to generate different size fingerprints; we can get rid of this requirement once we move to DCT-based fingerprints (or we can write a custom resampling algorithm to resize the fingerprints directly, but that seems wasteful when DCT fingerprints are a better solution).