Skip to content

Conversation

@leventov
Copy link
Member

@leventov leventov commented Dec 7, 2016

  • Deduplicate looking for bitset.nextSetBit() in BitSetIterator.next() and hasNext()
  • Along the way it makes this iterator to follow Iterator contract, should throw NoSuchElementException on calling next() after hasNext() = false. Before it was returning -1, that is bug-prone.
  • Add BitmapIterationTest which verifies Bitmap iteration implementations. BTW it fails for Roaring right now, but immediate fix is not possible, because it's a library dependency.

It makes iteration over a bitset 20-30% faster (before-after, benchmarks from #3751):

Benchmark                      (bitmapAlgo)  (prob)   (size)  Mode  Cnt        Score        Error  Units
BitmapIterationBenchmark.iter        bitset     0.0  1000000  avgt    5        4.052 ±      0.137  ns/op
BitmapIterationBenchmark.iter        bitset   0.001  1000000  avgt    5    36647.927 ±   1738.782  ns/op
BitmapIterationBenchmark.iter        bitset     0.1  1000000  avgt    5   699019.477 ± 158638.803  ns/op
BitmapIterationBenchmark.iter        bitset     0.5  1000000  avgt    5  3395422.304 ± 158315.241  ns/op
BitmapIterationBenchmark.iter        bitset    0.99  1000000  avgt    5  5875929.154 ± 241268.439  ns/op
BitmapIterationBenchmark.iter        bitset     1.0  1000000  avgt    5  5626511.995 ± 759176.569  ns/op

Benchmark                      (bitmapAlgo)  (prob)   (size)  Mode  Cnt        Score        Error  Units
BitmapIterationBenchmark.iter        bitset     0.0  1000000  avgt    5        3.221 ±      0.362  ns/op
BitmapIterationBenchmark.iter        bitset   0.001  1000000  avgt    5    26713.522 ±   6375.378  ns/op
BitmapIterationBenchmark.iter        bitset     0.1  1000000  avgt    5   521381.301 ±  16891.222  ns/op
BitmapIterationBenchmark.iter        bitset     0.5  1000000  avgt    5  2053421.442 ±  34196.751  ns/op
BitmapIterationBenchmark.iter        bitset    0.99  1000000  avgt    5  3969497.718 ± 410796.782  ns/op
BitmapIterationBenchmark.iter        bitset     1.0  1000000  avgt    5  4040926.926 ± 139893.245  ns/op

This is a copy of metamx/bytebuffer-collections#37

@leventov leventov changed the title Bitset iteration optimization and bug fix Bitset iteration optimization and improve safety Dec 7, 2016
@leventov
Copy link
Member Author

leventov commented Dec 7, 2016

Copy link
Contributor

@gianm gianm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 on the main code change but please adjust the test code as mentioned.

new BitSetBitmapFactory(),
new ConciseBitmapFactory()
// Exclude Roaring because it fails yet!
//new RoaringBitmapFactory()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What exactly fails here? Is it bad that it fails? If it's not bad, why are we testing it?

Please add a comment here answering those questions (and linking anything we're waiting for upstream in Roaring) or else remove the tests if they're not testing something we actually care about.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gianm done

@gianm gianm assigned fjy and gianm Dec 7, 2016
@gianm gianm added this to the 0.9.3 milestone Dec 7, 2016
@gianm
Copy link
Contributor

gianm commented Dec 7, 2016

thanks, LGTM

@fjy
Copy link
Contributor

fjy commented Dec 7, 2016

👍

@fjy fjy merged commit 949e651 into apache:master Dec 7, 2016
@leventov leventov deleted the bitset-iteration-optimization branch December 8, 2016 20:49
dgolitsyn pushed a commit to metamx/druid that referenced this pull request Feb 14, 2017
* Deduplicate looking for bitset.nextSetBit() in BitSetIterator.next() and hasNext()

* Add BitmapIterationTest

* More elaborate comment on why Roaring is not tested in BitmapIterationTest
@clambertus clambertus unassigned fjy and gianm Jul 6, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants