Topics as RwLock instead of Mutex

[Nereuxofficial]

Having benchmarked the performance of the broker i examined for possible performance gains.(Also IME it's the slowest when i've been fuzzing it)

I think a tokio's RwLock may increase the performance. I'd have to look into it, since it's been a while since i've been working with the broker, however if most accesses are Reads this could increase the performance significantly.

Do you think this is worthwhile? @mfloto

Edit: subscribing to the sender only requires read access

[mfloto]

I'd also have to look into it, but as far I can tell it would be beneficial as we only write to e.g. topics, if a subscription to a new topic is required.

[Nereuxofficial]

I since looked into it a bit. It does not in fact look good. Since channels use Mutexes internally, we do have to have a lot of mutable accesses(both reading from a channel as well as writing, as reading may remove an element) no matter what channel we use. Cloning the channel internally is beneficial, but only as long as multiple messages are received via it. For reduced internal complexity of the channel we could probably switch to a Watch since broadcast is many-to-many instead of single-to-many which is the only thing we use...

Implementing a structure for this ourselves would be difficult but possible. We could have a Ringbuffer or Vector, but would need to track the count of subscribers and remove messages after every one of them has received a message.

A fun approach though would be to use sharding for splitting the workload and let each thread handle a channel asynchronously, eliminating the need for shared state and Mutexes if well designed.(Though we would have to think about how to distribute them)

[Nereuxofficial]

One way this might work is by locking all channels behind a RwLock(or sharding them) and putting channels behind a Lock individually, so a lock would only have to be kept on one of the channels