Connection manager failure after configuration of RT replication

[cmeiklejohn]

I've identified another issue using basho/riak_test#470:

If you remove the wait_for_ring_convergence call after enabling and starting realtime replication, you can run into a situation which appears to be the following:

1.) Node 1 is leader, and knows about cluster B.
2.) Node 1 connection manger is killed, triggering cluster manager to restart because of the rest_for_all supervision configuration.
3.) Node 1 comes back online, and is re-elected as leader (but it appears as a new election as cluster manager has just started for the first time, triggering the notify fun to be called immediately after registration through riak_repl2_leader).
4.) Node 1 REPL ring contains no information about remote clusters it was previously connected to.

Both @kellymclaughlin and I have been able to reproduce this via the test by removing the wait calls, but it's unclear as to what the actual root cause is. We've decided to hold this issue back and push 1.4.4 along without attempting to fix it.

cc: @Vagabond @metadave @jonmeredith @jaredmorrow

[cmeiklejohn]

After about another 20 runs or so, yes, I know, we were able to reproduce the failure under what appears to be possibly related other conditions:

1.) riak-repl connect triggers the creation of a riak_core_cluster_conn process through it's supervisor.
2.) This process, upon a successful connection (or polling interval), asks the remote cluster for IP and name information.
3.) This process, then makes a async request to the cluster manager service (grep for cluster_updated) to persist this cluster in the ring.

Obviously, we have the potential for races here where a node dying or a process failing could cause the data to not be persisted to the ring. Leaving this comment so we know to come back and investigate this further.

References:

https://github.com/basho/riak_repl/blob/develop/src/riak_core_cluster_conn.erl#L94
https://github.com/basho/riak_repl/blob/develop/src/riak_core_cluster_mgr.erl#L368
https://github.com/basho/riak_repl/blob/develop/src/riak_core_cluster_mgr.erl#L559
https://github.com/basho/riak_repl/blob/develop/src/riak_core_cluster_mgr.erl#L528
https://github.com/basho/riak_repl/blob/develop/src/riak_core_cluster_mgr.erl#L543

[cmeiklejohn]

Moving to 2.1 as part of the scheduled connection manager work.