RSB does sometimes not activate participants when they were deactivated in advance.

[DivineThreepwood]

Target Version: rsb 0.18

Problem: RSB does sometimes not activate participants when they were deactivated in advance.

Reproducible via the example code provided by #28
when setting ENABLE_CONNECTION_SHARING = false;

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.protobuf.UnsafeUtil (file:/Users/divine/.m2/repository/com/google/protobuf/protobuf-java/3.0.2/protobuf-java-3.0.2.jar) to field java.nio.Buffer.address
WARNING: Please consider reporting this to the maintainers of com.google.protobuf.UnsafeUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Start run 1 with 20 participants...
Wait until pool is active...
Initiate Shutdown...
Shutdown progress: 0% 
Nov 08, 2019 4:19:02 PM rsb.introspection.IntrospectionParticipantObserver destroyed
INFO: Deactivating introspection protocol handler since no more participants exist
Shutdown progress: 100% (successful)
Start run 2 with 40 participants...
Wait until pool is active...
Initiate Shutdown...
Shutdown progress: 0% 
Nov 08, 2019 4:19:05 PM rsb.introspection.IntrospectionParticipantObserver destroyed
INFO: Deactivating introspection protocol handler since no more participants exist
Shutdown progress: 100% (successful)
Start run 3 with 60 participants...
Wait until pool is active...
Initiate Shutdown...
Shutdown progress: 0% 
Nov 08, 2019 4:19:07 PM rsb.introspection.IntrospectionParticipantObserver destroyed
INFO: Deactivating introspection protocol handler since no more participants exist
Shutdown progress: 100% (successful)
Start run 4 with 80 participants...
Wait until pool is active...
Nov 08, 2019 4:19:07 PM rsb.transport.spread.SpreadWrapperImpl makeConnection
WARNING: reoccuring SpreadException during connect to daemon
spread.SpreadException: write(): java.net.SocketException: Broken pipe (Write failed)
	at spread.SpreadConnection.sendConnect(SpreadConnection.java:491)
	at spread.SpreadConnection.connect(SpreadConnection.java:897)
	at rsb.transport.spread.SpreadWrapperImpl.makeConnection(SpreadWrapperImpl.java:147)
	at rsb.transport.spread.SpreadWrapperImpl.activate(SpreadWrapperImpl.java:251)
	at rsb.transport.spread.SpreadReceiver$StateInactive.activate(SpreadReceiver.java:232)
	at rsb.transport.spread.SpreadReceiver.activate(SpreadReceiver.java:344)
	at rsb.transport.spread.SpreadInPushConnector.activate(SpreadInPushConnector.java:80)
	at rsb.eventprocessing.RouteConfiguratorUtility.activate(RouteConfiguratorUtility.java:90)
	at rsb.eventprocessing.DefaultPushInRouteConfigurator.activate(DefaultPushInRouteConfigurator.java:69)
	at rsb.Listener$StateInactive.activate(Listener.java:123)
	at rsb.Listener.activate(Listener.java:191)
	at rsb.patterns.Method$StateInactive.activate(Method.java:109)
	at rsb.patterns.Method.activate(Method.java:202)
	at rsb.patterns.LocalMethod.activate(LocalMethod.java:162)
	at rsb.patterns.Server$StateInactive.activate(Server.java:103)
	at rsb.patterns.Server.activate(Server.java:218)
	at org.openbase.jul.extension.rsb.com.test.DeactivationOnEventLoadTest$1.run(DeactivationOnEventLoadTest.java:72)

rsb.InitializeException: Could not create spread connection host=localhost, port=4803
	at rsb.transport.spread.SpreadWrapperImpl.makeConnection(SpreadWrapperImpl.java:160)
	at rsb.transport.spread.SpreadWrapperImpl.activate(SpreadWrapperImpl.java:251)
	at rsb.transport.spread.SpreadReceiver$StateInactive.activate(SpreadReceiver.java:232)
	at rsb.transport.spread.SpreadReceiver.activate(SpreadReceiver.java:344)
	at rsb.transport.spread.SpreadInPushConnector.activate(SpreadInPushConnector.java:80)
	at rsb.eventprocessing.RouteConfiguratorUtility.activate(RouteConfiguratorUtility.java:90)
	at rsb.eventprocessing.DefaultPushInRouteConfigurator.activate(DefaultPushInRouteConfigurator.java:69)
	at rsb.Listener$StateInactive.activate(Listener.java:123)
	at rsb.Listener.activate(Listener.java:191)
	at rsb.patterns.Method$StateInactive.activate(Method.java:109)
	at rsb.patterns.Method.activate(Method.java:202)
	at rsb.patterns.LocalMethod.activate(LocalMethod.java:162)
	at rsb.patterns.Server$StateInactive.activate(Server.java:103)
	at rsb.patterns.Server.activate(Server.java:218)
	at org.openbase.jul.extension.rsb.com.test.DeactivationOnEventLoadTest$1.run(DeactivationOnEventLoadTest.java:72)
Caused by: spread.SpreadException: write(): java.net.SocketException: Broken pipe (Write failed)
	at spread.SpreadConnection.sendConnect(SpreadConnection.java:491)
	at spread.SpreadConnection.connect(SpreadConnection.java:897)
	at rsb.transport.spread.SpreadWrapperImpl.makeConnection(SpreadWrapperImpl.java:147)
	... 14 more

This for example prevent apps to enable a proper reconnection of participants when the rsb host or port is changed.

[scymtym]

This looks like a limitation regarding open file descriptors or the TCP accept backlog imposed by either the Spread daemon or the system.

By the way, when I try to reproduce the issue, the socket transport is used. This reports an error as well, though for 260 participants:

Start run 13 with 260 participants...
rsb.RSBException: Unable to create client socket for rsb.transport.socket.SocketOptions[address=localhost/127.0.0.1, port=55555, tcpNoDelay=true]
	at rsb.transport.socket.BusClientConnection.activate(BusClientConnection.java:79)
	at rsb.transport.socket.BusClient.activate(BusClient.java:75)
	…

This is not surprising since on my system the limit for open file descriptors per process is 1024 and which gets exhausted around 256 client-server pairs with one method each.

[DivineThreepwood]

By the way, when I try to reproduce the issue, the socket transport is used.
Sorry, I forget to mention that the spread transport has to be set in the local rsb config.

I thought that might be related to issue
openbase/bco.bcomfy#55
Hmm maybe we just close the issue and I will reopen it or create another one when I can reproduce the Broken pipe issue with the spread transport enabled.