Phabricator fails to reply for events with "max number of clients reached" #95
Description
As it can be seen in this case https://phabricator.services.mozilla.com/D167669 phab fails to get notified when the build status get updated.
The occurrence of this issue is about 2-3 times per 48hours.
This is due too many clients are connecting at once and we get a "max number of clients reached" in the connection to phab that we use. Retrying this for several times might prove to be useful but in the end it's a problem on the phab end and it becomes confusing and progression blocker for the engineers.
I see two solutions here:
- increase from the phab server the number of the accepted simultaneous users
- do a retry mechanism and retry in an exponential time, but this will translate that for each phab interrogation we should do this and phab api is not encapsulated in order to make things easier.