When no more jobs are idle on a particular group, the decision engine stops renewing the credentials #446
Description
The following issue has been known for quite some time, but now with scitokens that last just a couple hours it has become critical.
When no jobs are idle for a particular group in the decision engine, then it no longer advertises any glideclient classads
to the factory. Since glideclient classads are the means by which credentials are passed to the factory, it means
that the factory no longer has updated credentials to contact the remote resource, and therefore loses contact with any
glideins that may be still active.
This is also the case when the frontend is reconfigured but there are still glideins from the previous config that are active.. credentials don't get renewed for those either.
This is now a serious operational issue, we need to do something.
This was observed in decision engine 1.7.4 but it affects all versions because the behavior is in glideinwms/glide_frontend_element.py and the particular advertising logic of that routine hasn't changed since the DE was written.