Client inside Drydock api hung up.

[sb464f]

While performing bootactions drydock_provisioner throws errors
DRYDOCK LOGS:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/usr/local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 383, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/lib/python3.5/http/client.py", line 1198, in getresponse
    response.begin()
  File "/usr/local/lib/python3.5/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.5/http/client.py", line 266, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/usr/local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.5/site-packages/urllib3/util/retry.py", line 357, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.5/site-packages/urllib3/packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/usr/local/lib/python3.5/site-packages/urllib3/connectionpool.py", line 383, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/lib/python3.5/http/client.py", line 1198, in getresponse
    response.begin()
  File "/usr/local/lib/python3.5/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.5/http/client.py", line 266, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/keystoneauth1/session.py", line 589, in _send_request
    resp = self.session.request(method, url, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.5/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/requests/adapters.py", line 490, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./drydock_provisioner/objects/bootaction.py", line 185, in resolve_asset_location
    return ReferenceResolver.resolve_reference(asset_url)
  File "./drydock_provisioner/statemgmt/design/resolver.py", line 49, in resolve_reference
    return handler.__get__(None, cls)(design_uri)
  File "./drydock_provisioner/statemgmt/design/resolver.py", line 103, in resolve_reference_ucp
    resp = ks_sess.get(url)
  File "/usr/local/lib/python3.5/site-packages/keystoneauth1/session.py", line 667, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/local/lib/python3.5/site-packages/positional/__init__.py", line 108, in inner
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/keystoneauth1/session.py", line 555, in request
    resp = send(**kwargs)
  File "/usr/local/lib/python3.5/site-packages/keystoneauth1/session.py", line 599, in _send_request
    raise exceptions.ConnectFailure(msg)
keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://promenade-api.ucp.svc.cluster.local/api/v1.0/join-scripts?design_ref=deckhand%2Bhttp%3A//deckhand-int.ucp.svc.cluster.local%3A9000/api/v1.0/revisions/1/rendered-documents&hostname=worker01&ip=10.23.21.13&labels.dynamic=foo=bar

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./drydock_provisioner/control/bootaction.py", line 183, in do_get
    type_filter=asset_type_filter))
  File "./drydock_provisioner/objects/bootaction.py", line 84, in render_assets
    a.render(nodename, site_design, action_id, design_ref)
  File "./drydock_provisioner/objects/bootaction.py", line 166, in render
    data_block = self.resolve_asset_location(rendered_location)
  File "./drydock_provisioner/objects/bootaction.py", line 189, in resolve_asset_location
    str(ex)))
drydock_provisioner.error.InvalidAssetLocation: Unable to resolve asset reference promenade+http://promenade-api.ucp.svc.cluster.local/api/v1.0/join-scripts?design_ref=deckhand%2Bhttp%3A//deckhand-int.ucp.svc.cluster.local%3A9000/api/v1.0/revisions/1/rendered-documents&hostname=worker01&ip=10.23.21.13&labels.dynamic=foo=bar: Unable to establish connection to http://promenade-api.ucp.svc.cluster.local/api/v1.0/join-scripts?design_ref=deckhand%2Bhttp%3A//deckhand-int.ucp.svc.cluster.local%3A9000/api/v1.0/revisions/1/rendered-documents&hostname=worker01&ip=10.23.21.13&labels.dynamic=foo=bar
2018-02-08 08:05:30,647 - INFO - None - 8ca1f440-9421-479d-851c-c571de917d8e -  - http://10.24.20.100:31000/api/v1.0/bootactions/nodes/worker01/files - 500 Internal Server Error

The logs from promenade returns 200 OK.

req-d0b3310b-403d-4d4a-82e7-c497256899b2 req-d0b3310b-403d-4d4a-82e7-c497256899b2 8de5576a1f1a48bb8d06ca9a740e3087 promenade:process_response [116] GET http://promenade-api.ucp.svc.cluster.local/api/v1.0/join-scripts?design_ref=deckhand%2Bhttp%3A//deckhand-int.ucp.svc.cluster.local%3A9000/api/v1.0/revisions/1/rendered-documents&hostname=worker01&ip=10.23.21.13&labels.dynamic=foo=bar - 200 OK
Thu Feb  8 08:09:20 2018 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /api/v1.0/join-scripts?design_ref=deckhand%2Bhttp%3A//deckhand-int.ucp.svc.cluster.local%3A9000/api/v1.0/revisions/1/rendered-documents&hostname=worker01&ip=10.23.21.13&labels.dynamic=foo=bar (ip 10.97.244.229) !!!
Thu Feb  8 08:09:20 2018 - uwsgi_response_write_body_do(): Broken pipe [core/writer.c line 331] during GET /api/v1.0/join-scripts?design_ref=deckhand%2Bhttp%3A//deckhand-int.ucp.svc.cluster.local%3A9000/api/v1.0/revisions/1/rendered-documents&hostname=worker01&ip=10.23.21.13&labels.dynamic=foo=bar (10.97.244.229)

We suspect that the client inside Drydock API which is making request to Promenade hung up