*1119809 upstream sent too big header while reading response header from upstream / 502 Bad Gateway

[georgewfisher]

I have deployed EasyAuthForK8s via helm with the following options:

helm install nginx-ingress ingress-nginx/ingress-nginx --set rbac.create=true --set controller.config.large-client-header-buffers="8 32k"

However, in the nginx logs I get this massive error after authentication, which results in the user seeing "502 Bad Gateway".

Previously I was using oauthproxy2 whose solution to this issue was to enable Redis caching. I know the client header buffer increase should take care of this issue - but it does not appear to be sufficient.

2022/09/08 15:52:33 [error] 5393#5393: *1119809 upstream sent too big header while reading response header from upstream, client: 10.244.2.1, server: redacted.hostname, request: "GET /login?next=https%3A%2F%2Fredacted.hostname%2F%255Clogin%3Fnext%3Dhttps%253A%252F%252Fredacted.hostname%252F%25255Clogin%253Fnext%253Dhttps%25253A%25252F%25252Fredacted.hostname%25252F%2525255Clogin%25253Fnext%25253Dhttps%2525253A%2525252F%2525252Fredacted.hostname%2525252F%252525255Clogin%2525253Fnext%2525253Dhttps%252525253A%252525252F%252525252Fredacted.hostname%252525252F%25252525255Clogin%252525253Fnext%252525253Dhttps%25252525253A%25252525252F%25252525252Fredacted.hostname%25252525252F%2525252525255Clogin%25252525253Fnext%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Fredacted.hostname%2525252525252F%252525252525255Clogin%2525252525253Fnext%2525252525253Dhttps%252525252525253A%252525252525252F%252525252525252Fredacted.hostname%252525252525252F%25252525252525255Clogin%252525252525253Fnext%252525252525253Dhttps%25252525252525253A%25252525252525252F%25252525252525252Fredacted.hostname%25252525252525252F%2525252525252525255Clogin%25252525252525253Fnext%25252525252525253Dhttps%2525252525252525253A%2525252525252525252F%2525252525252525252Fredacted.hostname%2525252525252525252F%252525252525252525255Clogin%2525252525252525253Fnext%2525252525252525253Dhttps%252525252525252525253A%252525252525252525252F%252525252525252525252Fredacted.hostname%252525252525252525252F%25252525252525252525255Clogin%252525252525252525253Fnext%252525252525252525253Dhttps%25252525252525252525253A%25252525252525252525252F%25252525252525252525252Fredacted.hostname%25252525252525252525252F%2525252525252525252525255Clogin%25252525252525252525253Fnext%25252525252525252525253Dhttps%2525252525252525252525253A%2525252525252525252525252F%2525252525252525252525252Fredacted.hostname%2525252525252525252525252F%252525252525252525252525255Clogin%2525252525252525252525253Fnext%2525252525252525252525253Dhttps%252525252525252525252525253A%252525252525252525252525252F%252525252525252525252525252Fredacted.hostname%252525252525252525252525252F%25252525252525252525252525255Clogin%252525252525252525252525253Fnext%252525252525252525252525253Dhttps%25252525252525252525252525253A%25252525252525252525252525252F%25252525252525252525252525252Fredacted.hostname%25252525252525252525252525252F%2525252525252525252525252525255Clogin%25252525252525252525252525253Fnext%25252525252525252525252525253Dhttps%2525252525252525252525252525253A%2525252525252525252525252525252F%2525252525252525252525252525252Fredacted.hostname%2525252525252525252525252525252F%252525252525252525252525252525255Cfavicon.ico HTTP/2.0", upstream: "http://10.244.0.33:5000/\login?next=https%3A%2F%2Fredacted.hostname%2F%255Clogin%3Fnext%3Dhttps%253A%252F%252Fredacted.hostname%252F%25255Clogin%253Fnext%253Dhttps%25253A%25252F%25252Fredacted.hostname%25252F%2525255Clogin%25253Fnext%25253Dhttps%2525253A%2525252F%2525252Fredacted.hostname%2525252F%252525255Clogin%2525253Fnext%2525253Dhttps%252525253A%252525252F%252525252Fredacted.hostname%252525252F%25252525255Clogin%252525253Fnext%252525253Dhttps%25252525253A%25252525252F%25252525252Fredacted.hostname%25252525252F%2525252525255Clogin%25252525253Fnext%25252525253Dhttps%2525252525253A%2525252525252F%2525252525252Fredacted.hostname%2525252525252F%252525252525255Clogin%2525252525253Fnext%2525252525253Dhttps%252525252525253A%252525252525252F%252525252525252Fredacted.hostname%252525252525252F%25252525252525255Clogin%252525252525253Fnext%252525252525253Dhttps%25252525252525253A%25252525252525252F%25252525252525252Fredacted.hostname%25252525252525252F%2525252525252525255Clogin%25252525252525253Fnext%25252525252525253Dhttps%2525252525252525253A%2525252525252525252F%2525252525252525252Fredacted.hostname%2525252525252525252F%252525252525252525255Clogin%2525252525252525253Fnext%2525252525252525253Dhttps%252525252525252525253A%

My questions:

• The cookie AzAD.EasyAuthForK8s is very long, is that likely the issue? (711 characters)
• Is there anything other than controller.config.large-client-header-buffers="8 32k" which will circumvent this issue?

[jonlester]

Although we try to keep the cookie as small as possible, its contents are non deterministic, so there’s always a chance that it will be larger than the headers allowed. Adjusting the the nginx header values can fix this. The values we provided were as high as we felt comfortable using as a default value given the attack vectors it could create. Are you using a custom domain? Is it possible you have other, very large, wildcard cookies that your client is also sending? It custom domain, is your AAD tenant also using that custom domain? Do you have multiple user accounts within that tenant that you have logged in simultaneously in the same browser session? (This seems to be common when someone is testing). I can’t currently remember if we added telemetry around the headers to try to provide more useful troubleshooting information, but I will look at that when I investigate. For now any additional details you can provide about your context will be helpful. If you don’t feel comfortable posting here you can email details to easyauthfork8s@microsoft.com. Please remove PII, customer data, and anything else proprietary before sending.

[jonlester]

Will investigate possible mitigations

[wedevops1]

One way to work around this is to increase the size of the proxy-buffer-size to something bigger (16k, 32k or 64k) so nginx can handle larger headers from the upstream.

To do so, verify if the deployment for the nginx-controller has a configmap specified in the - args

# kubectl get deployment/ingres-nginx/controller -o yaml

...

    spec:
      containers:
      - args:
        - /nginx-ingress-controller
        ...
        - --ingress-class=nginx
        - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller # <<<
        ...

If there is not --configmap=... in the -args : of your deployment, add it like the example above.
If the configmap doesn't exist in the namespace, you can create one. If it already exists, you can edit it.

apiVersion: v1
kind: ConfigMap
name: ingress-nginx-controller
metadata:
  annotations:
....
data:
  ...
  proxy-buffer-size: 64k # << increase the buffer size

After the creation or update of your configmap, you can do a rollout restart of your nginx-controller to make this configuration effective

kubectl rollout restart deployment/ingress-nginx-controller

[jonlester]

That is the known workaround, which I’m hesitant to implement as the default or recommend it since it creates a higher risk for DOS attacks. We may add a note to the documentation for this solution along with the drawbacks so that people can make an informed decision.