Hop-by-hop Header Vulnerability in Go Standard Library Reverse Proxy

In this blog post, we explore potential vulnerabilities with reverse proxies that use hop-by-hop headers. We first explain hop-by-hop headers and their distinction from end-to-end headers. After that, we introduce the Go standard library's httputil.ReverseProxy. We explain its flawed handling of hop-by-hop headers. Finally, we will look into how you can fix the vulnerability by looking at how we fixed Ory Oathkeeper, a cloud native identity & access proxy.

Overall, we believe that the impact of this vulnerability will not be huge in most cases, but it could potentially be exploited by an attacker.

What are Hop-by-Hop Headers

HTTP/1.1 distinguishes between so-called hop-by-hop and end-to-end headers. Per spec, hop-by-hop headers are only meant for the next hop on the request path, in a cloud-native stack often some kind of reverse proxy. By default all headers are considered end-to-end, except a list of standard headers that only really make sense on a hop-by-hop basis, like Transfer-Encoding.

However, a client can mark a header to be treated as hop-by-hop by adding the header name to the Connection header. For example, to treat the Cookie header as hop-by-hop, one can send this request:

curl -H 'Cookie: value' -H 'Connection: close,cookie' https://some.url

According to the spec, the server receiving this request should remove the Cookie header before forwarding the request.

Go Stdlib httputil.ReverseProxy

Now that we looked at the client side, let’s see how a standard reverse proxy handles the Connection header. In general, Go comes with a great standard library and is widely used. Because reverse proxies are such a common thing in a modern, cloud-native stack, the Go standard library ships a simple yet powerful reverse proxy (httputil.ReverseProxy).

Without going into too many confusing details on how to write your own reverse proxy, I want to focus on the request forwarding part of it. The httputil.ReverseProxy allows you to set a so-called Director hook. It is a function that takes the incoming *http.Request and may modify it to set the upstream host, rewrite the path, add headers, rewrite the body, … the sky is the limit.

httputil.ReverseProxy handles incoming requests in this order:

2. Incoming request is cloned and passed to the Director hook
3. Hop-by-hop headers are removed
4. Request is forwarded to the upstream host

Hop-by-Hop Headers and httputil.ReverseProxy

You might have already spotted that the hop-by-hop headers are only removed after the director potentially added new headers, so it is totally possible that these are removed again if they were specified in the Connection header.

This leaves us with an inherently flawed design for the reverse proxy in the standard library. Fortunately we don’t have to stop here and present some third-party library that fixes this issue, but instead it was identified and fixed as a potential vulnerability already by the Go team. The fix landed with Go 1.20. Because the previous design was flawed, they needed to add a new hook supersede the Director which is now called Rewrite (way better naming as well).

We don't know why Director was not marked as deprecated, nor why no CVE was assigned to this. The issue apparently flew under most user's radars, and has not received any comments on GitHub as of the writing of this article.

How to Fix the Vulnerability

Ory Oathkeeper, our reverse proxy, was also affected. We fixed the issue by migrating to the new Rewrite hook, and we also issued a security advisory on this topic. In general, you can quite easily migrate the Director to the Rewrite hook by replacing all usages of the *http.Request with the *httputil.ProxyRequest.Out. See the PR in Oathkeeper for more details and examples.

We believe that even if the vulnerability is exploitable, the impact will not be huge in most cases, but it could potentially. It only takes one developer who decides that a request without an X-Forwarded-For header means it is an internal request -- a reasonable assumption --, and allow such a request to do all kinds of privileged calls. Think for yourself how much you trust your reverse proxies and if you make such assumptions.