In Cloudflare Workers, we've observed applications in the wild that continue to send messages after a Close message. This is incorrect, but it happens, and we end up logging a spurious error as we attempt to proxy the message through.

By removing this restriction, we can now proxy these WebSockets despite the incorrect usage.

There is currently no way to explicitly omit a Content-Length/Transfer-Encoding header on an HTTP response to a HEAD request. This is awkward for a proxy, which would ideally pass along responses as-is, even if they have no such headers.

This change allows an author to pass zero as the expected body length to HttpService::Response::send() to mean "do not set any body header." This means that a proxy might strip Content-Length: 0 headers, but will never add a Content-Length header where there was none before.

Some protocols, like Visual Studio Code's Language Server Protocol, have made the unfortunate decision to use HTTP-style message envelope even though they are not HTTP protocols. LSP, for example, sends each JSON-RPC messages as a Content-Length header followed by two CRLF followed by a JSON message of that length. I didn't want to rewrite HTTP parsing, so I extended the HTTP library to make this functionality reusable.

To safely use HttpHeaders::serialize*() with overridden connection-level headers, dependent code must stay up-to-date with any changes in the builtin header list. We now expose CONNECTION_HEADERS_COUNT and its friends to facilitate this.

This should be treated as the other end disconnecting, not an error.

This is not related to tables but is needed for the Cloudflare Workers runtime change I'm working on that also uses tables... so I'm throwing it into the same PR.

Printf debugging at its finest.

Before this change, if *either* the `from` or the `to` end of the pump threw a DISCONNECTED exception, we'd then try to call `to.disconnect()`. But if the exception had been thrown from `to` in the first place, then this would throw again, this time complaining about a previous write not having completed. But the whole point was always to propagate errors from the *receiving* end to the sending end. An error on the sending end should just propagate up the stack.

The gather-write and tryPumpFrom() member functions weren't encoding their chunk sizes in hex, and the gather-write case had an off-by-n error building the body data pieces.

It turns out I was testing it wrong, and it still doesn't work. :(

Theory is that maybe it did this intentionally, e.g. because it really just wanted to disconnect and it already dealt with any errors. The client should notice the stream ending prematurely so it should be possible for the developer to see the problem from the client side.

In theory KJ style should allow this to be caught (it doesn't throw if another exception was already thrown), but I've observed this producing an std::terminate() for some reason.

It turns out wrapping an HttpService in an HttpClient is considerably more complicated than vice versa, due to the need for pipes. This commit adds a WebSocket pipe implementation very similar to the recent byte-stream pipe (though considerably simpler since there's no need to deal with mismatched buffer sizes).

This speeds up the test somewhat, but more importantly, it tests the pipe implementation across a variety of usage patterns.

This actually uncovered a bug in the HTTP implementation: An HttpClient could inadvertently issue overlapping reads in cases where multiple concurrent (pipelined) requests are made.

Previously, the app could control Content-Length by passing `expectedBodySize`. This is great for enabling code that "just works" by handling GET and HEAD requests identically.

However, in somewhat more-complicated situations -- especilaly in proxies -- you end up having to write special-case hacks for HEAD requests to deal with the fact that the body is actually empty, but has a non-zero "expected" size.

We can make life easier for proxies by allowing the application to directly set the Content-Length and Transfer-Encoding headers in the case of HEAD responses, much like we allow applications to set WebSocket-related headers on non-WebSocket requests/responses.

This change actually fixes a bug in Cloudflare Workers where Content-Length is not passed through correctly for HEAD responses. No changes are needed on the Workers side (except adding a test).

Primarily, we want to support proxies passing through such requests blindly. So, I didn't add any new API for this, just added some hacks so that things "just work" when proxying.

This allows an application which calls drain() to potentially pass off HTTP connections to a new HttpServer afterwards. Without this, the application has no way to know if the connections are in an indeterminant state.

This change also makes it OK for an application to fail to read the whole request body. Previously, if an app returned a response without reading the whole request, an exception would eventually be thrown, but potentially not until the client had initiated a new request on the same connection. The client would then get a spurious 500 error.

Previously HttpService had two virtual methods: request() and openWebSocket(). Since it's legitimate to respond to a WebSocket request with a normal HTTP response, openWebSocket() actually had a default implementation that fell back to request().

In the new design, there is only request(). The HttpService detects a WebSocket request by checking the headers. A convenience method, HttpHeaders::isWebSocket(), is provided for this purpose.

The new approach makes life much easier for services composed of many layers. For example, you might write an HttpService implementation which performs some URL or header rewrite and then calls on to another HttpService. Previously, every such wrapper would have to separately handle regular requests and WebSockets, usually with near-identical code. Of course, you could factor out the common code, but in practice this often turned out pretty clunky. Worse, developers would often just omit the openWebSocket() implementation since implementing only request() seems to work fine -- until you need a WebSocket, and everything is broken. With the new approach, you have to go somewhat out of your way to write a wrapper layer that breaks WebSockets.

I did not apply the same logic to HttpClient because:

1. It's not as easy: HttpClient's methods return results rather than calling a callback on completion, so unifying the methods would have forced request()'s signature to change. Lots of code would need to be updated, and would likely become uglier, as request() would now have to return a `webSocketOrBody` variant type even when the caller isn't asking for a WebSocket.
2. People don't implement custom HttpClients nearly as often as they implement custom HttpServices.

This is particularly useful for implementing logic to grab the client's IP address and shove it in X-Real-IP.

The headers for multiple responses could end up merged because the call to headers.clear() was happening too early. This manifested as `Content-Length: 7, 13` observed in the second pipelined response.

Although applications in theory shouldn't care to see connection-level headers (e.g. `Transfer-Encoding`), higher-level specs like the JavaScript Fetch API often specify that these headers should be visible, and they can be useful for debugging. So, this change makes it so that the application can see the connection-level headers on incoming requests.

For outgoing requests, the application can provide an HttpHeaders object that specifies these headers (important especially for the pass-through case), but the HTTP implementation will ignore them.

Additionally, we can now allow the application to set WebSocket connection-level headers on non-WebSocket requests. This is useful for frameworks that emulate WebSocket over HTTP and assume the ability to set WebSocket headers (especially `Sec-WebSocket-Extension`) on regular non-WebSocket HTTP requests.

This leaks due to the old compiler bug:

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=33799
http://llvm.org/bugs/show_bug.cgi?id=12286

Previously we only closed immediately after a response finished.

If recvData is empty, recvData.begin() is a null pointer and binding a Header reference to it is invalid.

Detected by -fsanitize=null.

(Or if it didn't send the entire request body.)