• Kenton Varda's avatar
    Implement byte streams over Cap'n Proto. · 63c34d47
    Kenton Varda authored
    This implementation features path-shortening through pumps. That is, if an incoming Cap'n Proto stream wraps a KJ stream which ends up pumping to an outgoing Cap'n Proto stream, the incoming stream will be redirected directly to the outgoing stream, in such a way that the RPC system can recognize and reduce the number of network hops as appropriate.
    
    This proved tricky due to the features of KJ's `pumpTo()`, in particular:
    - The caller of `pumpTo()` expects eventually to be told how many bytes were pumped (e.g. before EOF was hit).
    - A pump may have a specified length limit. In this case the rest of the stream can be pumped somewhere else.
    - Multiple streams can be pumped to the same destination stream -- this implies that a pump does not propagate EOF.
    
    These requirements mean that path-shortening is not as simple as redirecting the incoming stream to the outgoing. Intsead, we must first ask the outgoing stream's server to create a "substream" object, and then redirect to that. The substream can have a length limit, swallows EOF, informs the original creator on completion, and can even redirect *back* to the original creator to allow the stream to now pump somewhere else.
    63c34d47
byte-stream.capnp 1.77 KB