1. 26 Jun, 2019 1 commit
  2. 22 Jun, 2019 1 commit
  3. 21 Jun, 2019 4 commits
  4. 20 Jun, 2019 2 commits
  5. 19 Jun, 2019 3 commits
  6. 18 Jun, 2019 20 commits
    • Kenton Varda's avatar
      Disallow initializing `Maybe<T>` from `T*`. · cded1f89
      Kenton Varda authored
      I think I imagined once upon a time that this would be a convenient way to deal with external interfaces that like to return nullable pointers. However, in practice it is used nowhere in KJ or Cap'n Proto, and it recently hid a bug in my code where I had assigned a `Maybe<T>` from an `Own<T>`. We can introduce a `fromNullablePointer()` helper or something if that turns out to be useful.
      cded1f89
    • Kenton Varda's avatar
      Really fix -Wall build. · f94b1a6f
      Kenton Varda authored
      f94b1a6f
    • Kenton Varda's avatar
      e2094eed
    • Kenton Varda's avatar
    • Kenton Varda's avatar
    • Kenton Varda's avatar
      Fix C++14 build. · ca0e85ce
      Kenton Varda authored
      ca0e85ce
    • Kenton Varda's avatar
      3ef62d15
    • Kenton Varda's avatar
      Fix typos. · 90667fbd
      Kenton Varda authored
      90667fbd
    • Kenton Varda's avatar
      CapabilityServerSet::getLocalServer() must wait for stream queue. · 4a4fe65c
      Kenton Varda authored
      Consider a capnp streaming type that wraps a kj::AsyncOutputStream.
      
      KJ streams require the caller to avoid doing multiple writes at once. Capnp streaming conveniently guarantees only one streaming call will be delivered at a time. This is great because it means the app does not have to do its own queuing of writes.
      
      However, the app may want to use a CapabilityServerSet to unwrap the capability and get at the underlying KJ stream to optimize by writing to it directly. However, before it can issue a direct write, it has to wait for all RPC writes to complete. These RPC writes were probably issued by the same caller, before it realized it was talking to a local cap. Unfortunately, it can't just wait for those calls it issued to complete, because streaming flow control may have made them appear to complete long ago, when they're actually still in the server's queue. How does the app make sure that the directly-issued writes don't overlap with RPC writes?
      
      We can solve this by making CapabilityServerSet::getLocalServer() delay until all in-flight stream calls are complete before unwrapping.
      
      Now, the app can simply make sure that any requests it issued over RPC in the past completed before it starts issuing direct requests.
      4a4fe65c
    • Kenton Varda's avatar
      Implement client RPC side of streaming. · 7a0e0fd0
      Kenton Varda authored
      7a0e0fd0
    • Kenton Varda's avatar
      Add client-side streaming hooks. · 56493100
      Kenton Varda authored
      Also, push harder on the code generator such that `StreamResult` doesn't show up in generated code at all.
      
      So now we have `StreamingRequest<Params>` which is like `Request<Params, Results>`, and we have `StreamingCallContext<Params>` which is like `CallContext<Params, Results>`.
      56493100
    • Kenton Varda's avatar
      Update bootstraps for previous commit. · 34481c85
      Kenton Varda authored
      34481c85
    • Kenton Varda's avatar
      Implement server side of streaming. · c3cfe9e5
      Kenton Varda authored
      There are two things that every capability server must implement:
      
      * When a streaming method is delivered, it blocks subsequent calls on the same capability. Although not strictly needed to achieve flow control, this simplifies the implementation of streaming servers -- many would otherwise need to implement such serialization manually.
      * When a streaming method throws, all subsequent calls also throw the same exception. This is important because exceptions thrown by a streaming call might not actually be delivered to a client, since the client doesn't necessarily wait for the results before making the next call. Again, a streaming server could implement this manually, but almost all streaming servers will likely need it, and this makes things easier.
      c3cfe9e5
    • Kenton Varda's avatar
      Regenerate bootstraps for streaming. · a784f2f7
      Kenton Varda authored
      Note: Apparently, json.capnp had not been added to the bootstrap test, and the checked-in bootstrap had drifted from the source file.
      a784f2f7
    • Kenton Varda's avatar
      Introduce new 'stream' keyword. · bd6d75ba
      Kenton Varda authored
      This can be used on a method to indicate that it is used for "streaming", like:
      
          write @0 (bytes :Data) -> stream;
      
      A "streaming" method is one which is expected to be called many times to transmit an ordered stream of items. For best throughput, it is often necessary to make multiple overlapping calls, so as not to wait for a round trip for every item. However, to avoid excess buffering, it may be necessary to apply backpressure by having the client limit the total number of overlapping calls. This logic is difficult to get right at the application level, so making it a language feature gives us the opportunity to implement it in the RPC layer.
      
      We can, however, do it in a way that is backwards-compatible with implementations that don't support it. The above declaration is equivalent to:
      
          write @0 (bytes :Data) -> import "/capnp/stream.capnp".StreamResult;
      
      RPC implementations that don't explicitly support streaming can thus instead leave it up to the application to handle.
      bd6d75ba
    • Kenton Varda's avatar
      Add Maybe<Own<T>>::emplace() to avoid assign-then-assert-nonnull. · 2ae7ca9b
      Kenton Varda authored
      I have this pattern:
      
          Maybe<Own<T>> foo;
      
          // ...
      
          foo = heap<T>();
          KJ_ASSERT_NONNULL(foo)->doSomething();
      
      The assertion feels non-type-safe.
      
      Now you can do:
      
          auto& ref = foo.emplace(heap<T>());
          ref.doSomething();
      2ae7ca9b
    • Kenton Varda's avatar
      Allow `kj::Absolute<T>` to be initialized to `kj::maxValue`. · 0bf96571
      Kenton Varda authored
      `kj::Quantity<T>` already supported this. I copied from it.
      0bf96571
    • Kenton Varda's avatar
      Fix `newAdaptedPromise<Promise<T>, Adapter>()`. · c786c444
      Kenton Varda authored
      This was failing to chain the promises, and so returning `Promise<Promise<T>>`.
      
      The idea here is you can create a PromiseAdapter which eventually produces another promise to chain to. The adapter is finished and should be destroyed at that point, but the final promise should then redirect to the new promise.
      c786c444
    • Kenton Varda's avatar
      Fix estimation of Return message sizes. · c826a71a
      Kenton Varda authored
      Apparently, Return messages with empty capability tables have been allocated one word too small all along, causing many Return messages to be split into two segments and allocate twice the memory they need. I never bothered to check whether this was happening...
      c826a71a
    • Kenton Varda's avatar
      Fix another AsyncPipe bug. · 889d7583
      Kenton Varda authored
      889d7583
  7. 17 Jun, 2019 7 commits
  8. 16 Jun, 2019 2 commits
    • Kenton Varda's avatar
      Calculate SO_VERSION in configure for compatibility with BSD make. · 8fdcad7a
      Kenton Varda authored
      BSD Make does not support `$(shell ...)`. It does support an alternative, `!=` assignments (which, confusingly, don't mean "not equal" but rather "evaluate the right in the shell before assignment"). GNU Make also supports `!=` as of version 4.0, released in 2013. Unfortunately, f***ing Apple ships GNU Make version 3.81, from 2006, with MacOS/XCode.
      8fdcad7a
    • Kenton Varda's avatar
      Add cheaper way to check size of RPC messages for flow control. · 76e35a7c
      Kenton Varda authored
      Way back in 538a767e I added `RpcSystem::setFlowLimit()`, a blunt mechanism by which an RPC node can arrange to stop reading new messages from the connection when too many incoming calls are in-flight. This was needed to deal with buggy Sandstorm apps that would stream multi-gigabyte files by doing a zillion writes without waiting, which would then all be queued in the HTTP gateway, causing it to run out of memory.
      
      In implementing that, I inadertently caused the RPC system to do a tree walk on every call message it received, in order to sum up the message size. This is silly, becaues it's much cheaper to sum up the segment sizes. In fact, in the case of a malicious peer, the tree walk is potentially insufficient, because it doesn't count holes in the segments. The tree walk also means that any invalid pointers in the message cause an exception to be thrown even if that pointer is never accessed by the app, which isn't the usual behavior.
      
      I seem to recall this issue coming up in discussion once in the past, but I couldn't find the thread.
      
      For the new streaming feature, we'll be paying attention to the size of outgoing messages. Again, here, it would be nice to compute this size by summing segments without doing a tree walk.
      
      So, this commit adds `sizeInWords()` methods that do this.
      76e35a7c