rpc.md 12.7 KB
Newer Older
Kenton Varda's avatar
Kenton Varda committed
1 2
---
layout: page
Kenton Varda's avatar
Kenton Varda committed
3
title: RPC Protocol
Kenton Varda's avatar
Kenton Varda committed
4 5 6 7
---

# RPC Protocol

Kenton Varda's avatar
Kenton Varda committed
8 9 10 11 12 13 14
## Introduction

### Time Travel! _(Promise Pipelining)_

<img src='images/time-travel.png' style='max-width:639px'>

Cap'n Proto RPC employs TIME TRAVEL!  The results of an RPC call are returned to the client
Kenton Varda's avatar
Kenton Varda committed
15
instantly, before the server even receives the initial request!
Kenton Varda's avatar
Kenton Varda committed
16 17 18 19 20 21 22 23 24

There is, of course, a catch:  The results can only be used as part of a new request sent to the
same server.  If you want to use the results for anything else, you must wait.

This is useful, however:  Say that, as in the picture, you want to call `foo()`, then call `bar()`
on its result, i.e. `bar(foo())`.  Or -- as is very common in object-oriented programming -- you
want to call a method on the result of another call, i.e. `foo().bar()`.  With any traditional RPC
system, this will require two network round trips.  With Cap'n Proto, it takes only one.  In fact,
you can chain any number of such calls together -- with diamond dependencies and everything -- and
25
Cap'n Proto will collapse them all into one round trip.
Kenton Varda's avatar
Kenton Varda committed
26 27 28 29 30 31 32 33

By now you can probably imagine how it works:  if you execute `bar(foo())`, the client sends two
messages to the server, one saying "Please execute foo()", and a second saying "Please execute
bar() on the result of the first call".  These messages can be sent together -- there's no need
to wait for the first call to actually return.

To make programming to this model easy, in your code, each call returns a "promise".  Promises
work much like Javascript promises or promises/futures in other languages:  the promise is returned
Kenton Varda's avatar
Kenton Varda committed
34 35
immediately, but you must later call `wait()` on it, or call `then()` to register an asynchronous
callback.
Kenton Varda's avatar
Kenton Varda committed
36 37

However, Cap'n Proto promises support an additional feature:
Kenton Varda's avatar
Kenton Varda committed
38
[pipelining](http://www.erights.org/elib/distrib/pipeline.html).  The promise
Kenton Varda's avatar
Kenton Varda committed
39 40 41 42 43 44 45 46
actually has methods corresponding to whatever methods the final result would have, except that
these methods may only be used for the purpose of calling back to the server.  Moreover, a
pipelined promise can be used in the parameters to another call without waiting.

**_But isn't that just syntax sugar?_**

OK, fair enough.  In a traditional RPC system, we might solve our problem by introducing a new
method `foobar()` which combines `foo()` and `bar()`.  Now we've eliminated the round trip, without
Kenton Varda's avatar
Kenton Varda committed
47
inventing a whole new RPC protocol.
Kenton Varda's avatar
Kenton Varda committed
48

49 50
The problem is, this kind of arbitrary combining of orthogonal features quickly turns elegant
object-oriented protocols into ad-hoc messes.
Kenton Varda's avatar
Kenton Varda committed
51 52 53 54 55 56

For example, consider the following interface:

{% highlight capnp %}
# A happy, object-oriented interface!

Kenton Varda's avatar
Kenton Varda committed
57
interface Node {}
Kenton Varda's avatar
Kenton Varda committed
58

Kenton Varda's avatar
Kenton Varda committed
59
interface Directory extends(Node) {
Kenton Varda's avatar
Kenton Varda committed
60 61 62 63 64 65 66 67 68 69 70 71
  list @0 () -> (list: List(Entry));
  struct Entry {
    name @0 :Text;
    file @1 :Node;
  }

  create @1 (name :Text) -> (node :Node);
  open @2 (name :Text) -> (node :Node);
  delete @3 (name :Text);
  link @4 (name :Text, node :Node);
}

Kenton Varda's avatar
Kenton Varda committed
72
interface File extends(Node) {
Kenton Varda's avatar
Kenton Varda committed
73 74 75 76 77 78 79
  size @0 () -> (size: UInt64);
  read @1 (startAt :UInt64, amount :UInt64) -> (data: Data);
  write @2 (startAt :UInt64, data :Data);
  truncate @3 (size :UInt64);
}
{% endhighlight %}

80
This is a very clean interface for interacting with a file system.  But say you are using this
Kenton Varda's avatar
Kenton Varda committed
81 82 83 84 85
interface over a satellite link with 1000ms latency.  Now you have a problem:  simply reading the
file `foo` in directory `bar` takes four round trips!

{% highlight python %}
# pseudocode
86 87 88 89
bar = root.open("bar");    # 1
foo = bar.open("foo");     # 2
size = foo.size();         # 3
data = foo.read(0, size);  # 4
Kenton Varda's avatar
Kenton Varda committed
90 91
# The above is four calls but takes only one network
# round trip with Cap'n Proto!
Kenton Varda's avatar
Kenton Varda committed
92 93 94 95 96 97 98 99 100 101 102
{% endhighlight %}

In such a high-latency scenario, making your interface elegant is simply not worth 4x the latency.
So now you're going to change it.  You'll probably do something like:

* Introduce a notion of path strings, so that you can specify "foo/bar" rather than make two
  separate calls.
* Merge the `File` and `Directory` interfaces into a single `Filesystem` interface, where every
  call takes a path as an argument.

{% highlight capnp %}
103
# A sad, singleton-ish interface.
Kenton Varda's avatar
Kenton Varda committed
104 105 106 107 108 109 110 111 112

interface Filesystem {
  list @0 (path :Text) -> (list :List(Text));
  create @1 (path :Text, data :Data);
  delete @2 (path :Text);
  link @3 (path :Text, target :Text);

  fileSize @4 (path :Text) -> (size: UInt64);
  read @5 (path :Text, startAt :UInt64, amount :UInt64)
Kenton Varda's avatar
Kenton Varda committed
113
       -> (data :Data);
Kenton Varda's avatar
Kenton Varda committed
114 115 116 117 118 119 120
  readAll @6 (path :Text) -> (data: Data);
  write @7 (path :Text, startAt :UInt64, data :Data);
  truncate @8 (path :Text, size :UInt64);
}
{% endhighlight %}

We've now solved our latency problem...  but at what cost?
Jo Liss's avatar
Jo Liss committed
121

Kenton Varda's avatar
Kenton Varda committed
122 123 124 125 126 127
* We now have to implement path string manipulation, which is always a headache.
* If someone wants to perform multiple operations on a file or directory, we now either have to
  re-allocate resources for every call or we have to implement some sort of cache, which tends to
  be complicated and error-prone.
* We can no longer give someone a specific `File` or a `Directory` -- we have to give them a
  `Filesystem` and a path.
Kenton Varda's avatar
Kenton Varda committed
128 129 130 131
  * But what if they are buggy and have hard-coded some path other than the one we specified?
  * Or what if we don't trust them, and we really want them to access only one particular `File` or
    `Directory` and not have permission to anything else.  Now we have to implement authentication
    and authorization systems!  Arrgghh!
Kenton Varda's avatar
Kenton Varda committed
132 133 134 135 136 137 138 139 140 141 142

Essentially, in our quest to avoid latency, we've resorted to using a singleton-ish design, and
[singletons are evil](http://www.object-oriented-security.org/lets-argue/singletons).

**Promise Pipelining solves all of this!**

With pipelining, our 4-step example can be automatically reduced to a single round trip with no
need to change our interface at all.  We keep our simple, elegant, singleton-free interface, we
don't have to implement path strings, caching, authentication, or authorization, and yet everything
performs as well as we can possibly hope for.

Kenton Varda's avatar
Kenton Varda committed
143 144
#### Example code

145
[The calculator example](https://github.com/sandstorm-io/capnproto/blob/master/c++/samples/calculator-client.c++)
Kenton Varda's avatar
Kenton Varda committed
146 147
uses promise pipelining.  Take a look at the client side in particular.

Kenton Varda's avatar
Kenton Varda committed
148 149 150 151
### Distributed Objects

As you've noticed by now, Cap'n Proto RPC is a distributed object protocol.  Interface references --
or, as we more commonly call them, capabilities -- are a first-class type.  You can pass a
Kenton Varda's avatar
Kenton Varda committed
152 153 154 155 156 157 158
capability as a parameter to a method or embed it in a struct or list.  This is a huge difference
from many modern RPC-over-HTTP protocols that only let you address global URLs, or other RPC
systems like Protocol Buffers and Thrift that only let you address singleton objects exported at
startup.  The ability to dynamically introduce new objects and pass around references to them
allows you to use the same design patterns over the network that you use locally in object-oriented
programming languages.  Many kinds of interactions become vastly easier to express given the
richer vocabulary.
Kenton Varda's avatar
Kenton Varda committed
159 160 161 162 163 164 165

**_Didn't CORBA prove this doesn't work?_**

No!

CORBA failed for many reasons, with the usual problems of design-by-committee being a big one.

166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190
However, the biggest reason for CORBA's failure is that it tried to make remote calls look the
same as local calls. Cap'n Proto does NOT do this -- remote calls have a different kind of API
involving promises, and accounts for the presence of a network introducing latency and
unreliability.

As shown above, promise pipelining is absolutely critical to making object-oriented interfaces work
in the presence of latency. If remote calls look the same as local calls, there is no opportunity
to introduce promise pipelining, and latency is inevitable. Any distributed object protocol which
does not support promise pipelining cannot -- and should not -- succeed. Thus the failure of CORBA
(and DCOM, etc.) was inevitable, but Cap'n Proto is different.

### Handling disconnects

Networks are unreliable. Occasionally, connections will be lost. When this happens, all
capabilities (object references) served by the connection will become disconnected. Any further
calls addressed to these capabilities will throw "disconnected" exceptions. When this happens, the
client will need to create a new connection and try again. All Cap'n Proto applications with
long-running connections (and probably short-running ones too) should be prepared to catch
"disconnected" exceptions and respond appropriately.

On the server side, when all references to an object have been "dropped" (either because the
clients explicitly dropped them or because they became disconnected), the object will be closed
(in C++, the destructor is called; in GC'd languages, a `close()` method is called). This allows
servers to easily allocate per-client resources without having to clean up on a timeout or risk
leaking memory.
Kenton Varda's avatar
Kenton Varda committed
191 192 193

### Security

Kenton Varda's avatar
Kenton Varda committed
194
Cap'n Proto interface references are
Kenton Varda's avatar
Kenton Varda committed
195 196 197 198 199 200 201 202
[capabilities](http://en.wikipedia.org/wiki/Capability-based_security).  That is, they both
designate an object to call and confer permission to call it.  When a new object is created, only
the creator is initially able to call it.  When the object is passed over a network connection,
the receiver gains permission to make calls -- but no one else does.  In fact, it is impossible
for others to access the capability without consent of either the host or the receiver because
the host only assigns it an ID specific to the connection over which it was sent.

Capability-based design patterns -- which largely boil down to object-oriented design patterns --
203 204 205 206
work great with Cap'n Proto.  Such patterns tend to be much more adaptable than traditional
ACL-based security, making it easy to keep security tight and avoid confused-deputy attacks while
minimizing pain for legitimate users.  That said, you can of course implement ACLs or any other
pattern on top of capabilities.
Kenton Varda's avatar
Kenton Varda committed
207

Kenton Varda's avatar
Kenton Varda committed
208 209 210
For an extended discussion of what capabilities are and why they are often easier and more powerful
than ACLs, see Mark Miller's
["An Ode to the Granovetter Diagram"](http://www.erights.org/elib/capability/ode/index.html) and
211
[Capability Myths Demolished](http://zesty.ca/capmyths/usenix.pdf).
Kenton Varda's avatar
Kenton Varda committed
212

Kenton Varda's avatar
Kenton Varda committed
213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234
## Protocol Features

Cap'n Proto's RPC protocol has the following notable features.  Since the protocol is complicated,
the feature set has been divided into numbered "levels", so that implementations may declare which
features they have covered by advertising a level number.

* **Level 1:**  Object references and promise pipelining, as described above.
* **Level 2:**  Persistent capabilities.  You may request to "save" a capability, receiving a
  persistent token which can be used to "restore" it in the future (on a new connection).  Not
  all capabilities can be saved; the host app must implement support for it.  Building this into
  the protocol makes it possible for a Cap'n-Proto-based data store to transparently save
  structures containing capabilities without knowledge of the particular capability types or the
  application built on them, as well as potentially enabling more powerful analysis and
  visualization of stored data.
* **Level 3:**  Three-way interactions.  A network of Cap'n Proto vats (nodes) can pass object
  references to each other and automatically form direct connections as needed.  For instance, if
  Alice (on machine A) sends Bob (on machine B) a reference to Carol (on machine C), then machine B
  will form a new connection to machine C so that Bob can call Carol directly without proxying
  through machine A.
* **Level 4:**  Reference equality / joining.  If you receive a set of capabilities from different
  parties which should all point to the same underlying objects, you can verify securely that they
  in fact do.  This is subtle, but enables many security patterns that rely on one party being able
Kenton Varda's avatar
Kenton Varda committed
235
  to verify that two or more other parties agree on something (imagine a digital escrow agent).
Kenton Varda's avatar
Kenton Varda committed
236 237
  See [E's page on equality](http://erights.org/elib/equality/index.html).

Kenton Varda's avatar
Kenton Varda committed
238 239 240 241 242
## Encryption

At this time, Cap'n Proto does not specify an encryption scheme, but as it is a simple byte
stream protocol, it can easily be layered on top of SSL/TLS or other such protocols.

Kenton Varda's avatar
Kenton Varda committed
243 244 245 246
## Specification

The Cap'n Proto RPC protocol is defined in terms of Cap'n Proto serialization schemas.  The
documentation is inline.  See
247
[rpc.capnp](https://github.com/sandstorm-io/capnproto/blob/master/c++/src/capnp/rpc.capnp).
Kenton Varda's avatar
Kenton Varda committed
248 249 250 251 252 253 254 255 256

Cap'n Proto's RPC protocol is based heavily on
[CapTP](http://www.erights.org/elib/distrib/captp/index.html), the distributed capability protocol
used by the [E programming language](http://www.erights.org/index.html).  Lots of useful material
for understanding capabilities can be found at those links.

The protocol is complex, but the functionality it supports is conceptually simple.  Just as TCP
is a complex protocol that implements the simple concept of a byte stream, Cap'n Proto RPC is a
complex protocol that implements the simple concept of objects with callable methods.