Document new features in 0.5: Generics and canonicalization.

doc/_posts/2014-06-17-capnproto-flatbuffers-sbe.md

@@ -4,7 +4,10 @@ title: Cap'n Proto, FlatBuffers, and SBE
 author: kentonv
 ---
 
-**Update:** I have made [some corrections](https://github.com/kentonv/capnproto/commit/e4e6c9076ae16804c07968cd3bdf6107155df7ee) since the original version of this post.
+**Update Jun 18, 2014:** I have made [some corrections](https://github.com/kentonv/capnproto/commit/e4e6c9076ae16804c07968cd3bdf6107155df7ee) since the original version of this post.
+
+**Update Dec 15, 2014:** Updated to reflect that Cap'n Proto 0.5 now supports Visual Studio and that
+Java is now well-supported.
 
 Yesterday, some engineers at Google released [FlatBuffers](http://google-opensource.blogspot.com/2014/06/flatbuffers-memory-efficient.html), a new serialization protocol and library with similar design principles to Cap'n Proto. Also, a few months back, Real Logic released [Simple Binary Encoding](http://mechanical-sympathy.blogspot.com/2014/05/simple-binary-encoding.html), another protocol and library of this nature.
 
@@ -37,13 +40,16 @@ Note: For features which are properties of the implementation rather than the pr
 <tr><td>Padding takes space on wire?</td><td class="pass">no</td><td class="warn">optional</td><td class="fail">yes</td><td class="fail">yes</td></tr>
 <tr><td>Unset fields take space on wire?</td><td class="pass">no</td><td class="fail">yes</td><td class="fail">yes</td><td class="pass">no</td></tr>
 <tr><td>Pointers take space on wire?</td><td class="pass">no</td><td class="fail">yes</td><td class="pass">no</td><td class="fail">yes</td></tr>
-<tr><td>C++</td><td class="pass">yes</td><td class="warn">GCC/Clang<br>(no MSVC)</td><td class="pass">yes</td><td class="pass">yes</td></tr>
-<tr><td>Java</td><td class="pass">yes</td><td class="warn">in progress</td><td class="pass">yes</td><td class="pass">yes</td></tr>
-<tr><td>Python</td><td class="pass">yes</td><td class="warn">yes (C ext)</td><td class="fail">no</td><td class="fail">no</td></tr>
-<tr><td>Other languages</td><td class="pass">lots!</td><td class="warn">6+ others</td><td class="warn">C#</td><td class="fail">no</td></tr>
+<tr><td>C++</td><td class="pass">yes</td><td class="pass">yes (C++11)*</td><td class="pass">yes</td><td class="pass">yes</td></tr>
+<tr><td>Java</td><td class="pass">yes</td><td class="pass">yes*</td><td class="pass">yes</td><td class="pass">yes</td></tr>
+<tr><td>C#</td><td class="pass">yes</td><td class="pass">yes*</td><td class="pass">yes</td><td class="pass">yes*</td></tr>
+<tr><td>Go</td><td class="pass">yes</td><td class="pass">yes</td><td class="fail">no</td><td class="pass">yes*</td></tr>
+<tr><td>Other languages</td><td class="pass">lots!</td><td class="warn">6+ others*</td><td class="fail">no</td><td class="fail">no</td></tr>
 <tr><td>Authors' preferred use case</td><td>distributed<br>computing</td><td><a href="https://sandstorm.io">platforms /<br>sandboxing</a></td><td>financial<br>trading</td><td>games</td></tr>
 </table>
 
+\* Updated Dec 15, 2014 (Cap'n Proto 0.5.0).
+
 **Schema Evolution**
 
 All four protocols allow you to add new fields to a schema over time, without breaking backwards-compatibility. New fields will be ignored by old binaries, and new binaries will fill in a default value when reading old data.
@@ -172,9 +178,19 @@ FlatBuffers also uses pointers, even though most objects are variable-width, pos
 
 **Platform Support**
 
-A really huge weakness of Cap'n Proto today is that it doesn't compile in Visual Studio, and therefore effectively doesn't support Windows (unless you count Cygwin, but most people don't).
+As of Dec 15, 2014, Cap'n Proto supports a superset of the languages supported by FlatBuffers and
+SBE, but is still far behind Protocol Buffers.
+
+While Cap'n Proto C++ is well-supported on POSIX platforms using GCC or Clang as their compiler,
+Cap'n Proto has only limited support for Visual C++: the basic serialization library works, but
+reflection and RPC do not yet work. Support will be expanded once Visual Studio's C++ compiler
+completes support for C++11.
+
+In comparison, SBE and FlatBuffers have reflection interfaces that work in Visual C++, though
+neither one has built-in RPC. Reflection is critical for certain use cases, but the majority of
+users won't need it.
 
-The problem initially was that Cap'n Proto makes liberal use of C++11 features, and MSVC has lagged behind in implementing them. We considered, but balked at, forking and backporting. It looks like [VS14 CTP1](http://blogs.msdn.com/b/vcblog/archive/2014/06/03/visual-studio-14-ctp.aspx) may finally be far enough to make a port practical, so now it's just a matter of finding the time. But that's a new problem: all my time is currently consumed by [Sandstorm.io](https://sandstorm.io), and while it is a major use of Cap'n Proto and will thus drive further development, it is entirely Linux-based, making it hard for me to justify spending my time on Windows support. What we need is a volunteer!
+(This section has been updated. When originally written, Cap'n Proto did not support MSVC at all.)
 
 ### Benchmarks?
 

doc/_posts/2014-12-15-capnproto-0.5-generics-msvc-java-csharp.md

+---
+layout: post
+title: "Cap'n Proto 0.5: Generics, Visual C++, Java, C#, Sandstorm.io"
+author: kentonv
+---
+
+Today we're releasing Cap'n Proto 0.5. We've added lots of goodies!
+
+### Finally: Visual Studio
+
+Microsoft Visual Studio 2015 (currently in "preview") finally supports enough C++11 to get Cap'n
+Proto working, and we've duly added official support for it!
+
+Not all features are support yet. The core serialization functionality sufficient for 90% of users
+is available, but reflection and RPC APIs are not. We will turn on these APIs as soon as Visual C++
+is ready (the main blocker is incomplete `constexpr` support).
+
+As part of this, we now support CMake as a build system, and it can be used on Unix as well.
+
+In related news, for Windows users not interested in C++ but who need the Cap'n Proto tools for
+other languages, we now provide precompiled Windows binaries. See
+[the installation page]({{site.baseurl}}install.html).
+
+I'd like to thank [Bryan Boreham](https://github.com/bboreham),
+[Joshua Warner](https://github.com/joshuawarner32), and [Phillip Quinn](https://github.com/pqu) for
+their help in getting this working.
+
+### C#, Java
+
+While not strictly part of this release, our two biggest missing languages recently gained support
+for Cap'n Proto:
+
+* [Marc Gravell](https://github.com/mgravell) -- the man responsible for the most popular C#
+  implementation of Protobufs -- has now implemented
+  [Cap'n Proto in C#](https://github.com/mgravell/capnproto-net).
+* [David Renshaw](https://github.com/dwrensha), author of our existing Rust implementation and
+  [Sandstorm.io](https://sandstorm.io) core developer, has implemented
+  [Cap'n Proto in Java](https://github.com/dwrensha/capnproto-java).
+
+### Generics
+
+Cap'n Proto now supports [generics](http://localhost:4000/capnproto/language.html#generic-types),
+in the sense of Java generics or C++ templates. While working on
+[Sandstorm.io](https://sandstorm.io) we frequently found that we wanted this, and it turned out
+to be easy to support.
+
+This is a feature which Protocol Buffers does not support and likely never will. Cap'n Proto has a
+much easier time supporting exotic language features because the generated code is so simple. In
+C++, nearly all Cap'n Proto generated code is inline accessor methods, which can easily become
+templates. Protocol Buffers, in contrast, has generated parse and serialize functions and a host
+of other auxiliary stuff, which is too complex to inline and thus would need to be adapted to
+generics without using C++ templates. This would get ugly fast.
+
+Generics are not yet supported by all Cap'n Proto language implementations, but where they are not
+supported, things degrade gracefully: all type parameters simply become `AnyPointer`. You can still
+use generics in your schemas as documentation. Meanwhile, at least our C++, Java, and Python
+implementations have already been updated to support generics, and other implementations that
+wrap the C++ reflection API are likely to work too.
+
+### Canonicalization
+
+0.5 introduces a (backwards-compatible) change in
+[the way struct lists should be encoded](http://localhost:4000/capnproto/encoding.html#lists), in
+order to support [canonicalization](http://localhost:4000/capnproto/encoding.html#canonicalization).
+We believe this will make Cap'n Proto more appropriate for use in cryptographic protocols. If
+you've implemented Cap'n Proto in another language, please update your code!
+
+### Sandstorm and Capability Systems
+
+[Sandstorm.io](https://sandstorm.io) is Cap'n Proto's parent project: a platform for personal
+servers that is radically easier and more secure.
+
+Cap'n Proto RPC is the underlying communications layer powering Sandstorm. Sandstorm is a
+[capability system](http://www.erights.org/elib/capability/overview.html): applications can send
+each other object references and address messages to those objects. Messages can themselves contain
+new object references, and the recipient implicitly gains permission to use any object reference
+they receive. Essentially, Sandstorm allows the interfaces between two apps, or between and app
+and the platform, to be designed using the same vocabulary as interfaces between objects or
+libraries in an object-oriented programming language (but
+[without the mistakes of CORBA or DCOM](http://localhost:4000/capnproto/rpc.html#distributed-objects)).
+Cap'n Proto RPC is at the core of this.
+
+This has powerful implications: Consider the case of service discovery. On Sandstorm, all
+applications start out isolated from each other in secure containers. However, applications can
+(or, will be able to) publish Cap'n Proto object references to the system representing APIs they
+support. Then, another app can make a request to the system, saying "I need an object that
+implements interface Foo". At this point, the system can display a picker UI to the user,
+presenting all objects the user owns that satisfy the requirement. However, the requesting app only
+ever receives a reference to the object the user chooses; all others remain hidden. Thus, security
+becomes "automatic". The user does not have to edit an ACL on the providing app, nor copy around
+credentials, nor even answer any security question at all; it all derives automatically and
+naturally from the user's choices. We call this interface "The Powerbox".
+
+Moreover, because Sandstorm is fully aware of the object references held by every app, it will
+be able to display a visualization of these connections, allowing a user to quickly see which of
+their apps have access to each other and even revoke connections that are no longer desired with
+a mouse click.
+
+Cap'n Proto 0.5 introduces primitives to support "persistent" capabilities -- that is, the ability
+to "save" an object reference to disk and then restore it later, on a different connection.
+Obviously, the features described above totally depend on this feature.
+
+The next release of Cap'n Proto is likely to include another feature essential for Sandstorm: the
+ability to pass capabilities from machine to machine and have Cap'n Proto automatically form direct
+connections when you do. This allows servers running on different machines to interact with each
+other in a completely object-oriented way. Instead of passing around URLs (which necessitate a
+global namespace, lifetime management, firewall traversal, and all sorts of other obstacles), you
+can pass around capabilities and not worry about it. This will be central to Sandstorm's strategies
+for federation and cluster management.
+
+### Other notes
+
+* The C++ RPC code now uses `epoll` on Linux.
+* We now test Cap'n Proto on Android and MinGW, in addition to Linux, Mac OSX, Cygwin, and Visual
+  Studio. (iOS and FreeBSD are also reported to work, though are not yet part of our testing
+  process.)

doc/cxx.md

@@ -371,6 +371,46 @@ implicitly convertible in this way.  Unfortunately, this trick doesn't work on G
 
 [Interfaces (RPC) have their own page.](cxxrpc.html)
 
+### Generics
+
+[Generic types](language.html#generic-types) become templates in C++. The outer type (the one whose
+name matches the schema declaration's name) is templatized; the inner `Reader` and `Builder` types
+are not, because they inherit the parameters from the outer type. Similarly, template parameters
+should refer to outer types, not `Reader` or `Builder` types.
+
+For example, given:
+
+{% highlight capnp %}
+struct Map(Key, Value) {
+  entries @0 :List(Entry);
+  struct Entry {
+    key @0 :Key;
+    value @1 :Value;
+  }
+}
+
+struct People {
+  byName @0 :Map(Text, Person);
+  # Maps names to Person instances.
+}
+{% endhighlight %}
+
+You might write code like:
+
+{% highlight c++ %}
+void processPeople(People::Reader people) {
+  Map<Text, Person>::Reader reader = people.getByName();
+  capnp::List<Map<Text, Person>::Entry>::Reader entries =
+      reader.getEntries()
+  for (auto entry: entries) {
+    processPerson(entry);
+  }
+}
+{% endhighlight %}
+
+Note that all template parameters will be specified with a default value of `AnyPointer`.
+Therefore, the type `Map<>` is equivalent to `Map<capnp::AnyPointer, capnp::AnyPointer>`.
+
 ### Constants
 
 Constants are exposed with their names converted to UPPERCASE_WITH_UNDERSCORES naming style

doc/cxxrpc.md

@@ -278,6 +278,9 @@ auto promise3 = promise2.then(
 });
 {% endhighlight %}
 
+For [generic methods](language.html#generic-methods), the `fooRequest()` method will be a template;
+you must explicitly specify type parameters.
+
 ### Servers
 
 The generated `Server` type is an abstract interface which may be subclassed to implement a
@@ -315,6 +318,11 @@ private:
 };
 {% endhighlight %}
 
+On the server side, [generic methods](language.html#generic-methods) are NOT templates. Instead,
+the generated code is exactly as if all of the generic parameters were bound to `AnyPointer`. The
+server generally does not get to know exactly what type the client requested; it must be designed
+to be correct for any parameterization.
+
 ## Initializing RPC
 
 Cap'n Proto makes it easy to start up an RPC client or server using the  "EZ RPC" classes,

doc/rpc.md

@@ -163,12 +163,31 @@ No!
 
 CORBA failed for many reasons, with the usual problems of design-by-committee being a big one.
 
-However, CORBA also had a critical technical flaw:  it did not implement promise pipelining.  As
-shown above, promise pipelining is absolutely critical to making object-oriented interfaces work
-in the presence of latency.  It is often said that object- and RPC-oriented protocols don't work
-because they try to pretend that a network call is equivalent to a local call.  In reality, this
-is not actually a problem with object protocols in general, but specifically CORBA and
-similarly-naive protocols that lack promise pipelining.  Promise pipelining is the missing link.
+However, the biggest reason for CORBA's failure is that it tried to make remote calls look the
+same as local calls. Cap'n Proto does NOT do this -- remote calls have a different kind of API
+involving promises, and accounts for the presence of a network introducing latency and
+unreliability.
+
+As shown above, promise pipelining is absolutely critical to making object-oriented interfaces work
+in the presence of latency. If remote calls look the same as local calls, there is no opportunity
+to introduce promise pipelining, and latency is inevitable. Any distributed object protocol which
+does not support promise pipelining cannot -- and should not -- succeed. Thus the failure of CORBA
+(and DCOM, etc.) was inevitable, but Cap'n Proto is different.
+
+### Handling disconnects
+
+Networks are unreliable. Occasionally, connections will be lost. When this happens, all
+capabilities (object references) served by the connection will become disconnected. Any further
+calls addressed to these capabilities will throw "disconnected" exceptions. When this happens, the
+client will need to create a new connection and try again. All Cap'n Proto applications with
+long-running connections (and probably short-running ones too) should be prepared to catch
+"disconnected" exceptions and respond appropriately.
+
+On the server side, when all references to an object have been "dropped" (either because the
+clients explicitly dropped them or because they became disconnected), the object will be closed
+(in C++, the destructor is called; in GC'd languages, a `close()` method is called). This allows
+servers to easily allocate per-client resources without having to clean up on a timeout or risk
+leaking memory.
 
 ### Security