More native code gen functionality.

Allow tables to be mapped to native types directly.  For example, a table
representing a vector3 (eg. table Vec3 { x:float; y:float; z:float; }) can
be mapped to a "mathfu::vec3" native type in NativeTables.  This requires
users to provide Pack and UnPack functions that convert between the
Table and native types.  This is done by adding the "native_type" attribute
to the table definition.

To support user-defined flatbuffers::Pack and flatbuffers::UnPack functions,
support a "native_include" markup that will generate a corresponding

Also add an UnPackTo function which allows users to pass in a pointer to
a NativeTable object into which to UnPack the Table.  The existing UnPack
function is now simply:

  NativeTable* UnPack() {
    NativeTable* obj = new NativeTable();
    Table::UnPackTo(obj);
    return obj;
  }

Finally, allow native types to be given a default value as well which are
set in the NativeTable constructor.  This is done by providing a
"native_default" attribute to the member of a table.

Change-Id: Ic45cb48b0e6d7cfa5734b24819e54aa96d847cfd

docs/source/CppUsage.md

@@ -85,7 +85,7 @@ convenient accessors for all fields, e.g. `hp()`, `mana()`, etc:
 
 *Note: That we never stored a `mana` value, so it will return the default.*
 
-## Object based API.
+## Object based API.  {#flatbuffers_cpp_object_based_api}
 
 FlatBuffers is all about memory efficiency, which is why its base API is written
 around using as little as possible of it. This does make the API clumsier
@@ -99,13 +99,79 @@ construction, access and mutation.
 To use:
 
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.cpp}
-    auto monsterobj = UnpackMonster(buffer);
+    // Autogenerated class from table Monster.
+    MonsterT monsterobj;
+
+    // Deserialize from buffer into object.
+    UnPackTo(&monsterobj, flatbuffer);
+
+    // Update object directly like a C++ class instance.
     cout << monsterobj->name;  // This is now a std::string!
     monsterobj->name = "Bob";  // Change the name.
+
+    // Serialize into new flatbuffer.
     FlatBufferBuilder fbb;
-    CreateMonster(fbb, monsterobj.get());     // Serialize into new buffer.
+    Pack(fbb, &monsterobj);
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
+The following attributes are specific to the object-based API code generation:
+
+-   `native_inline` (on a field): Because FlatBuffer tables and structs are
+    optionally present in a given buffer, they are best represented as pointers
+    (specifically std::unique_ptrs) in the native class since they can be null.
+    This attribute changes the member declaration to use the type directly
+    rather than wrapped in a unique_ptr.
+
+-   `native_default`: "value" (on a field): For members that are declared
+    "native_inline", the value specified with this attribute will be included
+    verbatim in the class constructor initializer list for this member.
+
+-   `native_type`' "type" (on a struct): In some cases, a more optimal C++ data
+    type exists for a given struct.  For example, the following schema:
+
+      struct Vec2 {
+        x: float;
+        y: float;
+      }
+
+    generates the following Object-Based API class:
+
+      struct Vec2T : flatbuffers::NativeTable {
+        float x;
+        float y;
+      };
+
+    However, it can be useful to instead use a user-defined C++ type since it
+    can provide more functionality, eg.
+
+      struct vector2 {
+        float x = 0, y = 0;
+        vector2 operator+(vector2 rhs) const { ... }
+        vector2 operator-(vector2 rhs) const { ... }
+        float length() const { ... }
+        // etc.
+      };
+
+    The `native_type` attribute will replace the usage of the generated class
+    with the given type.  So, continuing with the example, the generated
+    code would use |vector2| in place of |Vec2T| for all generated code.
+
+    However, becuase the native_type is unknown to flatbuffers, the user must
+    provide the following functions to aide in the serialization process:
+
+      namespace flatbuffers {
+        FlatbufferStruct Pack(const native_type& obj);
+        native_type UnPack(const FlatbufferStruct& obj);
+      }
+
+Finally, the following top-level attribute
+
+-   native_include: "path" (at file level): Because the `native_type` attribute
+    can be used to introduce types that are unknown to flatbuffers, it may be
+    necessary to include "external" header files in the generated code.  This
+    attribute can be used to directly add an #include directive to the top of
+    the generated code that includes the specified path directly.
+
 # External references.
 
 An additional feature of the object API is the ability to allow you to load

docs/source/Schemas.md

@@ -309,6 +309,10 @@ Current understood attributes:
     to be stored in any particular order, they are often optimized for
     space by sorting them to size. This attribute stops that from happening.
     There should generally not be any reason to use this flag.
+-   'native_*'.  Several attributes have been added to support the [C++ object
+    Based API](@ref flatbuffers_cpp_object_based_api).  All such attributes
+    are prefixed with the term "native_".
+
 
 ## JSON Parsing
 

include/flatbuffers/idl.h

@@ -459,6 +459,8 @@ class Parser : public ParserState {
     known_attributes_["cpp_type"] = true;
     known_attributes_["cpp_ptr_type"] = true;
     known_attributes_["native_inline"] = true;
+    known_attributes_["native_type"] = true;
+    known_attributes_["native_default"] = true;
   }
 
   ~Parser() {
@@ -573,6 +575,7 @@ private:
 
   std::map<std::string, bool> included_files_;
   std::map<std::string, std::set<std::string>> files_included_per_file_;
+  std::vector<std::string> native_included_files_;
 
   std::map<std::string, bool> known_attributes_;
 

samples/monster_generated.h

@@ -241,7 +241,8 @@ struct Monster FLATBUFFERS_FINAL_CLASS : private flatbuffers::Table {
            VerifyEquipment(verifier, equipped(), equipped_type()) &&
            verifier.EndTable();
   }
-  MonsterT *UnPack(const flatbuffers::resolver_function_t *resolver = nullptr) const;
+  MonsterT *UnPack(const flatbuffers::resolver_function_t *_resolver = nullptr) const;
+  void UnPackTo(MonsterT *_o, const flatbuffers::resolver_function_t *_resolver = nullptr) const;
   static flatbuffers::Offset<Monster> Pack(flatbuffers::FlatBufferBuilder &_fbb, const MonsterT* _o, const flatbuffers::rehasher_function_t *_rehasher = nullptr);
 };
 
@@ -371,7 +372,8 @@ struct Weapon FLATBUFFERS_FINAL_CLASS : private flatbuffers::Table {
            VerifyField<int16_t>(verifier, VT_DAMAGE) &&
            verifier.EndTable();
   }
-  WeaponT *UnPack(const flatbuffers::resolver_function_t *resolver = nullptr) const;
+  WeaponT *UnPack(const flatbuffers::resolver_function_t *_resolver = nullptr) const;
+  void UnPackTo(WeaponT *_o, const flatbuffers::resolver_function_t *_resolver = nullptr) const;
   static flatbuffers::Offset<Weapon> Pack(flatbuffers::FlatBufferBuilder &_fbb, const WeaponT* _o, const flatbuffers::rehasher_function_t *_rehasher = nullptr);
 };
 
@@ -418,19 +420,24 @@ inline flatbuffers::Offset<Weapon> CreateWeaponDirect(
 
 flatbuffers::Offset<Weapon> CreateWeapon(flatbuffers::FlatBufferBuilder &_fbb, const WeaponT *_o, const flatbuffers::rehasher_function_t *_rehasher = nullptr);
 
-inline MonsterT *Monster::UnPack(const flatbuffers::resolver_function_t *resolver) const {
-  (void)resolver;
+inline MonsterT *Monster::UnPack(const flatbuffers::resolver_function_t *_resolver) const {
   auto _o = new MonsterT();
+  UnPackTo(_o, _resolver);
+  return _o;
+}
+
+inline void Monster::UnPackTo(MonsterT *_o, const flatbuffers::resolver_function_t *_resolver) const {
+  (void)_o;
+  (void)_resolver;
   { auto _e = pos(); if (_e) _o->pos = std::unique_ptr<Vec3>(new Vec3(*_e)); };
   { auto _e = mana(); _o->mana = _e; };
   { auto _e = hp(); _o->hp = _e; };
   { auto _e = name(); if (_e) _o->name = _e->str(); };
   { auto _e = inventory(); if (_e) for (flatbuffers::uoffset_t _i = 0; _i < _e->size(); _i++) { _o->inventory.push_back(_e->Get(_i)); } };
   { auto _e = color(); _o->color = _e; };
-  { auto _e = weapons(); if (_e) for (flatbuffers::uoffset_t _i = 0; _i < _e->size(); _i++) { _o->weapons.push_back(std::unique_ptr<WeaponT>(_e->Get(_i)->UnPack(resolver))); } };
+  { auto _e = weapons(); if (_e) for (flatbuffers::uoffset_t _i = 0; _i < _e->size(); _i++) { _o->weapons.push_back(std::unique_ptr<WeaponT>(_e->Get(_i)->UnPack(_resolver))); } };
   { auto _e = equipped_type(); _o->equipped.type = _e; };
-  { auto _e = equipped(); if (_e) _o->equipped.table = EquipmentUnion::UnPack(_e, equipped_type(),resolver); };
-  return _o;
+  { auto _e = equipped(); if (_e) _o->equipped.table = EquipmentUnion::UnPack(_e, equipped_type(),_resolver); };
 }
 
 inline flatbuffers::Offset<Monster> Monster::Pack(flatbuffers::FlatBufferBuilder &_fbb, const MonsterT* _o, const flatbuffers::rehasher_function_t *_rehasher) {
@@ -440,25 +447,39 @@ inline flatbuffers::Offset<Monster> Monster::Pack(flatbuffers::FlatBufferBuilder
 inline flatbuffers::Offset<Monster> CreateMonster(flatbuffers::FlatBufferBuilder &_fbb, const MonsterT *_o, const flatbuffers::rehasher_function_t *_rehasher) {
   (void)_rehasher;
   (void)_o;
+  auto _pos = _o->pos ? _o->pos.get() : 0;
+  auto _mana = _o->mana;
+  auto _hp = _o->hp;
+  auto _name = _o->name.size() ? _fbb.CreateString(_o->name) : 0;
+  auto _inventory = _o->inventory.size() ? _fbb.CreateVector(_o->inventory) : 0;
+  auto _color = _o->color;
+  auto _weapons = _o->weapons.size() ? _fbb.CreateVector<flatbuffers::Offset<Weapon>>(_o->weapons.size(), [&](size_t i) { return CreateWeapon(_fbb, _o->weapons[i].get(), _rehasher); }) : 0;
+  auto _equipped_type = _o->equipped.type;
+  auto _equipped = _o->equipped.Pack(_fbb);
   return CreateMonster(
       _fbb,
-      _o->pos ? _o->pos.get() : 0,
-      _o->mana,
-      _o->hp,
-      _o->name.size() ? _fbb.CreateString(_o->name) : 0,
-      _o->inventory.size() ? _fbb.CreateVector(_o->inventory) : 0,
-      _o->color,
-      _o->weapons.size() ? _fbb.CreateVector<flatbuffers::Offset<Weapon>>(_o->weapons.size(), [&](size_t i) { return CreateWeapon(_fbb, _o->weapons[i].get(), _rehasher); }) : 0,
-      _o->equipped.type,
-      _o->equipped.Pack(_fbb));
+      _pos,
+      _mana,
+      _hp,
+      _name,
+      _inventory,
+      _color,
+      _weapons,
+      _equipped_type,
+      _equipped);
 }
 
-inline WeaponT *Weapon::UnPack(const flatbuffers::resolver_function_t *resolver) const {
-  (void)resolver;
+inline WeaponT *Weapon::UnPack(const flatbuffers::resolver_function_t *_resolver) const {
   auto _o = new WeaponT();
+  UnPackTo(_o, _resolver);
+  return _o;
+}
+
+inline void Weapon::UnPackTo(WeaponT *_o, const flatbuffers::resolver_function_t *_resolver) const {
+  (void)_o;
+  (void)_resolver;
   { auto _e = name(); if (_e) _o->name = _e->str(); };
   { auto _e = damage(); _o->damage = _e; };
-  return _o;
 }
 
 inline flatbuffers::Offset<Weapon> Weapon::Pack(flatbuffers::FlatBufferBuilder &_fbb, const WeaponT* _o, const flatbuffers::rehasher_function_t *_rehasher) {
@@ -468,10 +489,12 @@ inline flatbuffers::Offset<Weapon> Weapon::Pack(flatbuffers::FlatBufferBuilder &
 inline flatbuffers::Offset<Weapon> CreateWeapon(flatbuffers::FlatBufferBuilder &_fbb, const WeaponT *_o, const flatbuffers::rehasher_function_t *_rehasher) {
   (void)_rehasher;
   (void)_o;
+  auto _name = _o->name.size() ? _fbb.CreateString(_o->name) : 0;
+  auto _damage = _o->damage;
   return CreateWeapon(
       _fbb,
-      _o->name.size() ? _fbb.CreateString(_o->name) : 0,
-      _o->damage);
+      _name,
+      _damage);
 }
 
 inline bool VerifyEquipment(flatbuffers::Verifier &verifier, const void *obj, Equipment type) {

src/idl_gen_cpp.cpp

@@ -60,6 +60,11 @@ class CppGenerator : public BaseGenerator {
 
   void GenIncludeDependencies() {
     int num_includes = 0;
+    for (auto it = parser_.native_included_files_.begin();
+         it != parser_.native_included_files_.end(); ++it) {
+      code_ += "#include \"" + *it + "\"";
+      num_includes++;
+    }
     for (auto it = parser_.included_files_.begin();
          it != parser_.included_files_.end(); ++it) {
       const auto basename =
@@ -384,8 +389,12 @@ class CppGenerator : public BaseGenerator {
         return "std::vector<" + type_name + ">";
       }
       case BASE_TYPE_STRUCT: {
-        const auto type_name = WrapInNameSpace(*type.struct_def);
+        auto type_name = WrapInNameSpace(*type.struct_def);
         if (IsStruct(type)) {
+          auto native_type = type.struct_def->attributes.Lookup("native_type");
+          if (native_type) {
+            type_name = native_type->constant;
+          }
           if (invector || field.native_inline) {
             return type_name;
           } else {
@@ -487,7 +496,15 @@ class CppGenerator : public BaseGenerator {
                                           bool inclass) {
     return NativeName(struct_def.name) + " *" +
            (inclass ? "" : struct_def.name + "::") +
-           "UnPack(const flatbuffers::resolver_function_t *resolver" +
+           "UnPack(const flatbuffers::resolver_function_t *_resolver" +
+           (inclass ? " = nullptr" : "") + ") const";
+  }
+
+  static std::string TableUnPackToSignature(const StructDef &struct_def,
+                                            bool inclass) {
+    return "void " + (inclass ? "" : struct_def.name + "::") +
+           "UnPackTo(" + NativeName(struct_def.name) + " *" + "_o, " +
+           "const flatbuffers::resolver_function_t *_resolver" +
            (inclass ? " = nullptr" : "") + ") const";
   }
 
@@ -889,6 +906,17 @@ class CppGenerator : public BaseGenerator {
           }
           initializer_list += field.name;
           initializer_list += "(" + GetDefaultScalarValue(field) + ")";
+        } else if (field.value.type.base_type == BASE_TYPE_STRUCT) {
+          if (IsStruct(field.value.type)) {
+            auto native_default = field.attributes.Lookup("native_default");
+            if (native_default) {
+              if (!initializer_list.empty()) {
+                initializer_list += ",\n        ";
+              }
+              initializer_list +=
+                  field.name + "(" + native_default->constant + ")";
+            }
+          }
         } else if (cpp_type) {
           if (!initializer_list.empty()) {
             initializer_list += ",\n        ";
@@ -1159,6 +1187,7 @@ class CppGenerator : public BaseGenerator {
     if (parser_.opts.generate_object_based_api) {
       // Generate the UnPack() pre declaration.
       code_ += "  " + TableUnPackSignature(struct_def, true) + ";";
+      code_ += "  " + TableUnPackToSignature(struct_def, true) + ";";
       code_ += "  " + TablePackSignature(struct_def, true) + ";";
     }
 
@@ -1338,7 +1367,10 @@ class CppGenerator : public BaseGenerator {
       case BASE_TYPE_STRUCT: {
         const auto name = WrapInNameSpace(*type.struct_def);
         if (IsStruct(type)) {
-          if (invector || afield.native_inline) {
+          auto native_type = type.struct_def->attributes.Lookup("native_type");
+          if (native_type) {
+            return "flatbuffers::UnPack(*" + val + ")";
+          } else if (invector || afield.native_inline) {
             return "*" + val;
           } else {
             const auto ptype = GenTypeNativePtr(name, &afield, true);
@@ -1346,7 +1378,7 @@ class CppGenerator : public BaseGenerator {
           }
         } else {
           const auto ptype = GenTypeNativePtr(NativeName(name), &afield, true);
-          return ptype + "(" + val + "->UnPack(resolver))";
+          return ptype + "(" + val + "->UnPack(_resolver))";
         }
       }
       default: {
@@ -1395,7 +1427,7 @@ class CppGenerator : public BaseGenerator {
         code += "_o->" + field.name + ".table = ";
         code += field.value.type.enum_def->name + "Union::UnPack(";
         code += "_e, " + field.name + UnionTypeFieldSuffix() + "(),";
-        code += "resolver);";
+        code += "_resolver);";
         break;
       }
       default: {
@@ -1406,8 +1438,8 @@ class CppGenerator : public BaseGenerator {
           //    (*resolver)(&_o->field, (hash_value_t)(_e));
           //  else
           //    _o->field = nullptr;
-          code += "if (resolver) ";
-          code += "(*resolver)";
+          code += "if (_resolver) ";
+          code += "(*_resolver)";
           code += "(reinterpret_cast<void **>(&_o->" + field.name + "), ";
           code += "static_cast<flatbuffers::hash_value_t>(_e));";
           code += " else ";
@@ -1514,7 +1546,11 @@ class CppGenerator : public BaseGenerator {
       }
       case BASE_TYPE_STRUCT: {
         if (IsStruct(field.value.type)) {
-          if (field.native_inline) {
+          auto native_type =
+              field.value.type.struct_def->attributes.Lookup("native_type");
+          if (native_type) {
+            code += "flatbuffers::Pack(" + value + ")";
+          } else if (field.native_inline) {
             code += "&" + value;
           } else {
             code += value + " ? " + value + GenPtrGet(field) + " : 0";
@@ -1544,8 +1580,15 @@ class CppGenerator : public BaseGenerator {
     if (parser_.opts.generate_object_based_api) {
       // Generate the X::UnPack() method.
       code_ += "inline " + TableUnPackSignature(struct_def, false) + " {";
-      code_ += "  (void)resolver;";
       code_ += "  auto _o = new {{NATIVE_NAME}}();";
+      code_ += "  UnPackTo(_o, _resolver);";
+      code_ += "  return _o;";
+      code_ += "}";
+      code_ += "";
+
+      code_ += "inline " + TableUnPackToSignature(struct_def, false) + " {";
+      code_ += "  (void)_o;";
+      code_ += "  (void)_resolver;";
 
       for (auto it = struct_def.fields.vec.begin();
            it != struct_def.fields.vec.end(); ++it) {
@@ -1567,7 +1610,6 @@ class CppGenerator : public BaseGenerator {
         auto postfix = " };";
         code_ += std::string(prefix) + check + statement + postfix;
       }
-      code_ += "  return _o;";
       code_ += "}";
       code_ += "";
 
@@ -1582,17 +1624,42 @@ class CppGenerator : public BaseGenerator {
       code_ += "inline " + TableCreateSignature(struct_def, false) + " {";
       code_ += "  (void)_rehasher;";
       code_ += "  (void)_o;";
+
+      for (auto it = struct_def.fields.vec.begin();
+           it != struct_def.fields.vec.end(); ++it) {
+        auto &field = **it;
+        if (field.deprecated) {
+          continue;
+        }
+        code_ += "  auto _" + field.name + " = " + GenCreateParam(field) + ";";
+      }
+
       code_ += "  return Create{{STRUCT_NAME}}(";
       code_ += "      _fbb\\";
-
       for (auto it = struct_def.fields.vec.begin();
            it != struct_def.fields.vec.end(); ++it) {
         auto &field = **it;
         if (field.deprecated) {
           continue;
         }
+
+        bool pass_by_address = false;
+        if (field.value.type.base_type == BASE_TYPE_STRUCT) {
+          if (IsStruct(field.value.type)) {
+            auto native_type =
+                field.value.type.struct_def->attributes.Lookup("native_type");
+            if (native_type) {
+              pass_by_address = true;
+            }
+          }
+        }
+
         // Call the CreateX function using values from |_o|.
-        code_ += ",\n      " + GenCreateParam(field) + "\\";
+        if (pass_by_address) {
+          code_ += ",\n      &_" + field.name + "\\";
+        } else {
+          code_ += ",\n      _" + field.name + "\\";
+        }
       }
       code_ += ");";
       code_ += "}";

src/idl_parser.cpp

@@ -174,7 +174,8 @@ std::string Namespace::GetFullyQualifiedName(const std::string &name,
   TD(Include, 269, "include") \
   TD(Attribute, 270, "attribute") \
   TD(Null, 271, "null") \
-  TD(Service, 272, "rpc_service")
+  TD(Service, 272, "rpc_service") \
+  TD(NativeInclude, 273, "native_include")
 #ifdef __GNUC__
 __extension__  // Stop GCC complaining about trailing comma with -Wpendantic.
 #endif
@@ -432,6 +433,10 @@ CheckedError Parser::Next() {
             token_ = kTokenService;
             return NoError();
           }
+          if (attribute_ == "native_include") {
+            token_ = kTokenNativeInclude;
+            return NoError();
+          }
           // If not, it is a user-defined identifier:
           token_ = kTokenIdentifier;
           return NoError();
@@ -1857,6 +1862,10 @@ CheckedError Parser::DoParse(const char *source, const char **include_paths,
         (attribute_ == "option" || attribute_ == "syntax" ||
          attribute_ == "package")) {
         ECHECK(ParseProtoDecl());
+    } else if (Is(kTokenNativeInclude)) {
+      NEXT();
+      native_included_files_.emplace_back(attribute_);
+      EXPECT(kTokenStringConstant);
     } else if (Is(kTokenInclude) ||
                (opts.proto_mode &&
                 attribute_ == "import" &&