Added a "strict JSON" mode to the text generator and compiler

This will add quotes around field names, as required by the official
standard. By default it will leave quotes out, as it is more readable,
more compact, and is accepted by almost all JSON parsers.
The -S switch to flatc turns on strict mode.

As per rfc 7159.

Change-Id: Ibabe9c8162c47339d00ec581d18721a2ba40c6d0
Tested: on Windows.

docs/html/md__compiler.html

@@ -53,13 +53,15 @@ $(document).ready(function(){initNavTree('md__compiler.html','');});
 <div class="title">Using the schema compiler </div>  </div>
 </div><!--header-->
 <div class="contents">
-<div class="textblock"><p>Usage: </p><pre class="fragment">flatc [ -c ] [ -j ] [ -b ] [ -t ] file1 file2 ..
+<div class="textblock"><p>Usage: </p><pre class="fragment">flatc [ -c ] [ -j ] [ -b ] [ -t ] [ -o PATH ] [ -S ] file1 file2 ..
 </pre><p>The files are read and parsed in order, and can contain either schemas or data (see below). Later files can make use of definitions in earlier files. Depending on the flags passed, additional files may be generated for each file processed:</p>
 <ul>
 <li><code>-c</code> : Generate a C++ header for all definitions in this file (as <code>filename_generated.h</code>). Skips data.</li>
 <li><code>-j</code> : Generate Java classes.</li>
 <li><code>-b</code> : If data is contained in this file, generate a <code>filename_wire.bin</code> containing the binary flatbuffer.</li>
-<li><code>-t</code> : If data is contained in this file, generate a <code>filename_wire.txt</code> (for debugging). </li>
+<li><code>-t</code> : If data is contained in this file, generate a <code>filename_wire.txt</code> (for debugging).</li>
+<li><code>-o PATH</code> : Output all generated files to PATH (either absolute, or relative to the current directory). If omitted, PATH will be the current directory. PATH should end in your systems path separator, e.g. <code>/</code> or <code>\</code>.</li>
+<li><code>-S</code> : Generate strict JSON (field names are enclosed in quotes). By default, no quotes are generated. </li>
 </ul>
 </div></div><!-- contents -->
 </div><!-- doc-content -->

docs/html/md__schemas.html

@@ -84,7 +84,7 @@ root_type Monster;
 <p>Tables are the main way of defining objects in FlatBuffers, and consist of a name (here <code>Monster</code>) and a list of fields. Each field has a name, a type, and optionally a default value (if omitted, it defaults to 0 / NULL).</p>
 <p>Each field is optional: It does not have to appear in the wire representation, and you can choose to omit fields for each individual object. As a result, you have the flexibility to add fields without fear of bloating your data. This design is also FlatBuffer's mechanism for forward and backwards compatibility. Note that:</p>
 <ul>
-<li>You can add new fields in the schema ONLY at the end of a table definition. Older data will still read correctly, and give you the default value when read. Older code will simply ignore the new field. If you want to have flexibility to use any order for fields in your schema, you can manually assign ids (much like protocol buffer), see the <code>id</code> attribute below.</li>
+<li>You can add new fields in the schema ONLY at the end of a table definition. Older data will still read correctly, and give you the default value when read. Older code will simply ignore the new field. If you want to have flexibility to use any order for fields in your schema, you can manually assign ids (much like Protocol Buffers), see the <code>id</code> attribute below.</li>
 <li>You cannot delete fields you don't use anymore from the schema, but you can simply stop writing them into your data for almost the same effect. Additionally you can mark them as <code>deprecated</code> as in the example above, which will prevent the generation of accessors in the generated C++, as a way to enforce the field not being used any more. (careful: this may break code!).</li>
 <li>You may change field names and table names, if you're ok with your code breaking until you've renamed them there too.</li>
 </ul>
@@ -119,7 +119,7 @@ root_type Monster;
 <p>Attributes may be attached to a declaration, behind a field, or after the name of a table/struct/enum/union. These may either have a value or not. Some attributes like <code>deprecated</code> are understood by the compiler, others are simply ignored (like <code>priority</code>), but are available to query if you parse the schema at runtime. This is useful if you write your own code generators/editors etc., and you wish to add additional information specific to your tool (such as a help text).</p>
 <p>Current understood attributes:</p>
 <ul>
-<li><code>id: n</code> (on a table field): manually set the field id to <code>n</code>. If you use this attribute, you must use it on ALL fields of this table, and the numbers must be a contiguous range from 0 onwards. Additionally, since a union type effectively adds two fields, its id must be that of the second field (the first field is the type field and not explicitly declared in the schema). Once you've added id's, you can now order fields in any order in the schema, though new fields must still use the next available id when added.</li>
+<li><code>id: n</code> (on a table field): manually set the field identifier to <code>n</code>. If you use this attribute, you must use it on ALL fields of this table, and the numbers must be a contiguous range from 0 onwards. Additionally, since a union type effectively adds two fields, its id must be that of the second field (the first field is the type field and not explicitly declared in the schema). For example, if the last field before the union field had id 6, the union field should have id 8, and the unions type field will implicitly be 7. IDs allow the fields to be placed in any order in the schema. When a new field is added to the schema is must use the next available ID.</li>
 <li><code>deprecated</code> (on a field): do not generate accessors for this field anymore, code should stop using this data.</li>
 <li><code>original_order</code> (on a table): since elements in a table do not need to be stored in any particular order, they are often optimized for space by sorting them to size. This attribute stops that from happening.</li>
 <li><code>force_align: size</code> (on a struct): force the alignment of this struct to be something higher than what it is naturally aligned to. Causes these structs to be aligned to that amount inside a buffer, IF that buffer is allocated with that alignment (which is not necessarily the case for buffers accessed directly inside a <code>FlatBufferBuilder</code>).</li>

docs/source/Compiler.md

@@ -2,7 +2,7 @@
 
 Usage:
 
-    flatc [ -c ] [ -j ] [ -b ] [ -t ] file1 file2 ..
+    flatc [ -c ] [ -j ] [ -b ] [ -t ] [ -o PATH ] [ -S ] file1 file2 ..
 
 The files are read and parsed in order, and can contain either schemas
 or data (see below). Later files can make use of definitions in earlier
@@ -20,3 +20,10 @@ be generated for each file processed:
 -   `-t` : If data is contained in this file, generate a
     `filename_wire.txt` (for debugging).
 
+-   `-o PATH` : Output all generated files to PATH (either absolute, or
+    relative to the current directory). If omitted, PATH will be the
+    current directory. PATH should end in your systems path separator,
+    e.g. `/` or `\`.
+
+-   `-S` : Generate strict JSON (field names are enclosed in quotes).
+    By default, no quotes are generated.

include/flatbuffers/idl.h

@@ -290,14 +290,23 @@ class Parser {
   std::vector<uint8_t> struct_stack_;
 };
 
+// Container of options that may apply to any of the source/text generators.
+struct GeneratorOptions {
+  bool strict_json;
+  int indent_step;
+
+  GeneratorOptions() : strict_json(false), indent_step(2) {}
+};
+
 // Generate text (JSON) from a given FlatBuffer, and a given Parser
 // object that has been populated with the corresponding schema.
 // If ident_step is 0, no indentation will be generated. Additionally,
 // if it is less than 0, no linefeeds will be generated either.
 // See idl_gen_text.cpp.
+// strict_json adds "quotes" around field names if true.
 extern void GenerateText(const Parser &parser,
                          const void *flatbuffer,
-                         int indent_step,
+                         const GeneratorOptions &opts,
                          std::string *text);
 
 // Generate a C++ header from the definitions in the Parser object.
@@ -306,13 +315,15 @@ extern std::string GenerateCPP(const Parser &parser,
                                const std::string &include_guard_ident);
 extern bool GenerateCPP(const Parser &parser,
                         const std::string &path,
-                        const std::string &file_name);
+                        const std::string &file_name,
+                        const GeneratorOptions &opts);
 
 // Generate Java files from the definitions in the Parser object.
 // See idl_gen_java.cpp.
 extern bool GenerateJava(const Parser &parser,
                          const std::string &path,
-                         const std::string &file_name);
+                         const std::string &file_name,
+                         const GeneratorOptions &opts);
 
 }  // namespace flatbuffers
 

samples/sample_text.cpp

@@ -46,7 +46,8 @@ int main(int /*argc*/, const char * /*argv*/[]) {
   // to ensure it is correct, we now generate text back from the binary,
   // and compare the two:
   std::string jsongen;
-  GenerateText(parser, parser.builder_.GetBufferPointer(), 2, &jsongen);
+  GenerateText(parser, parser.builder_.GetBufferPointer(),
+               flatbuffers::GeneratorOptions(), &jsongen);
 
   if (jsongen != jsonfile) {
     printf("%s----------------\n%s", jsongen.c_str(), jsonfile.c_str());

src/flatc.cpp

@@ -25,7 +25,8 @@ namespace flatbuffers {
 
 bool GenerateBinary(const Parser &parser,
                     const std::string &path,
-                    const std::string &file_name) {
+                    const std::string &file_name,
+                    const GeneratorOptions & /*opts*/) {
   return !parser.builder_.GetSize() ||
          flatbuffers::SaveFile(
            (path + file_name + "_wire.bin").c_str(),
@@ -36,11 +37,13 @@ bool GenerateBinary(const Parser &parser,
 
 bool GenerateTextFile(const Parser &parser,
                       const std::string &path,
-                      const std::string &file_name) {
+                      const std::string &file_name,
+                      const GeneratorOptions &opts) {
   if (!parser.builder_.GetSize()) return true;
   if (!parser.root_struct_def) Error("root_type not set");
   std::string text;
-  GenerateText(parser, parser.builder_.GetBufferPointer(), 2, &text);
+  GenerateText(parser, parser.builder_.GetBufferPointer(), opts,
+               &text);
   return flatbuffers::SaveFile((path + file_name + "_wire.txt").c_str(),
                                text,
                                false);
@@ -54,7 +57,8 @@ bool GenerateTextFile(const Parser &parser,
 struct Generator {
   bool (*generate)(const flatbuffers::Parser &parser,
                    const std::string &path,
-                   const std::string &file_name);
+                   const std::string &file_name,
+                   const flatbuffers::GeneratorOptions &opts);
   const char *extension;
   const char *name;
   const char *help;
@@ -82,6 +86,7 @@ static void Error(const char *err, const char *obj, bool usage) {
     for (size_t i = 0; i < sizeof(generators) / sizeof(generators[0]); ++i)
       printf("  -%s      %s.\n", generators[i].extension, generators[i].help);
     printf("  -o PATH Prefix PATH to all generated files.\n"
+           "  -S      Strict JSON: add quotes to field names.\n"
            "FILEs may depend on declarations in earlier files.\n"
            "Output files are named using the base file name of the input,"
            "and written to the current directory or the path given by -o.\n"
@@ -110,6 +115,7 @@ std::string StripPath(const std::string &filename) {
 int main(int argc, const char *argv[]) {
   program_name = argv[0];
   flatbuffers::Parser parser;
+  flatbuffers::GeneratorOptions opts;
   std::string output_path;
   const size_t num_generators = sizeof(generators) / sizeof(generators[0]);
   bool generator_enabled[num_generators] = { false };
@@ -127,6 +133,9 @@ int main(int argc, const char *argv[]) {
           if (++i >= argc) Error("missing path following", arg, true);
           output_path = argv[i];
           break;
+        case 'S':
+          opts.strict_json = true;
+          break;
         default:
           for (size_t i = 0; i < num_generators; ++i) {
             if(!strcmp(arg+1, generators[i].extension)) {
@@ -165,7 +174,7 @@ int main(int argc, const char *argv[]) {
 
       for (size_t i = 0; i < num_generators; ++i) {
         if (generator_enabled[i]) {
-          if (!generators[i].generate(parser, output_path, filebase)) {
+          if (!generators[i].generate(parser, output_path, filebase, opts)) {
             Error((std::string("Unable to generate ") +
                    generators[i].name +
                    " for " +

src/idl_gen_cpp.cpp

@@ -486,7 +486,8 @@ std::string GenerateCPP(const Parser &parser, const std::string &include_guard_i
 
 bool GenerateCPP(const Parser &parser,
                  const std::string &path,
-                 const std::string &file_name) {
+                 const std::string &file_name,
+                 const GeneratorOptions & /*opts*/) {
     auto code = GenerateCPP(parser, file_name);
     return !code.length() ||
            SaveFile((path + file_name + "_generated.h").c_str(), code, false);

src/idl_gen_java.cpp

@@ -380,7 +380,8 @@ static bool SaveClass(const Parser &parser, const Definition &def,
 
 bool GenerateJava(const Parser &parser,
                   const std::string &path,
-                  const std::string & /*file_name*/) {
+                  const std::string & /*file_name*/,
+                  const GeneratorOptions & /*opts*/) {
   using namespace java;
 
   for (auto it = parser.enums_.vec.begin();

src/idl_gen_text.cpp

@@ -23,7 +23,8 @@
 namespace flatbuffers {
 
 static void GenStruct(const StructDef &struct_def, const Table *table,
-                      int indent, int indent_step, std::string *_text);
+                      int indent, const GeneratorOptions &opts,
+                      std::string *_text);
 
 // If indentation is less than 0, that indicates we don't want any newlines
 // either.
@@ -35,7 +36,8 @@ const char *NewLine(int indent_step) {
 // for a single FlatBuffer value into JSON format.
 // The general case for scalars:
 template<typename T> void Print(T val, Type /*type*/, int /*indent*/,
-                                int /*indent_step*/, StructDef * /*union_sd*/,
+                                StructDef * /*union_sd*/,
+                                const GeneratorOptions & /*opts*/,
                                 std::string *_text) {
   std::string &text = *_text;
   text += NumToString(val);
@@ -43,24 +45,25 @@ template<typename T> void Print(T val, Type /*type*/, int /*indent*/,
 
 // Print a vector a sequence of JSON values, comma separated, wrapped in "[]".
 template<typename T> void PrintVector(const Vector<T> &v, Type type,
-                                      int indent, int indent_step,
+                                      int indent, const GeneratorOptions &opts,
                                       std::string *_text) {
   std::string &text = *_text;
   text += "[";
-  text += NewLine(indent_step);
+  text += NewLine(opts.indent_step);
   for (uoffset_t i = 0; i < v.Length(); i++) {
     if (i) {
       text += ",";
-      text += NewLine(indent_step);
+      text += NewLine(opts.indent_step);
     }
-    text.append(indent + indent_step, ' ');
+    text.append(indent + opts.indent_step, ' ');
     if (IsStruct(type))
       Print(v.GetStructFromOffset(i * type.struct_def->bytesize), type,
-            indent + indent_step, indent_step, nullptr, _text);
+            indent + opts.indent_step, nullptr, opts, _text);
     else
-      Print(v.Get(i), type, indent + indent_step, indent_step, nullptr, _text);
+      Print(v.Get(i), type, indent + opts.indent_step, nullptr,
+            opts, _text);
   }
-  text += NewLine(indent_step);
+  text += NewLine(opts.indent_step);
   text.append(indent, ' ');
   text += "]";
 }
@@ -91,8 +94,10 @@ static void EscapeString(const String &s, std::string *_text) {
 
 // Specialization of Print above for pointer types.
 template<> void Print<const void *>(const void *val,
-                                    Type type, int indent, int indent_step,
-                                    StructDef *union_sd, std::string *_text) {
+                                    Type type, int indent,
+                                    StructDef *union_sd,
+                                    const GeneratorOptions &opts,
+                                    std::string *_text) {
   switch (type.base_type) {
     case BASE_TYPE_UNION:
       // If this assert hits, you have an corrupt buffer, a union type field
@@ -101,14 +106,14 @@ template<> void Print<const void *>(const void *val,
       GenStruct(*union_sd,
                 reinterpret_cast<const Table *>(val),
                 indent,
-                indent_step,
+                opts,
                 _text);
       break;
     case BASE_TYPE_STRUCT:
       GenStruct(*type.struct_def,
                 reinterpret_cast<const Table *>(val),
                 indent,
-                indent_step,
+                opts,
                 _text);
       break;
     case BASE_TYPE_STRING: {
@@ -123,7 +128,7 @@ template<> void Print<const void *>(const void *val,
           case BASE_TYPE_ ## ENUM: \
             PrintVector<CTYPE>( \
               *reinterpret_cast<const Vector<CTYPE> *>(val), \
-              type, indent, indent_step, _text); break;
+              type, indent, opts, _text); break;
           FLATBUFFERS_GEN_TYPES(FLATBUFFERS_TD)
         #undef FLATBUFFERS_TD
       }
@@ -135,18 +140,19 @@ template<> void Print<const void *>(const void *val,
 // Generate text for a scalar field.
 template<typename T> static void GenField(const FieldDef &fd,
                                           const Table *table, bool fixed,
-                                          int indent_step, int indent,
+                                          const GeneratorOptions &opts,
+                                          int indent,
                                           std::string *_text) {
   Print(fixed ?
     reinterpret_cast<const Struct *>(table)->GetField<T>(fd.value.offset) :
-    table->GetField<T>(fd.value.offset, 0), fd.value.type, indent, indent_step,
-                       nullptr, _text);
+    table->GetField<T>(fd.value.offset, 0), fd.value.type, indent, nullptr,
+                                            opts, _text);
 }
 
 // Generate text for non-scalar field.
 static void GenFieldOffset(const FieldDef &fd, const Table *table, bool fixed,
-                           int indent, int indent_step, StructDef *union_sd,
-                           std::string *_text) {
+                           int indent, StructDef *union_sd,
+                           const GeneratorOptions &opts, std::string *_text) {
   const void *val = nullptr;
   if (fixed) {
     // The only non-scalar fields in structs are structs.
@@ -158,16 +164,17 @@ static void GenFieldOffset(const FieldDef &fd, const Table *table, bool fixed,
       ? table->GetStruct<const void *>(fd.value.offset)
       : table->GetPointer<const void *>(fd.value.offset);
   }
-  Print(val, fd.value.type, indent, indent_step, union_sd, _text);
+  Print(val, fd.value.type, indent, union_sd, opts, _text);
 }
 
 // Generate text for a struct or table, values separated by commas, indented,
 // and bracketed by "{}"
 static void GenStruct(const StructDef &struct_def, const Table *table,
-                      int indent, int indent_step, std::string *_text) {
+                      int indent, const GeneratorOptions &opts,
+                      std::string *_text) {
   std::string &text = *_text;
   text += "{";
-  text += NewLine(indent_step);
+  text += NewLine(opts.indent_step);
   int fieldout = 0;
   StructDef *union_sd = nullptr;
   for (auto it = struct_def.fields.vec.begin();
@@ -178,16 +185,18 @@ static void GenStruct(const StructDef &struct_def, const Table *table,
       // The field is present.
       if (fieldout++) {
         text += ",";
-        text += NewLine(indent_step);
+        text += NewLine(opts.indent_step);
       }
-      text.append(indent + indent_step, ' ');
+      text.append(indent + opts.indent_step, ' ');
+      if (opts.strict_json) text += "\"";
       text += fd.name;
+      if (opts.strict_json) text += "\"";
       text += ": ";
       switch (fd.value.type.base_type) {
          #define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE) \
            case BASE_TYPE_ ## ENUM: \
               GenField<CTYPE>(fd, table, struct_def.fixed, \
-                              indent + indent_step, indent_step, _text); \
+                              opts, indent + opts.indent_step, _text); \
               break;
           FLATBUFFERS_GEN_TYPES_SCALAR(FLATBUFFERS_TD)
         #undef FLATBUFFERS_TD
@@ -196,8 +205,8 @@ static void GenStruct(const StructDef &struct_def, const Table *table,
           case BASE_TYPE_ ## ENUM:
           FLATBUFFERS_GEN_TYPES_POINTER(FLATBUFFERS_TD)
         #undef FLATBUFFERS_TD
-            GenFieldOffset(fd, table, struct_def.fixed, indent + indent_step,
-                           indent_step, union_sd, _text);
+            GenFieldOffset(fd, table, struct_def.fixed, indent + opts.indent_step,
+                           union_sd, opts, _text);
             break;
       }
       if (fd.value.type.base_type == BASE_TYPE_UTYPE) {
@@ -206,23 +215,23 @@ static void GenStruct(const StructDef &struct_def, const Table *table,
       }
     }
   }
-  text += NewLine(indent_step);
+  text += NewLine(opts.indent_step);
   text.append(indent, ' ');
   text += "}";
 }
 
 // Generate a text representation of a flatbuffer in JSON format.
 void GenerateText(const Parser &parser, const void *flatbuffer,
-                  int indent_step, std::string *_text) {
+                  const GeneratorOptions &opts, std::string *_text) {
   std::string &text = *_text;
   assert(parser.root_struct_def);  // call SetRootType()
   text.reserve(1024);   // Reduce amount of inevitable reallocs.
   GenStruct(*parser.root_struct_def,
             GetRoot<Table>(flatbuffer),
             0,
-            indent_step,
+            opts,
             _text);
-  text += NewLine(indent_step);
+  text += NewLine(opts.indent_step);
 }
 
 }  // namespace flatbuffers

src/idl_parser.cpp

@@ -669,7 +669,7 @@ void Parser::ParseDecl() {
   // Check if this is a table that has manual id assignments
   auto &fields = struct_def.fields.vec;
   if (!struct_def.fixed && fields.size()) {
-    int num_id_fields = 0;
+    size_t num_id_fields = 0;
     for (auto it = fields.begin(); it != fields.end(); ++it) {
       if ((*it)->attributes.Lookup("id")) num_id_fields++;
     }

tests/test.cpp

@@ -198,7 +198,8 @@ void ParseAndGenerateTextTest() {
   // to ensure it is correct, we now generate text back from the binary,
   // and compare the two:
   std::string jsongen;
-  GenerateText(parser, parser.builder_.GetBufferPointer(), 2, &jsongen);
+  flatbuffers::GeneratorOptions opts;
+  GenerateText(parser, parser.builder_.GetBufferPointer(), opts, &jsongen);
 
   if (jsongen != jsonfile) {
     printf("%s----------------\n%s", jsongen.c_str(), jsonfile.c_str());
@@ -406,7 +407,9 @@ void FuzzTest2() {
   TEST_EQ(parser.Parse(json.c_str()), true);
 
   std::string jsongen;
-  GenerateText(parser, parser.builder_.GetBufferPointer(), 0, &jsongen);
+  flatbuffers::GeneratorOptions opts;
+  opts.indent_step = 0;
+  GenerateText(parser, parser.builder_.GetBufferPointer(), opts, &jsongen);
 
   if (jsongen != json) {
     // These strings are larger than a megabyte, so we show the bytes around