Implemented the file identifier functionality for Java.

Also fixed flatc not outputting these identifiers for files
compiled on the command-line.

Bug: 16983987
Change-Id: I8b714cfea3a8e144fa52133f62b2f7eda6eb044a
Tested: on Linux

docs/html/md__compiler.html

@@ -64,7 +64,8 @@ $(document).ready(function(){initNavTree('md__compiler.html','');});
 <li><code>-b</code> : If data is contained in this file, generate a <code>filename.bin</code> containing the binary flatbuffer.</li>
 <li><code>-t</code> : If data is contained in this file, generate a <code>filename.json</code> representing the data in the flatbuffer.</li>
 <li><code>-o PATH</code> : Output all generated files to PATH (either absolute, or relative to the current directory). If omitted, PATH will be the current directory. PATH should end in your systems path separator, e.g. <code>/</code> or <code>\</code>.</li>
-<li><code>-S</code> : Generate strict JSON (field names are enclosed in quotes). By default, no quotes are generated. </li>
+<li><code>-S</code> : Generate strict JSON (field names are enclosed in quotes). By default, no quotes are generated.</li>
+<li><code>-P</code> : Don't prefix enum values in generated C++ by their enum type. </li>
 </ul>
 </div></div><!-- contents -->
 </div><!-- doc-content -->

docs/html/md__cpp_usage.html

@@ -104,6 +104,7 @@ assert(inv->Get(9) == 9);
 <p>if <code>ok</code> is true, the buffer is safe to read.</p>
 <p>Besides untrusted data, this function may be useful to call in debug mode, as extra insurance against data being corrupted somewhere along the way.</p>
 <p>While verifying a buffer isn't "free", it is typically faster than a full traversal (since any scalar data is not actually touched), and since it may cause the buffer to be brought into cache before reading, the actual overhead may be even lower than expected.</p>
+<p>In specialized cases where a denial of service attack is possible, the verifier has two additional constructor arguments that allow you to limit the nesting depth and total amount of tables the verifier may encounter before declaring the buffer malformed.</p>
 <h2>Text &amp; schema parsing</h2>
 <p>Using binary buffers with the generated header provides a super low overhead use of FlatBuffer data. There are, however, times when you want to use text formats, for example because it interacts better with source control, or you want to give your users easy access to data.</p>
 <p>Another reason might be that you already have a lot of data in JSON format, or a tool that generates JSON, and if you can write a schema for it, this will provide you an easy way to use that data directly.</p>

docs/html/md__java_usage.html

@@ -63,7 +63,9 @@ Vec3 pos = monster.pos();
 <p>Sadly the string accessors currently always create a new string when accessed, since FlatBuffer's UTF-8 strings can't be read in-place by Java.</p>
 <p>Vector access is also a bit different from C++: you pass an extra index to the vector field accessor. Then a second method with the same name suffixed by <code>Length</code> let's you know the number of elements you can access: </p><pre class="fragment">for (int i = 0; i &lt; monster.inventoryLength(); i++)
     monster.inventory(i); // do something here
-</pre><p>You can also construct these buffers in Java using the static methods found in the generated code, and the FlatBufferBuilder class: </p><pre class="fragment">FlatBufferBuilder fbb = new FlatBufferBuilder();
+</pre><p>If you specified a file_indentifier in the schema, you can query if the buffer is of the desired type before accessing it using: </p><pre class="fragment">if (Monster.MonsterBufferHasIdentifier(bb, start)) ...
+</pre><h2>Buffer construction in Java</h2>
+<p>You can also construct these buffers in Java using the static methods found in the generated code, and the FlatBufferBuilder class: </p><pre class="fragment">FlatBufferBuilder fbb = new FlatBufferBuilder();
 </pre><p>Create strings: </p><pre class="fragment">int str = fbb.createString("MyMonster");
 </pre><p>Create a table with a struct contained therein: </p><pre class="fragment">Monster.startMonster(fbb);
 Monster.addPos(fbb, Vec3.createVec3(fbb, 1.0f, 2.0f, 3.0f, 3.0, (byte)4, (short)5, (byte)6));
@@ -83,6 +85,8 @@ for (byte i = 4; i &gt;=0; i--) fbb.addByte(i);
 int inv = fbb.endVector();
 </pre><p>You can use the generated method <code>startInventoryVector</code> to conveniently call <code>startVector</code> with the right element size. You pass the number of elements you want to write. You write the elements backwards since the buffer is being constructed back to front.</p>
 <p>There are <code>add</code> functions for all the scalar types. You use <code>addOffset</code> for any previously constructed objects (such as other tables, strings, vectors). For structs, you use the appropriate <code>create</code> function in-line, as shown above in the <code>Monster</code> example.</p>
+<p>To finish the buffer, call: </p><pre class="fragment">Monster.finishMonsterBuffer(fbb, mon);
+</pre><p>The buffer is now ready to be transmitted. It is contained in the <code>ByteBuffer</code> which you can obtain from <code>fbb.dataBuffer()</code>. Importantly, the valid data does not start from offset 0 in this buffer, but from <code>fbb.dataStart()</code> (this is because the data was built backwards in memory). It ends at <code>fbb,capacity()</code>.</p>
 <h2>Text Parsing</h2>
 <p>There currently is no support for parsing text (Schema's and JSON) directly from Java, though you could use the C++ parser through JNI. Please see the C++ documentation for more on text parsing. </p>
 </div></div><!-- contents -->

docs/source/JavaUsage.md

@@ -38,6 +38,14 @@ suffixed by `Length` let's you know the number of elements you can access:
     for (int i = 0; i < monster.inventoryLength(); i++)
         monster.inventory(i); // do something here
 
+If you specified a file_indentifier in the schema, you can query if the
+buffer is of the desired type before accessing it using:
+
+    if (Monster.MonsterBufferHasIdentifier(bb, start)) ...
+
+
+## Buffer construction in Java
+
 You can also construct these buffers in Java using the static methods found
 in the generated code, and the FlatBufferBuilder class:
 
@@ -91,6 +99,16 @@ any previously constructed objects (such as other tables, strings, vectors).
 For structs, you use the appropriate `create` function in-line, as shown
 above in the `Monster` example.
 
+To finish the buffer, call:
+
+    Monster.finishMonsterBuffer(fbb, mon);
+
+The buffer is now ready to be transmitted. It is contained in the `ByteBuffer`
+which you can obtain from `fbb.dataBuffer()`. Importantly, the valid data does
+not start from offset 0 in this buffer, but from `fbb.dataStart()` (this is
+because the data was built backwards in memory). It ends at `fbb,capacity()`.
+
+
 ## Text Parsing
 
 There currently is no support for parsing text (Schema's and JSON) directly

include/flatbuffers/flatbuffers.h

@@ -665,8 +665,8 @@ template<typename T> const T *GetRoot(const void *buf) {
 
 // Helper to see if the identifier in a buffer has the expected value.
 inline bool BufferHasIdentifier(const void *buf, const char *identifier) {
-  return strncmp(reinterpret_cast<const char *>(buf) + 4, identifier,
-                 FlatBufferBuilder::kFileIdentifierLength) == 0;
+  return strncmp(reinterpret_cast<const char *>(buf) + sizeof(uoffset_t),
+                 identifier, FlatBufferBuilder::kFileIdentifierLength) == 0;
 }
 
 // Helper class to verify the integrity of a FlatBuffer

java/flatbuffers/Constants.java

+/*
+ * Copyright 2014 Google Inc. All rights reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package flatbuffers;
+
+// Class that holds shared constants.
+
+public class Constants {
+    // Java doesn't seem to have these.
+    static final int SIZEOF_SHORT = 2;
+    static final int SIZEOF_INT = 4;
+    static final int FILE_IDENTIFIER_LENGTH = 4;
+}
+

java/flatbuffers/FlatBufferBuilder.java

@@ -25,7 +25,7 @@ import java.nio.charset.Charset;
 // Class that helps you build a FlatBuffer.
 // See the section "Use in Java" in the main FlatBuffers documentation.
 
-public class FlatBufferBuilder {
+public class FlatBufferBuilder extends Constants {
     ByteBuffer bb;       // Where we construct the FlatBuffer.
     int space;           // Remaining space in the ByteBuffer.
     static final Charset utf8charset = Charset.forName("UTF-8");
@@ -36,10 +36,6 @@ public class FlatBufferBuilder {
     int num_vtables = 0;          // Number of entries in `vtables` in use.
     int vector_num_elems = 0;     // For the current vector being built.
 
-    // Java doesn't seem to have these.
-    final int SIZEOF_SHORT = 2;
-    final int SIZEOF_INT = 4;
-
     // Start with a buffer of size `initial_size`, then grow as required.
     public FlatBufferBuilder(int initial_size) {
         if (initial_size <= 0) initial_size = 1;
@@ -251,6 +247,17 @@ public class FlatBufferBuilder {
         addOffset(root_table);
     }
 
+    public void finish(int root_table, String file_identifier) {
+        prep(minalign, SIZEOF_INT + FILE_IDENTIFIER_LENGTH);
+        if (file_identifier.length() != FILE_IDENTIFIER_LENGTH)
+            throw new AssertionError("FlatBuffers: file identifier must be length " +
+                                     FILE_IDENTIFIER_LENGTH);
+        for (int i = FILE_IDENTIFIER_LENGTH - 1; i >= 0; i--) {
+            addByte((byte)file_identifier.charAt(i));
+        }
+        addOffset(root_table);
+    }
+
     public ByteBuffer dataBuffer() { return bb; }
 
     // The FlatBuffer data doesn't start at offset 0 in the ByteBuffer:

java/flatbuffers/Table.java

@@ -21,12 +21,10 @@ import java.nio.ByteBuffer;
 import java.nio.charset.Charset;
 
 // All tables in the generated code derive from this class, and add their own accessors.
-public class Table {
+public class Table extends Constants {
   protected int bb_pos;
   protected ByteBuffer bb;
 
-  final int SIZEOF_INT = 4;
-
   // Look up a field in the vtable, return an offset into the object, or 0 if the field is not
   // present.
   protected int __offset(int vtable_offset) {
@@ -75,4 +73,14 @@ public class Table {
     t.bb = bb;
     return t;
   }
+
+  protected static boolean __has_identifier(ByteBuffer bb, int offset, String ident) {
+    if (ident.length() != FILE_IDENTIFIER_LENGTH)
+        throw new AssertionError("FlatBuffers: file identifier must be length " +
+                                 FILE_IDENTIFIER_LENGTH);
+    for (int i = 0; i < FILE_IDENTIFIER_LENGTH; i++) {
+      if (ident.charAt(i) != (char)bb.get(offset + SIZEOF_INT + i)) return false;
+    }
+    return true;
+  }
 }

src/idl_gen_java.cpp

@@ -154,7 +154,8 @@ static void GenStructBody(const StructDef &struct_def, std::string *code_ptr,
   }
 }
 
-static void GenStruct(StructDef &struct_def, std::string *code_ptr) {
+static void GenStruct(const Parser &parser, StructDef &struct_def,
+                      std::string *code_ptr) {
   if (struct_def.generated) return;
   std::string &code = *code_ptr;
 
@@ -177,12 +178,21 @@ static void GenStruct(StructDef &struct_def, std::string *code_ptr) {
     code += "_bb.order(ByteOrder.LITTLE_ENDIAN); ";
     code += "return (new " + struct_def.name;
     code += "()).__init(_bb.getInt(offset) + offset, _bb); }\n";
+    if (parser.root_struct_def == &struct_def) {
+      if (parser.file_identifier_.length()) {
+        // Check if a buffer has the identifier.
+        code += "  public static boolean " + struct_def.name;
+        code += "BufferHasIdentifier(ByteBuffer _bb, int offset) { return ";
+        code += "__has_identifier(_bb, offset, \"" + parser.file_identifier_;
+        code += "\"); }\n";
+      }
+    }
   }
   // Generate the __init method that sets the field in a pre-existing
   // accessor object. This is to allow object reuse.
   code += "  public " + struct_def.name;
   code += " __init(int _i, ByteBuffer _bb) ";
-  code += "{ bb_pos = _i; bb = _bb; return this; }\n";
+  code += "{ bb_pos = _i; bb = _bb; return this; }\n\n";
   for (auto it = struct_def.fields.vec.begin();
        it != struct_def.fields.vec.end();
        ++it) {
@@ -321,6 +331,14 @@ static void GenStruct(StructDef &struct_def, std::string *code_ptr) {
     }
     code += "  public static int end" + struct_def.name;
     code += "(FlatBufferBuilder builder) { return builder.endObject(); }\n";
+    if (parser.root_struct_def == &struct_def) {
+      code += "  public static void finish" + struct_def.name;
+      code += "Buffer(FlatBufferBuilder builder, int offset) { ";
+      code += "builder.finish(offset";
+      if (parser.file_identifier_.length())
+        code += ", \"" + parser.file_identifier_ + "\"";
+      code += "); }\n";
+    }
   }
   code += "};\n\n";
 }
@@ -375,7 +393,7 @@ bool GenerateJava(const Parser &parser,
   for (auto it = parser.structs_.vec.begin();
        it != parser.structs_.vec.end(); ++it) {
     std::string declcode;
-    GenStruct(**it, &declcode);
+    GenStruct(parser, **it, &declcode);
     if (!SaveClass(parser, **it, declcode, path, true))
       return false;
   }

src/idl_parser.cpp

@@ -914,7 +914,8 @@ bool Parser::Parse(const char *source, const char *filepath) {
         if (builder_.GetSize()) {
           Error("cannot have more than one json object in a file");
         }
-        builder_.Finish(Offset<Table>(ParseTable(*root_struct_def)));
+        builder_.Finish(Offset<Table>(ParseTable(*root_struct_def)),
+          file_identifier_.length() ? file_identifier_.c_str() : nullptr);
       } else if (token_ == kTokenEnum) {
         ParseEnum(false);
       } else if (token_ == kTokenUnion) {

tests/JavaTest.java

@@ -85,7 +85,7 @@ class JavaTest {
         Monster.addTestarrayofstring(fbb, testArrayOfString);
         int mon = Monster.endMonster(fbb);
 
-        fbb.finish(mon);
+        Monster.finishMonsterBuffer(fbb, mon);
 
         // Write the result to a file for debugging purposes:
         // Note that the binaries are not necessarily identical, since the JSON
@@ -113,6 +113,8 @@ class JavaTest {
     }
 
     static void TestBuffer(ByteBuffer bb, int start) {
+        TestEq(Monster.MonsterBufferHasIdentifier(bb, start), true);
+
         Monster monster = Monster.getRootAsMonster(bb, start);
 
         TestEq(monster.hp(), (short)80);

tests/MyGame/Example/Monster.java

@@ -9,7 +9,9 @@ import flatbuffers.*;
 
 public class Monster extends Table {
   public static Monster getRootAsMonster(ByteBuffer _bb, int offset) { _bb.order(ByteOrder.LITTLE_ENDIAN); return (new Monster()).__init(_bb.getInt(offset) + offset, _bb); }
+  public static boolean MonsterBufferHasIdentifier(ByteBuffer _bb, int offset) { return __has_identifier(_bb, offset, "MONS"); }
   public Monster __init(int _i, ByteBuffer _bb) { bb_pos = _i; bb = _bb; return this; }
+
   public Vec3 pos() { return pos(new Vec3()); }
   public Vec3 pos(Vec3 obj) { int o = __offset(4); return o != 0 ? obj.__init(o + bb_pos, bb) : null; }
   public short mana() { int o = __offset(6); return o != 0 ? bb.getShort(o + bb_pos) : 150; }
@@ -57,5 +59,6 @@ public class Monster extends Table {
   public static void startTestnestedflatbufferVector(FlatBufferBuilder builder, int numElems) { builder.startVector(1, numElems, 1); }
   public static void addTestempty(FlatBufferBuilder builder, int testemptyOffset) { builder.addOffset(14, testemptyOffset, 0); }
   public static int endMonster(FlatBufferBuilder builder) { return builder.endObject(); }
+  public static void finishMonsterBuffer(FlatBufferBuilder builder, int offset) { builder.finish(offset, "MONS"); }
 };
 

tests/MyGame/Example/Test.java

@@ -9,6 +9,7 @@ import flatbuffers.*;
 
 public class Test extends Struct {
   public Test __init(int _i, ByteBuffer _bb) { bb_pos = _i; bb = _bb; return this; }
+
   public short a() { return bb.getShort(bb_pos + 0); }
   public byte b() { return bb.get(bb_pos + 2); }
 

tests/MyGame/Example/Vec3.java

@@ -9,6 +9,7 @@ import flatbuffers.*;
 
 public class Vec3 extends Struct {
   public Vec3 __init(int _i, ByteBuffer _bb) { bb_pos = _i; bb = _bb; return this; }
+
   public float x() { return bb.getFloat(bb_pos + 0); }
   public float y() { return bb.getFloat(bb_pos + 4); }
   public float z() { return bb.getFloat(bb_pos + 8); }

tests/monsterdata_test.json

@@ -32,5 +32,9 @@
       a: 30,
       b: 40
     }
+  ],
+  testarrayofstring: [
+    "test1",
+    "test2"
   ]
 }