Commit 01025fa2 authored by ncpenke's avatar ncpenke

Merge pull request #3 from google/master

Catchup
parents 17c0b677 a6a38f60
......@@ -75,12 +75,13 @@ $(document).ready(function(){initNavTree('md__internals.html','');});
<h3>Structs</h3>
<p>These are the simplest, and as mentioned, intended for simple data that benefits from being extra efficient and doesn't need versioning / extensibility. They are always stored inline in their parent (a struct, table, or vector) for maximum compactness. Structs define a consistent memory layout where all components are aligned to their size, and structs aligned to their largest scalar member. This is done independent of the alignment rules of the underlying compiler to guarantee a cross platform compatible layout. This layout is then enforced in the generated code.</p>
<h3>Tables</h3>
<p>These start with an <code>soffset_t</code> to a vtable. This is a signed version of <code>uoffset_t</code>, since vtables may be stored anywhere relative to the object. This offset is substracted (not added) from the object start to arrive at the vtable start. This offset is followed by all the fields as aligned scalars (or offsets). Unlike structs, not all fields need to be present. There is no set order and layout.</p>
<p>Unlike structs, these are not stored in inline in their parent, but are referred to by offset.</p>
<p>They start with an <code>soffset_t</code> to a vtable. This is a signed version of <code>uoffset_t</code>, since vtables may be stored anywhere relative to the object. This offset is substracted (not added) from the object start to arrive at the vtable start. This offset is followed by all the fields as aligned scalars (or offsets). Unlike structs, not all fields need to be present. There is no set order and layout.</p>
<p>To be able to access fields regardless of these uncertainties, we go through a vtable of offsets. Vtables are shared between any objects that happen to have the same vtable values.</p>
<p>The elements of a vtable are all of type <code>voffset_t</code>, which is a <code>uint16_t</code>. The first element is the size of the vtable in bytes, including the size element. The second one is the size of the object, in bytes (including the vtable offset). This size could be used for streaming, to know how many bytes to read to be able to access all fields of the object. The remaining elements are the N offsets, where N is the amount of fields declared in the schema when the code that constructed this buffer was compiled (thus, the size of the table is N + 2).</p>
<p>The elements of a vtable are all of type <code>voffset_t</code>, which is a <code>uint16_t</code>. The first element is the size of the vtable in bytes, including the size element. The second one is the size of the object, in bytes (including the vtable offset). This size could be used for streaming, to know how many bytes to read to be able to access all <em>inline</em> fields of the object. The remaining elements are the N offsets, where N is the amount of fields declared in the schema when the code that constructed this buffer was compiled (thus, the size of the table is N + 2).</p>
<p>All accessor functions in the generated code for tables contain the offset into this table as a constant. This offset is checked against the first field (the number of elements), to protect against newer code reading older data. If this offset is out of range, or the vtable entry is 0, that means the field is not present in this object, and the default value is return. Otherwise, the entry is used as offset to the field to be read.</p>
<h3>Strings and Vectors</h3>
<p>Strings are simply a vector of bytes, and are always null-terminated. Vectors are stored as contiguous aligned scalar elements prefixed by a 32bit element count (not including any null termination).</p>
<p>Strings are simply a vector of bytes, and are always null-terminated. Vectors are stored as contiguous aligned scalar elements prefixed by a 32bit element count (not including any null termination). Neither is stored inline in their parent, but are referred to by offset.</p>
<h3>Construction</h3>
<p>The current implementation constructs these buffers backwards (starting at the highest memory address of the buffer), since that significantly reduces the amount of bookkeeping and simplifies the construction API.</p>
<h3>Code example</h3>
......@@ -162,7 +163,29 @@ STRUCT_END(Vec3, 12);
}
</pre><p><code>CreateMonster</code> is a convenience function that calls all functions in <code>MonsterBuilder</code> above for you. Note that if you pass values which are defaults as arguments, it will not actually construct that field, so you can probably use this function instead of the builder class in almost all cases. </p><pre class="fragment">inline const Monster *GetMonster(const void *buf) { return flatbuffers::GetRoot&lt;Monster&gt;(buf); }
</pre><p>This function is only generated for the root table type, to be able to start traversing a FlatBuffer from a raw buffer pointer. </p><pre class="fragment">}; // namespace MyGame
}; // namespace Sample</pre> </div></div><!-- contents -->
}; // namespace Sample
</pre><h3>Encoding example.</h3>
<p>Below is a sample encoding for the following JSON corresponding to the above schema: </p><pre class="fragment">{ pos: { x: 1, y: 2, z: 3 }, name: "fred", hp: 50 }
</pre><p>Resulting in this binary buffer: </p><pre class="fragment">// Start of the buffer:
uint32_t 20 // Offset to the root table.
// Start of the vtable. Not shared in this example, but could be:
uint16_t 16 // Size of table, starting from here.
uint16_t 22 // Size of object inline data.
uint16_t 4, 0, 20, 16, 0, 0 // Offsets to fields from start of (root) table, 0 for not present.
// Start of the root table:
int32_t 16 // Offset to vtable used (default negative direction)
float 1, 2, 3 // the Vec3 struct, inline.
uint32_t 8 // Offset to the name string.
int16_t 50 // hp field.
int16_t 0 // Padding for alignment.
// Start of name string:
uint32_t 4 // Length of string.
int8_t 'f', 'r', 'e', 'd', 0, 0, 0, 0 // Text + 0 termination + padding.
</pre><p>Note that this not the only possible encoding, since the writer has some flexibility in which of the children of root object to write first (though in this case there's only one string), and what order to write the fields in. Different orders may also cause different alignments to happen. </p>
</div></div><!-- contents -->
</div><!-- doc-content -->
<!-- Google Analytics -->
<script>
......
......@@ -73,7 +73,10 @@ code.
### Tables
These start with an `soffset_t` to a vtable. This is a signed version of
Unlike structs, these are not stored in inline in their parent, but are
referred to by offset.
They start with an `soffset_t` to a vtable. This is a signed version of
`uoffset_t`, since vtables may be stored anywhere relative to the object.
This offset is substracted (not added) from the object start to arrive at
the vtable start. This offset is followed by all the
......@@ -88,7 +91,7 @@ The elements of a vtable are all of type `voffset_t`, which is
a `uint16_t`. The first element is the size of the vtable in bytes,
including the size element. The second one is the size of the object, in bytes
(including the vtable offset). This size could be used for streaming, to know
how many bytes to read to be able to access all fields of the object.
how many bytes to read to be able to access all *inline* fields of the object.
The remaining elements are the N offsets, where N is the amount of fields
declared in the schema when the code that constructed this buffer was
compiled (thus, the size of the table is N + 2).
......@@ -106,7 +109,8 @@ field to be read.
Strings are simply a vector of bytes, and are always
null-terminated. Vectors are stored as contiguous aligned scalar
elements prefixed by a 32bit element count (not including any
null termination).
null termination). Neither is stored inline in their parent, but are referred to
by offset.
### Construction
......@@ -249,4 +253,35 @@ start traversing a FlatBuffer from a raw buffer pointer.
}; // namespace MyGame
}; // namespace Sample
### Encoding example.
Below is a sample encoding for the following JSON corresponding to the above
schema:
{ pos: { x: 1, y: 2, z: 3 }, name: "fred", hp: 50 }
Resulting in this binary buffer:
// Start of the buffer:
uint32_t 20 // Offset to the root table.
// Start of the vtable. Not shared in this example, but could be:
uint16_t 16 // Size of table, starting from here.
uint16_t 22 // Size of object inline data.
uint16_t 4, 0, 20, 16, 0, 0 // Offsets to fields from start of (root) table, 0 for not present.
// Start of the root table:
int32_t 16 // Offset to vtable used (default negative direction)
float 1, 2, 3 // the Vec3 struct, inline.
uint32_t 8 // Offset to the name string.
int16_t 50 // hp field.
int16_t 0 // Padding for alignment.
// Start of name string:
uint32_t 4 // Length of string.
int8_t 'f', 'r', 'e', 'd', 0, 0, 0, 0 // Text + 0 termination + padding.
Note that this not the only possible encoding, since the writer has some
flexibility in which of the children of root object to write first (though in
this case there's only one string), and what order to write the fields in.
Different orders may also cause different alignments to happen.
......@@ -377,9 +377,10 @@ class CheckedError {
// Additionally, in GCC we can get these errors statically, for additional
// assurance:
#ifdef __GNUC__
#define CHECKED_ERROR CheckedError __attribute__((warn_unused_result))
#define FLATBUFFERS_CHECKED_ERROR CheckedError \
__attribute__((warn_unused_result))
#else
#define CHECKED_ERROR CheckedError
#define FLATBUFFERS_CHECKED_ERROR CheckedError
#endif
class Parser {
......@@ -437,54 +438,59 @@ class Parser {
// See reflection/reflection.fbs
void Serialize();
CHECKED_ERROR CheckBitsFit(int64_t val, size_t bits);
FLATBUFFERS_CHECKED_ERROR CheckBitsFit(int64_t val, size_t bits);
private:
CHECKED_ERROR Error(const std::string &msg);
CHECKED_ERROR ParseHexNum(int nibbles, int64_t *val);
CHECKED_ERROR Next();
FLATBUFFERS_CHECKED_ERROR Error(const std::string &msg);
FLATBUFFERS_CHECKED_ERROR ParseHexNum(int nibbles, int64_t *val);
FLATBUFFERS_CHECKED_ERROR Next();
bool Is(int t);
CHECKED_ERROR Expect(int t);
FLATBUFFERS_CHECKED_ERROR Expect(int t);
std::string TokenToStringId(int t);
EnumDef *LookupEnum(const std::string &id);
CHECKED_ERROR ParseNamespacing(std::string *id, std::string *last);
CHECKED_ERROR ParseTypeIdent(Type &type);
CHECKED_ERROR ParseType(Type &type);
CHECKED_ERROR AddField(StructDef &struct_def, const std::string &name,
const Type &type, FieldDef **dest);
CHECKED_ERROR ParseField(StructDef &struct_def);
CHECKED_ERROR ParseAnyValue(Value &val, FieldDef *field, size_t parent_fieldn);
CHECKED_ERROR ParseTable(const StructDef &struct_def, std::string *value,
uoffset_t *ovalue);
FLATBUFFERS_CHECKED_ERROR ParseNamespacing(std::string *id,
std::string *last);
FLATBUFFERS_CHECKED_ERROR ParseTypeIdent(Type &type);
FLATBUFFERS_CHECKED_ERROR ParseType(Type &type);
FLATBUFFERS_CHECKED_ERROR AddField(StructDef &struct_def,
const std::string &name, const Type &type,
FieldDef **dest);
FLATBUFFERS_CHECKED_ERROR ParseField(StructDef &struct_def);
FLATBUFFERS_CHECKED_ERROR ParseAnyValue(Value &val, FieldDef *field,
size_t parent_fieldn);
FLATBUFFERS_CHECKED_ERROR ParseTable(const StructDef &struct_def,
std::string *value, uoffset_t *ovalue);
void SerializeStruct(const StructDef &struct_def, const Value &val);
void AddVector(bool sortbysize, int count);
CHECKED_ERROR ParseVector(const Type &type, uoffset_t *ovalue);
CHECKED_ERROR ParseMetaData(Definition &def);
CHECKED_ERROR TryTypedValue(int dtoken, bool check, Value &e, BaseType req,
bool *destmatch);
CHECKED_ERROR ParseHash(Value &e, FieldDef* field);
CHECKED_ERROR ParseSingleValue(Value &e);
CHECKED_ERROR ParseIntegerFromString(Type &type, int64_t *result);
FLATBUFFERS_CHECKED_ERROR ParseVector(const Type &type, uoffset_t *ovalue);
FLATBUFFERS_CHECKED_ERROR ParseMetaData(Definition &def);
FLATBUFFERS_CHECKED_ERROR TryTypedValue(int dtoken, bool check, Value &e,
BaseType req, bool *destmatch);
FLATBUFFERS_CHECKED_ERROR ParseHash(Value &e, FieldDef* field);
FLATBUFFERS_CHECKED_ERROR ParseSingleValue(Value &e);
FLATBUFFERS_CHECKED_ERROR ParseIntegerFromString(Type &type, int64_t *result);
StructDef *LookupCreateStruct(const std::string &name,
bool create_if_new = true,
bool definition = false);
CHECKED_ERROR ParseEnum(bool is_union, EnumDef **dest);
CHECKED_ERROR ParseNamespace();
CHECKED_ERROR StartStruct(const std::string &name, StructDef **dest);
CHECKED_ERROR ParseDecl();
CHECKED_ERROR ParseProtoFields(StructDef *struct_def, bool isextend,
bool inside_oneof);
CHECKED_ERROR ParseProtoOption();
CHECKED_ERROR ParseProtoKey();
CHECKED_ERROR ParseProtoDecl();
CHECKED_ERROR ParseProtoCurliesOrIdent();
CHECKED_ERROR ParseTypeFromProtoType(Type *type);
CHECKED_ERROR SkipAnyJsonValue();
CHECKED_ERROR SkipJsonObject();
CHECKED_ERROR SkipJsonArray();
CHECKED_ERROR SkipJsonString();
CHECKED_ERROR DoParse(const char *_source, const char **include_paths,
const char *source_filename);
FLATBUFFERS_CHECKED_ERROR ParseEnum(bool is_union, EnumDef **dest);
FLATBUFFERS_CHECKED_ERROR ParseNamespace();
FLATBUFFERS_CHECKED_ERROR StartStruct(const std::string &name,
StructDef **dest);
FLATBUFFERS_CHECKED_ERROR ParseDecl();
FLATBUFFERS_CHECKED_ERROR ParseProtoFields(StructDef *struct_def,
bool isextend, bool inside_oneof);
FLATBUFFERS_CHECKED_ERROR ParseProtoOption();
FLATBUFFERS_CHECKED_ERROR ParseProtoKey();
FLATBUFFERS_CHECKED_ERROR ParseProtoDecl();
FLATBUFFERS_CHECKED_ERROR ParseProtoCurliesOrIdent();
FLATBUFFERS_CHECKED_ERROR ParseTypeFromProtoType(Type *type);
FLATBUFFERS_CHECKED_ERROR SkipAnyJsonValue();
FLATBUFFERS_CHECKED_ERROR SkipJsonObject();
FLATBUFFERS_CHECKED_ERROR SkipJsonArray();
FLATBUFFERS_CHECKED_ERROR SkipJsonString();
FLATBUFFERS_CHECKED_ERROR DoParse(const char *_source,
const char **include_paths,
const char *source_filename);
public:
SymbolTable<StructDef> structs_;
......
......@@ -43,7 +43,13 @@ static_assert(BASE_TYPE_UNION ==
static_cast<BaseType>(reflection::Union),
"enums don't match");
// Any parsing calls have to be wrapped in this macro, which automates
// handling of recursive error checking a bit. It will check the received
// CheckedError object, and return straight away on error.
#define ECHECK(call) { auto ce = (call); if (ce.Check()) return ce; }
// These two functions are called hundreds of times below, so define a short
// form:
#define NEXT() ECHECK(Next())
#define EXPECT(tok) ECHECK(Expect(tok))
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment