Merge pull request #112 from rw/python23-read-write-gen-port

Port FlatBuffers to Python.

.gitignore

@@ -48,3 +48,4 @@ FlatBuffers.xcodeproj/
 java/.idea
 java/*.iml
 java/target
+**/*.pyc

CMakeLists.txt

@@ -24,6 +24,7 @@ set(FlatBuffers_Compiler_SRCS
   src/idl_gen_cpp.cpp
   src/idl_gen_general.cpp
   src/idl_gen_go.cpp
+  src/idl_gen_python.cpp
   src/idl_gen_text.cpp
   src/idl_gen_fbs.cpp
   src/flatc.cpp

docs/source/PythonUsage.md

+# Use in Python
+
+There's experimental support for reading FlatBuffers in Python. Generate
+code for Python with the `-p` option to `flatc`.
+
+See `py_test.py` for an example. You import the generated code, read a
+FlatBuffer binary file into a `bytearray`, which you pass to the
+`GetRootAsMonster` function:
+
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.py}
+    import MyGame.Example as example
+    import flatbuffers
+
+    buf = open('monster.dat', 'rb').read()
+    buf = bytearray(buf)
+    monster = example.GetRootAsMonster(buf, 0)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Now you can access values like this:
+
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.py}
+    hp = monster.Hp()
+    pos = monster.Pos()
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To access vectors you pass an extra index to the
+vector field accessor. Then a second method with the same name suffixed
+by `Length` let's you know the number of elements you can access:
+
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.py}
+    for i in xrange(monster.InventoryLength()):
+        monster.Inventory(i) # do something here
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can also construct these buffers in Python using the functions found
+in the generated code, and the FlatBufferBuilder class:
+
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.py}
+    builder = flatbuffers.NewBuilder(0)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Create strings:
+
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.py}
+    s = builder.CreateString("MyMonster")
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Create a table with a struct contained therein:
+
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.py}
+    example.MonsterStart(builder)
+    example.MonsterAddPos(builder, example.CreateVec3(builder, 1.0, 2.0, 3.0, 3.0, 4, 5, 6))
+    example.MonsterAddHp(builder, 80)
+    example.MonsterAddName(builder, str)
+    example.MonsterAddInventory(builder, inv)
+    example.MonsterAddTest_Type(builder, 1)
+    example.MonsterAddTest(builder, mon2)
+    example.MonsterAddTest4(builder, test4s)
+    mon = example.MonsterEnd(builder)
+
+    final_flatbuffer = bulder.Output()
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Unlike C++, Python does not support table creation functions like 'createMonster()'.
+This is to create the buffer without
+using temporary object allocation (since the `Vec3` is an inline component of
+`Monster`, it has to be created right where it is added, whereas the name and
+the inventory are not inline, and **must** be created outside of the table
+creation sequence).
+Structs do have convenient methods that allow you to construct them in one call.
+These also have arguments for nested structs, e.g. if a struct has a field `a`
+and a nested struct field `b` (which has fields `c` and `d`), then the arguments
+will be `a`, `c` and `d`.
+
+Vectors also use this start/end pattern to allow vectors of both scalar types
+and structs:
+
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.py}
+    example.MonsterStartInventoryVector(builder, 5)
+    i = 4
+    while i >= 0:
+        builder.PrependByte(byte(i))
+        i -= 1
+
+    inv = builder.EndVector(5)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The generated method 'StartInventoryVector' is provided as a convenience
+function which calls 'StartVector' with the correct element size of the vector
+type which in this case is 'ubyte' or 1 byte per vector element.
+You pass the number of elements you want to write.
+You write the elements backwards since the buffer
+is being constructed back to front. Use the correct `Prepend` call for the type,
+or `PrependUOffsetT` for offsets. You then pass `inv` to the corresponding
+`Add` call when you construct the table containing it afterwards.
+
+There are `Prepend` functions for all the scalar types. You use
+`PrependUOffset` for any previously constructed objects (such as other tables,
+strings, vectors). For structs, you use the appropriate `create` function
+in-line, as shown above in the `Monster` example.
+
+Once you're done constructing a buffer, you call `Finish` with the root object
+offset (`mon` in the example above). Your data now resides in Builder.Bytes.
+Important to note is that the real data starts at the index indicated by Head(),
+for Offset() bytes (this is because the buffer is constructed backwards).
+If you wanted to read the buffer right after creating it (using
+`GetRootAsMonster` above), the second argument, instead of `0` would thus
+also be `Head()`.
+
+## Text Parsing
+
+There currently is no support for parsing text (Schema's and JSON) directly
+from Python, though you could use the C++ parser through SWIG or ctypes. Please
+see the C++ documentation for more on text parsing.
+

docs/source/doxyfile

@@ -750,6 +750,7 @@ INPUT = "FlatBuffers.md" \
         "CppUsage.md" \
         "GoUsage.md" \
         "JavaUsage.md" \
+        "PythonUsage.md" \
         "Benchmarks.md" \
         "WhitePaper.md" \
         "Internals.md" \

include/flatbuffers/idl.h

@@ -35,24 +35,24 @@ namespace flatbuffers {
 // Additionally, Parser::ParseType assumes bool..string is a contiguous range
 // of type tokens.
 #define FLATBUFFERS_GEN_TYPES_SCALAR(TD) \
-  TD(NONE,   "",       uint8_t,  byte,   byte,    byte) \
-  TD(UTYPE,  "",       uint8_t,  byte,   byte,    byte) /* begin scalar/int */ \
-  TD(BOOL,   "bool",   uint8_t,  boolean,byte,    bool) \
-  TD(CHAR,   "byte",   int8_t,   byte,   int8,    sbyte) \
-  TD(UCHAR,  "ubyte",  uint8_t,  byte,   byte,    byte) \
-  TD(SHORT,  "short",  int16_t,  short,  int16,   short) \
-  TD(USHORT, "ushort", uint16_t, short,  uint16,  ushort) \
-  TD(INT,    "int",    int32_t,  int,    int32,   int) \
-  TD(UINT,   "uint",   uint32_t, int,    uint32,  uint) \
-  TD(LONG,   "long",   int64_t,  long,   int64,   long) \
-  TD(ULONG,  "ulong",  uint64_t, long,   uint64,  ulong)  /* end int */ \
-  TD(FLOAT,  "float",  float,    float,  float32, float)  /* begin float */ \
-  TD(DOUBLE, "double", double,   double, float64, double) /* end float/scalar */
+  TD(NONE,   "",       uint8_t,  byte,   byte,    byte,   uint8) \
+  TD(UTYPE,  "",       uint8_t,  byte,   byte,    byte,   uint8) /* begin scalar/int */ \
+  TD(BOOL,   "bool",   uint8_t,  boolean,byte,    bool,   bool) \
+  TD(CHAR,   "byte",   int8_t,   byte,   int8,    sbyte,  int8) \
+  TD(UCHAR,  "ubyte",  uint8_t,  byte,   byte,    byte,   uint8) \
+  TD(SHORT,  "short",  int16_t,  short,  int16,   short,  int16) \
+  TD(USHORT, "ushort", uint16_t, short,  uint16,  ushort, uint16) \
+  TD(INT,    "int",    int32_t,  int,    int32,   int,    int32) \
+  TD(UINT,   "uint",   uint32_t, int,    uint32,  uint,   uint32) \
+  TD(LONG,   "long",   int64_t,  long,   int64,   long,   int64) \
+  TD(ULONG,  "ulong",  uint64_t, long,   uint64,  ulong,  uint64) /* end int */ \
+  TD(FLOAT,  "float",  float,    float,  float32, float,  float32) /* begin float */ \
+  TD(DOUBLE, "double", double,   double, float64, double, float64) /* end float/scalar */
 #define FLATBUFFERS_GEN_TYPES_POINTER(TD) \
-  TD(STRING, "string", Offset<void>, int, int, int) \
-  TD(VECTOR, "",       Offset<void>, int, int, int) \
-  TD(STRUCT, "",       Offset<void>, int, int, int) \
-  TD(UNION,  "",       Offset<void>, int, int, int)
+  TD(STRING, "string", Offset<void>, int, int, int, int) \
+  TD(VECTOR, "",       Offset<void>, int, int, int, int) \
+  TD(STRUCT, "",       Offset<void>, int, int, int, int) \
+  TD(UNION,  "",       Offset<void>, int, int, int, int)
 
 // The fields are:
 // - enum
@@ -61,12 +61,13 @@ namespace flatbuffers {
 // - Java type.
 // - Go type.
 // - C# / .Net type.
+// - Python type.
 
 // using these macros, we can now write code dealing with types just once, e.g.
 
 /*
 switch (type) {
-  #define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE) \
+  #define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE, PTYPE) \
     case BASE_TYPE_ ## ENUM: \
       // do something specific to CTYPE here
     FLATBUFFERS_GEN_TYPES(FLATBUFFERS_TD)
@@ -83,13 +84,13 @@ switch (type) {
 __extension__  // Stop GCC complaining about trailing comma with -Wpendantic.
 #endif
 enum BaseType {
-  #define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE) \
+  #define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE, PTYPE) \
       BASE_TYPE_ ## ENUM,
     FLATBUFFERS_GEN_TYPES(FLATBUFFERS_TD)
   #undef FLATBUFFERS_TD
 };
 
-#define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE) \
+#define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE, PTYPE) \
     static_assert(sizeof(CTYPE) <= sizeof(largest_scalar_t), \
                   "define largest_scalar_t as " #CTYPE);
   FLATBUFFERS_GEN_TYPES(FLATBUFFERS_TD)
@@ -464,6 +465,13 @@ extern bool GenerateJava(const Parser &parser,
                          const std::string &file_name,
                          const GeneratorOptions &opts);
 
+// Generate Python files from the definitions in the Parser object.
+// See idl_gen_python.cpp.
+extern bool GeneratePython(const Parser &parser,
+                           const std::string &path,
+                           const std::string &file_name,
+                           const GeneratorOptions &opts);
+
 // Generate C# files from the definitions in the Parser object.
 // See idl_gen_csharp.cpp.
 extern bool GenerateCSharp(const Parser &parser,

python/flatbuffers/__init__.py

+# Copyright 2014 Google Inc. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from .builder import Builder
+from .table import Table
+from .compat import range_func as compat_range

python/flatbuffers/compat.py

+""" A tiny version of `six` to help with backwards compability. """
+
+import sys
+
+PY2 = sys.version_info[0] == 2
+PY26 = sys.version_info[0:2] == (2, 6)
+PY3 = sys.version_info[0] == 3
+PY34 = sys.version_info[0:2] >= (3, 4)
+
+if PY3:
+    string_types = (str,)
+    binary_type = bytes
+    range_func = range
+    memoryview_type = memoryview
+    struct_bool_decl = "?"
+else:
+    string_types = (basestring,)
+    binary_type = str
+    range_func = xrange
+    if PY26:
+        memoryview_type = buffer
+        struct_bool_decl = "<b"
+    else:
+        memoryview_type = memoryview
+        struct_bool_decl = "?"
+
+# NOTE: Future Jython support may require code here (look at `six`).

python/flatbuffers/encode.py

+# Copyright 2014 Google Inc. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import ctypes
+
+from . import number_types as N
+from . import packer
+from .compat import memoryview_type
+
+
+def Get(packer_type, buf, head):
+    """ Get decodes a value at buf[head:] using `packer_type`. """
+    return packer_type.unpack_from(memoryview_type(buf), head)[0]
+
+
+def Write(packer_type, buf, head, n):
+    """ Write encodes `n` at buf[head:] using `packer_type`. """
+    packer_type.pack_into(buf, head, n)

python/flatbuffers/number_types.py

+# Copyright 2014 Google Inc. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import ctypes
+import collections
+import struct
+from ctypes import sizeof
+
+from . import packer
+
+
+# For reference, see:
+# https://docs.python.org/2/library/ctypes.html#ctypes-fundamental-data-types-2
+
+# These classes could be collections.namedtuple instances, but those are new
+# in 2.6 and we want to work towards 2.5 compatability.
+
+class BoolFlags(object):
+    bytewidth = 1
+    min_val = False
+    max_val = True
+    py_type = bool
+    name = "bool"
+    packer_type = packer.boolean
+
+
+class Uint8Flags(object):
+    bytewidth = 1
+    min_val = 0
+    max_val = (2**8) - 1
+    py_type = int
+    name = "uint8"
+    packer_type = packer.uint8
+
+
+class Uint16Flags(object):
+    bytewidth = 2
+    min_val = 0
+    max_val = (2**16) - 1
+    py_type = int
+    name = "uint16"
+    packer_type = packer.uint16
+
+
+class Uint32Flags(object):
+    bytewidth = 4
+    min_val = 0
+    max_val = (2**32) - 1
+    py_type = int
+    name = "uint32"
+    packer_type = packer.uint32
+
+
+class Uint64Flags(object):
+    bytewidth = 8
+    min_val = 0
+    max_val = (2**64) - 1
+    py_type = int
+    name = "uint64"
+    packer_type = packer.uint64
+
+
+class Int8Flags(object):
+    bytewidth = 1
+    min_val = -(2**7)
+    max_val = (2**7) - 1
+    py_type = int
+    name = "int8"
+    packer_type = packer.int8
+
+
+class Int16Flags(object):
+    bytewidth = 2
+    min_val = -(2**15)
+    max_val = (2**15) - 1
+    py_type = int
+    name = "int16"
+    packer_type = packer.int16
+
+
+class Int32Flags(object):
+    bytewidth = 4
+    min_val = -(2**31)
+    max_val = (2**31) - 1
+    py_type = int
+    name = "int32"
+    packer_type = packer.int32
+
+
+class Int64Flags(object):
+    bytewidth = 8
+    min_val = -(2**63)
+    max_val = (2**63) - 1
+    py_type = int
+    name = "int64"
+    packer_type = packer.int64
+
+
+class Float32Flags(object):
+    bytewidth = 4
+    min_val = None
+    max_val = None
+    py_type = float
+    name = "float32"
+    packer_type = packer.float32
+
+
+class Float64Flags(object):
+    bytewidth = 8
+    min_val = None
+    max_val = None
+    py_type = float
+    name = "float64"
+    packer_type = packer.float64
+
+
+class SOffsetTFlags(Int32Flags):
+    pass
+
+
+class UOffsetTFlags(Uint32Flags):
+    pass
+
+
+class VOffsetTFlags(Uint16Flags):
+    pass
+
+
+def valid_number(n, flags):
+    if flags.min_val is None and flags.max_val is None:
+        return True
+    return flags.min_val <= n <= flags.max_val
+
+
+def enforce_number(n, flags):
+    if flags.min_val is None and flags.max_val is None:
+        return
+    if not flags.min_val <= n <= flags.max_val:
+        raise TypeError("bad number %s for type %s" % (str(n), flags.name))
+
+
+def float32_to_uint32(n):
+    packed = struct.pack("<1f", n)
+    (converted,) = struct.unpack("<1L", packed)
+    return converted
+
+
+def uint32_to_float32(n):
+    packed = struct.pack("<1L", n)
+    (unpacked,) = struct.unpack("<1f", packed)
+    return unpacked
+
+
+def float64_to_uint64(n):
+    packed = struct.pack("<1d", n)
+    (converted,) = struct.unpack("<1Q", packed)
+    return converted
+
+
+def uint64_to_float64(n):
+    packed = struct.pack("<1Q", n)
+    (unpacked,) = struct.unpack("<1d", packed)
+    return unpacked

python/flatbuffers/packer.py

+"""
+Provide pre-compiled struct packers for encoding and decoding.
+
+See: https://docs.python.org/2/library/struct.html#format-characters
+"""
+
+import struct
+from . import compat
+
+
+boolean = struct.Struct(compat.struct_bool_decl)
+
+uint8 = struct.Struct("<B")
+uint16 = struct.Struct("<H")
+uint32 = struct.Struct("<I")
+uint64 = struct.Struct("<Q")
+
+int8 = struct.Struct("<b")
+int16 = struct.Struct("<h")
+int32 = struct.Struct("<i")
+int64 = struct.Struct("<q")
+
+float32 = struct.Struct("<f")
+float64 = struct.Struct("<d")
+
+uoffset = uint32
+soffset = int32
+voffset = uint16

python/flatbuffers/table.py

+# Copyright 2014 Google Inc. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from . import encode
+from . import number_types as N
+
+
+class Table(object):
+    """Table wraps a byte slice and provides read access to its data.
+
+    The variable `Pos` indicates the root of the FlatBuffers object therein."""
+
+    __slots__ = ("Bytes", "Pos")
+
+    def __init__(self, buf, pos):
+        N.enforce_number(pos, N.UOffsetTFlags)
+
+        self.Bytes = buf
+        self.Pos = pos
+
+    def Offset(self, vtableOffset):
+        """Offset provides access into the Table's vtable.
+
+        Deprecated fields are ignored by checking the vtable's length."""
+
+        vtable = self.Pos - self.Get(N.SOffsetTFlags, self.Pos)
+        vtableEnd = self.Get(N.VOffsetTFlags, vtable)
+        if vtableOffset < vtableEnd:
+            return self.Get(N.VOffsetTFlags, vtable + vtableOffset)
+        return 0
+
+    def Indirect(self, off):
+        """Indirect retrieves the relative offset stored at `offset`."""
+        N.enforce_number(off, N.UOffsetTFlags)
+        return off + encode.Get(N.UOffsetTFlags.packer_type, self.Bytes, off)
+
+    def String(self, off):
+        """String gets a string from data stored inside the flatbuffer."""
+        N.enforce_number(off, N.UOffsetTFlags)
+        off += encode.Get(N.UOffsetTFlags.packer_type, self.Bytes, off)
+        start = off + N.UOffsetTFlags.bytewidth
+        length = encode.Get(N.UOffsetTFlags.packer_type, self.Bytes, off)
+        return bytes(self.Bytes[start:start+length])
+
+    def VectorLen(self, off):
+        """VectorLen retrieves the length of the vector whose offset is stored
+           at "off" in this object."""
+        N.enforce_number(off, N.UOffsetTFlags)
+
+        off += self.Pos
+        off += encode.Get(N.UOffsetTFlags.packer_type, self.Bytes, off)
+        ret = encode.Get(N.UOffsetTFlags.packer_type, self.Bytes, off)
+        return ret
+
+    def Vector(self, off):
+        """Vector retrieves the start of data of the vector whose offset is
+           stored at "off" in this object."""
+        N.enforce_number(off, N.UOffsetTFlags)
+
+        off += self.Pos
+        x = off + self.Get(N.UOffsetTFlags, off)
+        # data starts after metadata containing the vector length
+        x += N.UOffsetTFlags.bytewidth
+        return x
+
+    def Union(self, t2, off):
+        """Union initializes any Table-derived type to point to the union at
+           the given offset."""
+        assert type(t2) is Table
+        N.enforce_number(off, N.UOffsetTFlags)
+
+        off += self.Pos
+        t2.Pos = off + self.Get(N.UOffsetTFlags, off)
+        t2.Bytes = self.Bytes
+
+    def Get(self, flags, off):
+        """
+        Get retrieves a value of the type specified by `flags`  at the
+        given offset.
+        """
+        N.enforce_number(off, N.UOffsetTFlags)
+        return flags.py_type(encode.Get(flags.packer_type, self.Bytes, off))
+
+    def GetSlot(self, slot, d, validator_flags):
+        N.enforce_number(slot, N.VOffsetTFlags)
+        if validator_flags is not None:
+            N.enforce_number(d, validator_flags)
+        off = self.Offset(slot)
+        if off == 0:
+            return d
+        return self.Get(validator_flags, self.Pos + off)
+
+    def GetVOffsetTSlot(self, slot, d):
+        """
+        GetVOffsetTSlot retrieves the VOffsetT that the given vtable location
+        points to. If the vtable value is zero, the default value `d`
+        will be returned.
+        """
+
+        N.enforce_number(slot, N.VOffsetTFlags)
+        N.enforce_number(d, N.VOffsetTFlags)
+
+        off = self.Offset(slot)
+        if off == 0:
+                return d
+        return off

python/setup.py

+from setuptools import setup
+
+setup(
+    name='flatbuffers',
+    version='0.1',
+    license='BSD',
+    author='FlatBuffers Contributors',
+    author_email='me@rwinslow.com',
+    url='https://github.com/google/flatbuffers/python',
+    long_description=('Python runtime library and code generator for use with'
+                      'the Flatbuffers serialization format.'),
+    packages=['flatbuffers'],
+    include_package_data=True,
+    requires=[],
+    description=('Runtime library and code generator for use with the '
+                 'Flatbuffers serialization format.'),
+)

src/flatc.cpp

@@ -64,6 +64,10 @@ const Generator generators[] = {
     flatbuffers::GeneratorOptions::kCSharp,
     "Generate C# classes for tables/structs",
     flatbuffers::GeneralMakeRule },
+  { flatbuffers::GeneratePython,   "-p", "Python",
+    flatbuffers::GeneratorOptions::kMAX,
+    "Generate Python files for tables/structs",
+    flatbuffers::GeneralMakeRule },
 };
 
 const char *program_name = NULL;

src/idl_gen_cpp.cpp

@@ -61,7 +61,8 @@ static std::string TranslateNameSpace(const std::string &qualified_name) {
 static std::string GenTypeBasic(const Parser &parser, const Type &type,
                                 bool real_enum) {
   static const char *ctypename[] = {
-    #define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE) #CTYPE,
+    #define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE, PTYPE) \
+      #CTYPE,
       FLATBUFFERS_GEN_TYPES(FLATBUFFERS_TD)
     #undef FLATBUFFERS_TD
   };

src/idl_gen_general.cpp

@@ -196,7 +196,7 @@ static std::string FunctionStart(const LanguageParameters &lang, char upper) {
 static std::string GenTypeBasic(const LanguageParameters &lang,
                                 const Type &type) {
   static const char *gtypename[] = {
-    #define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE) \
+    #define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE, PTYPE) \
         #JTYPE, #NTYPE, #GTYPE,
       FLATBUFFERS_GEN_TYPES(FLATBUFFERS_TD)
     #undef FLATBUFFERS_TD

src/idl_gen_go.cpp

@@ -616,7 +616,8 @@ static bool SaveType(const Parser &parser, const Definition &def,
 
 static std::string GenTypeBasic(const Type &type) {
   static const char *ctypename[] = {
-    #define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE) #GTYPE,
+    #define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE, PTYPE) \
+      #GTYPE,
       FLATBUFFERS_GEN_TYPES(FLATBUFFERS_TD)
     #undef FLATBUFFERS_TD
   };

src/idl_gen_text.cpp

@@ -159,7 +159,8 @@ template<> void Print<const void *>(const void *val,
       type = type.VectorType();
       // Call PrintVector above specifically for each element type:
       switch (type.base_type) {
-        #define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE) \
+        #define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE, \
+          PTYPE) \
           case BASE_TYPE_ ## ENUM: \
             PrintVector<CTYPE>( \
               *reinterpret_cast<const Vector<CTYPE> *>(val), \
@@ -225,7 +226,8 @@ static void GenStruct(const StructDef &struct_def, const Table *table,
       OutputIdentifier(fd.name, opts, _text);
       text += ": ";
       switch (fd.value.type.base_type) {
-         #define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE) \
+         #define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE, \
+           PTYPE) \
            case BASE_TYPE_ ## ENUM: \
               GenField<CTYPE>(fd, table, struct_def.fixed, \
                               opts, indent + Indent(opts), _text); \
@@ -233,7 +235,8 @@ static void GenStruct(const StructDef &struct_def, const Table *table,
           FLATBUFFERS_GEN_TYPES_SCALAR(FLATBUFFERS_TD)
         #undef FLATBUFFERS_TD
         // Generate drop-thru case statements for all pointer types:
-        #define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE) \
+        #define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE, \
+          PTYPE) \
           case BASE_TYPE_ ## ENUM:
           FLATBUFFERS_GEN_TYPES_POINTER(FLATBUFFERS_TD)
         #undef FLATBUFFERS_TD

src/idl_parser.cpp

@@ -25,14 +25,15 @@
 namespace flatbuffers {
 
 const char *const kTypeNames[] = {
-  #define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE) IDLTYPE,
+  #define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE, PTYPE) \
+    IDLTYPE,
     FLATBUFFERS_GEN_TYPES(FLATBUFFERS_TD)
   #undef FLATBUFFERS_TD
   nullptr
 };
 
 const char kTypeSizes[] = {
-  #define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE) \
+  #define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE, PTYPE) \
       sizeof(CTYPE),
     FLATBUFFERS_GEN_TYPES(FLATBUFFERS_TD)
   #undef FLATBUFFERS_TD
@@ -96,7 +97,7 @@ enum {
   #define FLATBUFFERS_TOKEN(NAME, VALUE, STRING) kToken ## NAME = VALUE,
     FLATBUFFERS_GEN_TOKENS(FLATBUFFERS_TOKEN)
   #undef FLATBUFFERS_TOKEN
-  #define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE) \
+  #define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE, PTYPE) \
       kToken ## ENUM,
     FLATBUFFERS_GEN_TYPES(FLATBUFFERS_TD)
   #undef FLATBUFFERS_TD
@@ -107,7 +108,8 @@ static std::string TokenToString(int t) {
     #define FLATBUFFERS_TOKEN(NAME, VALUE, STRING) STRING,
       FLATBUFFERS_GEN_TOKENS(FLATBUFFERS_TOKEN)
     #undef FLATBUFFERS_TOKEN
-    #define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE) IDLTYPE,
+    #define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE, PTYPE) \
+      IDLTYPE,
       FLATBUFFERS_GEN_TYPES(FLATBUFFERS_TD)
     #undef FLATBUFFERS_TD
   };
@@ -205,7 +207,8 @@ void Parser::Next() {
           attribute_.clear();
           attribute_.append(start, cursor_);
           // First, see if it is a type keyword from the table of types:
-          #define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE) \
+          #define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE, \
+            PTYPE) \
             if (attribute_ == IDLTYPE) { \
               token_ = kToken ## ENUM; \
               return; \
@@ -580,7 +583,8 @@ uoffset_t Parser::ParseTable(const StructDef &struct_def) {
       auto field = it->second;
       if (!struct_def.sortbysize || size == SizeOf(value.type.base_type)) {
         switch (value.type.base_type) {
-          #define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE) \
+          #define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE, \
+            PTYPE) \
             case BASE_TYPE_ ## ENUM: \
               builder_.Pad(field->padding); \
               if (struct_def.fixed) { \
@@ -593,7 +597,8 @@ uoffset_t Parser::ParseTable(const StructDef &struct_def) {
               break;
             FLATBUFFERS_GEN_TYPES_SCALAR(FLATBUFFERS_TD);
           #undef FLATBUFFERS_TD
-          #define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE) \
+          #define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE, \
+            PTYPE) \
             case BASE_TYPE_ ## ENUM: \
               builder_.Pad(field->padding); \
               if (IsStruct(field->value.type)) { \
@@ -648,7 +653,7 @@ uoffset_t Parser::ParseVector(const Type &type) {
     // start at the back, since we're building the data backwards.
     auto &val = field_stack_.back().first;
     switch (val.type.base_type) {
-      #define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE) \
+      #define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE, PTYPE) \
         case BASE_TYPE_ ## ENUM: \
           if (IsStruct(val.type)) SerializeStruct(*val.type.struct_def, val); \
           else builder_.PushElement(atot<CTYPE>(val.constant.c_str())); \

tests/MyGame/Example/Any.py

+# automatically generated, do not modify
+
+# namespace: Example
+
+class Any(object):
+    NONE = 0
+    Monster = 1
+

tests/MyGame/Example/Color.py

+# automatically generated, do not modify
+
+# namespace: Example
+
+class Color(object):
+    Red = 1
+    Green = 2
+    Blue = 8
+

tests/MyGame/Example/Stat.py

+# automatically generated, do not modify
+
+# namespace: Example
+
+import flatbuffers
+
+class Stat(object):
+    __slots__ = ['_tab']
+
+    # Stat
+    def Init(self, buf, pos):
+        self._tab = flatbuffers.table.Table(buf, pos)
+
+    # Stat
+    def Id(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(4))
+        if o != 0:
+            return self._tab.String(o + self._tab.Pos)
+        return ""
+
+    # Stat
+    def Val(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(6))
+        if o != 0:
+            return self._tab.Get(flatbuffers.number_types.Int64Flags, o + self._tab.Pos)
+        return 0
+
+    # Stat
+    def Count(self):
+        o = flatbuffers.number_types.UOffsetTFlags.py_type(self._tab.Offset(8))
+        if o != 0:
+            return self._tab.Get(flatbuffers.number_types.Uint16Flags, o + self._tab.Pos)
+        return 0
+
+def StatStart(builder): builder.StartObject(3)
+def StatAddId(builder, id): builder.PrependUOffsetTRelativeSlot(0, flatbuffers.number_types.UOffsetTFlags.py_type(id), 0)
+def StatAddVal(builder, val): builder.PrependInt64Slot(1, val, 0)
+def StatAddCount(builder, count): builder.PrependUint16Slot(2, count, 0)
+def StatEnd(builder): return builder.EndObject()

tests/MyGame/Example/Test.py

+# automatically generated, do not modify
+
+# namespace: Example
+
+import flatbuffers
+
+class Test(object):
+    __slots__ = ['_tab']
+
+    # Test
+    def Init(self, buf, pos):
+        self._tab = flatbuffers.table.Table(buf, pos)
+
+    # Test
+    def A(self): return self._tab.Get(flatbuffers.number_types.Int16Flags, self._tab.Pos + flatbuffers.number_types.UOffsetTFlags.py_type(0))
+    # Test
+    def B(self): return self._tab.Get(flatbuffers.number_types.Int8Flags, self._tab.Pos + flatbuffers.number_types.UOffsetTFlags.py_type(2))
+
+def CreateTest(builder, a, b):
+    builder.Prep(2, 4)
+    builder.Pad(1)
+    builder.PrependInt8(b)
+    builder.PrependInt16(a)
+    return builder.Offset()

tests/MyGame/Example/Vec3.py

+# automatically generated, do not modify
+
+# namespace: Example
+
+import flatbuffers
+
+class Vec3(object):
+    __slots__ = ['_tab']
+
+    # Vec3
+    def Init(self, buf, pos):
+        self._tab = flatbuffers.table.Table(buf, pos)
+
+    # Vec3
+    def X(self): return self._tab.Get(flatbuffers.number_types.Float32Flags, self._tab.Pos + flatbuffers.number_types.UOffsetTFlags.py_type(0))
+    # Vec3
+    def Y(self): return self._tab.Get(flatbuffers.number_types.Float32Flags, self._tab.Pos + flatbuffers.number_types.UOffsetTFlags.py_type(4))
+    # Vec3
+    def Z(self): return self._tab.Get(flatbuffers.number_types.Float32Flags, self._tab.Pos + flatbuffers.number_types.UOffsetTFlags.py_type(8))
+    # Vec3
+    def Test1(self): return self._tab.Get(flatbuffers.number_types.Float64Flags, self._tab.Pos + flatbuffers.number_types.UOffsetTFlags.py_type(16))
+    # Vec3
+    def Test2(self): return self._tab.Get(flatbuffers.number_types.Int8Flags, self._tab.Pos + flatbuffers.number_types.UOffsetTFlags.py_type(24))
+    # Vec3
+    def Test3(self, obj):
+        obj.Init(self._tab.Bytes, self._tab.Pos + 26)
+        return obj
+
+
+def CreateVec3(builder, x, y, z, test1, test2, Test_a, Test_b):
+    builder.Prep(16, 32)
+    builder.Pad(2)
+    builder.Prep(2, 4)
+    builder.Pad(1)
+    builder.PrependInt8(Test_b)
+    builder.PrependInt16(Test_a)
+    builder.Pad(1)
+    builder.PrependInt8(test2)
+    builder.PrependFloat64(test1)
+    builder.Pad(4)
+    builder.PrependFloat32(z)
+    builder.PrependFloat32(y)
+    builder.PrependFloat32(x)
+    return builder.Offset()

tests/PythonTest.sh

+#!/bin/bash -eu
+# Copyright 2014 Google Inc. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+pushd "$(dirname $0)" >/dev/null
+test_dir="$(pwd)"
+gen_code_path=${test_dir}
+runtime_library_dir=${test_dir}/../python
+
+# Emit Python code for the example schema in the test dir:
+${test_dir}/../flatc -p -o ${gen_code_path} monster_test.fbs
+
+# Syntax: run_tests <interpreter> <benchmark vtable dedupes>
+#                   <benchmark read count> <benchmark build count>
+interpreters_tested=()
+function run_tests() {
+  if $(which ${1} >/dev/null); then
+    echo "Testing with interpreter: ${1}"
+    PYTHONDONTWRITEBYTECODE=1 \
+    JYTHONDONTWRITEBYTECODE=1 \
+    PYTHONPATH=${runtime_library_dir}:${gen_code_path} \
+    JYTHONPATH=${runtime_library_dir}:${gen_code_path} \
+    COMPARE_GENERATED_TO_GO=0 \
+    COMPARE_GENERATED_TO_JAVA=0 \
+    $1 py_test.py $2 $3 $4
+    interpreters_tested+=(${1})
+    echo
+  fi
+}
+
+# Run test suite with these interpreters. The arguments are benchmark counts.
+run_tests python2.6 100 100 100
+run_tests python2.7 100 100 100
+run_tests python3 100 100 100
+run_tests pypy 100 100 100
+
+# NOTE: We'd like to support python2.5 in the future.
+
+# NOTE: Jython 2.7.0 fails due to a bug in the stdlib `struct` library:
+#       http://bugs.jython.org/issue2188
+
+if [ ${#interpreters_tested[@]} -eq 0 ]; then
+  echo "No Python interpeters found on this system, could not run tests."
+  exit 1
+fi
+
+# Run test suite with default python intereter.
+# (If the Python program `coverage` is available, it will be run, too.
+#  Install `coverage` with `pip install coverage`.)
+if $(which coverage >/dev/null); then
+  echo 'Found coverage utility, running coverage with default Python:'
+
+  PYTHONDONTWRITEBYTECODE=1 \
+  PYTHONPATH=${runtime_library_dir}:${gen_code_path} \
+  coverage run --source=flatbuffers,MyGame py_test.py 0 0 0 > /dev/null
+
+  echo
+  cov_result=`coverage report --omit="*flatbuffers/vendor*,*py_test*" \
+              | tail -n 1 | awk ' { print $4 } '`
+  echo "Code coverage: ${cov_result}"
+else
+  echo -n "Did not find coverage utility for default Python, skipping. "
+  echo "Install with 'pip install coverage'."
+fi
+
+echo
+echo "OK: all tests passed for ${#interpreters_tested[@]} interpreters: ${interpreters_tested[@]}."

tests/test.cpp

@@ -37,7 +37,7 @@ int testing_fails = 0;
 
 template<typename T, typename U>
 void TestEq(T expval, U val, const char *exp, const char *file, int line) {
-  if (expval != val) {
+  if (U(expval) != val) {
     auto expval_str = flatbuffers::NumToString(expval);
     auto val_str = flatbuffers::NumToString(val);
     TEST_OUTPUT_LINE("TEST FAILED: %s:%d, %s (%s) != %s", file, line,