Skip to content

Commit 48dfc69

Browse files
rwrw
committed
Port FlatBuffers to Python.
Implement code generation and self-contained runtime library for Python. The test suite verifies: - Correctness of generated Python code by comparing output to that of the other language ports. - The exact bytes in the Builder buffer during many scenarios. - Vtable deduplication correctness. - Edge cases for table construction, via a fuzzer derived from the Go implementation. - All code is simultaneously valid in Python 2.6, 2.7, and 3.4. The test suite includes benchmarks for: - Building 'gold' data. - Parsing 'gold' data. - Deduplicating vtables. All tests pass on this author's system for the following Python implementations: - CPython 2.6.7 - CPython 2.7.8 - CPython 3.4.2 - PyPy 2.5.0 (CPython 2.7.8 compatible)
1 parent 4d213c2 commit 48dfc69

35 files changed

Lines changed: 3608 additions & 35 deletions

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,3 +48,4 @@ FlatBuffers.xcodeproj/
4848
java/.idea
4949
java/*.iml
5050
java/target
51+
**/*.pyc

‎CMakeLists.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ set(FlatBuffers_Compiler_SRCS
2424
src/idl_gen_cpp.cpp
2525
src/idl_gen_general.cpp
2626
src/idl_gen_go.cpp
27+
src/idl_gen_python.cpp
2728
src/idl_gen_text.cpp
2829
src/idl_gen_fbs.cpp
2930
src/flatc.cpp

‎docs/source/PythonUsage.md‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
# Use in Python
2+
3+
There's experimental support for reading FlatBuffers in Python. Generate
4+
code for Python with the `-p` option to `flatc`.
5+
6+
See `py_test.py` for an example. You import the generated code, read a
7+
FlatBuffer binary file into a `bytearray`, which you pass to the
8+
`GetRootAsMonster` function:
9+
10+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.py}
11+
import MyGame.Example as example
12+
import flatbuffers
13+
14+
buf = open('monster.dat', 'rb').read()
15+
buf = bytearray(buf)
16+
monster = example.GetRootAsMonster(buf, 0)
17+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
18+
19+
Now you can access values like this:
20+
21+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.py}
22+
hp = monster.Hp()
23+
pos = monster.Pos()
24+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
25+
26+
To access vectors you pass an extra index to the
27+
vector field accessor. Then a second method with the same name suffixed
28+
by `Length` let's you know the number of elements you can access:
29+
30+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.py}
31+
for i in xrange(monster.InventoryLength()):
32+
monster.Inventory(i) # do something here
33+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
34+
35+
You can also construct these buffers in Python using the functions found
36+
in the generated code, and the FlatBufferBuilder class:
37+
38+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.py}
39+
builder = flatbuffers.NewBuilder(0)
40+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
41+
42+
Create strings:
43+
44+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.py}
45+
s = builder.CreateString("MyMonster")
46+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
47+
48+
Create a table with a struct contained therein:
49+
50+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.py}
51+
example.MonsterStart(builder)
52+
example.MonsterAddPos(builder, example.CreateVec3(builder, 1.0, 2.0, 3.0, 3.0, 4, 5, 6))
53+
example.MonsterAddHp(builder, 80)
54+
example.MonsterAddName(builder, str)
55+
example.MonsterAddInventory(builder, inv)
56+
example.MonsterAddTest_Type(builder, 1)
57+
example.MonsterAddTest(builder, mon2)
58+
example.MonsterAddTest4(builder, test4s)
59+
mon = example.MonsterEnd(builder)
60+
61+
final_flatbuffer = bulder.Output()
62+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
63+
64+
Unlike C++, Python does not support table creation functions like 'createMonster()'.
65+
This is to create the buffer without
66+
using temporary object allocation (since the `Vec3` is an inline component of
67+
`Monster`, it has to be created right where it is added, whereas the name and
68+
the inventory are not inline, and **must** be created outside of the table
69+
creation sequence).
70+
Structs do have convenient methods that allow you to construct them in one call.
71+
These also have arguments for nested structs, e.g. if a struct has a field `a`
72+
and a nested struct field `b` (which has fields `c` and `d`), then the arguments
73+
will be `a`, `c` and `d`.
74+
75+
Vectors also use this start/end pattern to allow vectors of both scalar types
76+
and structs:
77+
78+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.py}
79+
example.MonsterStartInventoryVector(builder, 5)
80+
i = 4
81+
while i >= 0:
82+
builder.PrependByte(byte(i))
83+
i -= 1
84+
85+
inv = builder.EndVector(5)
86+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
87+
88+
The generated method 'StartInventoryVector' is provided as a convenience
89+
function which calls 'StartVector' with the correct element size of the vector
90+
type which in this case is 'ubyte' or 1 byte per vector element.
91+
You pass the number of elements you want to write.
92+
You write the elements backwards since the buffer
93+
is being constructed back to front. Use the correct `Prepend` call for the type,
94+
or `PrependUOffsetT` for offsets. You then pass `inv` to the corresponding
95+
`Add` call when you construct the table containing it afterwards.
96+
97+
There are `Prepend` functions for all the scalar types. You use
98+
`PrependUOffset` for any previously constructed objects (such as other tables,
99+
strings, vectors). For structs, you use the appropriate `create` function
100+
in-line, as shown above in the `Monster` example.
101+
102+
Once you're done constructing a buffer, you call `Finish` with the root object
103+
offset (`mon` in the example above). Your data now resides in Builder.Bytes.
104+
Important to note is that the real data starts at the index indicated by Head(),
105+
for Offset() bytes (this is because the buffer is constructed backwards).
106+
If you wanted to read the buffer right after creating it (using
107+
`GetRootAsMonster` above), the second argument, instead of `0` would thus
108+
also be `Head()`.
109+
110+
## Text Parsing
111+
112+
There currently is no support for parsing text (Schema's and JSON) directly
113+
from Python, though you could use the C++ parser through SWIG or ctypes. Please
114+
see the C++ documentation for more on text parsing.
115+

‎docs/source/doxyfile‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -750,6 +750,7 @@ INPUT = "FlatBuffers.md" \
750750
"CppUsage.md" \
751751
"GoUsage.md" \
752752
"JavaUsage.md" \
753+
"PythonUsage.md" \
753754
"Benchmarks.md" \
754755
"WhitePaper.md" \
755756
"Internals.md" \

‎include/flatbuffers/idl.h‎

Lines changed: 28 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -35,24 +35,24 @@ namespace flatbuffers {
3535
// Additionally, Parser::ParseType assumes bool..string is a contiguous range
3636
// of type tokens.
3737
#define FLATBUFFERS_GEN_TYPES_SCALAR(TD) \
38-
TD(NONE, "", uint8_t, byte, byte, byte) \
39-
TD(UTYPE, "", uint8_t, byte, byte, byte) /* begin scalar/int */ \
40-
TD(BOOL, "bool", uint8_t, boolean,byte, bool) \
41-
TD(CHAR, "byte", int8_t, byte, int8, sbyte) \
42-
TD(UCHAR, "ubyte", uint8_t, byte, byte, byte) \
43-
TD(SHORT, "short", int16_t, short, int16, short) \
44-
TD(USHORT, "ushort", uint16_t, short, uint16, ushort) \
45-
TD(INT, "int", int32_t, int, int32, int) \
46-
TD(UINT, "uint", uint32_t, int, uint32, uint) \
47-
TD(LONG, "long", int64_t, long, int64, long) \
48-
TD(ULONG, "ulong", uint64_t, long, uint64, ulong) /* end int */ \
49-
TD(FLOAT, "float", float, float, float32, float) /* begin float */ \
50-
TD(DOUBLE, "double", double, double, float64, double) /* end float/scalar */
38+
TD(NONE, "", uint8_t, byte, byte, byte, uint8) \
39+
TD(UTYPE, "", uint8_t, byte, byte, byte, uint8) /* begin scalar/int */ \
40+
TD(BOOL, "bool", uint8_t, boolean,byte, bool, bool) \
41+
TD(CHAR, "byte", int8_t, byte, int8, sbyte, int8) \
42+
TD(UCHAR, "ubyte", uint8_t, byte, byte, byte, uint8) \
43+
TD(SHORT, "short", int16_t, short, int16, short, int16) \
44+
TD(USHORT, "ushort", uint16_t, short, uint16, ushort, uint16) \
45+
TD(INT, "int", int32_t, int, int32, int, int32) \
46+
TD(UINT, "uint", uint32_t, int, uint32, uint, uint32) \
47+
TD(LONG, "long", int64_t, long, int64, long, int64) \
48+
TD(ULONG, "ulong", uint64_t, long, uint64, ulong, uint64) /* end int */ \
49+
TD(FLOAT, "float", float, float, float32, float, float32) /* begin float */ \
50+
TD(DOUBLE, "double", double, double, float64, double, float64) /* end float/scalar */
5151
#define FLATBUFFERS_GEN_TYPES_POINTER(TD) \
52-
TD(STRING, "string", Offset<void>, int, int, int) \
53-
TD(VECTOR, "", Offset<void>, int, int, int) \
54-
TD(STRUCT, "", Offset<void>, int, int, int) \
55-
TD(UNION, "", Offset<void>, int, int, int)
52+
TD(STRING, "string", Offset<void>, int, int, int, int) \
53+
TD(VECTOR, "", Offset<void>, int, int, int, int) \
54+
TD(STRUCT, "", Offset<void>, int, int, int, int) \
55+
TD(UNION, "", Offset<void>, int, int, int, int)
5656

5757
// The fields are:
5858
// - enum
@@ -61,12 +61,13 @@ namespace flatbuffers {
6161
// - Java type.
6262
// - Go type.
6363
// - C# / .Net type.
64+
// - Python type.
6465

6566
// using these macros, we can now write code dealing with types just once, e.g.
6667

6768
/*
6869
switch (type) {
69-
#define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE) \
70+
#define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE, PTYPE) \
7071
case BASE_TYPE_ ## ENUM: \
7172
// do something specific to CTYPE here
7273
FLATBUFFERS_GEN_TYPES(FLATBUFFERS_TD)
@@ -83,13 +84,13 @@ switch (type) {
8384
__extension__ // Stop GCC complaining about trailing comma with -Wpendantic.
8485
#endif
8586
enum BaseType {
86-
#define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE) \
87+
#define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE, PTYPE) \
8788
BASE_TYPE_ ## ENUM,
8889
FLATBUFFERS_GEN_TYPES(FLATBUFFERS_TD)
8990
#undef FLATBUFFERS_TD
9091
};
9192

92-
#define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE) \
93+
#define FLATBUFFERS_TD(ENUM, IDLTYPE, CTYPE, JTYPE, GTYPE, NTYPE, PTYPE) \
9394
static_assert(sizeof(CTYPE) <= sizeof(largest_scalar_t), \
9495
"define largest_scalar_t as " #CTYPE);
9596
FLATBUFFERS_GEN_TYPES(FLATBUFFERS_TD)
@@ -464,6 +465,13 @@ extern bool GenerateJava(const Parser &parser,
464465
const std::string &file_name,
465466
const GeneratorOptions &opts);
466467

468+
// Generate Python files from the definitions in the Parser object.
469+
// See idl_gen_python.cpp.
470+
extern bool GeneratePython(const Parser &parser,
471+
const std::string &path,
472+
const std::string &file_name,
473+
const GeneratorOptions &opts);
474+
467475
// Generate C# files from the definitions in the Parser object.
468476
// See idl_gen_csharp.cpp.
469477
extern bool GenerateCSharp(const Parser &parser,

‎python/__init__.py‎

Whitespace-only changes.

‎python/flatbuffers/__init__.py‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# Copyright 2014 Google Inc. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
from .builder import Builder
16+
from .table import Table
17+
from .compat import range_func as compat_range

0 commit comments

Comments
 (0)