Binary Serialization with Google Protocol Buffers
April 25, 2017 Erik van Bilsen
April 25, 2017 Erik van Bilsen
[SHOWTOGROUPS=4,20]
Google’s Protocol Buffers are a flexible, compact and extensible mechanism for serializing structured data. We share our implementation that makes it easy to serialize Delphi records to a binary format that is 100% compatible with the Protocol Buffers specification.
Want to go straight to the source? You can find it on GitHub in our Для просмотра ссылки Войдиили Зарегистрируйся repository in the single unit Grijjy.ProtocolBuffers.
Protocol Buffers?
In Google’s own words: “Protocol buffers are Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler.”
You can use it is a binary interchange format to send data over the wire, to communicate with 3rd party applications that support Protocol Buffers, or even to create a light weight but extensible file format.
Protocol Buffers are:
или Зарегистрируйся, which can be used (among other things) to convert JSON to BSON (binary JSON) and vice versa. BSON is good for representing JSON data in a fast-to-parse format, but it is not designed to be compact. Also, formats like JSON and XML are good for representing unstructured data. Protocol Buffers require that your data is structured (although it is flexible enough to update the structure in the future).
At Для просмотра ссылки Войдиили Зарегистрируйся, we use Protocol Buffers to transmit data between our BAAS and our (mobile) apps.
To get a quick taste, this is all that is required to serialize a record:
Want to know more? Keep reading!
Default Implementation
Google designed the Protocol Buffers specification with language-neutrality in mind. To achieve this goal, they developed a language-independent protocol definition format to specify how information must be serialized. For example, to serialize person data using the example above, you would have to create a protocol definition file (.proto file) that could look like this:
It describes a message, which is just a collection of fields, where each field is uniquely identified with an integer Tag.
The next step would be to use a “protocol buffer compiler” to compile this file and generate source code for a specific language. This automatically generated source code can then be used to serialize and deserialize data in protocol buffer format.
To use protocol buffers with Delphi, this means that you would need a specific protocol buffer compiler that generates Delphi source code. There are some open source projects out there that do just this.
We use a different approach though…
Alternative Implementation
At Grijjy, we love the Protocol Buffer format and the efficiency and extensibility it brings. But the need for separate protocol definition files and a protocol compiler: not so much. It is yet another “language” for you to learn and an additional step in your build process.
We think we can achieve the same goal using the language you already know, by defining the protocol as regular Delphi records, decorated with attributes to customize behavior. For example:
These are plain vanilla Delphi records with just one difference: Each field must be preceded with a Serialize attribute with a single integer tag parameter that uniquely identifies the field.
In contrast to XML and JSON, Protocol Buffers use numeric tags instead of strings to associate values with corresponding fields. This allows for more compact serialization since integer values take up less space than string values.
The tags only need to be unique within the record. An exception will be raised when a record contains duplicate tags. Tags don’t have to be (and should not be) globally unique. In the example above, tag number 1 is used to identify both the Number field of the TPhoneNumber record and the Name field of the TPerson record. Even though the TPerson record also has a field of type TPhoneNumber, this doesn’t pose a problem.
Tags start at 1 and must be positive. You should reserve tags 1-15 for the most common fields, since these tags are stored most efficiently (using only a single byte). (In case you are wondering: tags 16-2047 use two bytes and other tags take more bytes).
Records are serialized in an extensible way. You can add, delete and reorder fields without breaking compatibility with older bit streams. However, you should never change the tag or data type of a field once bit streams are already “published”.
Our engine uses Run Time Type Information (RTTI) to determine the binary format from these record definitions, in a similar way as our Для просмотра ссылки Войдиили Зарегистрируйся. It does this in an efficient way however, so serialization and deserialization is pretty fast.
Supported Data Types
You can use a wide variety of Delphi data types for your serializable fields:
Sometimes, you can have integer data that contains random values across the entire 32-bit or 64-bit range. Common examples of these are CRC or Hash values and time stamps. In those cases, it is more efficient to store these integers as fixed 32-bit or 64-bit values. You can do this by declaring the field as one of 4 fixed integer types:
Serializing is very easy. You just fill your record with the values you want to serialize and call:
This is a generic method with a type parameter that must match the type of the record you are serializing.
Since Delphi is able to infer the generic type from the first parameter, you can also write this a little bit shorter:
You can serialize to a file, stream or TBytes array.
Deserializing is equally simple:
Because all fields in a record are optional, some fields may not be in the stream. To prevent the record from having unitialized fields after deserialization, the record is cleared before it is deserialized (that is, all fields are set to 0 or nil).
You can also provide your own means of initializing the record with default values. To do that, you have to add a parameterless Initialize procedure to your record. Then, the deserialization process will call that routine after clearing the record (so it will still clear any fields you don’t initialize yourself).
Changing the Protocol
The protocol you define is not set in stone. If it no longer fits you needs, but you still need to be able to read data in an older format, then you don’t have find some way to import older formats and convert them to a new format. You can just update your protocol definition (attributed records) without breaking existing code as long as you follow these few rules:
Go give it a try. Head over to GitHub to pull our Для просмотра ссылки Войдиили Зарегистрируйся repository. Among all the other goodies in there you will find the Grijjy.ProtocolBuffers unit. You can also check out the units tests (in the UnitTests sub-directory) for more examples of how to use the engine. There is Для просмотра ссылки Войди или Зарегистрируйся as well.
[/SHOWTOGROUPS]
Google’s Protocol Buffers are a flexible, compact and extensible mechanism for serializing structured data. We share our implementation that makes it easy to serialize Delphi records to a binary format that is 100% compatible with the Protocol Buffers specification.
Want to go straight to the source? You can find it on GitHub in our Для просмотра ссылки Войди
Protocol Buffers?
In Google’s own words: “Protocol buffers are Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler.”
You can use it is a binary interchange format to send data over the wire, to communicate with 3rd party applications that support Protocol Buffers, or even to create a light weight but extensible file format.
Protocol Buffers are:
- Flexible and extensible: You can update your data structure (or protocol) without breaking deployed programs that use an “older” format.
- Compact: The serialized format is compact, making it ideal for transfer over (wireless) networks. The output is 3 to 10 times smaller than corresponding data in XML or JSON format.
- Fast to parse and generate: The binary format can be generated and parsed very quickly. According to Google, it can be 20 to 100 times faster to parse compared to XML.
At Для просмотра ссылки Войди
To get a quick taste, this is all that is required to serialize a record:
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | type TPerson = record [Serialize(1)] Name: String; [Serialize(2)] Id: Integer; [Serialize(3)] Email: String; end; procedure SimpleSerialization; var Person: TPerson; Data: TBytes; begin Person.Name := 'John Doe'; Person.Id := 1234; Person.Email := '[email protected]'; Data := TgoProtocolBuffer.Serialize(Person); end; |
Want to know more? Keep reading!
Default Implementation
Google designed the Protocol Buffers specification with language-neutrality in mind. To achieve this goal, they developed a language-independent protocol definition format to specify how information must be serialized. For example, to serialize person data using the example above, you would have to create a protocol definition file (.proto file) that could look like this:
| 1 2 3 4 5 | message Person { required string name = 1; required int32 id = 2; optional string email = 3; } |
It describes a message, which is just a collection of fields, where each field is uniquely identified with an integer Tag.
The next step would be to use a “protocol buffer compiler” to compile this file and generate source code for a specific language. This automatically generated source code can then be used to serialize and deserialize data in protocol buffer format.
To use protocol buffers with Delphi, this means that you would need a specific protocol buffer compiler that generates Delphi source code. There are some open source projects out there that do just this.
We use a different approach though…
Alternative Implementation
At Grijjy, we love the Protocol Buffer format and the efficiency and extensibility it brings. But the need for separate protocol definition files and a protocol compiler: not so much. It is yet another “language” for you to learn and an additional step in your build process.
We think we can achieve the same goal using the language you already know, by defining the protocol as regular Delphi records, decorated with attributes to customize behavior. For example:
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | type TPhoneType = (Mobile, Home, Work); type TPhoneNumber = record public [Serialize(1)] Number: String; [Serialize(2)] PhoneType: TPhoneType; public procedure Initialize; end; type TPerson = record public [Serialize(1)] Name: String; [Serialize(2)] Id: Integer; [Serialize(3)] Email: String; [Serialize(4)] MainPhone: TPhoneNumber; [Serialize(5)] OtherPhones: TArray<TPhoneNumber>; public procedure Initialize; end; |
These are plain vanilla Delphi records with just one difference: Each field must be preceded with a Serialize attribute with a single integer tag parameter that uniquely identifies the field.
In contrast to XML and JSON, Protocol Buffers use numeric tags instead of strings to associate values with corresponding fields. This allows for more compact serialization since integer values take up less space than string values.
The tags only need to be unique within the record. An exception will be raised when a record contains duplicate tags. Tags don’t have to be (and should not be) globally unique. In the example above, tag number 1 is used to identify both the Number field of the TPhoneNumber record and the Name field of the TPerson record. Even though the TPerson record also has a field of type TPhoneNumber, this doesn’t pose a problem.
Tags start at 1 and must be positive. You should reserve tags 1-15 for the most common fields, since these tags are stored most efficiently (using only a single byte). (In case you are wondering: tags 16-2047 use two bytes and other tags take more bytes).
Records are serialized in an extensible way. You can add, delete and reorder fields without breaking compatibility with older bit streams. However, you should never change the tag or data type of a field once bit streams are already “published”.
Our engine uses Run Time Type Information (RTTI) to determine the binary format from these record definitions, in a similar way as our Для просмотра ссылки Войди
Supported Data Types
You can use a wide variety of Delphi data types for your serializable fields:
- UInt8 (Byte), UInt16 (Word), UInt32 (Longword/Cardinal), UInt64.
- Int8 (Shortint), Int16 (Smallint), Int32 (Longint/Integer), Int64.
- Single, Double.
- Boolean.
- Enumerated types, as long as the type does not contain any explicitly assigned values (since Delphi does not provide RTTI for these).
- Records (that is, your field can be of another record type).
- Strings (only Unicode strings).
- TBytes (for raw binary data).
- 1-dimensional dynamic arrays (TArray<>) of the types described above.
Sometimes, you can have integer data that contains random values across the entire 32-bit or 64-bit range. Common examples of these are CRC or Hash values and time stamps. In those cases, it is more efficient to store these integers as fixed 32-bit or 64-bit values. You can do this by declaring the field as one of 4 fixed integer types:
- FixedInt32, FixedUInt32
- FixedInt64, FixedUInt64
All other data types can not be used for serializable fields. An exception will be raised when an unsupported data type is encountered. However, you can still use these types for regular (non-serializable) fields. In particular, the following types are not supported:
- Extended, Comp, Currency.
- Class, Object, Interface.
- Enumerated types with explicitly assigned values.
- AnsiString, RawByteString, UTF8String, UCS4String etc.
- Static arrays.
- Multi-dimensional dynamic arrays.
Serializing is very easy. You just fill your record with the values you want to serialize and call:
| 1 | TgoProtocolBuffer.Serialize<TPerson>(MyPerson, 'Person.dat'); |
This is a generic method with a type parameter that must match the type of the record you are serializing.
Since Delphi is able to infer the generic type from the first parameter, you can also write this a little bit shorter:
| 1 | TgoProtocolBuffer.Serialize(MyPerson, 'Person.dat'); |
You can serialize to a file, stream or TBytes array.
Deserializing is equally simple:
| 1 | TgoProtocolBuffer.Deserialize(MyPerson, 'Person.dat'); |
Because all fields in a record are optional, some fields may not be in the stream. To prevent the record from having unitialized fields after deserialization, the record is cleared before it is deserialized (that is, all fields are set to 0 or nil).
You can also provide your own means of initializing the record with default values. To do that, you have to add a parameterless Initialize procedure to your record. Then, the deserialization process will call that routine after clearing the record (so it will still clear any fields you don’t initialize yourself).
Changing the Protocol
The protocol you define is not set in stone. If it no longer fits you needs, but you still need to be able to read data in an older format, then you don’t have find some way to import older formats and convert them to a new format. You can just update your protocol definition (attributed records) without breaking existing code as long as you follow these few rules:
- Don’t change the tag values for existing fields.
- Don’t change the data types for existing fields.
- You can always reorder fields.
- You can always add new fields with new tag values.
- You can also remove fields you don’t need anymore. Just don’t reuse its tag value for any new fields you add.
Go give it a try. Head over to GitHub to pull our Для просмотра ссылки Войди
[/SHOWTOGROUPS]
