kiel.protocol.messages¶

class kiel.protocol.messages.MessageSet(messages)[source]¶

Bases: object

Class representing a set of Message instances.

Kafka’s compression scheme works by taking a set of messages, compressing them with the chosen compression scheme, and then wrapping the result as the value of an envelope message, called the “message set”.

classmethod compressed(compression, msgs)[source]¶

Returns a MessageSet with the given messages optionally compressed.

If compression is used, the message set is rendered, compressed, and then a new message set is created with the raw output as a value in a single message.

If no compression is set, the returned instance will have a (<offset>, <message>) tuple for each given message, where the offset is -1.

render()[source]¶

Returns a tuple of format and data suitable for struct.pack.

Each (<offset>, <message>) tuple in self.messages is render()-ed and the output collected int a single format and data list, prefaced with a single integer denoting the size of the message set.

classmethod parse(buff, offset, size=None)[source]¶

Given a buffer and offset, returns the parsed MessageSet and offset.

Starts by determining the size of the raw payload to parse, and continuously parses the Int64 offset and Int32 size of a message then the Message itself.

If a parsed message’s flags denote compression, parse() is called recursively on the message’s value in order to unpack the compressed nested messages.

class kiel.protocol.messages.Message(**kwargs)[source]¶

Bases: kiel.protocol.part.Part

Basic Part subclass representing a single Kafka message.

Message =>
  crc => Int32
  magic => Int8
  attributes => Int8
  key => Bytes
  value => Bytes

parts = (('crc', <class 'kiel.protocol.primitives.Int32'>), ('magic', <class 'kiel.protocol.primitives.Int8'>), ('attributes', <class 'kiel.protocol.primitives.Int8'>), ('key', <class 'kiel.protocol.primitives.Bytes'>), ('value', <class 'kiel.protocol.primitives.Bytes'>))¶

render()[source]¶

Renders just like the base Part class, but with CRC32 verification.

classmethod parse(buff, offset)[source]¶

Given a buffer and offset, returns the parsed Message and new offset.

Parses via the basic Part parse method, but with added decompression support.

If a parsed message’s attributes denote that compression has been used, the value is run through the corresponding decompress() method.