This should provide acceptable memory usage and database locking
granularity, while making subscription and transport updates large
enough for the incremental update issue to be kicked into the long
grass.
Removed awareness of the serialisation format from the protocol
component wherever possible, and added tests to ensure that the
constants defined in the protocol package's API are compatible with
the serialisation format.
authenticate each frame before parsing its contents. Each connection
starts with a tag, followed by any number of frames, each starting
with the frame number (32 bits) and payload length (16 bits), and
ending with a MAC (256 bits).
Tags have the following format: 32 bits reserved, 16 bits for the
transport ID, 32 bits for the connection number, 32 bits (set to zero
in the tag) for the frame number, and 16 bits (set to zero in the tag)
for the block number. The tag is encrypted with the tag key in
ECB mode.
Frame numbers for each connection must start from zero and must be
contiguous and strictly increasing. Each frame is encrypted with the
frame key in CTR mode, using the plaintext tag with the appropriate
frame number to initialise the counter.
The maximum frame size is 64 KiB, including header and footer. The
maximum amount of data that can be sent over a connection is 2^32
frames - roughly 2^48 bytes, or 8 terabytes, with the maximum frame
size of 64 KiB. If that isn't sufficient we can add another 16 bits to
the frame counter.
arrays, check it before allocating the buffer, and always specify the
maximum length when reading untrusted data - otherwise
CountingConsumer will reject the packet, but not before we've tried to
allocate a buffer of the specified size (up to 2 GB).
depending on the value of the first byte, so that an object's initial
tag is included in the data seen by the ObjectReader. Digests and
signatures can therefore be calculated over objects by their readers
without any risk of ambiguity.
