[RFC] Standardize binary serialization crate when precise control over layout is needed

[RobertZ2011]

Background

Converting between a series of bytes and a struct is a commonly used feature in embedded development. Usually, the bytes have a specific format and layout dictated by a datasheet or spec. It's possible to do this manually with a combination of [repr(C)], and unsafe code, but it's not very ergonomic. There are numerous Rust crates that provide this functionality with a nice, safe API. Type-C code, in particular, currently uses bincode for this functionality, but it's marked as no longer maintained on Github, and wasn't ideal for the task. We want to migrate away and standardize on a new crate.

Proposal

This RFC proposes zerocopy as the standard crate. This crate provides traits like FromBytes, FromZeros, and TryFromBytes. These traits and their associated derive macros should provide the needed functionality. Zerocopy also supports endianness, which is desired to allow ODP to support a wide range of hardware. Endianness is also a property of the struct itself and I think that matches the typical use case better.

Alternatives Considered

• Bytemuck is very similar to zerocopy in the traits and API it provides but doesn't have support for endianness.
• Staying with Bincode is another option, but its default behavior is to serialize into its own binary format and custom code is frequently needed to match the required layout.
• Binary_serde is a strong contender. However, zerocopy is focused on this specific use case, while serde is a more general serialization framework. As a result, Zerocopy provides a richer set of traits that are better suited for binary data. Endianness is provided as an argument to decode/encode with binary_serde instead of being a property of the struct itself. binary_serde doesn't support dynamically sized types like &[T]. This ultimately makes it partially incompatible with the broader serde environment. Code would have to be specifically written for binary_serde, negating some of the benefit of using serde.

[kurtjd]

I'm on board with zerocopy. Though there are a few things I found to be a minor nuisance in this experiment: #678

Mainly the fact that zerocopy doesn't work well with unaligned fields in a struct. If you use #repr(packed) you run into potential issues if you try to take references to any of the structs fields in Rust code. The alternative is to use the provided U<N> types (which are also needed for endianness support anyway) but it bothers me a bit that this implementation detail leaks into the public interface of the types (unless we make fields private and only provide getter and setter methods which isn't great either).

I understand why these limitations are there though. If we want a struct in packed byte format without copying then I don't think there is any avoiding these.

[asasine]

I suggest bytemuck, primarily because it require an explicit repr(C), which avoids ambiguity that zerocopy glazes over.

#[derive(zerocopy::FromBytes)] does not require an explicit representation. While this is safe when roundtripping bytes to Rust types and back (assuming source and target are both Rust and the same versions of the Rust compiler), what about when interacting with external systems that are [explicitly or implicitly] using repr(C)? What should the value of padding bytes be serialized to? Their exact values aren't necessary for sound deserialization to the same repr(Rust) type, those padding bytes will leave the system and expose implementation details, and potentially having unintended side effects (e.g., writing to reserved bits in a register). repr(Rust) does not make many guarantees on the actual layout of bytes, and it may change between versions of the compiler; that lack of specificity is the exact opposite of what I want in a binary serialization format. bytemuck::bytes_of() requires NoUninit, whose #[derive] requires an explicit repr(C). You know exactly what you're gonna get.

[kurtjd]

Haven't looked into bytemuck much but how ergonomic is it for packing fields (as in, removing padding)? At least currently on our end for our comms we expect fields to be serialized into packed bytes with no padding. As I touched on above zerocopy doesn't do a great job at this but if bytemuck can do better that would be interesting.

[RobertZ2011]

bytemuck::bytes_of() requires NoUninit, whose #[derive] requires an explicit repr(C). You know exactly what you're gonna get.

@asasine That's a good point. I did a little more digging and bytemuck recommends pack1 to provide endian-aware wrapper types. This is the same thing that zerocopy does, it just includes those in the crate itself. I'd be in favor of using that combination over zerocopy.

If we want a struct in packed binary format without copying then I don't think there is any avoiding these.

@kurtjd I agree. We can't really avoid this because the presence of padding can open up its own can of worms, so we'd want to use packed structs anyway. Though I don't think the endian-aware wrapper types are really leaking any implementation details. At this level endianness is an important concern and I like that the wrapper types can be used to bring it front and center. When using bincode, it always felt weird to pass the endianness as an argument when that was fixed by the spec or datasheet.

[kurtjd]

bytemuck::bytes_of() requires NoUninit, whose #[derive] requires an explicit repr(C). You know exactly what you're gonna get.

@asasine That's a good point. I did a little more digging and bytemuck recommends pack1 to provide endian-aware wrapper types. This is the same thing that zerocopy does, it just includes those in the crate itself. I'd be in favor of using that combination over zerocopy.

If we want a struct in packed binary format without copying then I don't think there is any avoiding these.

@kurtjd I agree. We can't really avoid this because the presence of padding can open up its own can of worms, so we'd want to use packed structs anyway. Though I don't think the endian-aware wrapper types are really leaking any implementation details. At this level endianness is an important concern and I like that the wrapper types can be used to bring it front and center. When using bincode, it always felt weird to pass the endianness as an argument when that was fixed by the spec or datasheet.

To clarify what I meant by leaking implementation details, one of my goals is to try to make the ODP service message types reusable so other applications can use them. As an example, in the ec-test-app I reuse the ODP message types so I can just fill out the fields (https://github.com/kurtjd/ec-test-app/blob/291ab9fd870c2a2c51558967fbd9cb93baa0405e/rust/src/serial.rs#L286) and then call the serialize method (https://github.com/kurtjd/ec-test-app/blob/291ab9fd870c2a2c51558967fbd9cb93baa0405e/rust/src/serial.rs#L72).

When I first experimented with zerocopy for these, the ec-test-app then had to know about zerocopy in order to supply the correct field values (since the fields are no longer just plain u8, u32, etc, but zerocopy::U32, zerocopy::U8, etc). I don't think in this case it makes sense for the ec-test-app to be concerned with endianness, padding, alignment, etc. This can be mitigated though perhaps by creating a constructor that takes field values in regular types than converts to zerocopy types under the hood.