How to safely transfer data when resizing publisher slots?

[mikael-s-persson]

This is either a question or a feature request. Say I want to publish messages with some dynamically-sized zero-copy thing like flatbuffers which doesn't have a known size but a "realloc" hook to reallocate the memory that it aliases for storing the fields. The obvious way to use this on top of subspace is to use "GetMessageBuffer" to allocate the memory that flatbuffer fills with data as the message is built. And then, when we reach the capacity (slot_size), we can call "GetMessageBuffer" again to get a bigger slot (and resize the channel). But, as I understand it, the previously obtained message buffer is potentially invalidated (ref-count decremented, might get unmapped or closed), and so, the reallocation would need to first copy whatever data has already been put in the old message buffer to some temporary location (heap), then call "GetMessageBuffer" with the bigger size, and finally, copy the memory back.

Is that right?

If so, it might be nice to have some ref-counted scheme similar to the subscriber's "shared-ptr" messages to keep the old message buffer alive a bit longer to avoid the double-copy.

Of course, resizing would only happen a handful of times early on, so, it's not a huge problem.

[dallison]

It's actually safe to access the old memory as it will not be unmapped until you publish the message. The call to UnmapUnusedBuffers only occurs when PublishMessageInternal is called. That's when the old buffer is unmapped if there are no references. You can call GetMessageBuffer many times with an increasing size and you can keep the old buffer pointers. The memory is guaranteed to still be valid until you publish the message.

[dallison]

Oh, on the use of flatbuffers. I recently found out that you can't tell flatbuffers where to put its data and you have to copy it when it's complete. That was an AI generated response so it might be total crap. I am not a flatbuffers guy (I wrote my own version of Blueproto called Phaser which allows you to use the protobuf IDL and do transcoding, etc (just like Blueproto but better).

So was the AI wrong? Can you tell flatbuffers to put its data in an address you allocate?

[mikael-s-persson]

Thanks for the clarification. That sounds sufficient to make our flatbuffers thing work.

The flatbuffers uses a sort of double-sided growth thing (not sure what the point of that is, probably lack of imagination by its programmers). Basically, it grows an index of fields at one end and the field values at the other (I think). Anyways, you cannot give it a buffer to use (like in blueproto), but you can give it an allocator object (see allocator.h), so, it's pretty easy to make the "alloc" call GetMessageBuffer and the "realloc" call it again and copy over the already filled-out bytes from the previous buffer.