crabbedhaloablut has quit [Write error: Broken pipe]
crabbedhaloablut has joined #rust-embedded
crabbedhaloablut has quit [Remote host closed the connection]
crabbedhaloablut has joined #rust-embedded
GenTooMan has quit [Ping timeout: 255 seconds]
GenTooMan has joined #rust-embedded
<re_irc>
<ryan-summers> Is there a serde serializer/deserializer for serializing and deserializing in binary format without trying to mess with integer sizes? Postcard is _nearly_ what I want, but it automatically encodes all ints as varints, but MQTT's spec requires that _some_ integers have defined u32/u16/u8 sizes, and others are varints, so I want to manually specify _when_ things go to varint
<re_irc>
<ryan-summers> CC James Munns
<re_irc>
<ryan-summers> I want to have a struct that repr's MQTT control packets, maybe use some serde attributes and manual serialize/deserialize implementations, and then just get a binary slice out.
<re_irc>
<MathiasKoch> I have not been able to find anything like that, but if you do i would be very interested in a ping :)
<re_irc>
<ryan-summers> Coming to the conclusion I'll likely need to do the same. Largely because of the requirements of a given protocol
<re_irc>
<ryan-summers> I _guess_ you could leverage "postcard" and then for the types that you _need_ to be u32 32-bits, you could use "[serde(with = encode_int_to_be_bytes)]" sort of thing
<re_irc>
<ryan-summers> Or encode it as 4 u8s in a row
crabbedhaloablut has quit [Remote host closed the connection]
crabbedhaloablut has joined #rust-embedded
starblue has quit [Ping timeout: 260 seconds]
starblue has joined #rust-embedded
<re_irc>
<eldruin> nikomatsakis: cc dirbaio
<re_irc>
<firefrommoonlight> ryan-summers: Can you post an example of the struct?
<re_irc>
<firefrommoonlight> Eg for reading from a buf
<re_irc>
<firefrommoonlight> Has the downside of being verbose and tough to maintain
<re_irc>
<firefrommoonlight> eg if the protocol changes
<re_irc>
<ryan-summers> Those kind of APIs are really error-prone IMO, since it's easy to calculate indices wrong or typo one of them
<re_irc>
<James Munns> Yeah, I think we're on the same page, but Postcard isn't intended for fitting other binary protocols
<re_irc>
<ryan-summers> I'm mainly struggling on finding any kind of serde serializer/deserializer that allows you to conform with a predefined wire format
<re_irc>
<James Munns> The boilerplate for serde isn't bad, I should probably make a template for "so you want to build a binary format with serde"
<re_irc>
<ryan-summers> I was thinking I could use either postcard flavors or hacking in by encoding u8s etc, but that precludes borrowing data etc.
<re_irc>
<ryan-summers> Yeah, but the boilerplate is more than what I currently have
<re_irc>
<ryan-summers> 90% of postcard is what I need, I just want to change the other 10% a little bit for my usecase
<re_irc>
<James Munns> Yeah, you can always use serde_with or a custom ser/de impl on a wrapper type to override the varint stuff
<re_irc>
<ryan-summers> Yeah, but that doesn't work when you're trying to borrow a big &[u8] slice
<re_irc>
<ryan-summers> Because it still maps to a "deserialize_bytes()" call
<re_irc>
<James Munns> Not sure what you mean? if you make a "RyanSlice" type, you can have whatever custom behavior you want
dc740 has quit [Remote host closed the connection]
<re_irc>
<James Munns> (if you define your own ser/de impl for "RyanSlice")
<re_irc>
<ryan-summers> Yeah, but when I do a custom "impl Deserialize" for it, then I still need to call "deserializer.deserialize_bytes()" to get the actual byte buffer from postcard
<re_irc>
<ryan-summers> Since at that point the "Deserializer" is generic
<re_irc>
<ryan-summers> And that forces the varint header on the binary data
<re_irc>
<ryan-summers> There's no way to get borrowed binary data _without_ a header from what I can tell
<re_irc>
<James Munns> if you know ahead of time what the size is, you should be able to borrow it as a slice of array, but maybe not borrowed, lemme look
<re_irc>
<ryan-summers> In my case, binary data does have a u16 length, but it's always u16, not varint
<re_irc>
<ryan-summers> * length header,
<re_irc>
<James Munns> Yeah, that's probably true
<re_irc>
<ryan-summers> I was thinking if it would be possible to have some higher-level "flavor" on top of the postcard deserialization/serialization process. Then, you could have the default still be varints etc. as in the v1.0.0 spec, but then if you need to, you could customize how to encode the data
<re_irc>
<ryan-summers> So basically allow the user to impl "Deserializer" and "Serializer", but use postcard as the default impls I guess?
<re_irc>
<ryan-summers> Basically just customize certain "deserialize_<TYPE>" functions
<re_irc>
<ryan-summers> But at that point it's not really even postcard anymore
<re_irc>
<James Munns> my _gut_ says this is out of scope for postcard - and you're better off forking postcard and changing the impls you want. BUT if there does end up being a way to do this that doesn't negatively impact the complexity of postcard, I'd be open to discussing it?
<re_irc>
<James Munns> yeah
<re_irc>
<James Munns> Like I said, it wouldn't be too hard to make a "serde binary template" to use if you want to make your own format. It is boilerplate, but not really bad.
<re_irc>
<ryan-summers> In general, I'm just dissatisfied with using serde for conforming with arbitrary wire specs
<re_irc>
<James Munns> yeah, that's fair
<re_irc>
<James Munns> But I mean, for "an existing wire spec", you really need two things:
<re_irc>
- How that gets encoded/decoded
<re_irc>
- The structure of the data
<re_irc>
<James Munns> serde only really handles the latter
<re_irc>
<James Munns> whereas serde is meant to model ANY rust type to a given format
<re_irc>
<James Munns> I get the desire to use serde as a base though, it makes so many things so much easier :D
<re_irc>
<ryan-summers> I mean I think I want to use serde purely because its the defacto rust standard. however, comparing what I have to what I would have with a custom serde impl, the custom serde stuff looks far less maintainable
<re_irc>
<ryan-summers> Which is generally disheartening to see
<re_irc>
<firefrommoonlight> Of note: While I agree that manually parsing and writing a buffer is error-prone, the nice part is it's self-contained and universal. Ie, you don't need to know how to apply a library that might have a changing API, and can guarantee it's serializing in the correct format. I think weather you should reach for an abstraction like that depends on how much data you're serializing. Ie QCing indices on a small data...
<re_irc>
... chunk is feasible; doing so on a large one is not. So, you need to find the line
<re_irc>
<firefrommoonlight> * whether
<re_irc>
<James Munns> Enough people ask me how to get postcard to fit into XYZ format, there's probably a niche for a "serde based binary protocol toolkit", if that makes sense
<re_irc>
<firefrommoonlight> You can also use a helper fn that iterates appropriately if things get too verbose. But I would like an abstraction for bigger data structs; haven't found it yet
<re_irc>
<ryan-summers> Just reads 4 bytes, parses it as a u32, then bumps indices etc.
<re_irc>
<ryan-summers> It's basically serde, without the rest of the unneeded boilerplate
<re_irc>
<firefrommoonlight> Also of note, if you're dealing with integers, things get a lot simpler. Ie, here's 2 examples of serialization code (Oppposite of the sort I posted above. Notice how the one with floats is more verbose:
<re_irc>
impl From<&ChannelData> for [u8; CONTROLS_SIZE] {
<re_irc>
/// 4 f32s x 4 = 16, plus 2 u8s for arm and input mode.
<re_irc>
<James Munns> brb, making lunch. Honestly, if y'all can write up the kind of "degrees of freedom" you actually want, I think I could probably knock it out.
<re_irc>
<firefrommoonlight> * integers and enums,
<re_irc>
<James Munns> its not postcard, but a runtime/comptime configurable ser/de impl would be useful
<re_irc>
<James Munns> e.g. "choose endianness or whatever or whatever" for each kind of data
<re_irc>
<James Munns> "do you want your slices prepended with a u8/u16/u32/varint?"
<re_irc>
<ryan-summers> Choose endianness, and also specify potential custom layout (e.g. for a slice, maybe varint header, maybe u32 header)
<re_irc>
<firefrommoonlight> Are you thinking a proc-macro decorator?
<re_irc>
<ryan-summers> For MQTT, there's also properties, where the thing is <prop-id-varint><datatype based on prop-id>
dc740 has joined #rust-embedded
<re_irc>
<James Munns> firefrommoonlight: I tend to avoid proc macros, but I'd be happy to set it up in a way that would work with one, if someone else writes it :D
<re_irc>
<firefrommoonlight> I think there are a number of ways you could set up the API, but it would need to handle endianness as ryan-summer said, have sensible defaults, and customization. Support for spaces. Support for re-aligning indices
<re_irc>
<ryan-summers> Proc macros end up just being a ton of work and maintenance IMO
<re_irc>
<ryan-summers> Useful in some cases, but avoid when possible
<re_irc>
<firefrommoonlight> I would use a ser/deser lib in a number of projects if I found one I liked; currently using code like I posted above
<re_irc>
<firefrommoonlight> ryan-summers: Concur. Nice from user perspective; not from maintainer
<re_irc>
<ryan-summers> Honestly, I _think_ half the problem would be solved if "serde" had default "Error::NotImplemented" return codes for all the "deserialize_X" types
<re_irc>
<ryan-summers> * functions
<re_irc>
<ryan-summers> Then a user can build a serializer with just the types they care about
starblue has quit [Ping timeout: 268 seconds]
starblue has joined #rust-embedded
brazuca has joined #rust-embedded
<re_irc>
<James Munns> I'll think on it. I don't need it for anything, but seems like a fun experiment :)
<re_irc>
<firefrommoonlight> I'm just open to a less tediou way
<re_irc>
<firefrommoonlight> I do a lot of this stuff, and the code has a lot of inertia
<re_irc>
<firefrommoonlight> Ie if I want to change the protocol, I have to be careful the indices don't get messed up etc
<re_irc>
<firefrommoonlight> Desired, not required
<re_irc>
<diondokter> Maybe a weird question...
<re_irc>
Say I have two binaries on the same device that live in distinct memory regions. They could be a bootloader and application.
<re_irc>
Now, in one of the binaries I have an "extern "C"" function that I want to call from the other.
<re_irc>
I could invent a function pointer (if I got the address right) and call that and that should work.
<re_irc>
<diondokter> The worst part is that, ideally, this function calling can go both ways... So binary 1 can call binary 2 and vice-versa
<re_irc>
<diondokter> Btw, I'm doing cursed things so cursed solutions are also welcome :D
<re_irc>
<mabez> diondokter: Yes, but it might get tricky if you routinely change binary 1 (the one you want to call into). In esp32's there is built in ROM code will many helpful functions ready to use to save on code space, to use these functions we tell the linker there is a function at address X and it just works. You can see the use of these functions in the esp32c3 flash loader for probe-rs:...
<re_irc>
<mabez> Thing is, this code is burned into the chip at the factory and wont ever change, if you need to change binary 1 it will most likely invalidate the address used in binary 2
<re_irc>
<diondokter> Right yeah
<re_irc>
<diondokter> So both binaries will change during development
<re_irc>
<korken89> We did something similar once, what we did was to have a specific address hold the list to specific methods
<re_irc>
<diondokter> That was one path I was thinking about yeah
<re_irc>
<diondokter> Like a sort of vector table
<re_irc>
<mabez> In which case you could _probably_ hack something together to extract the addresses from each binary and pass to each linker
<re_irc>
<korken89> diondokter: Exactly like this is how we thought about it
<re_irc>
<mabez> Actually, if you go done with route and each binary can call into the other, you may get into a chicken and egg situation :D
<re_irc>
<diondokter> mabez: One issue I see there is that it's a circle. Binary 1 can't compile because binary 2 isn't compiled yet and no addresses are known
<re_irc>
<diondokter> Haha yeah
<re_irc>
<korken89> Ah I see one difference
<re_irc>
<mabez> Yes exactly, I'd go for the list route I think!
<re_irc>
<korken89> We made like a "soft device", so we only reflashed this part and it had the "vector table"
<re_irc>
<korken89> You want to look into the other binary
<re_irc>
<korken89> Hmm I see, that's more difficult indeed
<re_irc>
<diondokter> Yep
<re_irc>
<diondokter> This has to do with trustzone and everything is annoying :P
<re_irc>
<korken89> D:
<re_irc>
<diondokter> Ok, thanks for the tips! I'll try to persue the vector table idea.
<re_irc>
It's easy to hardcode one of the vectors so binary 2 can initialize its table
<re_irc>
<diondokter> I wish Rust had post-build scripts as well. Would make life a bit easier :D
<re_irc>
<James Munns> Yeah, I'd say either:
<re_irc>
- Use the linker script to build a vector table
<re_irc>
- Have a shared "vtable" struct that gets built by bin A, and a pointer to it is placed by A or passed to B
<re_irc>
The latter is close to what Neotron OS does, IIRC
<re_irc>
<James Munns> (e.g. the BIOS passes a vtable to the firmware)
<re_irc>
<diondokter> Right, that seems reasonable
<re_irc>
<mabez> Actually, if you go down this route and each binary can call into the other, you may get into a chicken and egg situation :D
<re_irc>
<James Munns> but you could have a shared linker script snippet
<re_irc>
<diondokter> Though there'd still need to be something that lets the application know which function corresponds to which index
<re_irc>
<James Munns> where you define all the function in the table in A, and use them in B.
<re_irc>
<boondocker> having some fun with LiteX, writing a kernel from the ground up, and custom Migen peripherals (in this case a memory -> memory DMA controller)