<re_irc>
<@firefrommoonlight:matrix.org> Maybe I should try Clion. I use PyCharm with Rust, and it slows down to a crawl or freezes periodically
<re_irc>
<@firefrommoonlight:matrix.org> I haven't tested thoroughly, but it seems to be something with the Rust plugin, or the way I'm using it specifically, compared to when I'm using it with Python or JS
<re_irc>
<@firefrommoonlight:matrix.org> I use VsCode for one-off files (as opposed to multi-file projects), and performance aside, IntelliJ Rust is so worth it
troth has quit [Ping timeout: 256 seconds]
troth has joined #rust-embedded
troth has quit [Ping timeout: 260 seconds]
troth has joined #rust-embedded
troth has quit [Ping timeout: 256 seconds]
troth has joined #rust-embedded
starblue has quit [Ping timeout: 260 seconds]
starblue has joined #rust-embedded
troth has quit [Ping timeout: 260 seconds]
troth has joined #rust-embedded
fabic_ has joined #rust-embedded
<re_irc>
<@xnorman:matrix.org> I'm trying to persist some data to/from a flash memory and I just added an Option to my data and now I'm running into issues.
<re_irc>
<@xnorman:matrix.org> I was serializing with postcard and the slice flavor, but I'm seeing that the size of the serialized data isn't fixed.. which screws up my storage algorithm because I have several "banks" of storage (all of the same type) and i want to be able to store/retrieve data from any of these banks.
<re_irc>
<@xnorman:matrix.org> So I've decided to try just transmuting my data struct into a slice and storing it into flash and reading it back out.. but i get segfaults cloning an Option<u8> on the data read out of flash
<re_irc>
<@xnorman:matrix.org> Should I be able to read/write to flash this way?
<re_irc>
<@xnorman:matrix.org> read/write worked well with postcard before I added Option items to the data I was serializing
PyroPeter has quit [Ping timeout: 240 seconds]
PyroPeter has joined #rust-embedded
troth has quit [Ping timeout: 246 seconds]
<re_irc>
<@xnorman:matrix.org> I figure I must be screwing something up in my transmute after I read from the flash
troth has joined #rust-embedded
vancz has quit [*.net *.split]
Amanieu has quit [*.net *.split]
Ekho has quit [*.net *.split]
mrkajetanp has quit [*.net *.split]
sauce has quit [*.net *.split]
sauce has joined #rust-embedded
Amanieu has joined #rust-embedded
vancz has joined #rust-embedded
mrkajetanp has joined #rust-embedded
Ekho has joined #rust-embedded
troth has quit [Ping timeout: 260 seconds]
troth has joined #rust-embedded
fabic_ has quit [Ping timeout: 260 seconds]
<re_irc>
<@ryan-summers:matrix.org> Most embedded serialization tools don't support Rust's complex types, like non-simple enums and options
<re_irc>
<@ryan-summers:matrix.org> E.g. how should you serialize `None` in a logical manner?
<Lumpio->
They don't?
<re_irc>
<@ryan-summers:matrix.org> serde-json-core supports Options (since JS has a concept of null), but doesn't support complex enums beyond unit types
<Lumpio->
xnorman: What are you using transmute for with postcard?
<re_irc>
<@ryan-summers:matrix.org> Since most other languages don't have a concept of complex enums
SanchayanMaity has quit [Ping timeout: 265 seconds]
<re_irc>
<@xnorman:matrix.org> irc_libera_lumpio-:psion.agg.io: I wasn't.. I ditched postcard because of the variable size of serialized data
<re_irc>
<@xnorman:matrix.org> so i was just writing into flash by converting my struct into a slice of bytes.. and then reading out of flash by reading into a slice of bytes and then transmuting slice.as_ptr() into a reference to my struct type, then cloning that. the clone is where I'm seeing the segfault
<re_irc>
<@dngrs:matrix.org> xnorman:matrix.org: as in, unknown upper bound on the MCU side, hence not straightforward to use with e.g. `heapless`?
<re_irc>
<@jamesmunns:beeper.com> One thing to note Alex Norman, Rust structs don't have stable order, which means struct field order and padding can change on any compile
<re_irc>
<@jamesmunns:beeper.com> If you want to just "memcopy to flash", you'd probably need to at least use `#[repr(C)]` on all your types, and potentially not use any rust-specific types, like enums, etc.
<re_irc>
<@xnorman:matrix.org> dngrs:matrix.org: yeah, well mostly because I want to store an "array" of them in flash.. so i want to compute the maximum size so i know what addresses to use for consecutive writes (padded to sector size)
<re_irc>
<@xnorman:matrix.org> I am using `repr(C)` but can I not use option in there?
<re_irc>
<@jamesmunns:beeper.com> I don't believe so, but I could be wrong.
<re_irc>
<@jamesmunns:beeper.com> You certainly could stagger your postcard messages, e.g. if they are typically up to 100 bytes, you could stagger them at 256 byte intervals per instance
<re_irc>
<@jamesmunns:beeper.com> Like dngrs (spookyvisiongithub) said, it can be possible to cap the maximum size, typically by using fixed-sized collections like heapless
<re_irc>
<@xnorman:matrix.org> okay, so to use option and enums i need some sort of serialization? thats fine, i guess I need to figure out the max size of the serialized data? I was wondering if postcard had a flavor that had a consistent size but I didn't see that..
<re_irc>
<@xnorman:matrix.org> yeah, i guess i just don't want to guess.. i want to compute the max size :)
<re_irc>
<@jamesmunns:beeper.com> I don't think there is one, though I've had some chats with korken89 on how to calculate the maximum fixed size
<re_irc>
<@jamesmunns:beeper.com> I sort of know how I'd do it, but haven't needed it enough to implement it :)
<re_irc>
<@jamesmunns:beeper.com> (basically, make a `#[derive(PostcardSized)]` proc macro, that calculates the size recursively)
<re_irc>
<@jamesmunns:beeper.com> so you could do something like:
<re_irc>
<@jamesmunns:beeper.com> ```rust
<re_irc>
<@jamesmunns:beeper.com> let max_size = MyPostCardStruct::get_max_size();
<re_irc>
<@xnorman:matrix.org> maybe i' over thinking it though.. i have a lot of flash, i could just use something like double or quadruple the size of my struct?
<re_irc>
<@jamesmunns:beeper.com> Yeah, that'll probably work as long as you don't expand it :)
<re_irc>
<@jamesmunns:beeper.com> you could always write a host-side unit test of a "very full" struct, and make sure the serialized size is <= your capacity
<re_irc>
<@xnorman:matrix.org> yeah, i'm also storing a 'version' header.. a uuid encoded as 16 bytes so i shouldn't accidentally read bad data
<re_irc>
<@jamesmunns:beeper.com> Yeah, as long as your start position is right, postcard will only "take" as many bytes as it needs from the buffer
<re_irc>
<@xnorman:matrix.org> cool on the 'very full'.. should it be much different between arm and x86_64?
<re_irc>
<@jamesmunns:beeper.com> In general: no. But also, I would not suggest using `usize` or `isize` in your serialized types
<re_irc>
<@jamesmunns:beeper.com> then there should be no difference at all.
<re_irc>
<@xnorman:matrix.org> ahh, good point! if i want to send the data between this MCU and another host, i shouldn't use usize/isize and.. they will have to agree on the encoding :) great!
<re_irc>
<@xnorman:matrix.org> .. eventually i want to be able to send it to my host computer so that i can update the dataschema and send it back after an upgrade.. but that is later on
<re_irc>
<@jamesmunns:beeper.com> Makes sense!
<re_irc>
<@dngrs:matrix.org> jamesmunns:beeper.com: ooooooh. that's a subtle but probably important design tip. (I have some structs where I store "index into some vec"… what's the canonical solution here, just cast to `u32`? Or some fancier data structure where I avoid even talking about indexed access at all?)
<re_irc>
<@xnorman:matrix.org> thanks all for the help!!!
<re_irc>
<@jamesmunns:beeper.com> proooooobably u32 will be fine, unless it isn't
<re_irc>
<@jamesmunns:beeper.com> sorta up to you on how many items you need to index :)
<re_irc>
<@jamesmunns:beeper.com> Unfortunately serde doesn't expose `usize` or `isize` as serialization targets - it silently converts them to `u32` or `u64` depending on your target
<re_irc>
<@jamesmunns:beeper.com> so I can't intelligently varint encode them like I do for array lengths (for slices, or things like Vecs)
<re_irc>
<@jamesmunns:beeper.com> So I either need to:
<re_irc>
<@jamesmunns:beeper.com> * encode ALL numbers as varints (safe, probably smaller, but probably slower)
<re_irc>
<@jamesmunns:beeper.com> * just avoid explicit usize/isizes in serialized types (simple, but footgunny)
<re_irc>
<@jamesmunns:beeper.com> Also annoying because it would make every value of `0xFF` as a u8 take up two bytes on the wire
<re_irc>
<@jamesmunns:beeper.com> so actually might even be a wire-size regression for many use cases.
<re_irc>
<@jamesmunns:beeper.com> I guess I could only do that on u16 or larger
<re_irc>
<@jamesmunns:beeper.com> (or even u32 or larger, but don't tell cr1901)
<re_irc>
<@dngrs:matrix.org> jamesmunns:beeper.com: honestly even `u16` would be excessive in most cases. Just wondering whether "index into a vec" is the best alternative anyway (I do need that info though: it's queues of animations and I store which one is currently active)
<re_irc>
<@jamesmunns:beeper.com> Yeah, up to you. In my forth parser, I just went with u16s as "good enough"
<re_irc>
<@jamesmunns:beeper.com> probably don't need more than a u8, honestly. If you're worried about the wire size, you could just use u32s and use rzcobs to smash down the leading zeros (also what I do in my forth)
<re_irc>
<@dngrs:matrix.org> jamesmunns:beeper.com: I'd say it's not 100% safe because it compromises the upper size bound calculation, however probably not so relevant in practice because in almost all cases `usize` on build host will be `>=` size on deployment host
<re_irc>
<@dngrs:matrix.org> jamesmunns:beeper.com: mostly worried about upper size bound inconsistencies
<re_irc>
<@jamesmunns:beeper.com> (so, postcard does this already, but ONLY for array lengths and enum discriminants)
<re_irc>
<@jamesmunns:beeper.com> so, that bridge has already been crossed :)
<re_irc>
<@jamesmunns:beeper.com> but most people don't notice because they don't have enums with > 127 variants, or send arrays with > 127 elements (and if they do, VERY rarely >32768 elements)
<re_irc>
<@jamesmunns:beeper.com> Oh, I guess the next break is at 16K elements, still, a lot.
<re_irc>
<@jamesmunns:beeper.com> introducing it for u32/u64s would actually probably make the average message size smaller, as long as you aren't sending values >= 0x4000_0000 very often.
<re_irc>
<@jamesmunns:beeper.com> but you'd spend more time encoding/decoding, since it's not a direct memcopy.
<re_irc>
<@jamesmunns:beeper.com> I should add that as a warning in the postcard docs though ("don't use usize/isize"). I thought I did, but I can't see it anywhere on a first glance.
<re_irc>
<@jamesmunns:beeper.com> might be faster than rzcobs + postcard though. Ho hum. Sounds like I need to write some benchmarks.
<re_irc>
<@ryan-summers:matrix.org> Generally if someone's doing something with more than 256 items, you're in the range of KB-MB, in which case performance is often an issue and there's a lot of special considerations anyways, so you're not using some OTS solution
<re_irc>
<@ryan-summers:matrix.org> Because just copying that much data on a micro starts having serious performance ramifications
<re_irc>
<@jamesmunns:beeper.com> Yeah, though I certainly have sent byte arrays bigger than that
<re_irc>
<@jamesmunns:beeper.com> (e.g. for OTA updates, etc)
<re_irc>
<@jamesmunns:beeper.com> but for u16 and above, I'd generally agree with you (though the breaking point to the second byte for varints is 127 items, not 255)
<re_irc>
<@jamesmunns:beeper.com> and for OTA, I tend to 0xFF-fill my data if it's a fixed window, to make flashing easier.
<re_irc>
<@jamesmunns:beeper.com> so those message sizes would double, but I think varint'ing a u8 is just universally a bad idea.
<re_irc>
<@ryan-summers:matrix.org> I will say from using MQTT, I personally despise varints, but that's somewhat unrelated :)
<re_irc>
<@jamesmunns:beeper.com> :)
<re_irc>
<@jamesmunns:beeper.com> I bbqueue's framed mode I started using vint64. a slightly different varint encoding style
<re_irc>
<@jamesmunns:beeper.com> (written by Tony Arcieri)
<re_irc>
<@jamesmunns:beeper.com> the difference is that it puts all the "length bits" in the first bytes, instead of one bit/byte
<re_irc>
<@jamesmunns:beeper.com> but still variable
<re_irc>
<@jamesmunns:beeper.com> I... don't remember why I didn't just use his impl, but there was some reason...
<re_irc>
<@jamesmunns:beeper.com> > This implementation makes assumptions that data larger than the platform's `usize::max()` will never be encoded/decoded, which is not true when sending between 32-bit and 64-bit platforms.
<re_irc>
<@jamesmunns:beeper.com> ahhhh, right
<re_irc>
<@jamesmunns:beeper.com> so I would want to use the "real" vint64 for postcard then.
<re_irc>
<@jamesmunns:beeper.com> (for bbqueue, this is okay because the producer and consumer are ALWAYS on the same system)
<re_irc>
<@jamesmunns:beeper.com> Though to be honest, if you're sending 4 billion of ANYTHING to a 32-bit system, you're gunna have a bad day anyway.
<re_irc>
<@jamesmunns:beeper.com> Right now, postcard uses protobuf-style varints (one bit at the top of each byte to signal the "last" byte)
fabic_ has joined #rust-embedded
<re_irc>
<@therdel:matrix.org> Does anyone know specific chat-rooms for any of these WASM runtimes for ES? (if they exist🤔)
<cr1901>
jamesmunns: Ahh good, you knew to tag me, knowing I'd find your 16-bit slander anyway :)
<re_irc>
<@jamesmunns:beeper.com> Hey! I'm keeping it real for 16 bit targets :)
<re_irc>
<@jamesmunns:beeper.com> I definitely *want* to support AVR/MSP430 targets, though I admit it might not be reasonable, depending on the RAM available.
<re_irc>
<@jamesmunns:beeper.com> (you're probably doing your own bit/byte packing to save space anyway, rather than serde's recursive ser/de approach)
<cr1901>
I haven't implemented RPCs yet, which was my original use case for postcard
<cr1901>
msp430 maxes out at ~8kB RAM, and ~48kB ROM
<cr1901>
(Okay, it's actually 10kB RAM and ~128kB of ROM, but it requires special link sections and Idk how to use them)
<cr1901>
(And I'm not sure TI knows how to use them either)
<re_irc>
<@jamesmunns:beeper.com> doesn't the MSP430X go up to some cursed >= 64K number?
<cr1901>
Nobody uses X, problem solved :P
<cr1901>
uhh it's 20-bit program counter
<re_irc>
<@jamesmunns:beeper.com> I used them at an old job
<re_irc>
<@jamesmunns:beeper.com> I know how cursed they are
<cr1901>
There's probably no way to implement this safely, but... I want a hypothetical proc macro
<cr1901>
that allows you to change the link_section for an entire crate
<cr1901>
that's it.
<cr1901>
even msp430 without x has a 20-bit PC internally. Just can't use it normally
troth has quit [Ping timeout: 268 seconds]
troth has joined #rust-embedded
troth has quit [Ping timeout: 260 seconds]
troth has joined #rust-embedded
<re_irc>
<@jamesmunns:beeper.com> Has anyone messed with the VL53L1 ToF sensors?
<re_irc>
<@jamesmunns:beeper.com> The datasheet says "no register defs for you, use the C driver", and that makes me angry