<re_irc>
<jbeaurivage> How do you all test and debug your software? I'm writing code for a device that's used in skydiving operations (absolutely nothing mission critial). I'm having transient crashes that pretty much only happen mid-flight, when you obviously can't just attach a debugger to diagnose the issue. Moreover, it's already hard to debug because the firmware only fits on my chips with a minimum "opt-level = 1", which makes using...
<re_irc>
... gdb super annoying. Anyway, I'm just looking for ideas on how y'all do post-mortem analysis for your embedded Rust projects
starblue has quit [Ping timeout: 256 seconds]
starblue has joined #rust-embedded
cyrozap has quit [Remote host closed the connection]
<re_irc>
<ian_rees> jbeaurivage: I don't have an answer, but love your question and am also interested in this space :)
<Darius>
debugging mid flight does sound tricky though...
<re_irc>
<jbeaurivage> I've made some attempts to yeet defmt logs onto an SD card but so far have been unsuccessful. One potential problem is keeping the elf that was used to produce those logs and the actual log file in sync
<re_irc>
<ian_rees> is there a good chat room somewhere for general embedded stuff like this, that's not necessarily Rust specific? For instance, I'm looking at a new-to-me embedded problem and just sorta wondering how other folks approach it: how to manage systems with a secure boot process
<re_irc>
<ian_rees> jbeaurivage: I like that approach - have some sort of circular buffer saving to nonvolatile memory. The executable-log sync problem is fundamental, I think - would have the same issue with core dumps
<re_irc>
<ian_rees> what would be cool, but probably device-specific and I guess quite hard, would be a handler that runs in the opposite end of RAM from the normal program, that could save state in to flash
<re_irc>
<jbeaurivage> ian_rees: Not a chat room per se, but you could always try the /r/embedded subreddit
<re_irc>
<ian_rees> yeah, I don't have a reddit account but it might be time to set one up... Recently found a comment on a Rust thread that's wrong - https://xkcd.com/386/
<re_irc>
<jbeaurivage> Pretty much, though when I tried setting that up I seemed to have been hitting some sort of UB. Of course, debugging an optimized executable makes it 10x harder
<re_irc>
<ian_rees> idle thought: a cool project might be a firmware that does the probe side of SWD, and just logs RTT to a SD card
<re_irc>
<ian_rees> so you could use that to make a little logging backpack that goes skydiving along with your project
<re_irc>
<jbeaurivage> Hey that actually sounds really cool! Could even use another transport like USB or UART, though then it's more machinery on the device side
<Darius>
WRT keeping your binaries in sync, I would suggest logging the git hash (or equivalent) first off
<Darius>
then you can at least build a binary to match the log stream after the fact
brazuca has quit [Ping timeout: 252 seconds]
<re_irc>
<ian_rees> perhaps separate files per logging stream, with RTT just being one of the stream sources. I've been thinking about a vaguely related big-computer app for capturing synchronised logs from different sources - at work we have endless problems getting QA to capture logs from a pair of devices that talk with each other...
<re_irc>
<James Munns> jbeaurivage: Just to note: /r/embedded _hates_ rust
<re_irc>
<James Munns> they might be good for answering other stuff, but I really don't like the vibe there
<re_irc>
<jbeaurivage> Idk if that app is written in rust, but I've found tracing and tracing-opentelemetry to be quite nice
<re_irc>
<ian_rees> James Munns: I mean, I can understand. I'd have a lot less job security if stuff just worked like it does in Rustland
<re_irc>
<jbeaurivage> Yeah, it definitely has an oldschool kind of vibe
<re_irc>
<James Munns> jbeaurivage: That's certainly a way to put it, lol
mightypork has quit [Ping timeout: 268 seconds]
mightypork has joined #rust-embedded
<re_irc>
<joris_van_den_berg> I am stuck at implementing a double buffer dma read from uart, but I suspect my actual problem is more general Rust use. Is it ok if I try to explain my issue here?
<re_irc>
<joris_van_den_berg> Or is this not the right place for that?
<re_irc>
<ryan-summers> Shoot your question :)
<re_irc>
<ryan-summers> This is definitely the place
<re_irc>
<joris_van_den_berg> I try to set up two buffers, a and b. While writing the contents of a to disk I recieve into b and vice versa. The attempted code looks like:
<re_irc>
<joris_van_den_berg> let rx_dma = rx.receive_with_dma(rx_buffer_a, chan0, |_| {});
<re_irc>
loop {
<re_irc>
let rx_dma = rx.receive_with_dma(rx_buffer_b, chan0, |_| {});
<re_irc>
let (chan0, rx, rx_buffer_a) = rx_dma.wait();
<re_irc>
<joris_van_den_berg> problem is that the receive_with_dma consumes rx (as it should), I get it back in the loop and then it goes out of scope
<re_irc>
<joris_van_den_berg> I do not know how to pass the rx and rx_buffers to the next loop iteration
<re_irc>
<ryan-summers> You could store them in an "Option<>" that exists in the outer loop
<re_irc>
<ryan-summers> Another option here is to change the DMA API to instead just take a mutable reference to the buffer for the duration of the transfer
<re_irc>
<ryan-summers> Then ownership of the buffer never changes, but you can still guarantee that DMA has exclusive access to it during the transfer
<re_irc>
<ryan-summers> Or are you using the HAL's DMA implementation here?
<re_irc>
<joris_van_den_berg> I am using the board support crate functions
<re_irc>
<ryan-summers> In general, it seems odd that the BSP takes _ownership_ of the buffer
<re_irc>
<ryan-summers> That's not how e.g. the STM32H7 HAL does it.
<re_irc>
<ryan-summers> Mutable borrow should be sufficient here imo
<re_irc>
<ryan-summers> Then all your scope issues go away
<re_irc>
<joris_van_den_berg> this is the atsamd
<re_irc>
<ryan-summers> e.g. it should be:
<re_irc>
let rx_dma = rx.receive_with_dma(&mut rx_buffer_b, chan0, |_| {});
<re_irc>
<ryan-summers> Then rx_dma only lives for as long as all the things are borrowed. "finish()" would consume "rx_dma", which ends the borrow of the buffer ( and would need to do this for the "chan" as well)
<re_irc>
let rx_dma = rx.receive_with_dma(&mut rx_buffer_b, &mut chan0, |_| {});
<re_irc>
rx_dma.finish();
<re_irc>
<joris_van_den_berg> I am not sure I am up to modifying the bsp yet
<re_irc>
<ryan-summers> But if you don't want to fix the DMA API, you can just use the "Option<>" method I proposed, where the "Option" lives outside the loop
<re_irc>
<ryan-summers> You're going to incure big copy overhead doing that though
<re_irc>
<ryan-summers> Because you have to copy into the option on like every iteration
<re_irc>
<ryan-summers> You're going to incur big copy overhead doing that though
<re_irc>
<ryan-summers> Also, don't be afraid of PRing into the BSP/HALs ;)
<re_irc>
<ryan-summers> It's usually not too bad
<re_irc>
<joris_van_den_berg> is there something like a stack allocated Box<>?
<re_irc>
<ryan-summers> Box always implies dynamic memory. You can look at something like a "heapless::Pool"
<re_irc>
<ryan-summers> It's basically a statically allocated pool of buffers that you can loan out from the pool
<re_irc>
<ryan-summers> * objects
<re_irc>
<ryan-summers> So you can make a pool of like 4 buffers and move them into/out of the pool as they get used/returned
<re_irc>
<joris_van_den_berg> modifying the bsp receive_with_dma would be in the lines of replacing (self, ...) with (&mut self, ...)?
<re_irc>
<ryan-summers> Yes, and you would also modify the other args to be &mut buffer, &mut channel
<re_irc>
<ryan-summers> Then it would store those references internally in a new object, then add a new api that does "fn finish(self)", which consumes self and drops the borrows
<re_irc>
<ryan-summers> I think that would just be rewritting "wait()" to consume self and not return anything
<re_irc>
<joris_van_den_berg> I guess I should be able to look it up in the STM32H7 version
<re_irc>
<joris_van_den_berg> ah, thats how it works, finish consumes self to release the borrows
<re_irc>
<joris_van_den_berg> thanks!
<re_irc>
<ryan-summers> No worries. Hope I gave you some ideas to work with :)
<re_irc>
<joris_van_den_berg> Contacting the bsp maintainer is also a good idea, I might learn some new stuff from their rational as well (assuming it is not a mistake)
<re_irc>
<ryan-summers> Is the BSP open-source here btw?
<re_irc>
<ryan-summers> Would be interesting to take a look at it
<re_irc>
<joris_van_den_berg> yes it is open source
<re_irc>
<joris_van_den_berg> I just looked at the dmac code, and I am not up to changing it myself (I am not sufficiently at ease with all the generics)
<re_irc>
<ryan-summers> I'd just open an issue to see what others have to say then :) Mention the problem you're having. They may have other ways around it as well
<re_irc>
<ryan-summers> Also, you can combine them into something like:
<re_irc>
let rx_dma = rx.receive_with_dma(buffer_holder_a.take().unwrap(), chan0, |_| {});
explore has joined #rust-embedded
dc740 has quit [Ping timeout: 268 seconds]
GenTooMan has quit [Ping timeout: 255 seconds]
<re_irc>
<mabez> Is there anyway to really guarantee the entire body of a function call is put in the right section (i.e ram)?
<re_irc>
<James Munns> mabez: not as far as I know. You might be able to make a static analysis tool that looks at this
<re_irc>
<James Munns> Honestly, the only REALLY deterministic way I could think of would be to compile a separate, standalone mini-binary
<re_irc>
<James Munns> (and load that to ram)
<re_irc>
<James Munns> The usual "oops" here is calling some function from "core", which doesn't get inlined (and is shared with the main binary), which lives in flash
<re_irc>
<James Munns> then surprise, your 99%-in-RAM function accidentally calls a function in flash
<re_irc>
<mabez> Yep that's exactly what I've just experienced. Just seen an interesting failure where a for loop desugared into an iterator and calls some flash functions
<re_irc>
<James Munns> Yeah, I think the most deterministic way to do it would be to make a standalone staticlib, which gets placed in RAM, and you have a single entry function in your main binary that passes any context necessary (with CFFI, sadly), and make sure the "trampoline" function gets put in RAM too.
<re_irc>
<James Munns> if the ONLY thing you do in your trampoline function is to call the CFFI function (also in RAM), I'd say it'd be pretty unlikely you ever end up in flash.
<re_irc>
<mabez> Thanks! I was also thinking about using "asm!" but that doesn't really scale well for larger functions
<re_irc>
<mabez> and its not portable so I'd need to write it twice, once for riscv & once for Xtensa 🙃
<re_irc>
<James Munns> Yeah, in this case the staticlib really just ensures you have total isolation, by making it a binary unto itself
<re_irc>
<James Munns> the one gotcha I can think of is that you will need a panic handler
<re_irc>
<James Munns> but, you could make that an extern symbol, and have a hook in your main binary that provides it
<re_irc>
<James Munns> which... I guess could end up in flash? but maybe you could also use panic_never hax lol
<re_irc>
<James Munns> (or just provide a "cpu reset" panic handler)
<re_irc>
<James Munns> depends on whether you are doing it for perf reasons (e.g. RAM ISRs), or for functional reasons (e.g. you are bootloading, and flash doesn't exist/is invalid/is inaccessible at the moment)
GenTooMan has joined #rust-embedded
<re_irc>
<mabez> Functional in this case, can't access flash when trying to map regions of it to address spaces
<re_irc>
<James Munns> Yeah, not saying this is the ONLY way to do this, but the only reliable way I am aware of :p
<re_irc>
<James Munns> seems fun though :D
<re_irc>
<mabez> It's annoying that in the case of for loops, you can't sprinkle "#[inline(always)]" atop of them, I think that would actually "fix" this issue (but not the general problem of course)
<re_irc>
<James Munns> I don't honestly know if there is a better way to do this in C? or if you just don't have something like a "core" lib that is as pervasive.
<re_irc>
<mabez> AFAIK the problem persists in C, there are no guarantees just like Rust
<re_irc>
<mabez> Which makes me sad, Rust should be better :(
<re_irc>
<James Munns> it would be able to have recursive control flow attributes
<re_irc>
<James Munns> seems like a hell of an RFC
<re_irc>
<James Munns> it would be cool to be able to have recursive control flow attributes
<re_irc>
edit: words
<re_irc>
<dirbaio> C has the same issue, but it's not as bad because it's less prone to calling functions you didnt explicitly call
<re_irc>
<Tom> This is sort of the core of the issue when writing firmware for imxrt, calling out to spi flash for XIP is... not fast
<re_irc>
<Tom> but sram isn't always abundent, imxrt1010 has I believe 128kb
<re_irc>
<Tom> like teensy-rs I think just loads everything to sram before running
<re_irc>
<Tom> but that has a lot more sram to work with
<re_irc>
<Tom> like teensy4-rs I think just loads everything to sram before running
<re_irc>
<newam> XIP isn't too bad to work with if you can memmap it and cache it
<re_irc>
<newam> well and if it has quad/octo SPI, and you don't need to deal with code signing... and now I'm realizing that this giant list does make it that bad to work with
dc740 has joined #rust-embedded
<re_irc>
<James Munns> Yeah, the one commercial project I did with the imxrt family I think I loaded everything into SRAM at boot
<re_irc>
<James Munns> but it was also super time critical, and we used the 1061/2, which had more SRAM IIRC
<re_irc>
<James Munns> oh yeah, it had 1MiB of SRAM
<re_irc>
<James Munns> so I think we split that into code and RAM sections, which was big enough at the time,
<re_irc>
<James Munns> * time.
<re_irc>
<James Munns> _taps my head_ don't need icache if your code is already in SRAM
<re_irc>
<newam> The SSDs I worked on loaded everything from SPI into SRAM then executed from SRAM to train DRAM, then copied everything over to DRAM.
<re_irc>
Meanwhile you had other CPUs frantically doing PCIe things because PCIe has merciless timing requirements.
<re_irc>
<Tom> haha, well you'd think that, but the sram is split into two regions itself, tcm and ocram
<re_irc>
<Tom> ocram requires a bus access, tcm does not as I understood it
<re_irc>
<Tom> can dma to from the ocram, not tcm, etc
<re_irc>
<James Munns> Yeah, I think we did code in tcm (and maybe stack?), and working mem in ocram?
<re_irc>
<James Munns> honestly can't remember though, it was a couple years back
<re_irc>
<James Munns> (and it's an ex-client project, so I can't go look)
<re_irc>
<Tom> M7's are just weird coming from m4's where I didn't think at all about flash access times
<re_irc>
<Tom> gotcha
<re_irc>
<Tom> I know you've said more than once I think you had a bit of a tough time with it, maybe I'm misremembering, but yeah its a complex chip with so many potential options on how to load and run firmware
<re_irc>
<James Munns> yeah, it was a _thing_
<re_irc>
<Tom> I got interested in it thinking I'd make a little synth with rust and imxrt (since it was stupid fast compared to any other mcu), my new thinking is use a fpga soc with linux and some programmable logic for the harder real time pieces instead
<re_irc>
<James Munns> Honestly, for a personal project, I've moved to using an MPU instead (a RISC-V RV64, single core 1ghz, 512M of dram), and it feels WAY EASIER than the imxrt
<re_irc>
<James Munns> Tom: You might want to come hang out in #anachro:matrix.org (https://matrix.to/#/#anachro:matrix.org), I'm planning to do exactly that with my homebrew OS, mnemOS.
<re_irc>
<jessebraham> Is that the Allwinner D1?
<re_irc>
<James Munns> jessebraham: Yup!
<re_irc>
<Tom> imxrt isn't that bad, and there's the super neat 685 with a hifi core, but yeah... just so many options, and you need to learn a lot and roll your own path sort of
<re_irc>
<James Munns> Allwinner D1 also has a hifi4 core :D
<re_irc>
<James Munns> haven't figured out how to use it tho
<re_irc>
<Tom> oh neat, now if only someone would enable rust on it with all the hifi extensions :-)
<re_irc>
If it's easier to run bare metal code than the raspberry pi I am very interested.
<re_irc>
<newam> Are you running linux the Allwinner D1 that or is it possible to program bare metal?
<re_irc>
<newam> -that
<re_irc>
<James Munns> newam: Bare metal! or rather, with my OS
<re_irc>
<Tom> writing in C with the hifi toolchain is... I dunno, not my normal thing lets say
<re_irc>
<boondocker> I have a few sipeed D1 boards I keep meaning to play with
<re_irc>
<James Munns> xfel is sort of like a "hardware debugger interface" built into the chip itself. You can hold an IO low on boot, and it will talk over USB to allow things like remote control of dram, loading code to RAM/flash, etc.
<re_irc>
<newam> oh that's pretty wild, I like that
<re_irc>
<James Munns> sort of like a superpowered version of ST's built-in ROM bootloader
<re_irc>
<James Munns> but over USB instead of UART/I2C/whatever
<re_irc>
<James Munns> (and not just code uploading)
<re_irc>
<newam> Every time I look at RISC-V it seems there's always a massive expansion in what is available.
<re_irc>
<James Munns> Yeah! Though xfel is an allwinner thing, not a risc-v thing. I think their arm chips have it too.
<re_irc>
<boondocker> Does it have SBI?
<re_irc>
<James Munns> yup!
<re_irc>
<James Munns> well, mnemos doesn't
<re_irc>
<James Munns> but rustsbi runs on the d1 already
<re_irc>
<James Munns> and you _could_ get mnemos working with that, I just haven't yet
<re_irc>
<James Munns> (right now in the code I linked, I use Machine interrupts for everything, but I could move that to using supervisor interrupts instead)
<re_irc>
<James Munns> (these are different "privilege rings" for RISC-V, basically, for anyone not familiar)
<re_irc>
<James Munns> theres Machine, Supervisor, and User
<re_irc>
<boondocker> SBI also provides some early hardware abstraction more like x86 BIOS as well IIRC
<re_irc>
<James Munns> Yeah, disclaimer, I'm SUPER new to RISC-V
<re_irc>
<James Munns> but it has been reasonable getting started with it
<re_irc>
<newam> Why are there so many cool things and so little time ðŸ˜
<re_irc>
<James Munns> but yeah, mnemos is aimed to be an OS between "hard realtime"/"RTOS" sort of apps, and something bigger like Linux
<re_irc>
<James Munns> (I'm personally looking for a company to work with so I can spend more time on it, but I'm also more than happy to mentor/accept PRs if other folks want to use it!)
<re_irc>
<boondocker> I’ve been playing with FPGA RV cores (vexriscv) for a while, but not with rust so far
<re_irc>
<James Munns> the goal is to be able to have independent applications, with a stable kernel interface. I even plan to support both MPUs (e.g. Cortex-A, RV64), as well as MCUs(e.g. Cortex-M, RV32)
<re_irc>
<James Munns> there might be some limitations on MCUs though (because no MMU, etc.), like you might only be able to run one app at a time, or you'd have to hardcode the run address for each app
<re_irc>
<James Munns> But either way, the Allwinner D1 is an awesome chip, and the Sipeed Pro Dock is a great little self contained system
<re_irc>
<James Munns> Campbell He even put together a PAC for it, so working with the hardware literally feels the same as on an MCU
<re_irc>
<James Munns> Even if you don't want to run mnemOS on it, it would be awesome to have more folks working on it :D
<re_irc>
Having 1GHz CPU and 512MiB of DDR3 feels like infinity resources :D
<re_irc>
<boondocker> I have a StarFive VisionFive as well with the SiFive chip
<re_irc>
<boondocker> I should really pull out all my RV boards and bring up some rust on them :)
<re_irc>
<James Munns> Yeah! Like I said, feel free to drop by #anachro:matrix.org (https://matrix.to/#/#anachro:matrix.org), there's a couple of us working on the D1 there.
<re_irc>
<James Munns> I actually chose the D1 specifically because it was still single core, and I haven't really wrapped my brain around bare metal multicore yet :D
<re_irc>
<jannic> Could it be an option to make RAM the default section and manually annotate the parts that can go to flash? Sure, that way many things will land in RAM by accident (especially parts from core), but perhaps you still get it small enough to fit?
explore has quit [Quit: Connection closed for inactivity]
<re_irc>
<James Munns> jannic: That's an interesting idea!
<re_irc>
<James Munns> I don't know if I've seen anyone try that, but it could be interesting
causal has joined #rust-embedded
causal has quit [Client Quit]
<re_irc>
<James Munns> explodingwaffle101: to be fair, I don't think it is "rust said it won't do this", but more rather "no one has asked"
<re_irc>
<James Munns> in my totally uninformed opinion, I think the harder question would be "how do you make sure that llvm does the right thing", or "how do you make rust live with the fact that there might be two instances of a lot of functions"?
<re_irc>
<James Munns> I think the actual goal is that you would want something like an "infectious" linker section attribute
<re_irc>
<James Munns> not just THIS function gets put in a linker section, but the whole CALL TREE gets put in that linker section
<re_irc>
<James Munns> (whether or not you treat that as taking precedence, or allow duplication, is probably something to discuss in the RFC)
<re_irc>
<James Munns> also, it's not just the function call graph, but any data dependencies. Things like string-constants are often put in .rodata, which lives in flash too
<re_irc>
<James Munns> (I'm not saying this to convince anyone _not_ to write this RFC, but rather to try and brain dump all the things that might be necessary to discuss. I hope someone out there does open up a pre-RFC/RFC for this, I just don't have it in me to do it atm)
<re_irc>
<James Munns> I think it's definitely _possible_, I just have no clue how much work it would actually be
dc740 has joined #rust-embedded
<re_irc>
<Tom> James Munns: It's not as bad as you might think to be honest
<re_irc>
<James Munns> One challenge at a time. I still haven't wrapped my head entirely around MMUs yet :D
<re_irc>
<James Munns> Definitely will get there, but I'm sort of growing my knowledge at the speed that I am implementing features in my kernel, so single core MCU -> single core MPU was a good logical step for me :)
<re_irc>
<Tom> if you can do single core with interrupt handling and you have rust semantics for ownership, smp is basically the same with usually some form of interprocess interrupt available to you, so you can poke other cores. Rust should make this significantly saner
<re_irc>
<Tom> like if you can basically treat each core as a thread, and use the ownership/mutability (Sync + Send)
<re_irc>
<James Munns> (fun fact, in bare metal, multicore CAN'T (in my opinion) be modeled by "Sync" alone)
<re_irc>
<James Munns> because you might not have the same resources available in each core, e.g. you might not have a cohesive set of peripherals or memory
<re_irc>
<James Munns> but I know what you mean :)
<re_irc>
<Tom> I haven't seen one where there _wasn't_ shared memory on a bus with a bunch of peripherals, but right some might be tied to only a subset of cores
<re_irc>
<Tom> I haven't seen that many though to be fair
<re_irc>
<James Munns> A simpler example is "multiprocess" programs on a desktop, vs "multithread" programs on a desktop
<re_irc>
<James Munns> sharing a reference from one _process_ to another would not be sound.
<re_irc>
<James Munns> (unless you, or the operating system, did extra unsafe work to make sure they are mapped in the same way, such as with mmap, but that's YOU doing the work, not Sync.)
<re_irc>
<James Munns> The reason this works for multiple _threads_, is that all threads of a given process exist within the same contextual environment, on a desktop. They share a single mapped memory environment, provided by the kernel
<re_irc>
<Lachlan> Also related, in some cases, to the performance/efficiency cores in some recent CPUs
<re_irc>
<Lachlan> Sometimes they have different capabilities
_whitelogger has joined #rust-embedded
dc740 has quit [Remote host closed the connection]
Lumpio- has quit [Read error: Connection reset by peer]