<whitequark>
in particular it kinda depends on (1) because once (1) is complete you will have a principled way to get the version of the toolchain utilities that are used to make the bitstream
<d1b2>
<attiegrande> sounds reasonable
<whitequark>
in the cache key, you need to have the SHA of the design, but also versions of all relevant Python dependencies (you can get those by looking up dependencies of installed glasgow package in importlib.metadata... and when we have OOT applets that will cover it too)
<whitequark>
and versions of the native components
<whitequark>
actually, come to think of it, that's overkill
<whitequark>
you only need to take native (yosys/nextpnr) toolchain versions into account in the cache key, since hash(RTLIL) already covers all of the Amaranth inputs
<whitequark>
potentially this will require changes to Amaranth so that it e.g. doesn't add line numbers everywhere for non-debug builds (this is a common cause of cache misses in glasgow)
<whitequark>
3. we REALLY need to finish migrating off amaranth.compat
<d1b2>
<attiegrande> ok, so key = SHA(design + yosys version + nextpnr version)
<whitequark>
+ icepack/ecppack version
<d1b2>
<attiegrande> ta
<whitequark>
oh, one tidbit
<whitequark>
the reason I still haven't implemented (1) is very silly
<d1b2>
<attiegrande> oh?
<whitequark>
you see, yowasp-yosys spits debug messages written by abc to the terminal, and there is nothing you can do about it, you cannot disable it in abc (it's not an option), and you cannot redirect abc output elsewhere (abc is built into yosys as object code)
<whitequark>
I actually tried to make a split abc.exe configuration work in yowasp
<whitequark>
this is very annoying to do with wasmtime, but I did it
<whitequark>
only to discover that abc crashes when built for WASM and it is basically impossible to find out why
<whitequark>
like... it doesn't crash when it shares its address space with yosys. but it crashes if it doesn't.
<whitequark>
clearly what is happening is an out of bounds memory access somewhere, but you don't have valgrind for wasm
<whitequark>
you're just fucked
<d1b2>
<attiegrande> ouch...
<whitequark>
and facing this discovery (i tried to get it to work on two separate occasions where i dropped everything else and just studied this problem) got me so depressed that i kind of just gave up on integrating yowasp
<whitequark>
because every time i would run it i would get reminded of what a pile of crap abc is
<d1b2>
<attiegrande> does this mean that #1 is still blocked? (by this), or is it ready to go
<whitequark>
without any way to disable it
<whitequark>
well, two years after first facing this task i realized that we can simply capture the output of yowasp-yosys in Glasgow if we're not running in debug mode
<d1b2>
<attiegrande> riight, well done for taking a break
<whitequark>
capture and discard, because yosys runs with -q anyway
<whitequark>
and none of what it prints to stdout is useful
<d1b2>
<attiegrande> nice, okay!
<whitequark>
if you get a synthesis failure it's usually in nextpnr, but if it's in yosys you have to look at the complete log and, usually, RTLIL, anyway
<whitequark>
so (1) is unblocked and the fix is pretty straightforward
<d1b2>
<attiegrande> great
<d1b2>
<attiegrande> I've had a skim through your link, I suspect I can take on #1
<d1b2>
<attiegrande> probably #2 as well, but let's wait for assigning that
<whitequark>
perfect. so handling #1 will improve our Windows story dramatically
<d1b2>
<attiegrande> i'd probably appreciate your help, but will do what I can without
<whitequark>
the remaining question is then, does esden ship Glasgow units flashed with firmware (that has a winusb descriptor) or are they blank?
<whitequark>
cause if they're blank that's Free Support Workload of explaining to people how to use zadig
<d1b2>
<attiegrande> this is a good question
<whitequark>
by the way, I considered packaging zadig as a pypi package
<whitequark>
hands down the most cursed thing i have tried to package so far
<d1b2>
<attiegrande> is that a specific tweak for Windows, or will other platforms accept winusb descriptors?
<d1b2>
<esden> O hey, sorry went off to do something else. I am back now... catching up now.
<d1b2>
<attiegrande> o/
<d1b2>
<attiegrande> re zadig: ... groan
<whitequark>
attiegrande: specific for windows. on linux you need a rules.d file no matter what (afaik), on macos... not sure, but i think it lets you just do the thing
<d1b2>
<attiegrande> @esden - the big one for you has just been mentioned... what state to ship units in
<whitequark>
because there is no kernel driver that binds to the device, yes?
<mwk>
oh, glasgow meeting, cool
<d1b2>
<attiegrande> o/
<whitequark>
mwk: was just looking at your PR
<mwk>
... I am actually in the middle of porting memory.floppy applet off amaranth.compat right now
<d1b2>
<attiegrande> @wq specific for windows, sure... but I was asking more if it will work in that state on other platforms too
<d1b2>
<attiegrande> ... or if we effectively have a "windows" firmware and "everything else" firmware
<whitequark>
so #308 is flagged as draft
<d1b2>
<attiegrande> @mwk - great! I'll assign that applet to you if that's okay?
<whitequark>
attiegrande: we have only one firmware that works on every revision and every platform
<whitequark>
which was a hard stance i took early on
<mwk>
attiegrande: sure
<d1b2>
<attiegrande> sensible - imo we should ship with windows compatibility then?
<d1b2>
<esden> I was planning to run glasgow factory on all the glasgows. As I have to test them anyways.
<d1b2>
<esden> I will not send untested hardware out.
<whitequark>
I have since softened on it (cc electronic_eel) and I think shipping e.g. per-major-rev firmware with #ifdefs is not fundamentally different to shipping 1 firmware with if()s; in either case you get the same conditionals and the same testing burden
<d1b2>
<esden> Which depends on having some kind of test that also tests the lvds connector.
<d1b2>
<esden> this is the biggest blocker at the moment
<whitequark>
esden: this is more a question of what state is the hardware after testing
<mwk>
whitequark: 308 is a draft because there were several ways to solve the problem of applets adding cruft to bus interface module, and I wanted confirmation that's the correct way
<whitequark>
as long as you run glasgow flash before packaging the board, everything will be OK
<d1b2>
<esden> Sure, we can make that happen. It would be good to have a document describing the shippable requirements.
<mwk>
but I ended up porting all applets anyway, so the PR is ready to go if you have no objections?
<whitequark>
mwk: oh! my stance on this today is "i do not find the diff offensive to senses so we should just ship it"
<mwk>
cool, marking it as ready for review
<d1b2>
<attiegrande> @esden - I was about to ask... about that / docs...
<whitequark>
I don't care about cruft, we just need to get off .compat no matter what
<d1b2>
<attiegrande> could you replicate / coordinate with me on a "factory testing & flash" page for the new docs?
<d1b2>
<esden> You also asked me a while back about serial numbers. I would like to address that at some point too. I have finally temperature and chemical resistent labels and printers to add visible serial numbers. But we need a format and template for those.
<mwk>
... do you want me to just go full angel girl and port everything
<d1b2>
<attiegrande> @mwk :3
<whitequark>
mwk: i would not interfere with you if you chose to do so
<whitequark>
and i would personally find it valuable
<d1b2>
<attiegrande> (I'd appreciate it too!)
<whitequark>
uh, floppies... i think all my floppies are still in storage due to moving
<d1b2>
<attiegrande> ... if you're ok with it, I'll give the "migrate from amaranth.compat" to you @mwk
<whitequark>
okay, 4. docs
<whitequark>
to get things rolling, I know what needs to be done: I need to write a document stating what is an acceptable documentation page (also, commit message, and code style). basically, we cannot possibly expect contributors to just divine these requirements, which exist but are unstated, out of thin air
<d1b2>
<attiegrande> I've started on the basics - gettings started, troubleshooting (collecting errors + resolutions)
<whitequark>
0.1% of contributors can and so far we've overrelied on those
<whitequark>
there's a sphinx documentation in the repo iirc
<mwk>
I'm actually stuck due to some weird electrical-level issues with the floppy interface right now, wherein glasgow -> floppy pins work, but floppy -> glasgow doesn't
<mwk>
but I'm at hswaw rn and I'm reasonably sure there's enough hardware and people here to debug this
<whitequark>
mwk: add 2k pullup to 5V on every of the latter
<d1b2>
<esden> @whitequark I have github action templates for those. If that is something that would be helpful.
<whitequark>
TTL uses termination and the floppy is not terminated
<mwk>
so... the --pulls option is not supposed to work?
<whitequark>
actually, wasn't there a flag in the applet to do that?
<mwk>
there is, and it still doesn't work
<mwk>
(inputs are stuck at 0 without the option, stuck at 1 with the option)
<d1b2>
<esden> I mean I have sphinx doc github actions set up in black magic probe. If that is useful I can adapt them and contribute them.
<whitequark>
mwk: wtf.
<whitequark>
Attie: I think I would like to see my Amaranth GH actions setup adapted
jstein has quit [Quit: quit]
<whitequark>
in general I want to harmonize somewhat between Glasgow and Amaranth repos
<d1b2>
<attiegrande> for docs? ok, i'll take a look
<d1b2>
<esden> Sure that makes sense to keep those similar.
<whitequark>
it has a useful feature that it pushes to gh-pages in the fork meaning the contributors can preview docs they submit in a PR without having to install sphinx
<mwk>
also FYI, I think you have issue #4: glasgow doesn't work on py3.11 due to asyncio.coroutine decorator being gone after 3 versions of deprecation
<mwk>
I intend to do it later, but the floppy stuff has priority, as I borrowed the hardware
<whitequark>
5. yes we need to figure out a stopgap measure for py3.11
<whitequark>
in the medium term, the existing simulator interface will be completely revamped
<whitequark>
this is one of the work items I have at Chipflow, it is on a roadmap with deadlines, so it will happen
<whitequark>
and in the meantime, we just need asyncio to shut up and run our tests
<d1b2>
<attiegrande> nice, i'll put that on you then? (@wq)
<whitequark>
uhhhhhhh
<whitequark>
yea sure
<d1b2>
<attiegrande> oh? (sorry, sounded like you are working on it already for Chipflow)
<whitequark>
oh, not on the bit that we need to stuff into glasgow
<d1b2>
<attiegrande> ah, gotcha
<whitequark>
i bet that one will be some awful workaround using private asyncio variables or something like that
<d1b2>
<attiegrande> was it ju
<d1b2>
<esden> As long as we can just run it on py3.11 asap ... any ugly hack is welcome
<d1b2>
<attiegrande> *ignore
<whitequark>
6. out of tree applets. I don't want to *support* them just yet for reasons discussed before, but we need to start building infra for them already
<d1b2>
<attiegrande> ok
<whitequark>
I am happy to delegate this to anyone skilled in the art, but I will be managing that fairly closely (unless they're really experienced with python packaging) because it is super easy to fuck it up in subtle ways
<d1b2>
<attiegrande> you'll have to forgive me, I forgot previous reasoning
<whitequark>
also, for everything that I mention I am happy to delegate, you can assume I will provide help as needed
<d1b2>
<attiegrande> thank you, much appreciated
<whitequark>
oh, the reasoning is that this means we now ship a stable API, and our current API is so bad we need to scrap it in entirety and rewrite it from scratch with more experience
<d1b2>
<attiegrande> i was going to suggest I could look into oot applets... but let's stick with aiming for yowasp (#1 & #2) first?
<whitequark>
7. rewriting applet API
<d1b2>
<attiegrande> ah, yes. thanks
<whitequark>
this is 3/4 of the way blocked by Amaranth improvements in coming months
<whitequark>
in fact I expect that a lot of what's currently in Glasgow will move to amaranth-{stdio,soc}
<d1b2>
<attiegrande> oooh
<whitequark>
we need streams, we need interfaces, we need data structures (I think it may make sense to deprecate the custom bitfield class in Glasgow in favor of amaranth.lib.data... or maybe not, in any case, that bitfield class should use variable annotations instead of that ugly list based syntax)
<whitequark>
we can start making progress before that, and that will come in the form of deciding on a new UI-side API
<whitequark>
for a new CLI and with enough hooks to add a GUI at some point
<whitequark>
basically, instead of the current cursed argparse shit, applets need to go declarative
<whitequark>
ideally to the point where applet metadata required to show --help can be cached in a JSON file somewhere so you don't have to import every applet for three minutes if you run it on raspberry pi
<d1b2>
<attiegrande> hah
<whitequark>
if we keep the current overall style, we need to formalize the notion of a hierarchical process through which a specific operation gets invoked
<whitequark>
from the CLI
<whitequark>
and the same formalism will be used in the CLI, in the GUI, and in the scripts
<whitequark>
I have some ideas, but that would be for another meeting
<whitequark>
8. we desperately need to replace bitarray
<whitequark>
a contributor sent a PR doing that, but the quality is unacceptable and I do not see a way to making it acceptable
<d1b2>
<attiegrande> ok
<d1b2>
<attiegrande> we should close / reject if that's the case
<whitequark>
all they needed to do is to copy the bits module, make it use array based split storage without adding mutation and ensuring the same tests pass, then add mutation in a next change
<whitequark>
I'll close
<d1b2>
<attiegrande> ok
<d1b2>
<attiegrande> anything else to discuss on bitarray?
<d1b2>
<attiegrande> when ready, i'd like to bring @esden back in on factory test/flash & documenting it
<_whitenotifier-6>
[GlasgowEmbedded/glasgow] mwkmwkmwk 034b015 - applet.interface.spi_controller: convert to native amaranth
<whitequark>
oh, re: OOT applets, one thing that would be valuable is splitting glasgow-contrib applets in their own python package
<whitequark>
in fact this is how we can trial OOT applet machinery
<d1b2>
<attiegrande> yes, agreed!
<whitequark>
cause there are lots of applets that are... subpar (written by me, generally) and while they are useful I don't want to claim they're good
<mwk>
whitequark: I had something in the queue, but I think it got preempted?
<mwk>
I don't remember tbh
<whitequark>
there was the xc9572xl bit map function
<whitequark>
but I can't find anything related in the git log
<whitequark>
happy to delegate to anyone who can write a basic JESD3 writer (very easy task) that follows the overall structure of the existing JESD3 reader
<whitequark>
or I would spend 15 minutes on it some other day I guess
<whitequark>
I think you can assign that to me, Attie