<neil>
look another victim :) (skip hails from Rocky land :) )
<skip77>
I'm one of those filthy unclean down-streamers :P
<davidlt[m]>
morning
davidlt has joined #fedora-riscv
<davidlt[m]>
rwmjones|CONF: I am kicking off non scratch build
<davidlt[m]>
annoying as that will take another 7 hours or so
<davidlt[m]>
neil nirik we will need Koshei or something like that. I was rebuilding some noarch packages for Perl and noticed that testsuites fail to run. That's NaN related, which something expected. So we need something that would attempt to install, rebuild, etc. in the background to ensure that we have a good product.
<davidlt[m]>
neil nirik we also have maybe an issue with rpmautospec. Today if I need to do a quick patch to the spec for Fedora/RISCV I remove %autorelease and %autochangelog stuff from the spec. Not sure how it should work between two Kojis with dist-git overlay (which we should get rid eventually, but we don't have powers to trigger release bump in upstream dist-git).
jcajka has joined #fedora-riscv
Guest27 has joined #fedora-riscv
Guest27 has quit [Client Quit]
Eighth_Doctor has quit [Quit: Bridge terminating on SIGTERM]
defolos has quit [Quit: Bridge terminating on SIGTERM]
acharles has quit [Quit: Bridge terminating on SIGTERM]
chuangzhu has quit [Quit: Bridge terminating on SIGTERM]
jim-wilson[m] has quit [Quit: Bridge terminating on SIGTERM]
davidlt[m] has quit [Quit: Bridge terminating on SIGTERM]
Esmil[m] has quit [Quit: Bridge terminating on SIGTERM]
FloGrauper[m] has quit [Quit: Bridge terminating on SIGTERM]
chuangzhu has joined #fedora-riscv
davidlt[m] has joined #fedora-riscv
jim-wilson[m] has joined #fedora-riscv
defolos has joined #fedora-riscv
Eighth_Doctor has joined #fedora-riscv
acharles has joined #fedora-riscv
FloGrauper[m] has joined #fedora-riscv
Esmil[m] has joined #fedora-riscv
<rwmjones|CONF>
davidlt[m]: ocaml build looks good now
<davidlt[m]>
rwmjones|CONF: yeah, just finished
<davidlt[m]>
rwmjones|CONF: there are 3 failures on the testsuites, but IIRC those are the same as on aarch64
<davidlt[m]>
I will be injecting ocaml-* packages soonish
<davidlt[m]>
I need to take a walk while there is a break in LPC
<skip77>
Had a question about some build experiments I'm running w/ larger packages, like the kernel
<skip77>
I'm running a 5.18.8-200 kernel build on an 8-core, 32 GB RAM qemu setup right now. Seems to be hanging at the very end, with xz processes running.
<skip77>
I'm wondering about performance of a qemu environment (such as this) vs. physical hardware. I've got a starfive visionfive I haven't set up yet
<davidlt[m]>
skip77: well, xz is really expensive on riscv64. So it might not be stuck :)
<davidlt[m]>
So, if you have a good recent gaming PC that will outperform the physical boards (the current ones).
<davidlt[m]>
But not in every case I would say. There will be some code that still seems to run faster on a physical hardware.
<davidlt[m]>
VisionFive V1 is no-go. I would be surprised if that gets upstream support.
<davidlt[m]>
V2, aka JH7110 (which is not a test chip) is what will be supported.
<davidlt[m]>
It is slightly faster compared to FU740/Unmatched.
<skip77>
Probably not stuck - just really really long. I noticed in the fedora-riscv koji, kernel build takes ~10 hours. What sort of hardware is it run on?
<davidlt[m]>
Today we finally switched away from QEMU and everything runs on SiFive Unmatched board (no more sold).
<davidlt[m]>
Those are to be replaced by way better (hopefully) Intel/SiFive board in 2022 (or 2023, who knows).
<skip77>
Well, right now I'm just looking for a proper platform to build packages on. Not necessarily build or support images lol
<davidlt[m]>
Once Horse Creek happens QEMU will not be needed :)
<skip77>
Yeah, I heard about the Sifive Intel connection, sounds exciting
<skip77>
Basically, I'm dipping my toe in this with an eye towards porting Rocky Linux
<davidlt[m]>
QEMU in that case is what you are looking, but VisionFive V2 and PINE64 Star64 (same SoC) is what you want soonish.
<skip77>
I'm not in a particular hurry - got bunch of other stuff going on
<davidlt[m]>
Yeah, I want Rocky Linux to happen soonish!
<davidlt[m]>
I haven't pinged neil about it, but should at some point.
<davidlt[m]>
Some form of Rocky Linux 9 for RISC-V would be nice to see. Similar how RHEL for ARM happened many years ago.
<davidlt[m]>
There are extra questions to discuss like ISA targets.
<davidlt[m]>
These SBCs are "e-waste" as they don't implement standards.
<davidlt[m]>
RVA22 is coming, and RVA23 is probably the next major RISC-V Profile (a set of ISA and various implementation bits).
<skip77>
I'll have to read up on the details lol. When they increment those profiles (I think they're called that), are they not backwards compatible? So if you compile on a newer processor revision, binaries won't work on the older one?
<davidlt[m]>
Yes, but basically RVA64GC (aka RVA20U64 profile) is garbage performance wise.
<skip77>
Sounds like qemu is the way to go for the time being then. I'm using 8-core because libvirt+qemu wouldn't let me go higher
<davidlt[m]>
Basically no server vendor would want RVA20U64 on their sever and no customer would want that too :)
<davidlt[m]>
QEMU had a hard limit of 8.
<davidlt[m]>
That was changed 1 or 2 versions ago IIRC.
<skip77>
That's what I figured. It's probably best to go with the max, I assume
<davidlt[m]>
The rest depends on U-Boot, Kernel. In Fedora that is set to max 32 cores (but there are patches to increase that).
<davidlt[m]>
Might not be, it doesn't scale in a good way IIRC.
<davidlt[m]>
At least during MTTCG (multi-threaded TCG) upstreaming days in QEMU were was a scalability issues after 8 cores.
<skip77>
What sort of setup does the current koji builders run on? When they were qemu, that is
<davidlt[m]>
So running it on 64-core ThreadRipper probably will not give a super epic improvement :)
<davidlt[m]>
There is also PINE64 Star64 version of this. It's bound to be popular one with Linux community.
<davidlt[m]>
Two BeagleBoards sadly got delayed :(
<skip77>
Just trying to plan it out from a hypothetical standpoint: Qemu is easier to get going, but if it's going to be ridiculously poor performance then we might have to consider getting some physical boards
<davidlt[m]>
Well, depends on Rocky Linux policies.
<skip77>
For initial bootstrap, it's "whatever works" I think ;-)
<davidlt[m]>
In Fedora (and beyond) native builds are preferred.
<skip77>
My thoughts were to grab the relevant versions of all packages from your koji: Equivalent(ish) to about Fedora 34. And use those as dependencies to get going
<davidlt[m]>
Yeah, I am trying to kill QEMU builds (in some cases that might be complicated), but that's what we need to do.
<davidlt[m]>
Yeah, I skipped Fedora 34 :)
<skip77>
Yeah, for actual official builds I agree we'd want physical boards. But I'm not even there yet. Not close
<davidlt[m]>
There is F33 + some Rawhide bits and F37 and soonish F38 (both are on rebuilding stage, so a mess)
<skip77>
I'm hoping that I can pick and choose the ones that get me closest to where the Rocky 9 package base would be. I don't think it needs to be perfect.
<skip77>
It will involve building a kind of "old snapshot" of the current koji though. From dependencies I need that are ancient history to you Fedora folks :-)
<skip77>
We in enterprise-land are slow, blundering, and conservative. But some people like it
<davidlt[m]>
Yeah, I know :)
<davidlt[m]>
Fedora/RISCV will got back it's glory days, and we will be ready for CentOS Stream 10.
<davidlt[m]>
But it would be cool to see Rocky Linux 9 before that as an experimental thing.
<davidlt[m]>
But again there is something to discuss longer-term, that would ISA (RVA20, 22, 23) and optimized binaries, like hwcaps-glibc stuff (but that requires support on infra side).
<davidlt[m]>
For example if the distro base is RVA20, but the hardware is RVA23. We could still install optimized set of packages for it.
<davidlt[m]>
Or just everything is buillt with RVA23 in might.
<davidlt[m]>
Note, that RVA23 doesn't exist yet in this document and not gonna happen this year. The target (in emails threads) is early 2023.
<skip77>
Sounds like things are really going to take off, non-embedded-wise, when new hardware and new specs get implemented. I'm going to keep my ambitions of Rocky 9 port modest for now ;-)
<davidlt[m]>
skip77: all the specs are basically currently working towards server support.
<davidlt[m]>
Priority is to ratify those parts.
<davidlt[m]>
The problem is that everything is happening in parallel.
<neil>
i have a terrible idea that would let us run on-demand qemu risc-v workers in Peridot, skip77 :P
<skip77>
Oh no....
<davidlt[m]>
There was just a talk in LPC with UEFI/ACPI for RISC-V.
<davidlt[m]>
I mean we already had RISC-V part of UEFI, ACPI, SMBIOS specs for years now. Even Tianocore EDK2 stuff.
<skip77>
Well, it's exciting getting it into some of the non-embedded space. That's where the Rocky Linux "angle" is :P
<neil>
one of the rocky releng/developers is at LPC in person. hoping he brings back lots of notes :)
<davidlt[m]>
RISC-V MC is happening right now.
<skip77>
Even if we just get some "practice" bootstrapping a bit of 9, I think it will do us good
<davidlt[m]>
nirik neil : any plans/ideas of how we solve issues and improve Fedora/RISCV long-termish?
<davidlt[m]>
Should we setup a meeting to kickstart some doc/discussions?
<neil>
I think that's a great idea. get us all in the same place/time to discuss
<davidlt[m]>
You can pick a time unless it's a middle of the night for me :)
* nirik
is in a meeting. I am still planning on starting a list thread, I just have been swamped.
<neil>
I'd been thinking an etherpad, but perhaps as nirik suggested it would be best to just send a high level overview out to the devel list and see who else may want to join a potential meeting?
* neil
also swamped unfortunately
<davidlt[m]>
Same here, but with Fedora/RISCV :D
<davidlt[m]>
I think we got 800+ packages build last week :)
<davidlt[m]>
Fighting systemd and pam stuff right now.
* nirik
released f37beta this morning.
<davidlt[m]>
It's possible to get Fedora/RISCV F37 disk images for Unmatched before F37 happens. I see a small window, but TBD.
<davidlt[m]>
Yeah, it's on time! That's wonderful!
<neil>
oh yes, i meant to say something! congrats on the release nirik :) i'm looking forward to upgrading soon (I need to get to f36 first though)
<davidlt[m]>
How to move forward into getting this more proper and better.
<neil>
i guess the less compelling part about aws right now is the unmatched boards you have locally, davidlt[m]
<davidlt[m]>
neil: well, that's builders.
<davidlt[m]>
Someone needs to host those and those aren't servers with BMCs, etc.
<davidlt[m]>
You can add BMC to those, but that's expensive (300-400 EUR per board).
<Eighth_Doctor>
davidlt: what's emulation performance like on AWS builders?
<neil>
perhaps some cache locally then, too, as is done for s390x it appears
<neil>
Eighth_Doctor: my understanding from earlier conversations is that qemu is less performant than the hardware davidlt[m] has (16x HiFive unmatched)
<Eighth_Doctor>
even qemu-user?
<Eighth_Doctor>
I'd be interested to see if emulating risc-v on aws aarch64 would be reasonably performant
<neil>
me too. skip77 and I had been talking about doing such a test
<neil>
I can setup a temporary box to do it
<neil>
re: qemu-user; I'm not sure on the exact details. what I understand is that there's an 8 core limitation on qemu and "12:08 <davidlt[m]> At least during MTTCG (multi-threaded TCG) upstreaming days in QEMU were was a scalability issues after 8 cores."
<Eighth_Doctor>
at some level, having more builders than beefier builders is more useful
<neil>
guessing davidlt[m] is offline now, but at some point the riscv koji had a whole bunch of qemu builders and it got the project so far
<skip77>
Yep. I can attest that my 8-core qemu setup is not going too well right now
<skip77>
This happened last time as well, I thought it might have been a memory exhaustion issue. So I fed it more RAM.
<skip77>
I'm on "Processing files: kernel-modules-5.18.8-200.0.riscv64.el9.riscv64" in the kernel compile, there are several xz processes going (not much cpu activity according to ps), and it's been there for 4+ hours
<skip77>
I might just try it with less cores, or attempt to emulate a "real" device. Right now it's pure qemu hardware
<neil>
i'll just make you a aarch64 vm in aws :)
<davidlt[m]>
We had up to 170 QEMUs at a time.
<davidlt[m]>
Today I am more interested in avoiding QEMU and doing native builds based on hardware.
<davidlt[m]>
I think that's doable. Relatively good performance (compared to what we had).
<davidlt[m]>
If/Once Intel/SiFive new board shows up QEMU probably will be a slower option for sure (expectation).
<davidlt[m]>
That's the reason why I have the boards. Native hardware builds.
<davidlt[m]>
(Not that I would fully trust the hardware, but I do trust it more than QEMU)
<davidlt[m]>
We don't have significant issues with RISC-V QEMU for years now, but still sometimes builds fail on QEMU, but work fine on real physical board.
<davidlt[m]>
Bugs are also a thing that exist within QEMU you want it or not. There are plenty of those in HW too, but hardware probably gets more testing during development.
<davidlt[m]>
Doing native builds now makes us closer to Fedora (in my opinion).
<davidlt[m]>
StarFive JH7110 will help a bit (but not significantly).
<davidlt[m]>
My hope is with Intel/SiFive Horse Creek with P550 (OoO) cores with Intel 4 (aka 7nm) with DDR and PCIe.
<davidlt[m]>
Combining OoO cores and 7nm should give us epic performance boost (assuming no crazy HW erratas).
<davidlt[m]>
That's probably the 1st Intel's 7nm product :)
<neil>
👀
<davidlt[m]>
Meteor Lake (MTL, 7nm) should be out late 2023.
<davidlt[m]>
Horse Creek is 2022 product IIRC.
<davidlt[m]>
(Sadly it's not P650 cores...)
<davidlt[m]>
The thing is that Koji infra is data movement challenge in some ways. I would say more than a compute challenge.
<davidlt[m]>
According to SMART stuff we moved 1+PB of data IIRC.
<davidlt[m]>
I see 30+T of data moved since the last reboot.
<skip77>
Yep. My goal is just to get off the ground first with qemu, because it's literally what I have in front of me. Will be doing "real" package builds on physical hardware in the future, but I need to iron out mock configs, bootstrap strategy, etc. first
<davidlt[m]>
skip77: I can help you out if needed :)
<davidlt[m]>
Rocky (like Alma and CentOS and RHEL and etc.) are way smaller package count wise.
<davidlt[m]>
With JH7110 boards, NVMe and a large swap drive (+ ./tmp on tmpfs disabled) you can do relatively cheap infra.
<davidlt[m]>
Those boards will go for <100 USD IIRC.
<davidlt[m]>
I bet the ones from PINE64 will be popular with FOSS community.
<neil>
i've got a handful of the jh7110 boards on the way :)
<davidlt[m]>
I believe more stuff is to arrive, that T-HEAD C910 should happen too.
<neil>
what's the best way to keep an ear to the ground about this all besides asking you? :D
<davidlt[m]>
Don't know :)
<davidlt[m]>
There are various bits everywhere.
<neil>
(:
<davidlt[m]>
There is also that crazy RISC-V laptop, that I highly against spending money on.
<skip77>
davidlt[m]: I 100% will have questions, I'm sure of it. In the short term, I'm going to do a rebuild + troubleshoot some key packages from the fedora koji (like kernel) for benchmark + troubleshooting purposes
<skip77>
When I'm pretty sure of my setup, I'll start yanking a copy of all risc-v build artifacts that I think will be compatible w/ the Rocky/RHEL versions. And begin building
<davidlt[m]>
Note, this one should have power and reset pins on the headers.
<davidlt[m]>
I am not sure about VisionFive V2 headers.
<davidlt[m]>
You can use eMMC modules from PINE64 store, but you also have PCIe x4 open-ended connector.
<davidlt[m]>
Thus M.2 adapter is needed, which costs extra.
<davidlt[m]>
Don't expect the chip to run 1.5GHz out of the box (that's typically not the case).
<davidlt[m]>
Also this chip is a bit more interesting. It has way newer U74-MC compared to FU740/Unmatched.
<davidlt[m]>
This means potential new IP blocks (like L2 prefetcher) and new extensions (e.g. bitmanip).
<davidlt[m]>
Which is basically not used, but I cannot imagine any future RISC-V SoC without bitmanip stuff. It's actually required in RVA22 and RVA23.
<davidlt[m]>
Sadly we don't have ifunc working yet.
<Eighth_Doctor>
and we still don't have server class riscv32 / riscv64 boards?
<davidlt[m]>
it's all about the right time.
<davidlt[m]>
Some specs are still WIP that are required to make a proper server.
<davidlt[m]>
That incl. new interrupt controller stuff, IOMMU and gazillions others.
<davidlt[m]>
But there are players going for it. Ventana especially.
<davidlt[m]>
You could make it today, but there would be a lot of vendor specific extensions, etc. That's not something we need for basic stuff.
<skip77>
Yep, sounds like lots of stuff coming. Hence the head-start to get RHEL (well, Rocky) ported and working on some of that sweet sweet server hardware :P