<bslsk05>
github.com: riscv-asm-manual/riscv-asm.md at master · riscv-non-isa/riscv-asm-manual · GitHub
<klange>
Grabbed a fresh microSD for the RPi, see if we can get one of them there barebones up and running, get meself a framebuffer, and go from there.
<geist>
those are magics for extracting particular fields of a symbols address
<dmh>
huh
<heat>
my percpu macros do real addition to tp but that's impossible on the trap entry
<geist>
so tprel_lo(symbol) may be kinda what you want
<zid`>
ahh that makes sense
<geist>
yah on arm64 you'll sometimes see that as :lo12:symbol or whatnot, that's their magic for it
pretty_dumm_guy has quit [Client Quit]
<geist>
i kepy reading that as 'lol 2' for a while until i realized it was 'low 12'
pretty_dumm_guy has joined #osdev
<heat>
i'm not liking the context switching logic between user/kernel
<heat>
it looks... weird
blockhead has joined #osdev
<klange>
Not familiar with riscv, what's the kernel-user switch look like (please elaborate beyond "weird")
<geist>
yah it's complicated. its what i was complaining about the other day, since it is complicated
<geist>
klange: the hardware simply drops you into supervisor mode with the same handler as everything else
<geist>
all the registers are user space's oyster
<heat>
you have no idea if you're in user-space or kernel-space when you enter the trap handler, so you need to swap a register with the sscratch register (and if it's 0, you're in kernel space, else user)
<geist>
(nothing is banked or saved for you) so it's even harder than arm64, beacuse there's no banked SP
<geist>
and yes, what heat just described seems tob e the standard solution
<geist>
the one control reg you have that you can use as your anchor is 'sscratch'
<heat>
that makes you need to swap again if you're in kernel space
<geist>
which you can swap with, which is pretty neat (and unique to riscv)
<geist>
so that's your breadcrumb you leave to get back to a working state
<heat>
then when you're making an interrupt frame you need to save some sort of sp, which you will need to restore in user space but not in kernel space
<geist>
i fought it for a while before i gave in and did the 'standard' solution
sprock has joined #osdev
<geist>
supervisor -> supervisor is easy because you can just push stuff on the stakc like you usually do
<klange>
I'm still thinking I'll avoid riscv until there's some viable commodity hardware for it.
<geist>
for running a gui OS like yours that's probalby not a bad strategy
<geist>
since it'll probably just be dissapoint to try to run a ui on TCG only
<geist>
heat: so sounds like you found the same solution: put the SP in sscratch when in user space
<geist>
and then use that as your 'did i come from user space or not' sign
<heat>
I use the tp because I need percpu storage
<geist>
but user space can trash tp
<geist>
oh and then you recuver the SP from the percpu field
<geist>
that should work too, yeah
<heat>
they can't trash sscratch so I keep the kernel tp in sscratch (like a swapgs kind of thing) and then recover the kernel's sp from percpu storage (what I was doing rn) and store the user sp to percpu storage
<geist>
i stored the percpu pointer on the stack itself which the code then basically pops off
<geist>
but either way works. the percpu->sp solution should be quite simple
<geist>
also dont forget to recover gp as well (if you're using it at all)
<geist>
i think i went for a while there forgetting to do it and it was trashing things funny
<heat>
yeah did that
<heat>
I've looked at your code + other OSes
<geist>
oh i think you asked te other day if there was a big difference between 32 and 64 bit and the answer is no
<geist>
it's basically identical arch, you can compile the same code for both, it's just the size offets and whatnot as you've seen in my code
<geist>
if you want to conditionalize it
<heat>
yeah I read that
<geist>
the ISA itself is surprisingly not entirely a subset of each other like i'd expect
<geist>
i guess that wasn't a goal. ie, you can't transparently run rv32 code on a rv64 without at least a mode switch
<geist>
since a couple of core opcodes are different
<geist>
iirc POWER/PPC has that property, which is kinda neat
<geist>
though i think theres still a mode bit there because of the way condition registers work...
<heat>
geist, are CSRs supposed to be fast to access or more like MSRs?
<geist>
i think they're pretty fast yeah
<geist>
i dont think the riscv spec is complex enough to specify what their memory model is yet though
<geist>
ie, do the serialize, can you rely on writing to one taking effect immediately, etc
<geist>
the ARM ARM has a lot of verbiage to that effect. because everything is complicated on ARM
<heat>
huh freebsd's trap code looks pretty similar to yours
<geist>
yah i think i ended up using that as the model
<geist>
it's kinda one of those 'cant be unseen' sort of things
<geist>
riscv there's basically one way to do it so you kinda end up trying to do it differently, but there's not a lot of options
<heat>
linux does it differently but it's honestly confusing me
<geist>
i think that's where i got the idea of stuffing a few things on the stack itself
<geist>
agreed
<geist>
as are all exception handling routines in linux, t's always hyper complicated
<nomagno>
Now that I have subroutines to manipulate stacks, but subroutines are stackless, I have the kinda funny issue of subroutines being completely prescindible
<nomagno>
I guess I should just keep it
<nomagno>
It's like goto
<nomagno>
Then I can use jmp and in-memory stacks for actual subroutines
Jari-- has joined #osdev
<geist>
hmm, what did you mean by 'prescindible'?
<geist>
i dont thik that's a word
<Jari-->
morning all from Finland, European Union
<gog>
mew
<heat>
Finland, Scandinavia, European Union, Europe, Earth
<Jari-->
heat, da ball
<nomagno>
geist;: dispensable, not necessary
<gog>
sol, orion arm, milky way galaxy, local cluster, universe
<gog>
universe might be a redundant line on the address
<heat>
u n i v e r s e
<nomagno>
Universe isn't really a physical place, it's a natural language thing to refer to "everything"
<gog>
hm true
<nomagno>
Hence the term "multiverse" only signifies a wrong use of "universe" in things like science fiction contexts
<gog>
the cosmos
<gog>
how about that
<heat>
u n i v e r s e
elastic_dog has quit [Ping timeout: 245 seconds]
<gog>
yes
elastic_dog has joined #osdev
<heat>
woohoo traps work i think
<heat>
need to add exception handling and irq handling
<geist>
woot
<heat>
does riscv have an ABI red zone?
toaru-aarch64 has joined #osdev
<toaru-aarch64>
oh good that works
<toaru-aarch64>
not that I really expected otherwise, it's a pretty straightforward pci device
<geist>
heat: not that i know of, no
<geist>
AFAIK it's only x86-64 that has it
<geist>
toaru-aarch64: woot, which one?
<toaru-aarch64>
e1000e
<heat>
congrats
<geist>
yah
<toaru-aarch64>
building a virtual bastard child of Intel and ARM here
<toaru-aarch64>
I think from the -input stuff I have what I need to do virtio-net as well, so I'll work on that
<toaru-aarch64>
but just wanted to make sure the network stack was working in the general case
<geist>
yah and it's not really even an impossible thing. i have an arm machie with real pcie cards and it really has a e1000e and a nvidia card in t
<geist>
and that's all totes mcgotes
<toaru-aarch64>
the ac'97 is a little more wrong, but it's equally so on the x86 machines I configure as well :)
<kazinsal>
yeah, intel NICs are so extremely prolific that you can shove 'em into anything with PCI(e) and it'll most likely work just fine
<toaru-aarch64>
still need to support module loading, these drivers are baked in right now
<toaru-aarch64>
which led to a minor oopsy as the e1000 driver does some sleeping and device init in its load function which is fine for a module loaded by userspace, but not so fine if I'm just calling that function from the kernel startup
<toaru-aarch64>
slipped it into a 'worker thread' and hello internets
<heat>
geist, do nvidia cards have arm64 firmware included?
toaru-aarch64 has quit [Quit: wheee]
<heat>
or does it not even run firmware?
cppimmo has joined #osdev
mahmutov has quit [Ping timeout: 240 seconds]
joe9 has joined #osdev
k8yun has joined #osdev
<kazinsal>
geist: you're a terrible influence. VAX and associated dongles will be arriving the first week of march.
dude12312414 has joined #osdev
<geist>
yay now i'm not goig to be the only one blabbing out it
<geist>
whatcha gonna do with it?
<kazinsal>
not sure, probably gonna set the scsi2sd up to have a few different images on it, one of which will be VMS, another probably some flavour of unix/unix clone, and maybe a third to fiddle around with for osdev
vdamewood has quit [Read error: Connection reset by peer]
vdamewood has joined #osdev
<geist>
reminds me, one weird thing i saw on that hardware: the serial ports are serviced by a single set of registers
<geist>
how you ask? the FIFO holds characters for both ports. when you read a char out of the fifo it has a field for which port it was
<kazinsal>
nice, all my VAX bits are expected to arrive the same week, except the SCSI2SD which is coming locally and should only be a few days
<kazinsal>
the VAX itself is coming from the US, as is the AUI transceiver, so hooray I get to suffer through eBay's global shipping program
<kazinsal>
console cable is coming from Germany, oddly enough
<vdamewood>
kazinsal" Dude, you're getting a VAX
* kazinsal
spontaneously grows a giant grey beard
<vdamewood>
How big is it? Full rack?
<kazinsal>
desktop, microvax-3100/40
<vdamewood>
Ah, not a mini model.
<vdamewood>
Cool.
<kazinsal>
yeah, this one's decked out with a whopping 32 megs of RAM too
<kazinsal>
built-in 10 megabit ethernet too!
<vdamewood>
Neato.
<geist>
yah it's pretty big. it's 'desktop' in the sense that they must have had big tables back then
<geist>
and it's of course made like a frickin tank
<kazinsal>
yeah, in reality it'll probably go under my desk next to my server and switch
<vdamewood>
So, it's not a compact little box like my SparcStation IPX was.
<geist>
oh no not at all. more like a sparcstation 10 or 20 but about 3 times as thick and not as deep
<geist>
it might be rack sized width wise, but it's probably a 3U or 5U or so?
<kazinsal>
something like that, yeah
<vdamewood>
So, a few pizzaboxes stacked on top of eachother.
<vdamewood>
(Who can guess which commercial Unix I had the most experience with.)
<vdamewood>
Oh, wait, damn. Second most. I just remembered my laptop runs a commercial Unix.
<geist>
i thnk i have mine literally sitting on top of a sparcstation 20 on the floor
<geist>
i should fire up the ss20 again too
<kazinsal>
hmm, I'm going to need another UPS or power bar...
<geist>
but both it and the vax use a non negligable amount of power so i've been hesitant to leave it on all the time
<geist>
though the ss20 burns a solid like 200W doing nothig, iirc the mvax is like 95W so it's fairly tolerable
<vdamewood>
I miss my SS IPX. I had to get rid of it because I lost the monitor in an unpaid-for storage unit.
<geist>
and since it's cmos i think i remember it actually using less when not active
<geist>
the ss20 though was apparently BiCMOS and just pulls max power constantly
<geist>
vdamewood: yah i had a huge 21" sun monitor for a while but i got rid of it ages ago because it was huge. but I do have a sparcstation LX and just use it over serial
<geist>
i tried to find a monitor that would sync on green with it, but haven't had any luck there
nyah has quit [Quit: leaving]
<geist>
and i only have a couple of LCDs that take VGA anyway, so i'm basically outta options unless i start buying displays which i dont want to
<vdamewood>
I didn't know enough about EE at the tome to build my own DB9-to-whatever-SSIPX-uses cable, and couldn't find any.
<vdamewood>
I'm not even suer if the SS IPX used RS232 or not. I assumed it did, though.
<geist>
13W3 i think is what it's called
<geist>
yah i'm sure it does
<geist>
the LX does and it's a bit later model, but AFAIK all the sparcstations have at least one DB25 or at least a RJ based serial port
<vdamewood>
Mine didn't.
<geist>
big annoying things with those old sparcstations: their cmos battery has almost certainly died by now
<geist>
and they have the annoying dallas-chip-wiht-integrated-battery thing
<geist>
iirc the mvax just takes a cr2032 i think
<vdamewood>
Mine had a dead chip. I bought a replacement and got it almost setup.
<vdamewood>
For some reason there was always a problem with it and I'd have to hit b to boot the thing up.
<geist>
yah my lx and ss20 are both in that category
<geist>
as a result they both think they're aa:aa:aa:aa:aa:aa ethernet
<geist>
but as long as i dont have both of them on at the same time..
<vdamewood>
I put the fractory chip back when I got rid of it.
<vdamewood>
Oh, I found instructions for changing the MAC address for mine.
<vdamewood>
Or rather, I extrapolated them from the instructions to replace the NVRAM chip.
dude12312414 has quit [Quit: THE RAM IS TOO DAMN HIGH]
<vdamewood>
Though, I remember the 3 high-order bytes being hard-coded on the system somewhere else, and only the three low-order bytes being changeable with the NVRAM.
<vdamewood>
I'm kind of sad that Solaris is no longer on the registry of Unix systems.
AeroNotix has quit [Quit: WeeChat 3.4]
<heat>
i think ld.lld on riscv is all sorts of broken
<heat>
ld.lld: error: arch/riscv64/image.S:30:(.boot+0x104C): relocation R_RISCV_PCREL_HI20 out of range: -1048329 is not in [-524288, 524287]; references __global_pointer$
<heat>
this links correctly with GNU ld
<Griwes>
I smell bug reports in the air
<heat>
weird smell
<Griwes>
at least it's no longer necessary to have had an account five years ago to file an llvm bug
<klange>
spinning up so many device drivers that were mucking with page tables simultaneously uncovered some insufficient locking
<klange>
happy to have this hvf environment to test on, as it really helps squeeze out these things
<klange>
still need to get all these static object relocations figured out so I can load modules, they are very different from the ones on x86, or the ones in shared objects
mim is now known as MiningMarsn
ElectronApps has joined #osdev
<heat>
ok I fixed it
<heat>
dunno if something in the link made the relocations not work
<heat>
for the .boot section(which runs at the physical load address) I made something that apparently is 32-bit ARM assembler-like
<heat>
it was having trouble relocating to kernel symbols in high memory
<heat>
actually I can ditch the inlining of .dword __global_pointer$
<heat>
still need the ld though
cppimmo has quit [Ping timeout: 256 seconds]
<geist>
you should be able to just do a ld gp, symbol
<geist>
but it will probably emit it as an auipc; add
<geist>
which is generally good, but may be pc relative which you dont want
<geist>
and worse, may try to use gp to load it, but you can disable that temporarily
<geist>
.option norelax
<geist>
er sorry, la
<geist>
la is a pseudoisntruction for load address
<geist>
for some reason i use lla in my code, and i forget precisely what the difference is
<heat>
it's not working though
<geist>
what are you trying to do?
<heat>
lld is finding it really difficult to relocate to anything in higher half (-2GB) from lower half (the default virt load address, +2GB and something)
<heat>
GNU LD does it easy pz
<geist>
oh. dunno there, was going to suggest using mcmodel=medany
<geist>
but you probaly already are?
<heat>
yeah
<heat>
i'm just getting issues in the tiny boot stub written in assembly
<geist>
that being said you should be able to use lla to get the gp and it would work
<geist>
though it will explicitly use PC relative setting, so maybe you have to fudge the GP once it is relocated
<heat>
guess not huh
<heat>
unless lld's overflow checking has a bug
<heat>
when I compile everything with clang and call riscv64-onyx-gcc to link, it links
<heat>
when I use clang + lld, it fails to link
<heat>
same object files, slightly different layout
<geist>
yah, though wasn't it already pretty established that lld has issues with riscv? i thought it was you a while back saying the GP stuff was basically unimplemented/broken on lld
<geist>
or in general loads of relaxations, and the GP stuff is a relaxation
<heat>
yeah I'm compiling with -mno-relax
<geist>
sp GP itself probably should just not work in that case
<geist>
are you passing -mno-relax to the linker?
<geist>
it's the one that's doing most of the work
<heat>
the linker doesn't look at mno-relax because it literally can't relax
<heat>
i tried it with and without -mno-relax
<heat>
(passed to the linker, but I'm always passing it to clang so lld can actually link)
<heat>
linux links the code that does early boot init mmu stuff and whatnot at -2GB huh
<heat>
guess all data references are relative?
<geist>
yah or at least can easily write asm to do that
<geist>
that's why in my start.S code in LK for example it's pretty careful to use lla pseudo instruction to load addresses of things
<geist>
since lla explicitly uses auipc, it'll always end up computing the physical address of the symbol
<geist>
and then once it enables the mmu and bounces it goes back and fixes up gp, etc to be correct by adding the offset
<heat>
x86 linux even calls C code while linked to a wrong address
<geist>
yep, we do the same thing in zircon too. it's risky in the sense that you're relying on the C code to emit PIE code, and it generally can as long as you stay away from certain constructs
<geist>
(like vtables, or grabbing pointers to globals, etc)
<geist>
but for fairly PC relative arches like riscv or arm64 it's pretty safe
k8yun has quit [Quit: Leaving]
<heat>
if I can get away with it in x86, riscv, arm64 that might be a more solid approach I guess
<heat>
at least in riscv it would solve this little issue
<heat>
yeah linking to a higher address works
<heat>
woohoo I guess
<heat>
i'd like a way to know it blew up at link time but that's probably hard (unless I scan the relocations in the object files)
sprock has quit [Ping timeout: 256 seconds]
mahmutov has joined #osdev
zaquest has quit [Remote host closed the connection]
heat has quit [Ping timeout: 250 seconds]
zaquest has joined #osdev
<geist>
yah link it where it wants to run and carefully make sure pre-mmu code can run PIC
<geist>
that's usually worked for me
elastic_dog has quit [Ping timeout: 240 seconds]
elastic_dog has joined #osdev
ThinkT510 has quit [Quit: WeeChat 3.4]
ThinkT510 has joined #osdev
mahmutov has quit [Ping timeout: 256 seconds]
<mrvn>
geist: any kind of address taking of globals will totaly screw up the PIC/PIE.
<mrvn>
like getting a pointer to the initial page table to initialize the MMU.
<geist>
right
<mrvn>
Makes absolutely no sense. At least with memory model small the compiler should emit all PC relative code.
<geist>
well, it wont always, but you gotta be careful
<geist>
function pointers for example are right out (hence vtable)
<mrvn>
any pointer in data is right out. Impossible to use with gcc/clang.
<geist>
well, no. i wouldn't say impossible. but it depends
<geist>
and yes, it's kinda dangerous. so usualyl what i do in this case is write helper C routines that take all their args from asm code that can do the right thing
<geist>
like, for example, building a page table with the args passed in to point at the page table, what you want it to set to, etc. and have the C routine only operate on what it was given
<mrvn>
a pointer in data will always be initialized statically to the address you linked at.
<geist>
well, not if it's compiled as PIE, and the hardware can support it
<geist>
might lea it effectively off PC relative
<mrvn>
geist: nope. gcc/clang will initialize it as data
<geist>
fine.
<mrvn>
it's totally more efficient. But breaks PIC/PIE.
<geist>
right. and if you compiled it as PIE then that's a bad compiler
<geist>
if you compiled it as PIE it should probably not do exactly what you just said
<mrvn>
geist: I'm taling things like: void *bla = &foo;
<mrvn>
If you have complex code when "void *p = complex_runtime_function();" will generate a .ctor entry. But anything the compiler can optimize ends up in .data. There is no flag to tell gcc/clang to always use .ctor to generate the address with PIC/PIE.
<geist>
yes. so dont do that
[itchyjunk] has quit [Read error: Connection reset by peer]
<bslsk05>
fuchsia.googlesource.com: zircon/kernel/arch/arm64/boot-mmu.cc - fuchsia - Git at Google
<geist>
i wrote that code in zircon to initialize the page tables while the cpu is still running in physical addrses space
<geist>
it works, because it explicitly doesn't do things like you said
<geist>
it only calls out to `boot_alloc_page_phys()` which is i believe defined in asm to work
<mrvn>
lambdas? Doesn't that compile in the absolute address of the code for that?
<mrvn>
at least potentially
<geist>
nope, because the _arm64_boot_map function is inline above
<geist>
potentially yes, but in practice no
<geist>
but that's just a detail, doesn't have to use a lambda. that's because the inner routine is shared by another routine later
<mrvn>
geist: how do you get gcc to not use memset to "optimize" line 159?
<geist>
the volatile ptr
<geist>
basuically it's a house of cards (we're replacing it with something more fancy soon) but it works and if the compiler decided to do something else it'd fail spectacularly, so it'd be easy to find
<klange>
just spent two hours chasing a red herring >_>
<geist>
it was all to avoid having to do this in asm which is a pain
<mrvn>
geist: I know. same problem for me.
<geist>
klange: did you at least learn something on the way?
<geist>
lots of times red herrings are at least educational
<klange>
mostly just that I'm a moron, I was chasing what looked like a bit of clobbered data because I was looking at the wrong debug symbols in gdb :)
<mrvn>
geist: For a non-PIC/PIE build I constructed my page table with constexpr and had it just end up in .data all initialized. That makes the bootstrap code rather small.
<geist>
word
<geist>
yaht he reason i had to do it more complicated on the arm64 build is because the code can't up front know where it'll be physically located
<geist>
so the start.S code computes where it is physically, and then calls this routine to construct a page table to map phys X -> virt Y
<klange>
for my kernel, I package symbols independently of the ELF stuff, mostly for old legacy reasons (eg. not trusting bootloaders to be able to provide them)
<klange>
and to make those symbol tables, I build without them, get a list of symbols with nm, generate some assembly, and then build the real kernel
<klange>
and generally the one with the symbol table will have different addresses, which doesn't matter for the symbol table itself (it's just a bunch of strings + actual symbol references, so it's resolved at link time)
<klange>
anyway, the initial symboltableless kernel sits around... and on x86-64 the same name is reused as part of a process of converting from a 64-bit ELF to a 32-bit one and stripping for size
<klange>
so on x86-64, I have this file 'misaka-kernel.x64' which is normally an unstripped 64-bit ELF, and I use that for gdbing
<klange>
but for aarch64... it's just the symbol table-less kernel... which has different addresses...
<geist>
so, i just discovered a really good reason not to use the 'struct as a way to access mmio registers'
<geist>
the virtio code i have doesn't work on 68k. because big endian
<geist>
and i was, at the time, using structs for mmio
<geist>
and so the code was left and right doing accesses via the struct, so it's really hard to endian swap without finding every single place
<mrvn>
what has one have to do with the other?
<klange>
w2b endianness as a thing you can specify on struct members
<geist>
so i got a bunch of reg = mmio_regs->the_thing
<geist>
that's easy enough because then i can just wrap it with a endian thing
<geist>
but also things like
<geist>
mmio_regs->the_thing |= 1; etc are annoying
<geist>
i mean it was lazy anyway, i generally do accessor functions/macros, but i guess this time i decided to do it as a struct because
<mrvn>
structs are a C hack to get a unique type so you don't accidentally pass the wrong type.
<geist>
also ugh, the same thing with the virtio ring. it has a bunch of in-memory descriptors that i fiddle with
<geist>
but need to endian swap
<mrvn>
In c++ the struct should make it simpler because you can make the "the_thing" private and add accessors that byte swap.
<geist>
yah i'm debating doing precisely that: converting the file from C to C++ and then doing that
<mrvn>
In C I would rename the struct/the_thing and then fix all the compile errors.
<ddevault>
but dumping the first sector into RAM and jumping to it is as simple as it gets
<geist>
if you're booting off something that has a sector, yes
<ddevault>
if only there were any good bootloaders to put in that first sector...
<kazinsal>
back in mah day we had to toggle the bootloader in with switches and we liked it! *shakes fist*
<moon-child>
kingoffrance: didn't mean to imply they were. uefi is a standardised interface and bloated committeeware. bios is neither
<kingoffrance>
:)
<moon-child>
kingoffrance: point is, to me, the fact that uefi is a standardised interface outweighs the fact that it's bloated committeeware
<ddevault>
has anyone booted their hobby OS with syslinux?
* ddevault
reads the wiki
<geist>
moon-child: agreed
<ddevault>
BIOS is semi-standardized
<geist>
after having dealt with lack of any standard at all over in the arm world, i'd take a bloated standard interface any day
<Belxjander>
ddevault: I've had to install windows off a network link using syslinux to launch the installer.. THAT was a complete nightmare and then some
<ddevault>
I solve this particular problem by not using Windows
<moon-child>
geist: they are slowly converging, right?
<geist>
moon-child: unclear. hobby stuff tends to just use uboot, so that's generally nice if you get a random board
<geist>
but consumer devices are still all over the place
<mrvn>
isn't everything relevant using linux's way to boot if not outright using linux?
<geist>
well thats sort of the backwards way of thinknig about it. as in via a set of ad hoc solutions they get linux into memory yes
<geist>
but that doesnt mean they're standardized
<ddevault>
u-boot with UEFI and a device tree is one of the sanest boot protocols I've seen
<ddevault>
which is really saying something -_-
<geist>
and how they get linux out of flash/disk/aether/etc is all over the place
<mrvn>
I was refering to having a device tree and call the kernel with that
<geist>
sure
<mrvn>
How you get to the point where the kernel gets called I mostly don't care about
<geist>
except if they dont. ie, apple products, things that run windows, et
<geist>
sure, but that's The Bootloader, which was the topic at hand
<j`ey>
servers
<ddevault>
bootboot is promising for my needs but why in seven hells does it use fasm
GeDaMo has joined #osdev
<ddevault>
and that osdev.org disclaimer in the readme does not fill me with confidence
<geist>
oh? what is this bootboot?
<kazinsal>
the core purpose of operating systems development as a hobby is to spite the newcomer at every turn
<kazinsal>
yeah he's eaten a few forum bans for being a bit unhinged
<mrvn>
One thing that would be nice in a bootloader protocol that uefi did right is providing boot services, like writing to console.
<ddevault>
I have a working riscv64 kernel booting via u-boot + UEFI, which is somewhat sane and straightforward
<klange>
i wish microsoft was paying me
<ddevault>
I want to port it to x86_64 and I am not having a good time
<geist>
ah
<kazinsal>
he's not full on terry a. davis (RIP) but he's definitely not firing on all cylinders
<j`ey>
I thought u-boot supported x86 too
<ddevault>
mrvn: well, I'm conflicted about this. The filesystem driver is kind of useful but also seems out of scope for a bootloader. Serial console is fine, but it would be better if it just initialized it and left it there for you to use. The rest of UEFI boot services seems out of scope imo
<ddevault>
j`ey: I don't actually want to use u-boot on x86
<ddevault>
it's not good, its purpose is just to provide UEFI on this one platform
<mrvn>
ddevault: who says the console is serial? Can be text mode or a framebuffer
<ddevault>
good point, mrvn
<mrvn>
ddevault: and as for fiulesystem drivers. If the bootloader can read filesystems it makes no sense not to provide that as service too.
<ddevault>
"if the bootloader can read filesystems" is where you probably lost me
<ddevault>
anyway, I'm just going to write a UEFI binary and give up
<geist>
bootloaders reading filesystems is very common
<mrvn>
ddevault: the bootloader needs some way to load the kernel from somewhere. If it's not just a fixed address in flash then it has to have filesystem drivers. And then why not let the kernel load it's initrd or plugins using the same code?
<ddevault>
see BIOS for my preferred approach
<geist>
right, lets you load your kernel out of /boot/foo or something (ie, what grub does, or NTLDR, or whatnot)
<geist>
well why aren't you just writing a bios based loader then?
<ddevault>
because of all the other things BIOS is
<geist>
you're porting to x86 right?
<geist>
oh?
<ddevault>
x86_64, rather
<ddevault>
the part which loads from the disk is fine, but all of the other bullshit on this platform is less good
<geist>
the other bullshit?
<mrvn>
ddevault: the part that loads from disk is horrible.
<ddevault>
ask me again in a week to enumerate the bullshit
<ddevault>
it's been a while
<geist>
heh, okay. honestly i'd just start with something like grub and move on
<geist>
there's really no reason to be hacking a new loader nowadays. or if so, UEFI i guess
<kazinsal>
the most notorious piece of bullshit about the x86-64 platform: you can find them ANYWHERE. how cruel is that?
<geist>
(on x86 PC that is)
<ddevault>
I have a problem where I like to use good software
<ddevault>
and if good software doesn't exist, it's a yak to shave
<mrvn>
e.g. you must load the kernel from a FAT12 from USB. But you must use FAT32 when loading from disk. So what if you put your disk in an USB enclosure?
<ddevault>
grub is not good software
<ddevault>
UEFI is also not good software, but at least it's not grub, so for now it'll have to do
<geist>
oh? it's not good?
<ddevault>
grub is the emacs of bootloaders
<geist>
and thats bad precisely because?
<ddevault>
not that emacs is so terrible, but I don't want emacs booting my system thank you very much
<zid`>
grub 0.97 for life
<ddevault>
bootloaders should be small and simple and should load the kernel and then bugger off imo
<geist>
but it's not emacs.
<kazinsal>
I'm sure there's an emacs.mod for grub2
<geist>
okay, but who gies a shit if its complicated as long as it does what you want
<ddevault>
me
<kazinsal>
why
<geist>
okay...
<zid`>
grub 0.97 is dead brainless
<moon-child>
just like me!
<geist>
LILO!
<klange>
while that mindset is an admirable one, it is not compatible with x86
<mrvn>
Seriously, what where they thinking with EFI there? Why have a block layer and a filesystem layer if you then only support specific combinations of each?
<Griwes>
except that there's complicated things that need to happen before a kernel can do its things (unless you have drivers for *everything*)
<geist>
+1 klange
<zid`>
100k file called stage2_eltorito and a 20 byte config file is all my OS uses
<ddevault>
I know, which is why for now I have resigned to using UEFI
<Griwes>
uefi is nice <shrug>
<geist>
interesting you were using it on riscv. what platform were you booting on there?
<zid`>
and the eltorito file can do all sorts of crazy magic I wouldn't wanna do myself
<ddevault>
hifive unmatched
<Griwes>
beats all the other things before uefi by a mile
<kazinsal>
I'm so tired of all these magic weird future systems that I bought a VAX
<geist>
ah. and you had the uefi build on it huh
<klange>
uefi is basically microsoft reimplementing grub in firmware anyway ;)
<geist>
usually it was grub based
<ddevault>
well, I flashed a u-boot build to the SPI chip
<ddevault>
which has UEFI support built in
<geist>
ah. okay that
<ddevault>
which for linux I use to load grub, but for my kernel I just boot directly
<geist>
probably didn't need uefi there. could just load yourself via grub as a .bin file
<ddevault>
and honestly this arrangement is pretty decent compared to most systems
<ddevault>
yeah, but I essentially wanted to build out a somewhat standard boot process
<geist>
er s/grub/uboot
<ddevault>
so that I could upstream a standard ISO build for Alpine Linux
<ddevault>
which you plug in via USB and boot via UEFI and everything Just Werks with no fuckery
<geist>
fair enough
<ddevault>
goal is to minimize fuckery
<kazinsal>
I guarantee you will spend twelve to eighteen months on this and then discover that nothing Just Works
<ddevault>
I have spent twelve to eighteen months on this and I am proud to say that it just werks but only thanks to months of beating it into submission thank you
<geist>
yah with booting on PC my suggestion is minimum fuckery and move on. dont write anything new that isn't already there
<g1n>
hello, i am reimplementing page frame allocator, currently i am not doing true bitmap'ing (i don't fully understand yet), but please let me know if i am getting closer to correct solution, here is code: https://ttm.sh/i_o.h - paging.h, https://ttm.sh/i_J.c - paging.c, thanks
<ddevault>
ah well on PC not so much
<geist>
g1n: does it work?
<mrvn>
g1n: bitmapping is stupid for it. so good for you.
<geist>
mrvn: do you have a suggestion?
<mrvn>
g1n: can't read your urls. That's C source and firefox can't show that.
<g1n>
geist: yes (i think), but i am not sure if it works correct and i am just using list of uint32_t
<zid`>
Your bitmap is a uint32_t btw
<g1n>
yes
<geist>
g1n: at first glance i'd say the kalloc_frame is a little weird
<mrvn>
g1n: for page frames? A linked list, stack or dequeue works well
<geist>
it tests for USED/FREE and then returns the value of the bitmap i-1
<geist>
which dosen't make any sense to me
<zid`>
It's a uint32_t map, not a bitmap
<mrvn>
g1n: don't use uint32_t for addr. make a separate type
<g1n>
geist: oh, thanks for noticing it was from previous implementation
<geist>
sure, but even if it's a waste of 31 bits, it's still returning a weird value
<geist>
cool, yeah i should be returning something like i * 0x1000
<g1n>
geist: is using unsigned char better?
<geist>
yes. and then better than that is packing it in so it's truly a bitmap
<geist>
bit baby steps. a 8 bit element will use 4x less ram
<zid`>
The idea of a bitmap is to use each bit, you've made like.. a bitmap that's 32x bigger than a bitmap.
<geist>
bit that's just an optimization. in broad strokes the allocator/freeerlooks okay
<geist>
though it's also probablyu unnecessarily using a |= instead of =
<g1n>
yes
<zid`>
char bitmap[2930]; bitmap[0] & 0x80 would be one bit, bitmap[0] & 0x40 another, bitmap[1] & 0x20 another, etc
<mrvn>
g1n: A bitmap has O(n) complexity and you want your page frame allocation to be always fast. Forget about bitmaps there.
<geist>
g1n: also since you're |= and FREE = 0.... it's not actually freeing it
<zid`>
err [0] & 0x20
<geist>
in kfree_frame()
<zid`>
&= ~FREE probably
<g1n>
ok
<zid`>
is what you want to unset a random mask
<mrvn>
zid`: ^= FREE?
<zid`>
that toggles
<geist>
you see why there? `|= 0` is a no-op
<zid`>
&= ~ is unset
<zid`>
|= is set
<geist>
doesn't do anything
<GeDaMo>
table[page_number>>3] & (1 << (page_number & 7)) for 8-bit array elements
<geist>
lets not confuse here
<zid`>
This is why I tried to teach him about cows last time but he rank away
<zid`>
ran*
<geist>
lets not focus on trying to optimize this thing before it works
<geist>
baby steps. char array, either set to USED or FREE
<GeDaMo>
"These cows are small but the ones out there are far away" :P
* mrvn
still wants to have a struct { uint64_t mask:1[64]; }
<g1n>
ok let me try to fix that
<geist>
sde note: there's no need to align the bitmap
<geist>
but the page table and page dir, yes
<mrvn>
g1n: Why don't you use a linkes list for page frames?
<mrvn>
linked list
<g1n>
mrvn: last time i tryed to implement it was not very success
* Griwes
doesn't want to even think about the levels of disambiguation mrvn's idea of bitfield arrays would require to be parseable at all
<geist>
yah lets keep it simple
<geist>
bitmaps are totally fine for just getting started
<mrvn>
geist: waste of time to implement them at that point and never a good solution
gog has joined #osdev
<mrvn>
and as we see again bitmaps aren't simple
<geist>
happy to see you again too mrvn
<geist>
we're arguing again just like old times
* Griwes
still has occasional flashbacks to that one time the c++ committee was deciding the disambiguation rules for default initializers for bitfields
<mrvn>
geist: sorry
<geist>
heh
<geist>
you never did check out that verilog code i wrote for you that one time!
<kazinsal>
makes more sense to get babby's first allocator working and do drop-in optimizations/rewrites later
<geist>
still sour about that
<mrvn>
Griwes: oh yeah, the syntax is horrible. But you get the idea.
<Griwes>
yeah, I get it
<kazinsal>
then eventually you become so jaded about optimizing things and fighting with platform emulators that you give up and become a hermit writing code for platforms that were obsolete 30 years ago
<geist>
yesssss
<Griwes>
I would *not* want to be the person to try and shepherd that through either of the iso committees
<geist>
kazinsal: ones where bitmap allocators are implemented in a single instruction
<kazinsal>
ha!
<kazinsal>
Bafflingly Complex Instruction Set Compute
<mrvn>
kazinsal: writing pong for the C64 in C++20?
<kazinsal>
r
<geist>
(i think there may actually be find bit set/clear in array instruction)
<Griwes>
more on topic tho, I'm definitely with mrvn here, bitmaps are anything but simple
<kazinsal>
mrvn: geist talked me into buying a microvax to hack on
<bslsk05>
'DooM E1M1 video running on a Commodore 64 (nuvie)' by ibm5155 (00:01:17)
<mrvn>
kazinsal: don't get the one with "Starkstrom".
<geist>
Griwes: not sure i agree with you there
<GeDaMo>
To me, bitmaps are much easier than linked lists
<j`ey>
Griwes: i think theyre simpler thank linked lists
<zid`>
yea bitmaps are way trivial
<geist>
obviously an optimized one is more trivial than not, but the kinda bitmap g1n has here where it's basically an array of bools is about as dumb as it gets
<zid`>
especially once paging gets involved
<GeDaMo>
I don't think I've ever seen a C linked list implementation that worked first time
<zid`>
paging has issues in that you can't actually unlink anything without mapping it first
<Griwes>
well, people keep getting the bit fiddling wrong, as evidenced before
<kazinsal>
on i386 you have, what, bt/bts/btr?
<j`ey>
you can get linkedlists wrong :p
<geist>
kazinsal: yah though i dont think there's a 'keep searching subsequent bytes' version
<geist>
ie, string versions of it
<zid`>
I think a *very* noob person tends to get linked list /code/ wrong for a long time
<geist>
j`ey: not if you're using VAX!
<Griwes>
well, you don't actually need a "linked list" for this
<zid`>
but will ultimately get a working bitmap
<mrvn>
zid`: a stack as single linked list? Just push and pop. That's less code than a bitmap.
<kazinsal>
yeah no idea if there's a "bit scan"
<zid`>
mrvn: pop requires it to be mapped.
<zid`>
I just said that
<geist>
mrvn: push/pop also requires code that does the push/pop
<Griwes>
like, actual linked lists are hard because it's easy to fsck up setting prev/next
<Griwes>
agreed there
<mrvn>
Griwes: that would be a doubly-linked list. Much harder.
<zid`>
push pop for a linked list is really easy, but arriving at that easy code is often fairly hard for amateurs
<kazinsal>
there's a "bsf" for "bit scan forward" but it operates on a single r/m8 I think
<kazinsal>
no idea if you can slap a rep prefix on it
<Griwes>
but since a long time ago my frame stack is just a pointer
<zid`>
removing from the middle of a list is the thing that kills all the beginners
<Griwes>
it points to the next free frame, and that frame contains the next pointer at offset 0
<mrvn>
zid`: if you need that then a single linked list isn't the right data structure.
<zid`>
I use a linked list and just mapped the entire list into memory elsewhere so I can pop it for free, mega lazy :P
<kazinsal>
"If the content of the source operand is 0, the content of the destination operand is undefined." well that's just funky
<zid`>
mrvn: Incorrect
<Griwes>
and yes, that requires either the full physical memory to be mapped, or to have a (per-core) window where you do temporary mapping
<Griwes>
but honestly neither of those is all that hard to do
<moon-child>
kazinsal: can rep cmps
<moon-child>
but that is slow
<zid`>
Basically, amateurs never realize you need to struct node ** when dealing with linked lists so that you can modify the head, so they constantly have issues trying
<zid`>
not realizing the head has been passed by value
<mrvn>
zid`: you are thinking way to complicated and optimized here
<Griwes>
again, you don't actually want linked lists here for the *simple* frame allocator
<geist>
gog!
* moon-child
pets god
<Griwes>
so all this is moot
* klange
has been doing just fine with a bitmap for years
<kazinsal>
geist: ooh, yeah, that looks useful!
<mrvn>
zid`: all we need is a stack implemented as linked pages.
* kazinsal
headpats gog
<zid`>
I've been in C support channels, 99% of issues with "I can't implement a basic linked list, it crashs sometimes at random" is because they used a single pointer and tried to free the head, which was passed by value
<Griwes>
moon-child, put that god back where you found it
<zid`>
absolutely nothing to do with optimization, which you already told me
<Griwes>
zid`, noone is talking about using linked lists
<zid`>
I already told you* brainfart
<zid`>
Except all the people who are
<geist>
kazinsal: yeah it's subtle. it uses the 8 byte value to index into a table which it then ands with the mask operand
<Griwes>
*sigh*
<mrvn>
zid`: using ** is an optimization
<zid`>
It's not an optimization.
<g1n>
hi gog
<Griwes>
okay, read whatever the hell you want from that messages
<zid`>
It's *mandatory* to be able to change the head
<geist>
so it seems like you could build some table that causes it to skip 0xff and tell you which is the first bit set maybe
<Griwes>
it's just not what is being said
<Griwes>
you need 0 "linked list" structs to do a frame stack or however you want to call it
<zid`>
correct
<klange>
'free stack'
<mrvn>
zid`: all we have is a global "void *next_free_page;"
<kazinsal>
yeah, I'm trying to think of other interesting was you could attempt to quasi-hand-optimize a bitmap allocator in various assemblers
<kazinsal>
ways*
<zid`>
mrvn: which you need a pointer to
<Griwes>
???
<mrvn>
zid`: no. it's a global.
<Griwes>
no
<mrvn>
or static
<zid`>
yea making it a global works
<gog>
clz
<mrvn>
or a member if you use c++
<gog>
or __builtin_ffs
<kazinsal>
geist: I love the parallel conversations going on here. one late night theorycrafting about hand-rolled optimized bitmap allocators using bizarre CISC instructions, one argument about the best way to implement linked lists in C
<geist>
yuo said member
<zid`>
make all your locals globals so that you can not have to learn about pointers :p
<bslsk05>
github.com: reaveros/pmm.cpp at main · griwes/reaveros · GitHub
<geist>
i think it's as lively as i've seen it in a long time
<gog>
zid`: didn't you say pointers are evil
<zid`>
kazinsal: It's not even the best way, it was a discussion about how amateurs always get linked list code wrong
<zid`>
and a bunch of people jumped on me for other shit
<Griwes>
you started talking about how people get linked lists wrong, and we're saying this ain't even linked lists
<geist>
i'm always a fan of figuring out the asker's skill level and tune the response to it
<zid`>
Great, I am not talking about a this.
<mrvn>
zid`: I think you also are thinking about the wrong ** thing. You make the prev pointer in the doubly linked list a **
<Griwes>
`ret->next` linked above is spelled that way only because that makes everything easier to type
<zid`>
I am talking about amateurs implementing linked lists.
<zid`>
mrvn: No, and no.
<mrvn>
stupic type hack in C
<GeDaMo>
The best way to implement a linked list is in an array :P
<kazinsal>
yeah, it's important to figure out if we're dealing with "hey, I've implemented $thing but I want to make it better, is $otherThing a good way of doing it"
<Griwes>
okay, so I have no idea where did this particular discussion sprang up from then
<zid`>
GeDaMo: that's called a zidlist :p
<kazinsal>
or if it's a case of "my $thing isn't working and I've stared at it for six hours please help"
<g1n>
geist: i can't understand how (i * 0x1000), it just returns 0 on first element and why return bitmap[i] would be bad?
<geist>
i dont know! it's going critical mass
<Griwes>
because if you aren't talking about this in context of frame stack, I have no clue why we're even on linked lists lol
<mrvn>
GeDaMo: that's called an allocator
<geist>
g1n: return bitmap[i] returns the value of the bitmap. and in this case would be USED
<gog>
g1n: because the value in the bitmap isn't the address of the page
<geist>
which isn't useful. you want that function to return the address you just allocated
<geist>
which in this case is (i * 0x1000)
<geist>
since i is the index into the bitmap you found a slot, and each index represents a page
<zid`>
It stemmed from the discussion about whether 'writing a bitmap' or 'writing a linked list' is 'easier'. I said amateurs often completely fuck up linked lists for a long time (years), but eventually will end up with a working bitmap.
<geist>
zid`: agreed
<g1n>
so just return (i * 0x1000); , but how it will be even releted to that bitmap?
<geist>
g1n: well, then back tot eh basics. what is the bitmap *for*?
<gog>
g1n: ok so say you have a bitmap with 32 entries
<geist>
gog has it
<Griwes>
zid`, okay, so you took "frame stack is easier than bitmap" and turned that into "linked lists are easier than bitmap"
<mrvn>
zid`: frankly if they can't write a linked list they should not be doing kernel work. So maybe that is a good thing.
<gog>
representing 32 pages
* geist
tag teams with gog and jumps out the ring
<Griwes>
which is not at all equivalent
<g1n>
gog: ok
<gog>
then each bit of the entry represents a page
<zid`>
mrvn: most people here are incapable of writing a linked list to my standard, let's not be TOO harsh about their proficiency at their hobbies :p
* kazinsal
hands geist a stiff drink
<g1n>
gog: so it will be 32*8 pages?
<gog>
no
<gog>
each bit represents a page
<gog>
the first bit represents 0 * 0x100
<gog>
the second 1*0x1000
<gog>
etc
<g1n>
oh
<gog>
so you keep a counter of the bit at which the next free page is stored
* Griwes
hands gog a well earned fish
<kazinsal>
(aside: I feel like we should have some kind of "crash course on bitwise arithmetic" like they teach in networking classes for subnetting so people know how to frob bits easier)
<zid`>
Hmm I swear I wrote some linked list code for advent, but I can't find it
<gog>
so for a C array of a bitmap of uint32_t's, it would be at bitmap[index/32] & 1 << (index %32)
<mrvn>
gog: that's potentially UB
<gog>
mrvn: you're right
<gog>
1 << 32 is ub
<Griwes>
in other news, I'm building llvm 13.0.1 outside of a container and it installs a builtins archive into a 13.0.1 dir, but inside a container it installs into a 13.0.0 dir
<mrvn>
gog: 1 << 31 is UB
<zid`>
0-31 are the valid indicies
<g1n>
gog: so in malloc for example i need to have access to that bitmap?
<gog>
no
<mrvn>
zid`: nope, not for int32_t
<gog>
no g1n
<zid`>
it's uint
<mrvn>
zid`: it's 1
<g1n>
gog: then how to access it?
<g1n>
gog: *addr?
<Griwes>
1u << 31 is fine, 1 << 31 is not (assuming sizeof(int) * CHAR_BIT == 4)
<zid`>
oh right *that* side
* Griwes
wants to invent a time machine and make C have integer literals like haskell
<zid`>
C's integer literals is definitely one thing I'd fix
<mrvn>
Even worse is 1 >> x; you can't do an arithmetic shift in C all all without it being implementation defined.
<zid`>
Like, do I have to write 70000L? really?
<Griwes>
all the arithmetic and promotion nonsense too
<mrvn>
zid`: strictly speaking: yes
<gog>
g1n access what?
<zid`>
promotion is absolutely vital, you cannot tear it away from me
<zid`>
mrvn: yea it was rhetorical
<mrvn>
zid`: Not that it helps. L can be 16bit.
<Griwes>
promotions suck
<zid`>
promotions are *vital*
<g1n>
gog: or how to access memory?
<Griwes>
va_args is an awful thing
<zid`>
fuck off am I going to cast everything to long
<kazinsal>
promotions mean you get paid more, and sucking every dollar you can out of your corporate overlords is good
<GeDaMo>
There are also the fixed size macros e.g. UINT64_C(70000)
<zid`>
just so I can add 1 to my stuff
<gog>
g1n: we're not talking about access, we're talking about tracking right now
<gog>
accessing memory is trivial
<mrvn>
zid`: funnily "long" is the one C type that is completly broken and thereby useless.
<Griwes>
"the one" C type that is completely broken?
<zid`>
I don't remember what 6.7.2 says about long
<zid`>
I assumed >= int
<kazinsal>
that's basically it yeah
<Griwes>
yes
<g1n>
gog: ok, i am not really understand it (maybe i need to be more abstract), but let that be
<g1n>
gog: thanks
<mrvn>
zid`: it's 32bit on windows, 64bit on 64bit systems !windows
<zid`>
I know, yes, I don't care
<Griwes>
anyway I'd rewrite everything that touches integral types in C if I could
<zid`>
I wanna know what C says
<gog>
LP64/LLP64
<kazinsal>
presumably there's some ABI that someone's using for some reason that's ILP64
<Griwes>
make it make sane decisions about arithmetic conversions
<mrvn>
zid`: use int or long long. Or join the 21th century and use C99 and uint64_t.
<kazinsal>
but thankfully LP64 is p much standard
<Griwes>
no more uint16_t + uint16_t => int
<zid`>
minimum value for an object of type long int
<zid`>
-2147483647 // −(231 − 1)
<zid`>
LONG_MIN
<kazinsal>
and yeah if your machine supports 64-bit your compiler supports C99 or newer
<zid`>
yea so long is just >= int
<zid`>
so long is perfectly fine for 70000 exactly like I thought
<Griwes>
int can be 16 bits
<zid`>
either the platform has the range or it doesn't
<mrvn>
kazinsal: MSVC doesn't or didn't until quite recently.
<Griwes>
there's a quote of me yelling something like "BECAUSE INT IS FSCKING SHORTER TO TYPE THAN SHORT" in #c++
<Griwes>
back in the old days
<zid`>
What's annoying is that it won't warn or not compile unless it ends up being part of a constant initializer
<zid`>
it'll just.. wrap
<GeDaMo>
char can be 32 bits :P
<kazinsal>
yeah, MSVC leaned harder into the C++ standards because that's what Microsoft was using primarily
<Griwes>
anyway, everything related to arithmetic types is hell in C
<zid`>
msvc didn't even support C for aaaages
<Griwes>
and because of that, it's also hell in C++
<kazinsal>
a lot of low level NT stuff is still written in C mind you
<mrvn>
zid`: Hah, that's another funny thing with C++ now. Stuff like that us UB. Except constexpr doesn't allow UB. So any code you can get to compile as constexpr you get UB errors, the rest it silently breaks at runtime.
<Griwes>
and while I could be fine with it being hell in C, because C is hell anyway, I'd like C++ to not be hell when it comes to arithmetic types
<zid`>
the type promotions and type rules are mostly fine, it's the constants alone that suck
<mrvn>
Griwes: constexpr to the rescue :)
<zid`>
I don't wanna have to do special shit to add 1 to a char
<Griwes>
no, promotions are not fine
<kazinsal>
MSVC is tailored less to "write whatever damn bastard code you want" so I see their reasoning behind their various standards compliance levels is
<kazinsal>
whereas GCC/clang are explicitly meant to enable aforementioned damn bastard code
<gog>
i love writing damn bastard code though
<zid`>
go write me int a = 4; a += 1; without type promotion, so whatever fancy grammar you think is "better" than type promotion, go:
<mrvn>
kazinsal: I think MSVC was tailored to: it should compile windows and MS Word and such.
<Griwes>
uint16_t + uint16_t gives you a fucking int
<zid`>
long int a*
<Griwes>
that's *not* fine
<kazinsal>
I think as of MSVC 2019 they have explicit modes for C11 and C17 but no explicit C99 mode
<mrvn>
Griwes: only if int is large enough iirc
<zid`>
when is it not fine?
<mrvn>
zid`: when int is 16bit
<Griwes>
int a = 4; is a conversion, not a promotion
<Griwes>
oh right, it's worse, because if int is 16 bit, the result of that is an *unsigned int*
<zid`>
Griwes: long int a = 4; a += 1; the 1 is an int
<zid`>
the a is a long int
<Griwes>
...and?
<zid`>
*someone* has to do a conversion there, the programmer or the compiler
<Griwes>
...yes?
<gog>
g1n: i'm gonna throw somethig together for you to look at and try to grasp
<Griwes>
i'm not saying ban conversions
<zid`>
So answer the question?
<mrvn>
Griwes: the result of uint16_t + uint16_t is such that all results are valid.
<g1n>
gog: ok
<zid`>
What do you *want* it to look like
<Griwes>
I'm saying (1) kill promotions and (2) make usual arithmetic conversions sane
<Griwes>
long long a = 1; is fine
<mrvn>
Griwes: that it becomes a larger type is an optimization for compilers.
<zid`>
what's 'better' than what C does? 1_as_long_int? Now that code is unusable as a macro. blah blah
<Griwes>
uint16_t + uint16_t being an int is not fine
<zid`>
i want to see what the 'correct' code would look like to you
<Griwes>
dude
<Griwes>
long long a = 1;
<Griwes>
a += 1;
<g1n>
gog: oh, when i am trying to return i * 0x1000 from kalloc_frame, it should be uint32_t, but bitmap unsigned char array?
<mrvn>
Griwes: it becomes a long.
<gog>
no
<zid`>
okay so you *do* want type promotions
<Griwes>
you're latching onto a thing I'm consistently saying is fine
<Griwes>
no
<Griwes>
that's not a promotion
<zid`>
1 is promoted to long long
<Griwes>
a promotion is specifically a conversion to specifically int in very specific situations
<zid`>
you mean integer promotion
<zid`>
not promotion
<Griwes>
...
<Griwes>
yes
<mrvn>
Griwes: what is your problem with uint16_t + uint16_t becoming a larger type?
<zid`>
Okay then, yea, a few of those rules are silly
<Griwes>
we're talking about integers
<gog>
g1n well, sort of
<Griwes>
mrvn, because it's hell? because it's surprising?
<zid`>
it doesn't surprise me
<Griwes>
someone adds those, saves them in an auto variable
<mrvn>
Griwes: where does it surprise you?
<zid`>
C doesn't have auto
<Griwes>
now they have ub on overflow when they expected unsigned arithmetic
<kazinsal>
ooh, neat, they may be opening the border back up at the end of the month
<g1n>
gog: hmm, then to what size i should change it?
<Griwes>
mrvn, me? it doesn't, because I know the spec
<gog>
g1n: hang on lemme throw this thing together and comment it carefully to see if you can grasp it
<g1n>
gog: ok
<mrvn>
Griwes: auto is rather new and horrible for this stuff.
<Griwes>
but that puts me in like .01% of C and C++ programmers
<Griwes>
people don't know this shit
<mrvn>
Griwes: most problems like that are legacy cruft.
<Griwes>
the rules are horrendeous
<zid`>
That most people aren't qualified to operate a C source file isn't exactly news :p
<mrvn>
Griwes: the rules for auto are horrible too
<Griwes>
it's not their fault, it's C's
<zid`>
It really isn't
<Griwes>
rules for auto are "this variable has the decayed type of the expression"
<j`ey>
'decayed' :P
<mrvn>
Griwes: except not, like && and const
<Griwes>
it's easy to explain while avoiding the word decayed
<zid`>
It does *precisely* what you'd want it to do, if what you wanted to do was what aligned with what C was designed to do, let you write effectively, macro assembly for writing systems code for a few similar processors
<Griwes>
C never does what I want it to when I work with integers
<zid`>
It does almost exactly what I want it to do all of the time
<mrvn>
what was that nice example, let me think: auto i = {1}; I think. What type is that?
<Griwes>
mrvn, all those rules follow consistently from what the rules for templates have always been
<mrvn>
Griwes: yeah, horrible. :)
<Griwes>
that one is a fault of initializer_list, not of auto
<zid`>
I want to be able to make macros, so I need things to type promote. I want char c = blah; a + 4 to give int, but I want it to silently be a char again if I re-assign it to a char, blah blah
<mrvn>
zid`: No, I want that to give an error. Arithmetic on non intgeger type.
<Griwes>
why would you want a + 4 to give you int and not whatever a's type is?
<zid`>
because I don't want it to wrap
<zid`>
that'd be fucking insane
<Griwes>
that's bonkers
<clever>
i sometimes want it to wrap, but i usually % or & to force that
<Griwes>
anyway, we seem to have arrived at an agreement that we have the polar opposite ideas on what is insane
<kingoffrance>
^^^
<zid`>
I'm writing an emulator, I decode an relative jump and read my char rel8 operand. I write the code PC + rel8
<kingoffrance>
my interuppted ^^^ was to clever
<Griwes>
which I think is a nice place to end this discussion
<zid`>
If that ends up as a *char* I'm fucking mad
<mrvn>
zid`: if you reassign it to char you have UB
<zid`>
It's not UB it's converted.
* Griwes
is out
<mrvn>
zid`: signed integer overflow if char is signed on your arch
<zid`>
it doesn't overflow a char
<mrvn>
Griwes: I have to agree with you on the implicit conversions and promotion stuff. It's horrible in a lot of places. I would expect "a + 1 ", "++a" and "a += 1" all to be the same.
<mrvn>
What does this do/return anway: uint16_t a; ... ++a; ? uint16_t + 1 ==> int and truncate when assigning back to a?
<zid`>
(The answer is that it's IDB, btw)
<zid`>
C++20 makes it congruent to the modulo I think
<Griwes>
assigning back to the unsigned has always been fine
<Griwes>
but if you assign back to signed, you've got overflow
<zid`>
you've got IDB, it *might* overflow, but then it'd be legal
<mrvn>
Griwes: int b = ++a; Will that give a != b when it truncates?
<kingoffrance>
i think there needs to be a customary dangerous c, without any fixed size types, just the sizeof(int) >= sizeof(short), etc. and then the version that tries to fix them, and then whatever c++ wants to use. its the trying to twist them all into eachother that causes all the problems
<kingoffrance>
too many cooks
<mrvn>
kingoffrance: how about not allowing arithmatic with different types. If you want that then cast yourself.
<Griwes>
zid`, oh right, we've got the stupid rule where it's ub for float to integral but not for integral to integral <facepalm>
<Griwes>
it *should* all just be ub
<mrvn>
except literals don't have a strict type
<Griwes>
this way you have a chance to catch it with ubsan
<zid`>
If you want that to be something the programmer has to specify, I think that's what Rust is
<zid`>
I prefer my fast and loose conversions so I can actually write some code, in exchange for having to remember not to assign 23902 to char if I want it not to do weird suff
<kingoffrance>
mrvn, makes sense. and probably what zid` says too
<Griwes>
your fast and loose conversions would be fine if they were consistent which they aren't
<zid`>
I can think and consider the rules faster than I can write the annotations that it'd need, basically
<mrvn>
Griwes: they are historically grown and made to benefit compilers
<Griwes>
as mrvn said, if `a + 1`, `++a`, and `a += 1` don't produce the same values of the same type, your thing is fucked up
<mrvn>
Griwes: what does int b = ++a; result in?
<Griwes>
zid`, I'd appreciate if you stopped talking about some strawman where you need to annotate all conversions
<Griwes>
mrvn, what's a?
<mrvn>
Griwes: uint16_t from above
<zid`>
Griwes: That's the alternative to it happening automatically. I asked you what you want to happen or what you wanted it to do and you refused to tell me.
<zid`>
You don't get to tell me I am being difficult here.
<Griwes>
mrvn, then it does a mod2^16 arithmetic and assigns
<zid`>
I even gave you a partially shaded dinosaur to complete in case it was too much work for you that I wasn't telepathic.
<Griwes>
zid`, ...what.
<mrvn>
Griwes: so ++ doesn't promote like +?
<Griwes>
I mean at least in C++ ++a returns an lvalue so it can't promote
<Griwes>
zid`, I really don't know what more to tell you than "make a + 1 give you a value of the type of a"
<Griwes>
"for a + b, convert to the larger of the two if that's lossless" (otherwise you do want to actually explicitly annotate so that the meaning is obvious)
<mrvn>
Griwes: That looks like a promotion to int to me
<bslsk05>
gist.github.com: exercise for g1n · GitHub
<gog>
please note this code is not safe
<gog>
it's just an example
<gog>
DO NOT ACTUALLY USE THIS CODE
<zid`>
it's ovulating
<Griwes>
mrvn, you're converting to an int, yes? but `++x` itself is an uint16_t lvalue, I'm pretty sure?
<mrvn>
oh wait, movzx is the masking to 16 bit. never mind.
<kazinsal>
I am concerned at the concept of ovulating code
<Griwes>
zid`, you can read wg21.link/p1467 which contains something as close to the rules I'd like for arithmetic types in C and C++ as possible when the pre-existing arithmetic types, well, exist
<bslsk05>
redirect -> www.open-std.org: P1467R8: Extended floating-point types and standard names
<zid`>
I don't know any of the rules for C++, are they relevent?
<Griwes>
not for reading what the proposed rules are
<zid`>
kk
<Griwes>
but the rules are generally almost identical in C++ as they are in C
<mrvn>
Griwes: more of that backward compatibility crap
<zid`>
oh all the examples are C++ though
<zid`>
and almost all of this seems to be about floats?
<Griwes>
this is about floats, yes
<Griwes>
but the same philosophy of conversions would ideally apply to integers
<zid`>
yea I don't care enough about float stuff to have an informed opinion
<zid`>
so I don't know what it's changing to what
<Griwes>
the examples are C++, but the only C++ in most of them is std:: lol
<Griwes>
it shows what's legal and what's not
<mrvn>
Anyone using user-defined-literals like 100_hz?
<zid`>
I have a friend who keeps posting me code using them
<zid`>
saying how I should totally learn C++
<Griwes>
and ideally integers would have the same rules as what this paper proposes for extended floating point
<zid`>
He's doing it to troll me partially, but he genuinely writes all his code like that
<g1n>
gog: thanks
<Griwes>
also by some miracle it turns out that C's incoming rules for their new floating point types aligned with most of what we proposed in the linked paper before ever looking at the C stuff
<Griwes>
there was one tweak we needed to make I think
masoudd has joined #osdev
<mrvn>
Griwes: i wish fixed point math would be better supported in C/C++
<Griwes>
if my employer decides to care about it for standardization, you may see my name attached to a potential proposal, but I don't think it's on anyone's mind right now
<mrvn>
*onder* Will c++ ever drop the ones-complement for ints?
<kazinsal>
I still need to find some way to throw together a C toolchain targeting flat binaries on PC/ATs
<mrvn>
+p
<zid`>
I thought C already had dropped ones-comp in later standards?
<mrvn>
kazinsal: what is wrong with default gcc?
<zid`>
My knowledge of C17 is near non-existant, C11 is sketchy :p
<kazinsal>
mrvn: doesn't actually properly support 8088s
<Griwes>
hmm, maybe we actually did remove the ub from shifts
<Griwes>
any changes here would've come in C++17
<mrvn>
Griwes: That fixes the UB on both << and signed >>
<zid`>
Leaving it UB lets the compiler optimizes, I like 'the compiler optimizes'.
<Griwes>
I think
<Griwes>
> Right-shift on signed integral types is an arithmetic right shift, which performs sign-extension.
<mrvn>
Griwes: breaks 8088 support there
<mrvn>
Griwes: E1 << E2 using E2 % 2^width needs extra code on 8088
<zid`>
I wonder if <<32 would be defined if x86 could have encoded it at the time :P
* Griwes
shrugs
<mrvn>
zid`: It was defined without the encoding but then the encoding changed it so it became UB.
<zid`>
intel have never had a cpu that could encode that have they?
<mrvn>
zid`: all modern cpus have a << opcode that does multiple bit shifts at once.
<zid`>
pardon?
<mrvn>
while (--i) shift(a); vs. shift(a, i)
<zid`>
I'm *really* lost here man
<mrvn>
zid`: On 8088 you have to shift in a loot one bit at a time. Then <<32 becomes 0. Modern cpus shift interally and only look at the lower bits.
<GeDaMo>
Is this about the 8086 originally only having shifts by 1?
<kazinsal>
and you may ask yourself, how did I get here?
<zid`>
mrvn: okay great, I can also do two 16s
<zid`>
I still can't directly encode 32
<GeDaMo>
# This is not my beautiful house. This is not my beautiful wife. #
<mrvn>
But I think there are some cpus out there where the "shl x,y" opcode for y >= 32 results in 0.
<mrvn>
zid`: shift by register, not immediate
<zid`>
no, it still masks it to 5 bits
<mrvn>
zid`: it does on x86, which is why it had to be UB.
<zid`>
...
<zid`>
That's where this convo *started*
<zid`>
"I wonder if C would have defined it, if x86 defined it."
<zid`>
That statement is about how x86 doesn't define it
<mrvn>
or it was UB already from some other CPU so x86 decided to implement it that way becuase it uses less transistors.
<zid`>
I assume they just went "nobody needs that", and so did everybody else
<zid`>
so it's common that nobody can do it
<mrvn>
zid`: it just doesn't generate from a barrel shifter circuit.
<mrvn>
You would need a bunch of extra gates and probably extra cpu cycles to evaluate x << 32 to 0
<zid`>
exactly, so why bother, if no customer is going to want it
<mrvn>
A lot of C stuff is like that. If we don't need it or hardware doesn't do it then make it UB.
<mrvn>
Another one of those funny things is that [] is just syntactic suggar for +. That's why -1[x] works.
<zid`>
Yup, that's why it's useful
<zid`>
I keep seeing people being annoyed about various things being UB or whatever
<zid`>
and my reaction is always Nooooo I need that to be UB
<zid`>
else my compiler is going to spit out 3 jumps and some cmps for every single add
<mrvn>
zid`: No, I need that to be well defined but still optimize like it does now.
<zid`>
Where's my "tough titties, reality bites" projector slide
<mrvn>
zid`: how about "memcpy(NULL, NULL, 0)"?
<GeDaMo>
Maybe you could have a compiler feature to emit checks so you can explicitly ask for them
<mrvn>
GeDaMo: or different types. int doesn't overflow but int16_t can.
<kazinsal>
clearly the solution here is rust
* kazinsal
runs away
<mrvn>
kazinsal: haskell
<mrvn>
go
<kazinsal>
I have no dog in the fight of modern bizarre "look at me daddy I'm a systems programmer" languages tbh
<zid`>
rust is what I was thinking of
<kazinsal>
but I'm reading an interesting article about discord switching from go to rust because of issues with the garbage collector
<zid`>
But it can't do any of these tricks around TUs so it's kinda poopy
<mrvn>
Did rust have linear types?
<zid`>
It'll try to do some static analsysis so it can omit bounds checks and stuff, but has no way to transmit that analysis outside of that instance of rustcc or whatever their compiler is called
<j`ey>
rustc
<mrvn>
What I know of go comes down to "It's designed to compile fast"
<kazinsal>
iirc go's design originates from rob pike realizing that the average fresh CS grad wasn't capable of writing sanity-checked C
<kazinsal>
but je ne suis pas un googler
<moon-child>
unlike most people in this channel
<j`ey>
kazinsal: googleur
<GeDaMo>
Sanity is over-rated :P
<mrvn>
moon-child: i wouldn't say that.
<kazinsal>
j`ey: I'm on the wrong coast to be able to french properly
<mrvn>
A lot of people here, including me, can't write valid C/C++ code (on the first try)
<kingoffrance>
are we allowed to make topic requests "a systems programmer will know what to do when society breaks down, because the systems programmer already lives in a world without law."
<zid`>
j`ey: Why is kazinsal trying to kiss you with tongues?
<j`ey>
zid`: we're good pals :P
<zid`>
ah okay
* kazinsal
waggles his eyebrows
<zid`>
like sapho
<kazinsal>
"just gals bein' pals"
<zid`>
sappho*
<zid`>
turns out sapho is someone else
<mrvn>
kingoffrance: system programmers only work with a rigidly defined area of doubt and uncertainty
<kazinsal>
but yes I am no googler, as my actual field of professional knowledge is not something they need and I am not a classically trained programmer so I cannot do programming interview crud
<j`ey>
they need networking people?
<kazinsal>
entities as large as google don't care about the nitty-gritty of traditional enterprise networking and datacenter hardware stuff
<j`ey>
you mean cos their stuff is beyond that?
<kazinsal>
google's not buying off-the-shelf compute solutions, they've got a factory somewhere on contract that spits out exactly what they need with exactly the firmware images and baked-in software they need
<j`ey>
right
<mrvn>
At a certain scale you can just order custom build hardware
<moon-child>
cf tpus
<mrvn>
Like PSUs with build in UPS.
<kazinsal>
yeah, and the big FAANG style entities are well beyond that
<j`ey>
they came a long way from off the shelf, or out of the garbage components
<gog>
did somebody say gals bein pals
bradd_ has quit [Ping timeout: 240 seconds]
<zid`>
Yep, absolutely just great pals.
<kazinsal>
nothing special there, just two gals who experience closeness and tragedy and re-unity as good pals
bradd has joined #osdev
Jari-- has quit [Ping timeout: 256 seconds]
<g1n>
gog: i have a bit ported your code, but have questions: what initial value of bitmap should be? (i set it to 1 using loop, seems working), also wdym by it is not safe? (i have added if statement after loops, it is safer now?), and what values should i receive? (i receive 0 on first try and 131072 on second)
<zid`>
it gives weird results if you don't feed it good inputs
<zid`>
or memory is already all allocated
<zid`>
if your testing was limited to "calling it twice" I recommend you do more.
<zid`>
Add a main() and compile it on your regular computer, make it allocate a bunch of stuff, add free and make it free some stuff, then allocate more stuff etc
<zid`>
get a feel for what it's doing and how it works
<zid`>
what the problems and limitations are
<zid`>
you know, do some work :P
<zid`>
Make a function that dumps the bitmap to the screen as a series of 1s and 0s, do some allocations and frees, make sure the bits do what you expect, etc etc
<zid`>
And if you get into trouble, you can always come back with "<x> here on line 32, I print the value of 'foo' and I get 0x20, but I was expecting 0x40, why is this?' or something reasonable, and not just "it no work??!"
Affliction has quit [*.net *.split]
duckworld has quit [*.net *.split]
<g1n>
zid`: oh ok
nyah has joined #osdev
<g1n>
zid`: thanks, i think i found something, zeroes (marking as used in this implementation) only first bit of uint32_t, seems i need to add a bit more checks and understand bit manipulation better
<g1n>
i think i found one more thing - i am not 1'ing all bits in uint32_t lol
<GeDaMo>
~0
<zid`>
~0ULLLLLLL
<g1n>
GeDaMo: bitmap[i] |= ~0, will work?
<zid`>
why or?
<g1n>
so just =?
<zid`>
seems like extra work to me, reading it first, oring it to make all 1s, then writing it back, compared to just.. writing all 1s
<GeDaMo>
If you want to set all bits to 1 then just = ~0 should work
<g1n>
ok thanks
<GeDaMo>
You don't need to bitwise 'or' it
<GeDaMo>
|= would usually be used to set a subset of bits to 1
pretty_dumm_guy has joined #osdev
<g1n>
GeDaMo: ok
<GeDaMo>
UINT32_C(~0) maybe, if you're using inttypes
<zid`>
That's why I suggested ~0ULLLLLLLLLLLLLL :p
<zid`>
but it won't matter, given this is for paging code for a certain cpu that only does paging where int is 32bit
<g1n>
GeDaMo: same result lol
<GeDaMo>
Will it actually accept all those Ls? :P
<g1n>
lol
<GeDaMo>
g1n: what result?
<zid`>
It better if it knows what's good for it, all I'm sayin
<g1n>
GeDaMo: at least it is not 0, but with every new bit it is getting more and more and then in last bit overflow (as i understood)
<bslsk05>
arxiv.org: [1603.06549] Consistently faster and smaller compressed bitmaps with Roaring
<mrvn>
https://godbolt.org/z/Y4bKfcKvo Is there a way to write the bitmap search so the compiler optimizes that to check 32bit at a time and use the count-leading-zero opcode?
<bslsk05>
godbolt.org: Compiler Explorer
<GeDaMo>
gcc has builtins for clz
<mrvn>
GeDaMo: that just avoids having to write asm(), doesn't optimize it for me.
<GeDaMo>
I'd be more inclined to use a table lookup anyway
<gog>
mrvn: __builtin_ffs
<mrvn>
GeDaMo: what?
<mrvn>
gog: not the question
<GeDaMo>
A table which shows the lowest (or highest) bit set in a byte
<gog>
oh
<mrvn>
GeDaMo: a table for uint32_t? That would be rather big.
<GeDaMo>
byte, 8 bits
<mrvn>
and about 200 times slower
<GeDaMo>
4 lookups for a 32 bit
<mrvn>
for 8 bit a table would be feasable. But why when the cpu has an opcode for it?
<GeDaMo>
Why not use a builtin when the compiler won't generate the code you want?
<mrvn>
anyway, that wasn't the question. The question was if one can write the loop so the compiler will check "map[i / 32] == 0 ==> i += 32;"
<GeDaMo>
i += !(map[i / 32] == 0) * 32
<mrvn>
GeDaMo: that's you doing it, not the compiler
<GeDaMo>
I'm smarter than a compiler! :P
<GeDaMo>
Or am I? I might have the logic the wrong way around there :|
<mrvn>
also you won't benefit from the cpu being 64bit or using SSE.
<GeDaMo>
It's more likely to vectorise something without branches than with
<mrvn>
With SSE3 I could skip 1024 bit at a time, right?
<GeDaMo>
SSE3 still only goes up to 128 bit
<mrvn>
4 or 5 then
<GeDaMo>
I don't think it's until AVX that large vector register sizes came in
<g1n>
yet another question: how should i extract address to bit from address? i mean current kalloc_frame returns position in list of bits * 0x1000, i will extract index using addr / 0x1000, but how to extract bit?
<gog>
leaving that to you to figure out
<kazinsal>
throw some math at it
<mrvn>
g1n: index / sizeof(T) and index % sizeof(T)
<gog>
^
<mrvn>
*8
<mrvn>
ir CHAR_BITS to be correct
<g1n>
mrvn: ok
<mrvn>
g1n: doing this correctly adds a ton of boilerplate stuff
<gog>
i'm more concerned with conceptual correctness than programmatic correctness for g1n at the moment
<kazinsal>
aye
<mrvn>
g1n: for start you can do /32 and %32
<g1n>
ok
<kazinsal>
personal understanding + slow code > fast code + no understanding
<g1n>
kazinsal: indeed lol
<mrvn>
kazinsal: oh, we haven't even started on optimizing speed yet. It's all been correctness and storage size so far.
<gog>
let's not confuse the boy with any of that rn
<gog>
:p
<mrvn>
g1n: Here is my tip for you: Forget about free, forget about the bitmap, declare a fixed array of 64 pages in your code and count how many are allocated. Use that to setup page tables and initialize your memory management properly.
<g1n>
i think i have tryed to do it like that, but it went wrong....
<mrvn>
g1n: you are investing time into code that runs once at boot and then never again.
<g1n>
mrvn: but i need it for memory managment, that will be used a lot
<gog>
see: me and my prior bootloader experiments
<mrvn>
g1n: for memory management you need something dynamically sized. You don't know how much memory you have yet. To get there you need a small bit of memory.
<mrvn>
g1n: the fixed array of 64 pages is just to bootstrap the memory management itself.
<mrvn>
g1n: allows you to create page tables and parse the memory map from the bios/bootloader and such.
<g1n>
mrvn: ok, but maybe would be better on first, i finish and understand that i am already have?
<g1n>
s/, i/ to
<g1n>
s/am//
<mrvn>
g1n: if you want. But you are far from a actually usable bitmap and it's the wrong data structure for your problem.
<g1n>
:(
<GeDaMo>
*opinions may vary
<mrvn>
long term usable that is
<g1n>
mrvn: ok i will try
<mrvn>
g1n: do you have any code that calls free_page()?
<kazinsal>
it is best to get something unoptimized but replaceable in to get things working then improve it as best as possible iteratively
<gog>
mrvn: none of this is for long-term purposes anyway, it's intro to concepts necessary for osdevving and even programming in general. g1n is still a newbie to coding iirc
gareppa has joined #osdev
<mrvn>
gog: where "necessary" is an opinion
<kazinsal>
eg. if you can have your virtual memory allocator call a GetNextFreeFrame() function then you can drop in a new GetNextFreeFrame() and associate allocator whenever you want so long as it fulfills the same argument/return requirements
<gog>
mrvn: fair
<kazinsal>
I would teach a new guitarist who knowns only basic open chords the blues before I taught them how to play a song by Dream Theater
<mrvn>
My opinion is that a bitmap is never a good choice for page frames. You can use that for virtual memory, not physical.
gareppa has quit [Remote host closed the connection]
<j`ey>
which is why you hide the details behind an API
<GeDaMo>
Pay no attention to the code behind the curtain! :P
<kazinsal>
correct
* gog
shoves the mess under the rug
<gog>
nothing to see here
<gog>
just a clean abstraction
<gog>
nevermind the leak
<kazinsal>
a basic but functional implementation will always be able to support higher level concepts in the stack
<mrvn>
that imho only appplies when the compiler also generates no asm for the code.
<mrvn>
(or close to that)
<kazinsal>
an unoptimized allocator that is logically solid will always *work*
<kazinsal>
it may be a bottleneck, but the system won't spontaneously destabilize
<kazinsal>
so long as the underlying implementation is logically solid
<mrvn>
Best argument I have against bitmaps is their memory cost: When you run out of memory the bitmap will be all 0, have no informational content and still use a big chunk of memory.
Affliction has joined #osdev
duckworld has joined #osdev
<kazinsal>
yeah that's why I've in the past used a "tag array" instead of a plain bitmap
<mrvn>
Many kernels later also have a struct per_page instead of just a bit.
<kazinsal>
8 bits per frame is generally useful enough assuming a reasonable frame size
<gog>
ugh this reminds me i need to still expand my tag array thing to be refcounted and link back to the owning object
<mrvn>
gog: hehe, I have either or. A page can be owned by a process or be refcounted when it is cache.
<gog>
mrvn: yeah i haven't worked out the details of how that's all gonna fit. i have a lot of missing pieces yet :p
<kazinsal>
I have more or less given up on the actual osdev process
<gog>
osdev machine broke
<kazinsal>
I had a goal but it was too lofty despite being relatively realistic as far as out-there osdev goals go
<kazinsal>
it was focused but still too broad
<kazinsal>
and now I just want to do post-ironic gimmick osdev stuff'
<gog>
i'm an osdev hipster too
<GeDaMo>
Like esolangs but for osdev? :P
<gog>
i want to dig back into that old CS paper i read about JIT-optmization in-kernel
<GeDaMo>
Which one? Synthesis?
<gog>
yeah
<mrvn>
I want my hello-world kernel image to run on any qemu -M<X> for ARM
<gog>
i think it's less having a whole compiler than being able to find and elide unnecessary chains of calls
<gog>
folding up indirect references and such
<kazinsal>
my goal was "functional modern-connection-capable router/firewall OS" and it's such a significant development investment that I'm just going to revert to "bad unix clones" but this time for uncommon platforms
<bslsk05>
en.wikipedia.org: Printed electronics - Wikipedia
<kazinsal>
(historical lesson: vintage Cisco routers had an nvram:/ filesystem that was located on a battery-backed memory device that was originally an optional order item. later devices virtually implemented the nvram:/ filesystem as a hidden partition of the compactflash device that held the flash:/ filesystem, and current devices do the same but on the on-motherboard SSD.)
<zid`>
basically all routers still work like taht
<zid`>
wanna see my nvram? *cocquettish wink that sort of looks like a lazy eye*
<zid`>
kazinsal: You can buy SRAM for PCs btw
<mrvn>
Griwes: template<T, U, V> requires not is_same<U, V> T operator +(U, V) = delete; template<T> T operator +(T x, T y) { return x+y; } Wouldn't that solve your uint16_t + uint16_t issue?
<kazinsal>
yeah, it's not like battery-backed nvram for PCs is *uncommon*
<kazinsal>
it's just a shame there was never a standard for it established in the 80s
<mrvn>
kazinsal: battery backed ram is quite expensive
<kazinsal>
I think for a while the battery-backed CMOS NVRAM has been on Super I/O chips so it's kind of disappointing that manufacturers never expanded that RAM space to a handful of kilobytes
<zid`>
HYPERDRIVE 5M, good name for a card
<kazinsal>
once you reach the stage of "standardized ASIC that every x86 machine has one of onboard" you could do all sorts of stuff like that
<kazinsal>
but no one decided to
<kazinsal>
having 16-32K of OS-specific NVRAM on every machine would be nice
<mrvn>
kazinsal: kilobytes? I'm talking gigabytes / terrabytes here
<kazinsal>
oh yeah no I just care about configuration
<zid`>
I just use my ethernet card's eeprom
wolfshappen has quit [Ping timeout: 256 seconds]
<mrvn>
efi has a few kb. But if you fill it you bricked your PC.
<bauen1>
can't you put that much into UEFI variables on modern systems ?
<kazinsal>
you can squeeze a pretty dense modern large org router's config into ~100 KB after compression
wolfshappen has joined #osdev
<gog>
speaking of using ethernet cards for fun things, i saw a necroware video where he used an ethernet card to carry a firmware image for XTIDE
<gog>
so he could initialize a slightly broken ATA controller
<kazinsal>
and by "large org" I mean "full BGP tables router"
pretty_dumm_guy has quit [Remote host closed the connection]
pretty_dumm_guy has joined #osdev
<mrvn>
Is there a hack to make "(Foo*)0x12345678" a constexpr?
<gog>
i don't think so
[itchyjunk] has joined #osdev
<mrvn>
gog: wrapping it in a class and hiding the cast in operator ->, * and/or . might be a solution.
<gog>
i think it's recommended to not overload those operators though
<mrvn>
we are way past recommended with this anyway.
<gog>
yeah i suppose so lo
<gog>
my c++ expertise is not really enough to weigh in on that in any meaningful way
<mrvn>
Casting a literal to pointer is UB.
<gog>
yes
<mrvn>
The only allowed literal you can cast safely is 0.
<zid`>
IDB
<mrvn>
which is why we can do it at all, true.
<zid`>
and every implementation WOULD define it anyway, even if the standard didn't
<zid`>
same as memmove, you just can't really write them without defining things outside of the spec
<mrvn>
zid`: A compiler that includes range checking in the pointers fails when you try to use a pointer initialized that way
<zid`>
err, them = useful things
<mrvn>
zid`: well, regarding memmove. If your malloc() just returns memory out of a huge empty array of bytes then memmove is well defined.
<zid`>
no because that isn't what the standard means by object
<mrvn>
which is what sbrk() and friends do.
<mrvn>
zid`: the whole memory is one object and you return pointers to inside that one global object in malloc(). All well defined.
<clever>
mrvn: but if you malloc a chunk above a certain size, it wont use the sbrk heap, but just mmap a new chunk, at least on linux glibc
<clever>
to avoid creating a 10mb hole in the sbrk heap that it cant close because 4 bytes are used after the hole
<mrvn>
clever: implementation details. you don't have to do it that way.
<clever>
yeah
<zid`>
mrvn: But isn't infact, how any malloc implementation *actually* works. So they define various UB things to allow memmove to function etc.
<mrvn>
zid`: malloc and memmove don't exist in isolation. My point was that you can define your malloc in C so that memmove is well defined.
<zid`>
sure, you can, but that's tangential to what I was talking about
<zid`>
hence me saying, "no, implementations don't do that"
<zid`>
they just define various bits of ub/idb such that it all works out in the end, way easier :D
<mrvn>
technically they all kind of do. It's how memory works in modern CPUs.
<mrvn>
which is why the "I just don't care" solution works for memcpy and such.
<gog>
tfw the entire modern world relies on undefined behavior working
<mrvn>
gog: only on the implementation defined behavior though. We should be more precise there
<mrvn>
my bad
<gog>
fair
<zid`>
meh you need actual UB to do memmove how we *actually* write it
<mrvn>
zid`: huh? where?
<zid`>
unless you wanna claim that 'the entire address space is one huge valid object'
<zid`>
which is.. a weird take
<gog>
valid
<mrvn>
zid`: while(--n) { *dst++ = *src++; } where is that UB?
<zid`>
That's not a valid impl
<mrvn>
the only tricky part is overlap detection and that is implementation defined.
<zid`>
hope I never have to use that version of memmove anywhere :p
<zid`>
no it's UB
<mrvn>
zid`: cast to uintptr_t and check for overlaps.
<zid`>
yea that dodges it into IDB
<mrvn>
as I was saying. :)
<zid`>
so we'll skip all that and implement it with <
<mrvn>
zid`: blows up nicely on AVR
<mrvn>
can't memcpy from flash to ram.
<zid`>
how's it manage that, if the idb version works?
<clever>
you need a special opcode to read from avr flash
<mrvn>
zid`: flash and ram are different address spaces.
<clever>
yep, that
<zid`>
no, I get how it doesn't work
<zid`>
how does it *ever* work, if the IDB solution does
<mrvn>
zid`: so pointers to flash and ram really can't be compared.
<clever>
there is usually an _P variant of functions, like memcpy_P
<clever>
that expects one arg to be a flash pointer
<zid`>
You said it like "You can't just use < because it breaks AVR", but presumably so does the uinptr_t method
<zid`>
(also glibc apparently casts to unsigned long? heh)
<mrvn>
zid`: as clever said. Left to the programmer to call the right function.
<zid`>
So you're saying.. < is fine
<zid`>
because it's broken *either* way
<mrvn>
zid`: Just saying that there are CPUs out there where you can't just < pointers.
xenos1984 has quit [Read error: Connection reset by peer]
<zid`>
Yes but you *also* can't uinptr_t them
<zid`>
so it's completely irrelevent
<zid`>
Considering the discussion is whether you can just use < and define it to work
<mrvn>
zid`: that depends on the implementation. uintptr_t could be struct { enum namespace; int addr; }
<mrvn>
kind of
<mrvn>
I whish C/C++ had better support for far pointers / multiple address spaces.
<zid`>
It can't
<zid`>
standard says 'integer type'
<zid`>
not compound type
<mrvn>
zid`: "kind of". mangle the bits into a single integer type and you are fine.
<zid`>
maybe you could typedef it to ull that's 64bit and your pointers are 32bit and address space tag it that way though
<zid`>
yea
<zid`>
but that's even more idb, because now you're relying on uintptr_t to be linear wrt to addresses
<mrvn>
zid`: AVR has 16bit addresses
<zid`>
Nothing's stopping me in theory implementing void * to uinptr_t as 'xor the pointer with 0xDEADBEEF' and uintptr_t as 'xor with 0xDEADBEEF' I don't think?
<clever>
mrvn: gcc has generated code that used 24bit addressing, i ran into trouble when my avr emulator lacked support for that
<mrvn>
zid`: with one exception: 0 must cast to nullptr and back.
<zid`>
nullptr isn't C so I couldn't tell you
<zid`>
all C cares about is that 0 compares equal to the null pointer constant
<mrvn>
zid`: NULL then, for where that is defined as pointer and not integer
<zid`>
which can have any actual pointer value
<zid`>
and uintptr is also only valid for 'valid pointer's
<mrvn>
NULL and 0 don't have to be the same bit pattern though. They just have to convert into each other.
<zid`>
correct
<zid`>
>all C cares about is that 0 compares equal to the null pointer constant
<zid`>
you're not supposed to memset compound types with pointers in them for precisely that reason
<zid`>
initialization has special wording that allows it to assign other bitpatterns
<mrvn>
Which actually makes this UB: struct { int x; void* next; ...} foo; bzero(foo, sizeof(foo));
<zid`>
you're slow :p
<mrvn>
or IDB
<mrvn>
zid`: :)
<sortie>
Yeah needs memset to be defined
<sortie>
= { 0 }; curiously is defined in cases where memset is not
<zid`>
assignment is the easy case
<mrvn>
Do you know any code that actually initializes pointers in structs to 0 instead of memset/bzero the struct?
<zid`>
and initialization
<sortie>
In any case, you cowards, build your machine of doom
<zid`>
yea, all of mine?
<zid`>
I just do = {0}
<zid`>
it's less code anyway
<mrvn>
sortie: assignment is fine. that's the compilers job to get right then.
<zid`>
C allowing NULL to be 0 or (void *)0 is one of my pet peeves btw
<sortie>
I'm up here on my hill in my fortress assuming your doomsday device does not exist fight me
<zid`>
the former is retarded and breaks varargs
rorx has quit [Ping timeout: 250 seconds]
<sortie>
zid`, no ableist language people
<sortie>
*please
<zid`>
huh
<zid`>
am I getting euphamism treadmilled
<mrvn>
zid`: but that would make existing compilers from the 60s non-compliant.....
<zid`>
phemism*
<zid`>
compliant with a standard that didn't exist until 1989?
<zid`>
Unsurprising
<mrvn>
Does modern C actually still have that wording or is NULL a pointer now?
<zid`>
there's modern C!?
<zid`>
Someone find it and kill it
<mrvn>
zid`: more modern than c89 at least
<zid`>
I don't *think* they ever changed it
<GeDaMo>
C compiler? 60s?
<zid`>
but anyone doing the former should be shot into the sun
<mrvn>
If NULL is 0 then why bother writing NULL at all? 0 is three letters shorter.
<sortie>
0 is four letters shorter
<zid`>
Okay I checked wikipedia, The compiler author would have to have an intellectual disability to pick 0 over (void *)0
<mrvn>
zid`: the reason why 0 is allowed is because back then existing compilers did just that
<zid`>
I mean, good for them, it's still imbecilic
<mrvn>
saves 3 bytes of memory during compile and such arguments.
<zid`>
By 1999 I'm sure we had enough space
<mrvn>
I really warming up to this kind of default construction: class Foo { int x{0}; void *p{nullptr}; double f{1.23};};
<zid`>
If you teach my C compiler to read that I will be *very* upset.
<mrvn>
zid`: the class keyword should tell you that it is c++
<gog>
c += 1
<gog>
pls
<mrvn>
d?
<zid`>
you *know* I know that mrvn
<zid`>
and you saw me type that I also know my C compiler won't accept that
<zid`>
It was to me, *very clearly* a joke
<zid`>
gog: I thought all the cool kids did ++c anyway
xenos1984 has joined #osdev
<gog>
count++
<mrvn>
My point was rather that now the class has defined content for all it's member variables. You can even do: Foo(int x_) : x(x) { } and it's still all initialized.
<zid`>
ngl if I see preincrement in your source I start worrying
<gog>
i use it in a few places but understand why
<zid`>
It's like uncovering a half-exposed landmine
<bauen1>
c++ is in some ways an improvement, but it also does so many things wrong and makes sure you use them :(
<zid`>
Maybe it's there to stop those pesky man eating rhinos, but.. you're probably just in a minefield
<bauen1>
could you combine pre and post increment ?
<zid`>
combine now
<zid`>
how*
<bauen1>
like a monster of myarray[++i++] ?
<zid`>
no because ++i isn't an lvalue
<zid`>
it's just a value
<zid`>
it'd be like saying 57++
<zid`>
where's it going to write it back to? :)
<mrvn>
I wan't the subset of c++ that is allowed for constexpr, make const default and mutable required for everything not and drop other legacy crap.
<mrvn>
bauen1: ++ is only allowed once per sequence point.
<mrvn>
bauen1: is that ++(i++) or (++i)++?
<zid`>
left
<zid`>
then right
<zid`>
It's defined dw :P
<zid`>
also a+++++b is illegal, to my great regret
<zid`>
it parses as a++ ++ +b; instead of a++ + ++b;
<mrvn>
zid`: left can't compile, as you say it's just a value.
<zid`>
C parser is a greedy boi
<zid`>
(defined to be)
<mrvn>
zid`: got to love ocaml for that one. let (+++++) x y = incr x; incr y; !x + !y
<bauen1>
ocaml is also a bit of a weird language, weirdly functional but also not
<mrvn>
zid`: In ocaml anything that starts with an infix operator is also an infix operator.
<mrvn>
C++ lambdas give us local / nested functions. So why doesn't the standard simply allow nested functions too?
ElectronApps has quit [Remote host closed the connection]
<gog>
gnu extensions
<gog>
:D
<mrvn>
(as in just reuse the function syntax for lambdas)
<mrvn>
gog: yes, gcc already supported it and it works. No need to invent something new.
dude12312414 has quit [Remote host closed the connection]
dude12312414 has joined #osdev
<kingoffrance>
"I whish C/C++ had better support for far pointers / multiple address spaces" it did, wasnt that function pointers ? :D
<mrvn>
kingoffrance: it has them but you can't use that address space, it's compiler only.
<g1n>
ok, i tested kalloc_frame and kfree_frame and it is working properly! yay! so now i am going to start memory manager :) (i found this page as guide - https://wiki.osdev.org/Writing_a_memory_manager, is it good?)
<bslsk05>
wiki.osdev.org: Writing a memory manager - OSDev Wiki
<mrvn>
g1n: do you care about userspace at this point?
<g1n>
mrvn: yes, i think
<g1n>
mrvn: ok, i will rewrite it, but i wanted to understand basic concepts lol
<mrvn>
g1n: I would stick with handling page sized objects at this point. It will be a long time before you need to handle smaller objects inside the kernel.
<mrvn>
g1n: Are you going to do x86_64 or ARM64?
<mrvn>
g1n: In my OS I use the page table itself to locate free space in the address space and to track the memory a process holds. I just have a pointer to the end of the last allocation so I basically always find empty space to put memory on the first try. And pages are kept in a stack. Makes the whole thing really simple.
<mrvn>
malloc, slabs, slubs, or whatever then runs on top of that as needed.
mahmutov has joined #osdev
<mrvn>
g1n: Consider having 3 separate components: 1) the physical memory (pages), 2) the virtual address space, 3) the MMU that maps/unmaps physical pages into/from the address space.
<g1n>
mrvn: hmmm, ok, i am currently targeting i386 (but already ported makefile to be able to handle other archs)
<mrvn>
g1n: i386 is kind of dead, no point learning the obsolete stuff unless you have to.
<g1n>
i just had cross compiler for it, didn't want to recompile it
<mrvn>
g1n: does -m64 not work?
<g1n>
mrvn: wdym?
<g1n>
for now i have 0 x86_64 code
<mrvn>
g1n: gcc -m64 on your cross compiler to compile for x86_64.
<g1n>
mrvn: also, if doing linked list, if i free page, then how would i delete it from there? point previous page to next page?
<mrvn>
to free a page you would push it to the top of the stack: So page->next = free_pages; free_pages = page;
<mrvn>
So you point the newly freed page to the older pages and remember the new page.
<g1n>
ok, i will try
<g1n>
s/ / /
<g1n>
mrvn: also, it will work without "loading page dir"? (cuz i am currently doing physical) and will it work in future to "load" page?
<mrvn>
g1n: the page has to be mapped. Most people have all memory mapped somewhere all the time. Otherwise it gets more complex because you have to map the page before you can link it.
<g1n>
so on "alloc", i make page (it is struct of what? of used flag and pointer?), then map it, then point it to NULL, and turn on used flag?
<g1n>
mrvn: ^
<mrvn>
g1n: A page inside the allocator is struct Page { Page *next; }; or just void ** it. In alloc you do; if (free_pages) { void *res = free_pages; free_pages = free_pages->next; return res; }
<mrvn>
There is no tracking if a page is used at that stage. Just free pages.
<g1n>
oh ok
<g1n>
what is free_pages?
<mrvn>
g1n: your pointer to the top of the stack of pages / begining of the linked list
<g1n>
mrvn: so it should be in page_init func?
<mrvn>
g1n: global. init, alloc and free have to access it
<g1n>
mrvn: i should init it like struct page *free_pages; ?
<g1n>
(cuz without malloc it is a bit strange lol)
<mrvn>
g1n: you can init it to nullptr and then free all the pages that should be free.
<g1n>
mrvn: if it is a linked list, how would i know that i shouldn't add more pages?
<g1n>
i mean i can do loop in init page = free_pages free_pages->page?
<g1n>
(loop as many times as PAGE_MAX?)
<mrvn>
g1n: if all your memory is allocated then free_pages will be nullptr again. That's how you start at boot to init it. Everything is allocated, then free pages that are free.
<mrvn>
g1n: yes, you loop over all (free) pages.
<g1n>
ok, let me try to reimplement this whole thing one more time :)
<g1n>
mrvn: why not just NULL (or as in c++ nullptr)?
<mrvn>
Carefull on the ++page there. If you use something like: struct Page { Page *next; char buf[PAGE_SIZE - sizeof(Page*)]; } then that will work to add 4096 to the address
<mrvn>
g1n: why not just NULL where?
<g1n>
mrvn: NULL instead of _kernel_end?
<mrvn>
g1n: that would mark the early memory and kernel as free pages.
<g1n>
mrvn: but, it should not be accessed isn't it?
heat has joined #osdev
<mrvn>
g1n: you certainly don't write your page table over your kernel code. That wouldn't work.
rorx has joined #osdev
<mrvn>
and everything below 1MB on x86/x86_64 is best left alone unless you know what's where.
<g1n>
mrvn: hmmm, ok, let me check if i have that kernel_end thing in my linker script (or how it called?)
<heat>
erm do you not have the memory map?
<mrvn>
heat: does your memory map have everything below 1MB included correctly?
<heat>
sure as hell hope so
<heat>
if not, it's a firmware bug
<mrvn>
heat: the chunk of memory the memory map itself uses?
<g1n>
hi heat, i have it (i hope, last time i checked, it was working)
<heat>
mrvn, that's the kernel's concern
<mrvn>
heat: yes. and complicated. If you ignore everything below 1MB and up to kernel_end then it's much simpler to get started.
<heat>
the memory map is not guaranteed to be below 1MB
<heat>
same with any other piece of memory the bootloader/EFI gives you
<mrvn>
heat: have you gotten a memory map above 1MB? I don't think grub ever does that.
<heat>
yup lots of times
<heat>
every pointer I look at is usually above 1MB
<mrvn>
bios or efi?
<heat>
usually I test with the BIOS
<mrvn>
heat: grub should create the memory map before it loads the kernel to see if the kernel can even be loaded at the right address. And before it has loaded the kernel it wouldn't know where above 1MB to place the memory map so it's after the kernel. Seems rather strange.
<mrvn>
g1n: check the address of your memory map from the bootloader. If it's above 1MB you have to skip that chunk of memory in init. Never had that case but see heats comments.
cppimmo has joined #osdev
<g1n>
mrvn: in my linked.ld i have . = 1M;, it is how it should work right?
<mrvn>
g1n: that's the usual place on x86/x86_64.
<g1n>
mrvn: also in your example you do page < PAGE_MAX; ++page, how this will work at all? maybe i should use counter and do page->next = free_pages; manually in loop?
<mrvn>
g1n: you have to figure out the address of the last page there.
<mrvn>
g1n: usualy the memory map has multiple blocks of usable memory and you would free each block in turn. Not just one continous block.
<g1n>
but i don't have anything for free(), i am making it now
<mrvn>
all you need is the address of the page to free
<g1n>
mrvn: ok, but still how that comparision will work? it is different types at least
<g1n>
++page not gives any warnings
<mrvn>
g1n: replace PAGE_MAX with the address of the last page (in the segment you want to free). As a first hack just say 16MB there.
heat has quit [Remote host closed the connection]
heat has joined #osdev
<g1n>
mrvn: so it will be &free_pages?
<mrvn>
(Page*)(0x1000000) for example
<g1n>
mrvn: ok :) thanks
<mrvn>
g1n: as mentioned think about the "++page". I made my struct Page exactly 4096 byte so that it works. Otherwise you have to calculate the address of the next page yourself.
<g1n>
oh, ok
<heat>
mrvn, just built my x86_64 kernel, I get the multiboot 2 info at 0x23b5f0 (~2MB)
<mrvn>
you can do; for (int i = 0x100; i < 0x1000; ++i) free((Page *)(i * 0x1000)); or so if you think that is easier to read.
<heat>
probably at the end of the kernel?
<mrvn>
heat: what is your kernel_end symbol at?
<mrvn>
heat: I will have to watch out for that when I go back to doing x86_64.
<heat>
this makes no sense unless grub is insane
<heat>
the mb2 info is in the middle of my kernel
<heat>
ahhh
<heat>
it placed it on a gap between the ELF file's sections
<heat>
it found a little cute gap between rodata and .data and it placed it right there
<mrvn>
ouch, insane.
<mrvn>
should be save there though.
mahmutov_ has joined #osdev
mahmutov has quit [Ping timeout: 256 seconds]
<heat>
i guess it takes a look at the final memory map (with the ELF executable) and tries to place the info as close to the kernel as it can
<gog>
i gather boot data after the kernel is loaded and slap a few pages after kc_image_end
dennis95 has quit [Quit: Leaving]
<mrvn>
heat: it's actually nice if it can stick the map into a hole. Frees up one more page.
mahmutov_ is now known as mahmutov
freakazoid333 has joined #osdev
<heat>
bash wizards, I want to list files in my current directory with a wildcard and then get the wildcard. how?
<gog>
get the wildcard?
<heat>
I have "for sysroot in minimal-sysroot-*.tar.zst; do" but I also need the wildcard
<heat>
preferably without regex
<gog>
hmm
<gog>
wondering if that's possible without putting it single-quoted into a variable and then eval() ing
<zid`>
line below it, also do the command over it again the stick it in a var
<zid`>
evaled and not
<g1n>
heat: i would do that like FILES=`ls minimal-sysroot-*.tar.zst` maybe
<g1n>
or $( )
<zid`>
for(i = 0; i < 10; i++) f(##a); f(a); I don't know bash so :P
<heat>
maybe I should rewrite it in python
<heat>
i bet you can do something like this with awk or or grep or something like that
<heat>
but I don't know regex
<gog>
ok
<mrvn>
#, ##, % and %% will help you
<GeDaMo>
Can't you capture the list of files in a variable?
<g1n>
mrvn: i am doing for kfree_frame these: free_pages->next = page; free_pages = page; page->next = free_pages, is it correct?
<mrvn>
can't you do it in Makefile?
<g1n>
GeDaMo: that what i thought lol
<mrvn>
g1n: no.
<heat>
probably (but I don't see how that helps me) and no this is not part of the build system
<g1n>
mrvn: oh :(
<mrvn>
g1n: just the last and the middle one
<g1n>
mrvn: oh yes! thanks :)
<mrvn>
g1n: if you think of it as a linked list then you are inserting the new page at the front of the list.
<g1n>
ok
<g1n>
mrvn: wait but what alloc should do now?
<mrvn>
g1n: remove the first page from the list and return it
<bslsk05>
www.geeksforgeeks.org: Implement a stack using singly linked list - GeeksforGeeks
<g1n>
mrvn: ok
<mrvn>
g1n: drawing some picture of the operations helps a lot for understanding it
<g1n>
mrvn: hmm ok
<gog>
aha ok
<gog>
so if you store the string with the glob in a variable
<gog>
the globs are retained
<gog>
i.e. it doesn't expand them
<gog>
if it's the RHS of an assignment
<heat>
grep -Po '(?<=minimal-sysroot-)[^.]*'
<heat>
this is it
<zid`>
oh right yea
<zid`>
blahbah-(.*|\*)-blah
<GeDaMo>
FILES=*(ls minimal-sysroot-*.tar.zst)
<zid`>
any characters any number of times, or a single *
<GeDaMo>
for f in $FILES; do echo $f; done
myon98 has quit [Read error: Connection reset by peer]
<zid`>
it should match it anyway though shouldn't it?
<zid`>
a file with a * in it matches a file with anything
<GeDaMo>
Sorry, there shouldn't be ls in there :P
<GeDaMo>
FILES=*(minimal-sysroot-*.tar.zst)
<mrvn>
$ PRE="minimal-sysroot-"; POST=".tar.zst"; for F in ${PRE}*${POST}; do T=${F#$PRE}; T=${T%$POST}; echo $F: $T; done
<mrvn>
minimal-sysroot-1.tar.zst: 1
<mrvn>
minimal-sysroot-2.tar.zst: 2
<mrvn>
and if no file exist it finds '*' wrongly
<mrvn>
$ PRE="minimal-sysroot-"; POST=".tar.zst"; for F in $(find . -maxdepth 1 -name "${PRE}*${POST}"); do T=${F#$PRE}; T=${T%$POST}; echo $F: $T; done if you need to detect missing files
<g1n>
mrvn: i think i have made smth, should kalloc_frame return Page* and how should i test it?
<mrvn>
g1n: .oO(Does it boot?)
<mrvn>
g1n: you can write a few unit tests
<g1n>
mrvn: it boots
<g1n>
also i have tryed this: page_t *page = kalloc_frame(); warn("%x\n", &page);
<mrvn>
g1n: allocate and free some pages.
<g1n>
mrvn: i have allocated several, all gives 109 if &page
<mrvn>
g1n: 109 isn't a multiple of a page. print out the page and top in free_frame() and check how it constructs the list.
<g1n>
mrvn: i just need to print page or &page or smth else?
<mrvn>
page, &page would be the address of the variable pointing to the page
<mrvn>
&page should be on your stack. Are you initializing the stack pointer?
freakazoid333 has quit [Read error: Connection reset by peer]
<g1n>
mrvn: where should i do that?
<mrvn>
in free_frame()
<g1n>
mrvn: i mean init stack pointer
<mrvn>
g1n: at the start
<mrvn>
boot.S
<g1n>
let me check
<g1n>
mrvn: mov esp, stack_top (i am using nasm)
<g1n>
mrvn: stack_top is just label in the end of boot.asm for me
freakazoid333 has joined #osdev
doppler has quit [Quit: doppler]
dude12312414 has quit [Ping timeout: 276 seconds]
dude12312414 has joined #osdev
freakazoid333 has quit [Ping timeout: 240 seconds]
* g1n
just understood that he needed to implement this in not his os lol
sprock has quit [Ping timeout: 256 seconds]
<zid`>
[11:59] <zid`> Add a main() and compile it on your regular computer
<zid`>
welcome to 7 hours ago? :D
<gog>
:D
<g1n>
zid`: lol yes
<mrvn>
zid`: static_assert works even better
<zid`>
better than main()?
<zid`>
I don't see the connection
<mrvn>
zid`: with main() you still have to run the code and static_assert will find UB
<zid`>
okay? still non-plussed.
<gog>
i'm plus-plussed
<zid`>
Well you're a cat
<mrvn>
zid`: having the compiler fail to build when you have for example a signed integer overflow in your unit tests is nice.
<zid`>
okay? Still have no idea what you're on about
<zid`>
Like absolutely none
<zid`>
not "I'm pretending I don't because I think he's being silly", like, genuinely no clue
<mrvn>
zid`: instead (or besides) of running your unit tests in main() you can static_assert them if they are constexpr.
<zid`>
"Don't do development for programming exercises that you're not sure how to solve inside your OS, do them on your desktop" "no, use statc_assert"
<g1n>
lol
<g1n>
mrvn: static_assert is some c++?
<mrvn>
g1n: yes, constexpr too
<mrvn>
g1n: it's a compile time assertion
<g1n>
mrvn: then i can't use it lol
<gog>
zid hates c++ don't talk to him about it
<g1n>
lol
<sortie>
++c
<gog>
yes
<zid`>
I hate it in that I think it's dumb, you can talk to me about how dumb it is np
<gog>
cool
<sortie>
Hey zid did you hear the one about smart pointers
<mrvn>
it fixes a few C screwups at least
<zid`>
And adds 800 more
<zid`>
Bad trade
<kingoffrance>
https://en.wikipedia.org/wiki/C* if it's good enough for jurassic park it's good enough for me, what could go wrong
<bslsk05>
en.wikipedia.org: C* - Wikipedia
<sortie>
I'm sure there's a reasonable subset of C++
<g1n>
mrvn: btw, on my machine, kalloc_frame works, but free not
<zid`>
sortie :D
<mrvn>
g1n: can't really be since you need free() to init
<g1n>
mrvn: i am not free'ing on start, cuz it gives me strange results
<mrvn>
then alloc should fail with out-of-memory on the first call
<GeDaMo>
Sourceforge was bought over and stooped all the dodgy stuff some time ago, not sure what they're like now
<gog>
i still don't trust microsfts intentions wrt open source
xenos1984 has joined #osdev
<mrvn>
what does that do? mine a bitcoin before the page loads?
<gog>
embrace, extend, extinguish
<mrvn>
g1n: ever used godbolt.org?
<g1n>
mrvn: no
<mrvn>
why is there a left and right side? And is there a download link?
<g1n>
mrvn: idk, you can edit in left side
<gog>
the right side is messages
<gog>
and output
<zid`>
google are worse than microsoft ever were, atm
<gog>
yeah
<gog>
yeah i think i agree with that
<mrvn>
error: 'uint32_t' does not name a type 13 | typedef uint32_t page_addr_t;
<g1n>
mrvn: oh, i am not using it now, cuz have page_t
<zid`>
godbolt started working agin for a few months in my browser and then they broke it again
<zid`>
I'm too lazy to open my stock chrome install just to use it
<mrvn>
g1n: page_t kind of is an internal thing of the allocator. Probably better not to expose that to the outside world.
<g1n>
mrvn: check new version, it is my executable code (i mean i put it in my testing program)
<g1n>
ok, sorry, i need to go to sleep now, hope to see you tommorow
<mrvn>
g1n: you might want to return (phys_addr_t)((uintptr_t)page - VIRT_TO_PHYS); or something later.
<g1n>
huge thanks to all of you for help! :)
<g1n>
mrvn: oh ok
<g1n>
bye
* geist
yawns
<geist>
good afternoon everyone
<sham1>
Good evening
GeDaMo has quit [Remote host closed the connection]
* mrvn
ponders how to mock in stuff into unit tests for the kernel.
<CompanionCube>
not the type of code you can run in userspace, then?
<geist>
it's doable via fairy standard mock things for at least a lot of code, it's a matter of how much code duplication (for testing) or rutime overhead (unnecessary virtuals) you want to tolerate
<geist>
we have a fairly large set of kernel unit tests in zircon that work reasonably well
<geist>
what's hard to mock is lower level details like exception handlers or context siwtching or whatnot
<geist>
but i generally feel that those get fairly well tested via stress testing and general system use
<geist>
whereas more peripheral library things are generally more straightforward to unit test
<geist>
but, for example, we have cdoe that creates an address space object, runs through a series of mmu operations adding/removing pagese from the page tables, and then some code that manually verifies that the structure got built a certain way
<geist>
or another pile of code that feeds the cpuid parsing code a set of fake cpuid values so it can run through detecting various different cpus and the feature set
<mrvn>
yeah, I want to mock hardware basically
<geist>
we have some of that in fuchsia user space via having a whole mocked hardware interface. it's pretty complicated, but i thin kthe first thing you have to do is run all hardware acceseses through some sort of intermediate routines
<geist>
so you can mock it there
<geist>
ie, register acceseses, etc
<mrvn>
Not sure if I need any extra virtuals. I was thinking of inserting the fake hardware per device tree
<gog>
i wonder what it'd take to make qemu useful for unit testing
<gog>
like making a test binary, having a vm that's configured to work with it, then test
<mrvn>
gog: too much overhead
<bauen1>
^ if you're faking at the hardware level you could also be writing those things for qemu and get a simulator for your platform
<mrvn>
bauen1: qemu already has them at a much lower level
<mrvn>
geist: if I go down to register access and to running the original driver then I would have to mmap/mprotect the peripherals pages and then decode every segfault. Seems like a lot of work.
<mrvn>
next level up would be diverting the struct MMIO_Reg
<mrvn>
That could probabl be #ifdef-ed without too much trouble and make every register access call emulation code.
<geist>
mrvn: well that's the point, you'd eed to probably stop using the struct mmio_reg stuff and move every access through some sort of intermediate object
<geist>
or whatnot. it's easier in C++
<geist>
lik,e if you had some sort of object that generically represents an aperture of registers and then some sort of 'read/write 32bit value at offset X' then you can build a mock version that just pretends to be the device
<geist>
that's basically what fuchsia does
masoudd has quit [Ping timeout: 256 seconds]
<mrvn>
geist: overloading operator -> and * or .() in C++ allows for easily catching access through the struct mmio_reg
<geist>
sure
<geist>
but you get my drift, the general idea is to redirect things at the register level and then have code that pretends to be a device
<geist>
i'm not personally sold that the effort is worth it, but some folks think so
heat has joined #osdev
<mrvn>
I'm not sure I'm any better at writing the emulation for e.g. the UART than driving the UART in the first place.
<geist>
my general experience is most driver bugs are either a misunderstandig of how the device works, not handling an edge case, or buggy hardware
<mrvn>
me too
<geist>
none of which writing a mock hardware will really catch
<zid`>
(That's my view on unit tests in general)
<zid`>
if I was smart enough to write a test for it, I was smart enough to not fuck it up in the code
<geist>
or i suppose highly multithreaded races, etc which mock testing also probably wont catch
<mrvn>
That's why was thinking of replacing the UART driver with one that just calls puts
<geist>
zid`: right. the general idea there is it's future refactors, new people working o thins later, etc
<bauen1>
unit tests are quite nice if you've found a bug or corner case, then you can write a simple test for it (a few function calls and asserts usually) and then develop until you pass that test, as that is usually faster than testing on hardware
<geist>
so there's *some* value there, but i think for a personal hobby project i dont see it as a good use of time
<zid`>
It just slows down changing the APIs for years until it catches an obvious bisectable typo, imo
<bauen1>
oh and if you're refactoring code, even a few simple unit tests are amazing
<zid`>
It's great for web stuff
<mrvn>
I just need something so any kprintf() call in a unit still compiles in the unit test
<zid`>
cus that shit is so random
<zid`>
and interpreted
<zid`>
You hit the path every time you compile it in a statically compiled lang, but only when ran in interp
<zid`>
so unit tests are just a replacement for not having a compiler there
<geist>
i think pure lib stuff is fairly good, or things that interpret external code (ACPI, cpuid, device tree, etc) are an obvious place to run unit tests
<mrvn>
The biggest problem with unit tests is that you only test known bugs. Unit testing doesn't find bugs. It only helps noticing when one comes back again.
<geist>
since you can ubild a corpus of input data and feed it through to make sure future refactors dont break it
<bauen1>
geist: it depends, testing every single getter method ? please find a better use of your time, a few examples that also assert the output as tests, those are helpful even to yourself if you're revisiting the code some time later and you'll probably be writing these examples/test anyway, doesn't take much to polish them a bit
<zid`>
I wouldn't consider that an actual unit test
<zid`>
That's a test harness with a test database or whatever, test environment, but not unit test suite
<mrvn>
unit tests are nice while developing the code though.
<geist>
yah i am not terribly sold on unit tests aggressively, but i've been mostly dragged into it kicking and screaming at work and once you have some it is marginally useful
<zid`>
unit tests tend to be "Is this value still an int? Because I don't know because this is javascript"
<mrvn>
Debuggin e.g. a linked list module by watching a raspberry Pi crash or not crash is hard.
<geist>
if the time spent writing them is on someone elses dime (ie, the company paying you to do it) then i think the calculus is a bit different
<zid`>
It's something to do rather than needlessly fucking up some code I guess :p
<geist>
but i do agree there are useful cases, and i'm slowlky writing more on my personal projects other than 'does it seem to run?'
<bauen1>
geist: yes, i was asked to write a unit test for 2 getter methods, that do literally nothing else than call another function, it was to date the most useless code i've ever produced
<zid`>
I generally test aggressively when I write it, sometimes I leave the #if TEST int main() .. #endif at the bottom
<geist>
bauen1: hah yeah. OTOH tests for future refactors is baically an elaborate assert() which i do believe in
<dmh>
muh coverage%
<zid`>
Nothing's going to *break* my linked list impl 20 years after I wrote it unless someone changed it
<geist>
ie assert early and often and make your code very brittle to future refactors has saved my bacon toooons of time
<bauen1>
dmh: at that point in time, i don't think coverage was even tracked, or that particular file was excluded for some reason lol
<zid`>
invariants <3
<dmh>
ahah
<geist>
even in code that doesn't expect to change, like linked list
<zid`>
compiled languages just offer way way way way way more invariants
<geist>
since those can then act as a canary for other breakage
<bauen1>
yes, asserting your invariants is so much more useful than unit tests
<zid`>
The compiler does like 400 unit tests on your invariants for you every time you compile it
<zid`>
"yep, this is an int not a double"
<bauen1>
geist: especially in code you don't expect, because somebody (yes that will be youself) will manage to call that code and break some of your invariant
<bauen1>
*don't expect to change
<zid`>
"yep, your if() is not actually an assignment" blah blah that'd all be unit tests in an enterprise web project
<geist>
one of the thigs i do love about more modern hardware is it over time seems like the cost of asserts has gone down to the point where you can basically just leave them on all the time and not really see a speed decreates
<mrvn>
For thinks like linked lists I like to have validation code for the contract. E.g. when asked to insert an item from the list first validate the list, then insert and then validate it again after.
<geist>
decrease. (obviously YMMV)
<mrvn>
geist: a predicatble jump is nearly always free.
<geist>
exactly
<zid`>
Especially because it won't be in the hot path
<heat>
mrvn, yeah and that's obviously useful for more complex data structures like trees and whatever
<mrvn>
calculating the bool that you assert on can be costly though
<zid`>
sort of definitionally
<zid`>
And can often be completely deferred and hidden inside memory reads etc anyway, so literally free except for the decode, which again, is probably hidden inside a different memory access
<bauen1>
a validation function for data structures is usally very easy to write and makes any debugging 200x easier
<mrvn>
heat: that is oh so useful. You can't imagin how many hours debugging you will save when your list data structure bombs the first time it gets called with a corrupted list instead of spewing all over memory.
<geist>
i've been generally sold on 'add a canary value to the top of all major data structures' too
<geist>
it has caught a fair amout of stuff in my code and by proxy zircon, since it was derived from the same code base
<zid`>
Don't forget to change all your bools to 5555555 btw
<bauen1>
implementing linked lists with a debugger seems such a chore when you can just assert that you're list fullfills its invariants and if it doesn't and you know which invariant you've broken it's usally very easy to find the code that did it
<geist>
especially kernel code, which even if it's not broken, tends to be a canary for systemwide data corruptions
<mrvn>
geist: and if you turn up debugging make the canary a checksum
<zid`>
Google got hacked that way once, random bitflip from a false to a true made someone an admin ;)
<mrvn>
bauen1: that assert of the invariant is what I'm talking about.
<geist>
and random bitflips are a thing for reals
<heat>
mrvn, the checksum are way more complex
<geist>
one nice thing: modern cpus have built in crc32s that are *faaast*
<mrvn>
heat: that's why you make them optional.
<heat>
you'd need getters and setters for everything (lots of times, not the case)
<geist>
basically the cost of reading the memory in
<zid`>
My friend is basically the ECC squad, he posts me all silly stories about bitflips and customers annoying him telling him their service is not 99.99% as advertized because of a bitflip in the customer's ethernet card's ram
<mrvn>
heat: yes you do. only way to catch corruption early.
<geist>
one thing we see sometimse in fuchsia via kernel crashes is driver doing stray DMA over kernel. that's always a fun one
<mrvn>
It's too bad you can't have decorators in c++, like @checksum_this \n def insert_sorted(self, item): ....
<heat>
i remember I read something on IRC (maybe #llvm?) about a game dev that was trying to detect bitflips or bad memory before saving the game because periodically someone complained their save game got corrupted with impossible data
<geist>
and one we've seen on a few devices is bad ram. you get say bit 27 that's a ittle sticky and generates fun kernel adresses like
<geist>
0xffdf800000000000
<geist>
boom, kernel gets a fatal page fault, looks like corrupt kernel bug
<geist>
and then you data mine it a bit and notice the exact same device is reporting that a lot
<heat>
isn't stray DMA mitigated by IOMMU?
<geist>
heat: if only iommus were really implemented ubiquotously
<mrvn>
geist: That's why I like microkernel where drivers have separate address spaces. You can verify the addresses beforte submitting them to DMA hardware.
<geist>
alas. that doesn't help without iommu
<zid`>
does anyone actually.. configure their iommu that defensively?
<zid`>
like, I have an iommu, but I don't think windows actually *uses* it
<geist>
i think linux does if it detects it yes
<geist>
and we do on fuchsia if we detect one, but currently only intel iommu
<heat>
mrvn, fuchsia is also a microkernel
<heat>
doesn't help
<heat>
(apparently)
<geist>
right, without iommu you cant control what the driver puts in their DMA hardware
<mrvn>
heat: it doesn't validate address spaces like that
<mrvn>
geist: take the DMA hardware away from the driver.
<geist>
cant
<geist>
that's not an option. the hardware is as the hardware is
<geist>
if the device can initiate dma, as many do, you can't take that feature away from the device
<geist>
that's what iommus are for
<mrvn>
I mean separate the code. DMA hardware is pretty similar and you can have a common validation framework that every driver then has to go through.
<geist>
dma hardware is most definitely *not* pretty similar
<geist>
i cant disagree with you more there
<geist>
random one off standalone dma controllers maybe, but anything more sophisticated has its own mechanism for queuing io that's hiiiighly specific to the device
<mrvn>
geist: You have source and dest, stripes and size and such that the validation layer can understand and validate before the DMA driver puts it into the actual regs.
<geist>
that's again for trivial dma controllers, of which that's like 5% of the pie
<geist>
and sure, you can mitigate that by having a separate dma controller driver process that yo then.. i dunno, look at more closely
<mrvn>
For the queuing IO I agree with you. But so far the address stuff I found pretty common.
<geist>
but anything more complex has its own built in mechanism. io rins, etc
<mrvn>
there aren't that many ways to specify a memcpy() after all.
<geist>
oi'm thinking about basically all of the io on any device i've dealt with at work in the last 10 years or so. almost all of it is some sort of built in dma controller (SD, MMC, ethernet, wifi, GPU, etc), or trivial (uart, i2c, spi)
<geist>
the trivial stuff you just dont use DMA unless you have to, or you use a standalone dma controller which probably is its own driver since it'd need to be shared among multiple trivial devices
<mrvn>
geist: any of them not have buffers described as start, stripe and size?
<geist>
no
<geist>
most of the complex ones are ring based transfer descriptor things
<geist>
ethernet, xhci, SDMMC, etc
<geist>
ie, you build a chain of descirptors somewhere, insert it in a ring. or the ring is a list of transfer descriptors
<mrvn>
geist: exactly. And the descriptors you can run through a address validation layer between the driver wanting to do DMA and the driver filling in the descriptor bit pattern.
<geist>
i fail to see how that is even remotely practical. since the descriptors are highly device descriptor, living in ram themselves, etc
<geist>
you'd have to do something like hand off the ring buffer stuff to the kernel, with some BPF like language describing how the descriptor is formatted
<mrvn>
geist: think a bit higher. You start by having C pointers to buffers. Those you validate and only then the driver creates the bit patterns for the descriptors
<geist>
basically 'i'm virtio, my rings look like this, here's some interpreted code to manipulate it for me because i can't be trusted'
<geist>
yeah i think we're on vastly different pages here
<geist>
anyway it's all moot. this is what iommus are for
<geist>
and hopefully as things get more mature that'll start showing up more and more
<heat>
we should do DMA on the blockchain
<heat>
if every client agrees it's done, if not, no DMA for you
<heat>
every problem ever just got solved
<mrvn>
At the moment my drivers validate addresses in the virt-to-phys translation. You can't do DMA on some memory that doesn't belong to the driver because then the virt-to-phys fails to find the physical address.
<geist>
in general the main problem with stray dma is not that it put in a bad address, it's that there was a race with the pages being freed and reused
<geist>
and the dma wasn't cancelled in order, etc
<heat>
don't you pin the pages?
<geist>
we do, but then drivers fuck that up
<mrvn>
geist: oh totally, that is a completly different bug scenario.
<klange>
never underestimate the ability of software to fuck something up
<geist>
or a fun one is drivers crash
<heat>
or the hardware ;)
<geist>
so we have a whole mechanism in the kernel for that: when you pin pages to a 'BTI' and the process crashes, the kernel will keep the pages pinned until a new instance of the driver comes up and reclaims it (presumably after it parks the hardware)
<mrvn>
geist: a free and reuse is probably a far more likely cause for the DMA to overwrite stuff than the driver actually messing up the address from the start. You kind of notice the later fast.
<geist>
agreed
<geist>
yah most of the messed up address has been stuff like 4GB wrapping math issues or whatnot
<geist>
and that gets fixed quickly
<geist>
or, the thing o the other side of the driver is actually another piece of software that is itself buggy
joe9 has quit [Ping timeout: 256 seconds]
<geist>
anyway, going to piddle with virtio a bit
<heat>
which device?
<geist>
all of them. i gotta modernize this dfiver i wrote 7 years ago
<geist>
and make it work on both PCI and MMIO
<geist>
it currently only deals with virtio-mmio
<heat>
yeah
<klange>
speaking of drivers, today I will try to get object relocations implemented for aarch64 so I can get these device drivers I've been fixing up back into modules...
<heat>
my virtio drivers only work with PCI
<heat>
klange, I think on linux arm64 modules are just shared objects
<geist>
heat: yeah not a tremendous amount of difference, just need to more generically abstract the low level interface and discovery
<heat>
there's a handful of architectures that do that
<klange>
I saw the code as it came up while googling relocations I don't think they are
<klange>
As they have all of the static relocations and none of the shared ones
<geist>
and might have to add MSI-X since i thik a few of the devices in qemu by default only publish MSI-X
<heat>
hmm I remember that X86 was one of the exceptions
<heat>
maybe arm64 is too
<heat>
geist, the ones I've seen always expose regular IRQs + MSI-X
<klange>
There's only like five or six relocations to implement, they're just a bit trickier than the x86 ones since they rewrite specific bits within instructions so I have to read the ABI manual a bit more than just "plop some bytes here"
<geist>
heat: yeah. i think you can force it to also expose MSI, but i think in practice any OS that supports MSI also supports MSI-X so it hasnt' been a problem
<mrvn>
klange: got some docs about arm64 relocations in elf?
<bslsk05>
github.com: abi-aa/aaelf64.rst at 320a56971fdcba282b7001cf4b84abb4fd993131 · ARM-software/abi-aa · GitHub
<geist>
yah, i just looekd at a .ko on my arm box and they definitely apepar to be what i had seen from x86
<gog>
today is saturday
<gog>
i thought
<geist>
they're raw .o files, un-dynamically linked
<geist>
gog: klange lives in the future!
<gog>
oh that's right
<klange>
it is 7:30am on Sunday
<klange>
it's going to rain today
<gog>
damn i'd just go back to bed
<gog>
:D
<gog>
it's going to be 7:30am sunday here and i don't wanna be awake for that
<geist>
oooh Spoon released new album
<heat>
spoon(2)
dude12312414 has quit [Ping timeout: 276 seconds]
dude12312414 has joined #osdev
<mrvn>
I like .ko.xz files.
<klange>
I guess I could gzip modules, but that would serve little purpose; usually they're on a gzipped ramdisk anyway
<mrvn>
2.5G /boot/
<geist>
wow, that seems a lot
<mrvn>
yeah, over time kernels collect. Have to clean that more often.
<mrvn>
the grml image is a big part too
<mrvn>
137M/lib/modules/5.10.0-11-amd64
<mrvn>
126M/lib/modules/5.7.0-0.bpo.2-amd64
<mrvn>
... Debian doesn't compress .ko files.
<heat>
grml?
<mrvn>
heat: Linux live CD
<mrvn>
rescue system
<sortie>
finish him
<klange>
hello sortie how are sortix things?
<sortie>
klange, um, so I set up this IRC network running on Sortix, and very convincing human like nicks have appeared, so I gave a couple of them the commit bit, and now commits are making themselves in the master branch
<sortie>
I'm pretty sure they are actual real life humans and not some sort of emergency anomalous entities
<sortie>
At least gitlab is enforcing mandatory code review :)
<sortie>
klange, nothing like me catching covid makes me suddenly power through on making it a community project
<sortie>
(I think I'm basically recovered as of tonight)
<klange>
remind me, does your irc network still have non-TLS access?
<sortie>
Yeah in case anyone has some old school tech that doesn't do TLS such as the old school tech that I have that doesn't do TLS
<sortie>
Def need to rewrite my ports with libtls or something
<sortie>
irc(1) in the base system is trickier cuz I don't have any TLS in the base system
<sortie>
But yeah port 6667 is still go in the classic sense
<heat>
get libressl onto the base system?
<heat>
it's a pretty essential feature tbf
<sortie>
The base system is fine without it. It's in the minimal ports set.
<geist>
oh huh. didn't realize that virtio 1.1 added a new format for the virtio queues
<geist>
questino is of course what supports it and thus is it worth supporting
<heat>
the packed thing?
<geist>
yah
<geist>
previously when i wrote this code it was virtio .9 or 1.0 so looking at what changed
<sortie>
klange, but yeah, going community project is the big real thing on my side. nortti's been a great contributor for a while, and put them in charge of the random newcomers so I don't have to deal with their merge requests and training junior devs.
<sortie>
Honestly that's been the hard thing to solve about becoming a community project, how to do it without becoming a bottleneck, or wanting to be one, right? Letting other people do stuff, and offloading responsibilities to others, so you don't get overloaded by well meaning newbies
<sortie>
So basically I made a 'senior developer' role that essentially boils down to being a great contributor and being in charge of dealing with the new contributors and junior developers.
<sortie>
The whole point of this is to offload me, so I only mostly interact with the contributions that solid (enough) on the first try, while someone else trains the newbies in the style and design
<sortie>
I mean I def could just let Sortix stagnate whenever I move on in my life, like I've done a few times, but you know, I actually like it a bunch, so I'm happy to be having people play with it
<sortie>
klange, how about you, what have you been up to lately?
<geist>
oh hey also interesting note: legacy virtio interface says the rings are all guest endian, but the new format says it's little endian
<geist>
so yay i dont hae to futz with the actual ring code to work on 68k
<sortie>
Hehe klange going toaru arm in my IRC :D
<klange>
aarch64 port is working nicely, got some PCI(e) device drivers working so audio and networking is available in QEMU
<klange>
regularly testing with HVF on a Mac
<klange>
going to start working on the RPi4 platform support soon enough
<sortie>
Sweeeet!
<mrvn>
geist: so if I use modern virtion on m68k I have to endian convert everything twice instead of just DMAing it?
<geist>
well, when futzing with the virtio ring structure, yes
<geist>
since its apparently explicitly in little endian format
<mrvn>
geist: is that just the control structure or all the data?
<geist>
control structure
<heat>
and probably virtio commands no?
<geist>
it's clear the new virtio ring structure is designed to be mroe efficient. the original structure is okay, but too many moving parts that have to be updated
<geist>
yah probably the structure of commands too
<mrvn>
what's that? 8 fields?
<klange>
I should try to submit some of my QEMU patches upstreams. The Cocoa frontend is absolute garbage. It started behind my terminal, doesn't have any scaling options, and it prompts you with a horrible modal when you go to close it
<geist>
it's a ring and some transfer descriptors, with a few fields a pice
<geist>
a combination of 16, 32, and 64 bit fields
<geist>
per device there are structures transferred throught he virtio interface that are probaly also LE
<klange>
I patched out the modal and fixed the window ordering, and I should add scaling as I want to put some effort into a high-dpi mode for my UI
<heat>
geist, you should get virtio-scsi if you don't have it yet
<heat>
for the GCP goodness
<geist>
i should
<mrvn>
heat: is that better than block?
<heat>
no clue but GCP uses virtio-scsi only
<geist>
i think in general virtio-scsi is better than blk yes
<heat>
it's fancier though, you speak SCSI through virtio instead of the very limited command set virtio-blk speaks
<geist>
more features, being developed more, etc
<geist>
i get the feeling that blk is being mostly dropped on the floor
<mrvn>
does it support the device sending a resize notification?
<geist>
possibly.i think it's also easier for hosts to translate scsi commands to their native stack
joe9 has joined #osdev
joe9 has quit [Quit: leaving]
joe9 has joined #osdev
dequbed has quit [Quit: bye!]
dennis95 has quit [Quit: Leaving]
dequbed has joined #osdev
dude12312414 has quit [Quit: THE RAM IS TOO DAMN HIGH]
heat has quit [Remote host closed the connection]
heat has joined #osdev
myon98 has joined #osdev
heat has quit [Read error: Connection reset by peer]