klange changed the topic of #osdev to: Operating System Development || Don't ask to ask---just ask! || For 3+ LoC, use a pastebin (for example https://gist.github.com/) || Stats + Old logs: http://osdev-logs.qzx.com New Logs: https://libera.irclog.whitequark.org/osdev || Visit https://wiki.osdev.org and https://forum.osdev.org || Books: https://wiki.osdev.org/Books
<gog> gin and juice
<zid> sipping on?
<gog> yes
<zid> Dr. Gog
<zid> wait, was that dre?
<gog> dre and snoop dogg i think
<zid> dogg and dre yea thinking about it it was definitely dogg singing
<zid> any silly ideas for VBE?
YuutaW has quit [Quit: WeeChat 3.1]
immibis_ has joined #osdev
qookie has quit [Ping timeout: 255 seconds]
immibis has quit [Ping timeout: 246 seconds]
<kingoffrance> i saw the gcc "global register variables" on mips supposedly lets you use coprocessor registers...so, i dont know, incorporate that into your psx port or patch gcc until you can
<kingoffrance> and there are also "local register variables" but it says you can only pass to and fro asm()
<kingoffrance> local as in a function
<kingoffrance> that feature sounds more useful when you are using <registers that code will not use>
<kingoffrance> i.e. coprocessor, other devices, etc.
<kingoffrance> but i didnt see any instance except mips that was ever the case according to docs
<kingoffrance> maybe it will be expanded someday
vdamewood has joined #osdev
<geist> sure. that's actually useful on other arches too
<geist> for example, on riscv the ABI calls out the 'gp' and 'tp' registers as being fixed. f you're the OS you can easily set them and/or access them directly via a global register
<geist> or on arm64 you can set the x18 register (also free via the ABI) for your own purposes
<geist> register current_thread *ct asm("x18"); for example
<kingoffrance> i wrote a silly compile test to see what it allows, for ia32 it seems it can do 8-bit [a-d][hl] 16-bit [a-d]x 32-bit e[a-d]x, e[sd]i, e[bs]p those will compile anyhow. really, i think it likes ah or al but not both etc.
<moon-child> ah is evil and should be destroyed
<kingoffrance> it is, 8008 A, B-E . what is this 16-bit nonsense
<kingoffrance> funny enough i can do register unsigned char foo[1] or 2 or 4 but 3 or 5 it gets made for e.g. eax. but you can access as "byte arrays" apparently (only did compile test of "declarations" -- no initializing allowed either)
<kingoffrance> *gets mad foo[1] asm("eax")
<kingoffrance> if you are doing this, you should know how big short/int/long etc. are
<kingoffrance> so i dont know why you'd do that, but hey
<kingoffrance> it doesnt want you pretending its 24-bit lol
<kingoffrance> this might also be a problem if tehy started letting you do registers from multiple devices
<kingoffrance> somehow has to "translate" to c types for your main cpu
<kingoffrance> or, youd end up with e.g. "this int is type x, this other int is actually type y" etc.
dormito has joined #osdev
SGautam has joined #osdev
vdamewood has quit [Quit: Life beckons]
iorem has joined #osdev
<kingoffrance> i wonder if would let you stick overlapping things in a union lol
<moon-child> thanks, I hate it
gog has quit [Ping timeout: 250 seconds]
<kingoffrance> its infinitely more useful if you have code to everything, then you can tell code not to touch things etc.
<kingoffrance> *have source
<kingoffrance> hey you said "what if custom callign convention" well...
<kingoffrance> thats a crude form
<zid> Interesting
<bslsk05> ​github.com: Quasar32 · GitHub
<zid> someone made a github account just to send me a Makefile s/rm/$(RM) patch
dude12312414 has joined #osdev
<kingoffrance> the docs say dont need to make declaration included/visible everywhere, instead use ffixed-<reg> whatever to warn other code, which i think is another way of saying "relies on the honor system"
<kingoffrance> i.e. sounds like the linker will not stop you from doing something conflicting
<geist> kingoffrance: right
<geist> or writing asm, etc. the linker dont know what the code does
<kingoffrance> well thats not bad necessarily, gets out of your way. you just wont know until you crash at runtime if you messed up :)
<kingoffrance> it means it is your responsiblity to make sense
<kingoffrance> its the ghost of unix past lol
<doug16k> kingoffrance, on x86_64, you can use the "l" on all 8 registers. if the instruction has a rex prefix, you can't use any "h" register. It would have a rex prefix if it has 64 bit operand size or uses any of the added registers r8-r15 and [xyz]mm8-[xyz]mm15
<doug16k> all 16
<doug16k> r15l
<doug16k> so compilers don't have to screw around with 8 bit values not working in esi edi ebp esp
<doug16k> skylake has errata with h registers. you should stop using them altogether
<kingoffrance> youll be 64 by the time i get to 64-bit
<doug16k> they can't fix it in microcode
<doug16k> lol
<doug16k> actually I think they may have a fix
<doug16k> I'm curious how much it cripples it
<doug16k> haskell was hammering h registers and they had failures with their codegen
<doug16k> you might as well consider the h registers gone in long mode, since rex will annoyingly cause issues with being able to use them
freakazoid333 has quit [Read error: Connection reset by peer]
sts-q has joined #osdev
<doug16k> you want your code to never pick at registers. ideally you create completely new full width results every time
<doug16k> movzbl is probably going to run better than mov al,
<doug16k> because picking at registers makes a sequential section. if you create a whole new register, it breaks dependency of destination and starts a brand new dependency chain
<doug16k> it could be able to start it far ahead of time
<doug16k> point being, byte registers are bad
<doug16k> for store of low 8 bits, it's fine - internally you are depending on that whole register
isaacwoods has quit [Quit: WeeChat 3.2]
riposte has quit [Ping timeout: 256 seconds]
<kingoffrance> makes sense. well i try not to hardcode anything, so im not sure i want to use gcc feature
<kingoffrance> i just wondered waht registers it would allow
freakazoid333 has joined #osdev
<doug16k> realistically you would use a callee saved register, so you can link with code that is oblivious about it, and it still works
<doug16k> ebx esi edi would be your options
<doug16k> bl bh I guess
riposte has joined #osdev
<doug16k> if you used fomit-frame-pointer, you could use ebp
<moon-child> doug16k: you don't need a rex prefix for *mm8-15, if you use the vex prefix instead
ElectronApps has joined #osdev
nshp has joined #osdev
nyah has quit [Ping timeout: 255 seconds]
dude12312414 has quit [Quit: THE RAM IS TOO DAMN HIGH]
YuutaW has joined #osdev
vdamewood has joined #osdev
jjuran has quit [Quit: Killing Colloquy first, before it kills me…]
jjuran has joined #osdev
ElectronApps has quit [Remote host closed the connection]
rorx has quit [Ping timeout: 245 seconds]
doug16k has quit [Remote host closed the connection]
doug16k has joined #osdev
doug16k has quit [Remote host closed the connection]
doug16k has joined #osdev
SGautam has quit [Ping timeout: 258 seconds]
SGautam has joined #osdev
mctpyt has joined #osdev
MarchHare has joined #osdev
MarchHare has quit [Ping timeout: 240 seconds]
SGautam has quit [Ping timeout: 255 seconds]
SGautam has joined #osdev
mcfrdy has quit []
immibis_ has quit [Ping timeout: 258 seconds]
GeDaMo has joined #osdev
Burgundy has joined #osdev
immibis__ has joined #osdev
immibis__ is now known as immibis
mcfrdy has joined #osdev
<klange> ay, we got kernel modules again
<sham1> Nice
<sham1> Those are good to hvae
<zid> can you [*] and [M] them or just [M]
<klange> only supports runtime loading, not compile time linking, mostly because I did the minimum to have a clean implementation of the object linker and get a couple of drivers moved out of the core kernel
<klange> going to leave a lot more in the main kernel image this time around; toaru32 had critical features in modules so you had to load them at boot time; sticking with stuff I can load from the ramdisk during startup now
<klange> hacked up a quick little thing to check a list of PCI device IDs, so at the moment this is the 'device manager': https://github.com/klange/toaruos/blob/master/base/etc/startup.d/04_modprobe.sh
<bslsk05> ​github.com: toaruos/04_modprobe.sh at master · klange/toaruos · GitHub
<klange> Doesn't even support modules depending on other modules yet; the old one sort of did - at least it could bind symbols from other loaded modules, so I'll add that eventually...
<klange> I have plans... like sticking PCI IDs into a special section so they can be automagically checked by a userspace device manager rather than that hand-written shell script.
<klange> This linker is also way less terrible than the old one... https://github.com/klange/toaruos/blob/master/kernel/misc/elf64.c#L59
<bslsk05> ​github.com: toaruos/elf64.c at master · klange/toaruos · GitHub
dennis95 has joined #osdev
zaquest has quit [Remote host closed the connection]
rorx has joined #osdev
zaquest has joined #osdev
sortie has joined #osdev
<doug16k> klange, you can make a program header entry to jump straight to it
<doug16k> I make a .driver output section and put it in `... } :driver` program header
<doug16k> which pulls in *(.driver) from __attribute__((__section__(".driver"))) struct variable
<doug16k> from define that takes device id or whatever
<j`ey> doug16k: can you link an example driver?
<bslsk05> ​github.com: dgos/usb_xhci.cc at master · doug65536/dgos · GitHub
<bslsk05> ​github.com: dgos/kmodule.h at 978105c459d98cac3281da7b83dcb93c38ae4386 · doug65536/dgos · GitHub
<doug16k> yes
<doug16k> not master but probably close enough or same
<j`ey> and where's the code that uses these drivers?
<klange> > program header
<klange> those aren't a thing in relocatable object files
<j`ey> (driver sections)
<doug16k> oh
<klange> not that it matters, one would just look for the actual section by name
<doug16k> you already signed up to be a linker then :)
<klange> aye
<doug16k> j`ey, that part isn't finished yet
<doug16k> 99% there though
<j`ey> ah :)
<doug16k> have all the syscalls to do the probes and I proved I can read them
<doug16k> let me check
<doug16k> nope
<doug16k> it just loads everything right now: https://github.com/doug65536/dgos/blob/master/user/init/init.cc#L888
<bslsk05> ​github.com: dgos/init.cc at master · doug65536/dgos · GitHub
<j`ey> nice, so those are on the initramfs?
<doug16k> yes
<doug16k> this becomes the blitter thread
<j`ey> return start_framebuffer();
<doug16k> yeah plus dumping ground for testing newer syscall stuff
<doug16k> will move to proper home eventually :)
<doug16k> I thoroughly tested that you can modify the context properly in signals, for some reason, lol
<doug16k> plan was to make it suitable for debugging
<zid> is there sys_zelda
<doug16k> what would it do?
<j`ey> start zelda
<doug16k> wouldn't nintendo send hitmen after me?
<zid> It requires you provide zelda.o
<doug16k> tyrian syscall should be no problem
<doug16k> or doom
<zid> I'd like to find out why perf is so bad, but not enough to actually do anything
ElectronApps has joined #osdev
gw has joined #osdev
<zid> It runs about realtime speed, but without any syncing
<zid> it's just an infinite loop that writes framebuffer pixels
<doug16k> I got over 1000 fps in kvm
<zid> yea it gets 1200fps on my desktop
<doug16k> full memory bandwidth basically
<doug16k> need to profile it
<zid> It might just be crappy fb perf in general and the emu just gets that long between qemu code
<doug16k> can add your own hardware perf counter profiler with a perf counter set on cycles. set counter to -100000 and configure perf INT in LAPIC to get interrupt every 100000 cycles
<doug16k> return address is sample
<doug16k> reset perf counter to -100000 or whatever before returning
<zid> I think the perf loss is imaginary though
<doug16k> then blast them to serial? script to lookup function from nm dump?
<zid> and I don't have lapic :P
<doug16k> you have timer irq?
<doug16k> just collect the return addresses
<zid> okay but that will just show 100% cpu use inside my code, is the point
<zid> I think the cpu loss is inside qemu's memcpy to its texture buffer or whatever
<doug16k> yes but you remember them in a giant array and write them to output, then analyze all the saved PC against nm symbol addresses
<doug16k> then you can calculate percent of samples in each function
<doug16k> then sort by that descending
<zid> and they will add to 100%
<doug16k> then fix perf of top one
<zid> and tell me nothing
<zid> I know the perf of the *program*
<zid> The question is why is the perf of the qemu-program system bad
<doug16k> ...then analyze all the saved PC against nm symbol addresses...
<zid> and they will add to 100%
<zid> and tell me nothing
<doug16k> it will tell you which *function* it was in the most
<zid> you're not listening
<zid> I now which function it's in the most
<doug16k> how do you know?
<zid> because it's the same program as I run on my desktop
<doug16k> what if it is taking way more cycles in something else?
<zid> and I wrote all the code
<zid> what something else
<doug16k> what if some port I/O is 1000x slower
<zid> It doesn't use any i/o ports
<zid> and it's in userspace.
sortie has quit [Quit: Leaving]
<doug16k> why is it in qemu?
<zid> because my OS is hosting it.
sortie has joined #osdev
<doug16k> what I said applied to the OS
pg12 has quit [Ping timeout: 268 seconds]
<zid> good news, the OS isn't running
<zid> >it's just an infinite loop that writes framebuffer pixels
<doug16k> so the zelda emulation was just to mislead
<doug16k> make me think it's an emulator
<zid> You're trolling yourself and blaming me
<zid> It is an emulator.
<zid> The emulator is an infinite loop, that writes pixels to a framebuffer.
<zid> It uses 100% cpu inside the guest, NO MATTER WHAT
pg12 has joined #osdev
<zid> If you profile it, you will get precisely: 100% cpu use in 'the code' and the split will be 'what zelda does'
<zid> it's completely useless to measure it, when we're asking about *qemu* performance
SGautam has quit [Ping timeout: 255 seconds]
SGautam has joined #osdev
<zid> neat
<zid> what does your vmware module support doing?
<klange> vmport mouse, which is why it's loaded in qemu
<zid> ahh, is it tied into the vmware balloon stuff, or just the name you gave to vmware's mouse?
<klange> also there's some commented out code for doing automatic modesetting in vmware itself that I should probably uncomment
<zid> I know vmware has a bunch of backchannel stuff
<klange> I don't know how the balloon stuff works but presumably it's the same interface
<klange> oh i remember why that was commented out, it was waiting on kernel tasklets
<klange> those have existed for ages, but I wanted the absolute mouse support in qemu really early on
<zid> MIce? Where we're going we don't need mice!
<zid> I don't even have keyboard support
<zid> I do have utf-8 text rendering in my dmesg though, in case some wants a lovely monospaced katakana
<zid> My plan was always to interface with the kernel over http though
<zid> so like a modem or whatever, where you web configure it and then it just does stuff in the bg
<klange> I'm spoiled.
<klange> I got really annoyed about having to un-grab the mouse in qemu during early testing once I got my GUI up on the new kernel, so I _had_ to get the vmport mouse implemented or I was gonna go apeshit.
<zid> oh is that how that works
<zid> it lets you make it act like a window?
<zid> I'm using qemu in vmware so I'm double-nested in mouse shenanigans
<klange> qemu also supports a usb tablet, but that obviously needs a usb stack
<klange> the vmport mouse also works with the vnc frontend, which is really important
<zid> my shit does boot on VMware, but I used the bochs-vbe device for my test geraffiks
<zid> what doc do I need for vmware graphics?
<klange> search 'vmware svga' and you might find info
<klange> it's roughly the same process as the bochs vbe, but different ports
<zid> not that I touch ports
<zid> I do it over the pci-e mmio region
<zid> it emulates it as ports though
<klange> Careful with that, virtualbox doesn't support the mmio mode.
doug16k has quit [Quit: Leaving]
<zid> vbox hadn't even entered my mind
<zid> not sure I've ever used it
<klange> Welcome to osdev, test your shit in everything :)
<zid> My real hw is an e1000 I'm tempted to try booting it and pinging it
<zid> the problem is, I'd have nothing to ping it /with/
<klange> Don't forget to have a ten year old laptop constantly doing nothing in a corner of the room...
<klange> 64 bytes from icmp_seq=1 ttl=64 time=0.940 ms
<klange> :3
<zid> try, you may need to bring your own ethernet cable
mhall has quit [Quit: Connection closed for inactivity]
<klange> oh right, if I load my i965 modesetting driver during init, then it'll change the resolution after the splash screen has initialized...
<zid> [*] time
<klange> and the splashscreen doesn't handle display size changes...
<zid> I thought briefly about how I'd support [*] and decided "it's hard"
<zid> I guess like.. x-macros to make an array of init functions to call
<klange> the compositor does - there's an interface to send a signal to the display "owner" when the resolution changes, just the splash screen doesn't do anything with it :)
<zid> which should make the linker happy about not tossing it away
<klange> so I get garbled text for a bit before the compositor comes up, lol
gw has quit [Quit: Konversation terminated!]
dennis95 has quit [Remote host closed the connection]
dennis95 has joined #osdev
qookie has joined #osdev
ZetItUp has joined #osdev
yomon has joined #osdev
Burgundy has quit [Ping timeout: 255 seconds]
yomon has quit [Ping timeout: 252 seconds]
immibis_ has joined #osdev
immibis has quit [Ping timeout: 255 seconds]
gog has joined #osdev
nyah has joined #osdev
Oli has joined #osdev
Oli has quit [Client Quit]
tacco has joined #osdev
sortie has quit [Remote host closed the connection]
sortie has joined #osdev
brynet has quit [Quit: leaving]
MarchHare has joined #osdev
brynet has joined #osdev
immibis_ is now known as immibis
myon98 has joined #osdev
Skyz has joined #osdev
andreas303 has quit [Ping timeout: 268 seconds]
isaacwoods has joined #osdev
andreas303 has joined #osdev
opios2 has quit [Ping timeout: 268 seconds]
rorx has quit [Ping timeout: 268 seconds]
dh` has quit [Ping timeout: 268 seconds]
meisaka has quit [Ping timeout: 268 seconds]
meisaka has joined #osdev
rorx has joined #osdev
nismbu has quit [Ping timeout: 256 seconds]
Mutabah has quit [Ping timeout: 268 seconds]
Mutabah has joined #osdev
ElectronApps has quit [Read error: Connection reset by peer]
froggey has quit [Ping timeout: 256 seconds]
froggey has joined #osdev
vdamewood has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
qookie has quit [Quit: leaving]
iorem has quit [Quit: Connection closed]
Robbe has joined #osdev
zoey has joined #osdev
sortie has quit [Remote host closed the connection]
sortie has joined #osdev
<Skyz> Klange: Do you want people to use your OS?
<Skyz> I would suggest making an iso that someone could try loading into virtual box
froggey has quit [Ping timeout: 246 seconds]
<j`ey> Skyz: there already is an iso
froggey has joined #osdev
<Skyz> Where? I don't see it
<bslsk05> ​github.com: Releases · klange/toaruos · GitHub
<Skyz> Finding that is not intuitive
<clever> thats where every major project on github hosts the files
<j`ey> the README probably could mention it
<Skyz> It boots but the screen is messed up with the scan lines
<Skyz> It might have to load
<Skyz> I mean if you want people to use it, you would put it somewhere where people can find it
<j`ey> yes, in the release section on github :P
<gog> that seems like a perfectly reasonable place to find releases
<gog> obvious, even
<Skyz> The iso file should be easier to find
<Skyz> There should be a link to the releases atleast
<Skyz> In the readme
<Skyz> And going to the releases, even then you got to scroll down and hit assets
<Skyz> It won't load to the desktop with VMware
<Skyz> I got something on Virtualbox
mcfrdy has quit [Ping timeout: 255 seconds]
<bslsk05> ​imgur.com <no title>
<bslsk05> ​imgur.com: Imgur: The magic of the Internet
mcfrdy has joined #osdev
<j`ey> I had a similar issue in virtualbox
<j`ey> I messed around with the display settings -> grapics controller
<Skyz> Oh awesome, it works on VMware
dude12312414 has joined #osdev
<bslsk05> ​imgur.com: Imgur: The magic of the Internet
<j`ey> nice
<Skyz> I like this feature
<Skyz> You write Hello into the command prompt and it returns hello world
dh` has joined #osdev
pony has quit [Ping timeout: 265 seconds]
<Skyz> Can't get packages because of an error
<bslsk05> ​imgur.com <no title>
<Skyz> imgur is giving me some problems
pony has joined #osdev
<bslsk05> ​redirect -> doc-10-2s-docs.googleusercontent.com <no title>
jjuran has quit [Ping timeout: 265 seconds]
<dzwdz1> are there any executable formats simpler than a.out? ones that don't require any relocation
jjuran has joined #osdev
<clever> dzwdz1: do raw .bin files count?
<dzwdz1> but that would require the executable to be built as a PIE, right?
<clever> you could just set a rule that its always loaded to a given addr
<zid> Yea the opposite of relocatable just non-relocatable :P
<clever> as an example, the bootcode.bin file on an rpi, just gets copied to 0x8000_0000 physical, and then the rom jumps to 0x8000_0200
<clever> no relocation, no patching
<zid> tbh though, a non-relocatable ELF is a pretty damn simple format
<GeDaMo> Does source code which is compiled at load time count as an executable format? :P
<zid> you can just read a certain offset pair and you're done
<dzwdz1> wouldn't i need to load everything into rw memory if i was using a .bin file?
<clever> dzwdz1: yeah, since there is no way to define what permissions go where
<dzwdz1> how do microkernels that don't have an exec() usually handle loading the first executable?
<dzwdz1> do they have an elf parser just for it?
<zid> My OS jsut has two images
<zid> one loads to -2GB ones loads to 2MB
<zid> and the kernel jumps to 2MB
<dzwdz1> my idea is that the kernel loads an executable which has the initrd drivers
<dzwdz1> and which also has the initrd embedded in itself
<GeDaMo> Why can't the executable just be loaded to RO memory?
<dzwdz1> and the elf loader, since i don't see the point in putting it in the kennel
<dzwdz1> .data?
<clever> GeDaMo: stack
<dzwdz1> unless there's a way to prevent the compiler from using .data at all
<clever> it needs some kind of rw memory, to be able to make a few func calls to fixup its own permissions
<GeDaMo> A default stack space doesn't seem like a big problem
<clever> yeah, that could work around it
<dzwdz1> ^
<dzwdz1> it's only .data which is the issue
dzwdz1 is now known as dzwdz
<clever> and you have to define the size of .bss somewhere
<GeDaMo> The executable requests RW memory and copies across the data initial values
<GeDaMo> Or changes the permissions on part of itself which has the data values
<clever> a lot of MCU's go the copy route
<clever> the .text and a template of .data are all in rom (or flash that isnt modified)
<clever> and on startup, it copies .data to ram
<GeDaMo> Yeah, that's what I was thinking of
<dzwdz> does gcc support that?
<clever> dzwdz: yep, its handled by the linker script
<zid> As in, it's nothing to do with gcc
<clever> you can setup the linker script so the virtual addr (used by compiled code) is in ram, but the physical address (only used by the loader/programmer) is in flash/rom
<clever> then whatever is generating the .bin file for your rom, has to respect physical addresses in the .elf
<clever> but the linker uses virtual when filling things in on the asm gcc provided
<dzwdz> that seems more complicated than just rolling my own executable format tbh
<clever> dzwdz: you would rather that every user has to have a custom binutils?
<zid> I mean, that is how you'd make your own executable format
<clever> the above process, lets end-users program the MCU with stock tools
<dzwdz> clever: it would only be used for a single executable, which would be the same in most setups
<clever> dzwdz: sounds like an MCU, it runs a single binary on startup, and it never changes
<zid> . = VIRT; .text { * (.*); } is a pretty normal linker script for something like that
<zid> which just makes all the code expect to be loaded to `VIRT`
<zid> and all the data and stuff be included int he output
<dzwdz> doesn't the compiler expect a writeable .data section?
<zid> If you want permissions on various pieces of memory, you're going to need a proper image format
<zid> in which case you may as well just use ELF
<bslsk05> ​github.com: pico-sdk/memmap_default.ld at master · raspberrypi/pico-sdk · GitHub
<clever> __data_start__ and __data_end__ mark the virtual addresses that .data begins/ends at
<clever> } > RAM AT> FLASH
<clever> the AT keyword, says that the physical address, goes into the region called FLASH
<clever> lines 26/27 define what RAM and FLASH are
<zid> https://github.com/zid/boros/blob/master/linker.ld and my dirt simple kernel linker script :P
<bslsk05> ​github.com: boros/linker.ld at master · zid/boros · GitHub
<dzwdz> oh, thanks!
<dzwdz> so do i have to implement copying .data into the correct location myself?
<clever> yeah
<zid> unless the hardware is going to do it for you.. yes?
Oli has joined #osdev
<zid> it isn't going to end up there any other way I can imagine
<clever> let me find an example...
<dzwdz> nah, i get it
<dzwdz> i can find one myself
<zid> My elf code is literally
<zid> p = (struct program_header *)(u32)(e->e_phoff + (u32)e); for(i = 0; i < e->e_phnum; i++) map_section(&p[i], (u32)e);
<dzwdz> is it seriously that simple
<dzwdz> wow
<bslsk05> ​github.com: pico-sdk/crt0.S at master · raspberrypi/pico-sdk · GitHub
<clever> 229 gets the address of a set of things to copy
<clever> 236 copies one thing, and then it repeats
<zid> ELF is just a main header that tells you where to find the program headers, and how many of them there are
<zid> and each program header tells you where and how many bytes to copy
<clever> 278-295 is a set of source/start/end pointers
<dzwdz> it just seemed like a complicated format
<zid> everything else is for linkers and debuggers and stuff
<clever> the linker script says to put the .data into flash after .text
<clever> so the .S file i linked, can safely assume that .data begins at __etext
<clever> and because .text stays in flash, __etext is a flash based addr
<clever> which jsut happens to be where .data begins in flash
<zid> try readelf -l on some binary
<zid> It's a 6 field array
<zid> offset/virt/phys/fileoffset/size/flags
Skyz has quit [Quit: Client closed]
ZipCPU has quit [Ping timeout: 272 seconds]
ZipCPU has joined #osdev
* geist yawns
<geist> good afternoon folks
<nur> afternoon sir
<dzwdz> hi o/
<nur> although it is 3am here
lucf117 has joined #osdev
<dzwdz> why are you still up? that's not healthy
<geist> huh 3am what time zone is that?
<geist> i thought you were in europe
rorx has quit [Ping timeout: 240 seconds]
<geist> east asia i guess, or west asia
<nur> Malaysia
<nur> I'm up because well, the rest of my people live opposite the world and this is when _they_ are up
<nur> why did you think I was in Europe
rorx has joined #osdev
<geist> i dunno, i just thought for some reason you had mentioned you were in scandinavia before
<geist> so many people i've talked to here, easy to get them mixed up
<geist> also maybe i have a bias with words like 'nur' sounding like 'north' and thus scandinavia comes into mind
mcfrdy has quit [Ping timeout: 255 seconds]
<Oli> exit
<Oli> whoops
mcfrdy has joined #osdev
Oli has quit [Quit: leaving]
Oli has joined #osdev
Oli has quit [Quit: leaving]
Oli has joined #osdev
<sortie> Tonight I'm poking a bit on my build infrastructure. A bunch of the release logic, like generating the html manual, is actually not part of the ‘make release’ command but are done by an unversioned CI script
<sortie> So I'm basically merging stuff from that CI script into the main repository, plus making sure it works natively. Turns out grep -r --include='*.html' crashes with a libc assertion failure in mbsnrtowcs(3)
<sortie> One thing I never tried is cross-compiling my OS under itself, i.e. building the 32-bit release under the 64-bit release, with a proper cross-compiler. It should work in theory, but some ports might have stupid kinds of issues, who knows. Certainly it'd simplify things for me if I could do this.
<sortie> Anyways I'm starting to finally get over this nasty cold and feel somewhat productive again and I have coffee :)
<sortie> What are people up to tonight?
<gog> gonna smoke a bowl and watch a thing about black holes
srjek has joined #osdev
<sortie> Fun :)
jjuran has quit [Ping timeout: 250 seconds]
jjuran has joined #osdev
isaacwoods has quit [Read error: Connection reset by peer]
GeDaMo has quit [Quit: Leaving.]
vai has joined #osdev
<vai> howdy all
<vai> sortie:
<sortie> Hiya vai
<sortie> How are you?
mcfrdy has quit [Ping timeout: 255 seconds]
mcfrdy has joined #osdev
silverwhitefish has quit [Ping timeout: 258 seconds]
heat has joined #osdev
<heat> warm greetings fellow humans
freakazoid333 has quit [Read error: Connection reset by peer]
<mjg> heat after all
<clever> "2021-07-17 19:11:27 bedroom temp: 26.00c(78.80f), kitchen: 26.38c(79.47f), living room: 25.75c(78.35f), outdoor: 24.00c(75.20f), server: 27.12c(80.83f) VCC: over 4.5 volts portb: 00000000"
<clever> heat: then i have you to thank for this weather? :D
<heat> :D
<heat> today peaked at around 30C around here
<heat> not too bad, but not great
<mjg> how hot is it in russia?
<heat> 29 in saint petersburg
<mjg> if it's anything above 15C they are getting cooked alive
<heat> 25 and raining in vladivostok
<zid> >20C still outside at midnight, bleh
<zid> it's probably like 30C in here
dude12312414 has quit [Quit: THE RAM IS TOO DAMN HIGH]
<heat> so I had this idea, what if I wrote a new build system for building other projects that have separate build systems
dude12312414 has joined #osdev
<heat> it would work really well for an OS monorepo that has lots of different random projects/build systems/packages
<heat> has anyone tried something similar?
<zid> Yes, I sneak into the houses of people who use ant/ninja/cmake/etc and stab them to death.
<zid> I find it more effective ovrall
<heat> what happens to kati users?
<heat> stab or no stab?
<zid> no idea what that is, I can't be everywhere at once sadly
<heat> android thing that takes a makefile and outputs a ninja file
<zid> android people are already insane, so them doing insane things is to be expected
<heat> ah I see
<heat> you must be one of those autoconf users
<kazinsal> autotools is heretical
<clever> most of my stuff builds with just a makefile, no autotools/autoconf
<clever> only if i'm targetings multiples arches, do i get cmake involved
<clever> like windows/linux/mac
<heat> yeah but the issue here is: what do you do when you have a monorepo?
<clever> my most complex monorepo is rpi-open-firmware
<heat> regular build systems don't understand when other build systems need to rebuild/have rebuilt
<clever> it builds for linux-arm, baremetal-arm, baremetal-vpu
<clever> and that is all done with plain makefiles (and the nix package manager)
<clever> nix handles the rebuilding for you
<clever> basically, every input dependency is at a path like /nix/store/hash-name
<zid> Packaging is for distros
<clever> and you must install to /nix/store/hash-name, which is a hash of all of your input strings
<zid> we honestly don't need to be detecting if various packages are installed in ./configure
<zid> gentoo shouldn't let you emerge the ebuild until it has the deps, or the ebuild is broken
<heat> zid, you do if they're optional
<clever> yeah, my makefile just assumes that -l and #include can find things
<clever> its up to the package manager to make it discoverable and installed
<zid> heat: You still have to specify that, why bother specifying it to configure
<clever> zid: nix is basically gentoo, but better
<zid> like, I'm surprised it isn't make -DOPTION instead of ./configure --with-OPTION
<zid> or such
<clever> as an example, if you build something with gentoo, it can find things you didnt enable with a USE flag
<clever> and then link against them
<clever> but because you didnt set the USE flag, gentoo thinks its not a dependency
<zid> USE should set ./configure --with
<clever> then things break when you uninstall it
<zid> if not, ebuild bug
<clever> nix cant have that bug
<zid> good for nix
<clever> so its better then gentoo
<clever> /usr/lib doesnt exist, /lib doesnt exist
<zid> gentoo is shit because python
<clever> the only way to find a package, is for the package manager to tell you where it is
<zid> not because of how portage works
<clever> if you dont ask for something, youll never know its installed
<heat> anyway, my idea here was to create some sort of meta-build-system language that describes the build and do the rest in Go/Rust (to force myself to learn one of those)
<clever> heat: sounds like nix :P
<heat> that would fix the Make/ninja/gn/cmake/bazel stupid
<clever> nix describes the dep tree, what binary to run to kick-off a build, and what env vars to set during the build, and thats it
<zid> heat: you just invented ebuilds, except awful awful python
<clever> built ontop of that, is a framework to automate ./configure && make && make install, but other frameworks exist for other languages
<heat> clever: does it know how to check if something was rebuilt?
<clever> heat: basically, the product gets installed to /nix/store/$(hash depa-1.0 depb-2.0)-result
<bslsk05> ​github.com: gentoo/gcc-4.7.3-r1.ebuild at master · ClusterHQ/gentoo · GitHub
<zid> random ebuild google gave me
<clever> heat: if the version of depa changes, the hash changes, so the build product isnt out of date, its just missing
<clever> heat: every time you ask for a given package, it computes that hash, and then checks if you have a copy locally, matching that exact build plan
<zid> It's just a bash script that sets up configure and make
<zid> based on your system settings
<clever> if ANY part of the build plan differs, its considered a different result, and rebuilds what it needs to
<heat> 1) crazy duplication? 2) yeah but what happens for changes in the actual build
dennis95 has quit [Quit: Leaving]
<clever> heat: the hash covers the build directions, and the hash for every input library
<heat> package foo depends on A and B, A and B didn't change but foo's source has changed
<clever> changing the source impacts the hash
<heat> they also hash the source?
<clever> yes
<heat> that's erm... possibly not practical
<clever> gentoo does it too
<clever> the hash of the .tar.gz is in the package description
<clever> in nix, the hash for the entire build plan, is rooted on those .tar.gz hashes, and the build instructions defined in the nix files
<clever> and at download time, if a given .tar.gz doesnt have the expected hash, its a fatal error, and the build aborts
<zid> just ship binaries lol
<heat> yes but I'm not dealing with tarballs and if something that isn't built changes, I don't care
<clever> zid: because of how reproducable nix is, you can now share binaries between machines
<heat> and if the build config changes, I *do* care
<clever> heat: something local?
<zid> great, now get rid of all that mess
<zid> and just provide binaries
<heat> clever: hm?
<clever> heat: if your not dealing with a tarball, how are you getting the source?
<heat> monorepo
<clever> heat: ah, one sec
<bslsk05> ​github.com: rpi-open-firmware/default.nix at master · librerpi/rpi-open-firmware · GitHub
<clever> heat: this tells nix to make a copy of ./. but ignore .o files as its copying
<clever> it will then hash that copy
<clever> and if that hash changes, rebuild
<clever> zid: nix allows you to reuse the pre-built binaries from a cache, but then override anything you want, and it will seamlessly rebuild what is needed
<clever> zid: so i can trivially add a patch to udev to fix a bug, and not need to worry about what i have to rebuild afterwards
<clever> and still benefit from the pre-built binaries in the cache
<zid> neat, neither do I
<heat> thinking about this is hard
<clever> heat: basically, the build instructions for a single package get compiled down to 3 things
<clever> 1: a binary to run to start the build (usually bash)
<clever> 2: the args for that binary (usually a bash script)
<clever> 3: a set of env vars to be set when launching that
<clever> that whole thing gets serialized and hashed, to compute the output dir, /nix/store/${HASH}-${NAME}
<clever> if you depend on another library, the ${HASH} of that libraries build plan, will be in one of the env vars
<clever> so changing that library in any way, changes the env vars, and changes your own HASH
<clever> to get rid of insanity like rebuilding when the version of curl changes, the package to download a URL for a .tar.gz has slightly different rules
<clever> the HASH is based on the expected hash of the .tar.gz, not the binaries ran to fetch it
freakazoid333 has joined #osdev
<heat> what I was thinking of was something like: you provide build configuration, platform/target/board/configuration, etc; build gets divided into stages (so boot stage (for things that go on the boot partition), system stage (for general system stuff)); you can't have dependencies between stages and a package that is used in multiple stages needs to be build N times
<clever> nix is split up, and can do stages like that, but it can also reuse when it makes sense
<heat> the build tool (with a cool name of course) would invoke the build of every package required by reading the metadata and having a general query command and build command(s)
<bslsk05> ​github.com: rpi-open-firmware/default.nix at master · librerpi/rpi-open-firmware · GitHub
<clever> heat: as an example, common on line 37 gets built in 3 different configurations, always as a static library
<clever> baremetal arm, baremetal vpu, and linux arm
<clever> tlsr (a heap library) on 27 gets built for just baremetal arm
<clever> tlsf*
<clever> chainloader on 72 gets build for baremetal-arm
<heat> build output would get sent to something like out/$target-$board-$platform/$stage_name/path_to_pkg/$pkgname; and they would populate some sort of sysroot at something like out/$blahblah/$stage_name/sysroot
<clever> firmware on 95 gets built for baremetal-vpu, and depends on the bare-arm build of chainloader
<clever> bootdir on 175, depends on firmware
<clever> and also compiles the dts files
<clever> so if i ask for the bootdir, nix will built tlsf, common, chainloader, and firmware, in roughly that order
<heat> in the end the build tool would take care of any stripping that may be required, and would assemble the partitions; it could also be needed to discard some files that were only needed by the build(like libc.a in some libc-devel kind of thing)
<clever> diskImage on 196, will then make a .img file, containing an entire OS install, ready to boot
<clever> with nix, libc.a is in the -dev outputs
<clever> and nix searches your build product for paths, to see if you refer to something
<clever> if you dont refer to -dev, then -dev doesnt get included in the disk image
silverwhitefish has joined #osdev
<heat> right
<heat> but shouldn't -dev packages be required to be explicitly installed by whoever is configuring the build/doing sysadmin?
<clever> not with nix
<clever> -dev isnt a seperate package
<heat> hmm
<clever> a package can have multiple outputs
<clever> [ "/nix/store/hp8wcylqr14hrrpqap4wdrwzq092wfln-glibc-2.32-37" "/nix/store/mbxi7rhg9pj2mfnzr7dk9cd46xdhnrf9-glibc-2.32-37-bin" "/nix/store/f8ahd6cxx9ia4fzdiyczmhcv3c2qmd42-glibc-2.32-37-dev" "/nix/store/5win19g9qga6fgf5wiwc6qhlfx3m2768-glibc-2.32-37-static" "/nix/store/12nhr313dlk4mbkgmwrgjcz830fh1cxa-glibc-2.32-37-debug" ]
<clever> heat: in this example, glibc has 5 outputs, [ "out" "bin" "dev" "static" "debug" ]
<clever> when you build glibc, it must create those 5 directories, and populate them with something
<heat> can you trigger nix to do something like keep /lib/libc.a but discard /usr/include/stdio.h?
<clever> in this example, libc.a is in -static, and stdio.h would be in -dev
<clever> since they are seperate outputs, yes
<clever> /nix/store/5win19g9qga6fgf5wiwc6qhlfx3m2768-glibc-2.32-37-static/lib/libc.a
<clever> /nix/store/f8ahd6cxx9ia4fzdiyczmhcv3c2qmd42-glibc-2.32-37-dev/include/bits/stdio.h
<clever> heat: when you define glibc as a dependency, the static folder gets its /lib added to the -L path, so gcc can find it with you `-lc`
<heat> oh yeah something completely random: I found a decent reason to actually have a usr/ and have the root lib,bin symlink to it: the inode/block allocation is different in non-root so everything in usr/ will be closer together
<clever> and the -dev folder gets its /include added to -I, so #include just works
<clever> AFTER your build is done, nix will search your resulting binary, for both 5win19g9qga6fgf5wiwc6qhlfx3m2768 and f8ahd6cxx9ia4fzdiyczmhcv3c2qmd42
<clever> and only if you have one of those hashes in your binary, do you depend on it
<clever> if you dont delete on it, then it can be deleted or not copied
<heat> like a pure binary grep?
<clever> yeah
<heat> that seems a bit less useful
<clever> if 5win19g9qga6fgf5wiwc6qhlfx3m2768 doesnt appear in your result, there is ZERO way for you to ever find that libc.a
<heat> why do those different outputs have different hashes?
<clever> so you cant use it at runtime
<clever> the hash also includes the output name, static/dev
sav has joined #osdev
dude12312414 has quit [Quit: THE RAM IS TOO DAMN HIGH]
<clever> and if you cant use it at runtime, you wont care/notice if the path is missing at runtime
<heat> interesting
<clever> so nix is free to delete the entire -static during garbage collection
<clever> or just omit it from the disk image
mcfrdy has quit [Ping timeout: 268 seconds]
<heat> nix is certainly a lot closer to something I would want as a build-my-os tool
<clever> nix also has native support for cross-compiling
<clever> so you can just define how to build a package for any arch, then ask for 3 copies of it, for 3 different arches
<bslsk05> ​github.com: rpi-open-firmware/default.nix at master · librerpi/rpi-open-firmware · GitHub
<clever> heat: line 98 says that firmware depends on common and notc
<clever> "buildInputs": "/nix/store/5p2az54f4j4kb31ahkqiyx99gdjlnapi-common-vc4-elf /nix/store/hjkchjr6b6rxzxlh22syp5hvr5bg4p6j-notc-vc4-elf",
<clever> heat: that results in this env var being set at build time
<heat> how do you know how to invoke the build?
vin has quit [Quit: WeeChat 2.8]
<clever> [clever@amd-nixos:~/apps/rpi/rpi-open-firmware]$ nix build -f . vc4.firmware
<heat> e.g having a autoconf package and needing to pass a different ./configure based on the arch
<heat> not that :P
<clever> ah, that
<clever> thats all handled by the stdenv, one sec
mcfrdy has joined #osdev
<bslsk05> ​github.com: nixpkgs/setup.sh at master · NixOS/nixpkgs · GitHub
<clever> heat: by default, nix will run the configurePhase on line 966, to get configure ran
<clever> and setup hooks in various places, can inject things into $configureFlags, to extend it further
<heat> where's the cross compilation?
<clever> let me find an example
<heat> like --host=$cpu-linux-gnu
<clever> some of it, is just done via $CC and $LD, which contain the cross compiler/linker
<clever> so if you just blindly $CC and $LD things, you cross-compile without even having to try
<heat> yes but you need to pass correct configure options I think
<heat> unless you don't and I've been wasting my time all along
SGautam has quit [Ping timeout: 255 seconds]
<clever> $ nix-shell '<nixpkgs>' -A pkgsCross.aarch64-multiplatform.hello
<clever> this will drop me into a shell, with all of the env vars and tools required, to cross-compile hello to aarch64
<clever> *looks*
<clever> CC=aarch64-unknown-linux-gnu-gcc
<clever> configureFlags=--build=x86_64-unknown-linux-gnu --host=aarch64-unknown-linux-gnu
<clever> heat: yep, there it is, one of the setup hooks added that to $configureFlags
<clever> now i can search the src for that...
<clever> stdenv/generic/make-derivation.nix: ++ optional (elem "host" configurePlatforms) "--host=${stdenv.hostPlatform.config}"
vin has joined #osdev
<bslsk05> ​github.com: nixpkgs/make-derivation.nix at master · NixOS/nixpkgs · GitHub
<clever> heat: and there is the source behind it
<clever> lines 257-259 convert the configureFlags from the package into a uniform style (an array of strings)
<clever> 260-262 then append elements to the list, if certain conditions are met
radens has joined #osdev
<clever> configurePhase in setup.sh, then runs ./configure, with $configureFlags as flags
<radens> Hey, is there a good datasheet to cite in my code for the programming interface for the serial/COM/UART ports for x86 PCs?
<radens> Also, what's a good standard name for this interface? The OS Dev wiki has similar information on pages labelled COM, Serial, and UART.
<clever> heat: its also extensible, if you add cmake to your runtime deps, it will (by default) hijack configurePhase, and modify it to do `mkdir build ; cd build ; cmake ..` instead
<heat> is there any requirement for out-of-tree builds?
<heat> or can packages do as they please?
<clever> heat: for most packages, its just a dumb `./configure && make && make install`, but cmake by design allows for more out-of-tree stuff, so it defaults to using a build dir
<clever> the temp dir it builds in, gets deleted automatically once the build is done
<heat> also are there implicit dependencies on glibc/gcc libs/llvm libs/rust stuff?
<heat> or are you required to specify that
<clever> heat: gcc and glibc are in the default scope, but there are variants of the default env