<geist>
if nothing else because once your kernel gets serious enough, things like jtag dont really work right, sicne the context gets switched around, it fucks up timing of your int handlers, etc
<heat>
firmware people also use LEDs
<heat>
get it blinking through stages
<geist>
but i'm also extremely biased against jtag because i've seen entire companies fall prey to 'need jtag or can't work' and then they get themselves fucked up
<geist>
because they get so rliant on the tools that no one understands anything
<heat>
if on x86 you could also use port 0x80
<heat>
plus one of those fancy mobos that print hex codes
<geist>
more to the point, try *not* using jtag for a while. i bet you'll find it's not so bad
<geist>
and again, qemu. it's this really ppowerful tool right in front of you
<rpnx_>
hum... I mean. I could develop for qemu... but then what happens when there are differences between qemu and real hardware?
<rpnx_>
Can QEMU emulate a gpu, for example
<geist>
you're putting the cart in front of the horse here.
<geist>
jtag can't break into the gpu either
<geist>
look, when you're gettings tarted, almost everything you're going to futz with for some time is just pure ARM stuff
<geist>
things like getting the cpu into the mode you want, getting the C/rust/etc environment running, getting paging working, getting interrupts working
<geist>
that works exactly the same everywhere
<geist>
debug that where it's easiest to do it, emu
<geist>
qemu
<geist>
then once you are comfortable there, know that the shared things work
<rpnx_>
hum, ok. I just hope they aren't too dissimilar.
<geist>
try to get it booting on real hardware and write whatever drivers you need to achieve parity
<geist>
or you can go right in, but sounds like you're a bit worried it'll be too hard
<geist>
answer is probably not, but i have no idea what your skill level is
<geist>
of course nothing keeps you from just diving int the deep end, i'm just giving you my somewhat hard earned advice on how to approach these things. it's about biting off one piece at a time, so you can build on what you already have
<geist>
but it might not sound fun to you, so. <shrug>
<geist>
about 50% of the people at this point just yolo it anyway, so you wouldn't be the first
<geist>
but dont get soo fixated on your particular rpi board. it's just one of many, what you'll find is in the ARM world things are *far* more wild west. every board is different, different drivers, different bootloadres
<heat>
klange, what's that post-it about? :P
<geist>
given that there is a sea of boards, and every one is different, i already recommend qemu virt first because it's the simplest board with an emulator attached
<rpnx_>
I'll buy another board if there are suggestions
<rpnx_>
But I want to also try with real hardware
<heat>
you have one in your super powerful PC, -machine virt
<heat>
get something *working* first
<clever>
rpnx_: i can still help with rpi stuff in #raspberrypi if you want to keep using it
<heat>
the rpi is the most cursed fucking board out there
<heat>
get virt, then move on to real hardware
<heat>
it'll be substantially easier
<geist>
again it's not bad to want to get started on rpi. if so then just go directly to that if you want
<heat>
you will in fact kinda know what you're doing at that point
<geist>
it's just you're giving yourself a huge handicap
<rpnx_>
I was thinking of doing both virtual/qemu and real hardware at the same time.
<rpnx_>
That's why I was hoping for a board that matched qemu
<geist>
and if so, then go for it. make sure you have a serial uart hooked up, and the first thing you do in your asm is start printing characters
<geist>
rpnx_: you can't always get what you want
<geist>
but why does the qemu -M rasp3b not do what you want?
<clever>
geist: while we where debugging `qemu -M rasp3b` earlier, i discovered how -kernel foo.elf deviates from the rpi firmware in 3 different ways, but -kernel foo.bin doesnt
<geist>
anywya clever can help you
<geist>
then use foo.bin
<geist>
actually i'd generally assume you shouldn't use -kernel on that machine?
<rpnx_>
geist, I wasn't able to get DTB loaded in qemu for some reason when it's in rasp3b mode
<clever>
and yeah, we got stuck at that
<geist>
probably supposed to boot on the sd card and not use -kernel?
<geist>
ah, well, no idea here
<clever>
geist: nope, qemu cant read from the disk drive
<clever>
it expects a -kernel or -firmware
<geist>
oh, so it's really not a very good emulation then
<clever>
geist: the mailbox emulation is also broken on the latest linux kernel, so no graphics anymore
<geist>
then yep, if you are hell bent on running on raspberry pi, either do what we suggest (qemu virt first, then move to pi) or go in on pi
<rpnx_>
yes, I'm open to buying another board if there is one that qemu can emulate more closely
<rpnx_>
I don't really care if it's the PI or not
<clever>
rpnx_: i have some more ideas on how to debug the dtb on rpi, if you want to keep emulating rpi
<geist>
then what *do* you care? it's arm and is physical?
<klange>
< klange> -M rasbpi is barely emulating a raspberry pi to begin with!
<geist>
is it the physical part that matters to you?
<rpnx_>
yes, arm64 + physical
<clever>
klange: yep
<rpnx_>
But having qemu at the same time I think is also nice
<geist>
again you can't always get what you want
<heat>
get a server board? :P
<rpnx_>
I want to test things like power management and so on
<heat>
>power management
<heat>
that's so late game
<geist>
i mean i think we've pretty much laid out the red carpet thus far, and i can tell you this is a hell of a lot easier now than 5 or 10 or 15 or 20 years ago
<heat>
also, QEMU doesn't emulate that
<rpnx_>
then check how much power different ways of setting scheduler and etc use
<wxwisiasdf>
gotta save up those watts
<clever>
rpnx_: i would add some #ifdef RPI3 and #ifdef QEMU_VIRT, and then modify things like uart to work on both, so you can run under either machine in qemu
<geist>
i think you're getting way ahead of yourself. if you're just gettings tarted off, it'll be *years* until you're fiddling with that
<geist>
maybe a year if you're totally bad ass
<rpnx_>
clever, yes I think I will do that
<rpnx_>
It's that hard? hum
<heat>
at the end of the day, you're probably not messing around with that
<geist>
by then whatever board you pick will be so irrelevant becuase you'll know so much stuff you currently dont know you dont know
<geist>
like, this is sort of like buying a toyota prius now, worrying about the future when you modify it to race in the formula 1 races
<heat>
it took linux like what, 15 years to get decently working power management?
<geist>
like, uh, worry about learning how to drive now
<wxwisiasdf>
do not try to do power management
<heat>
you can try, but it requires a decent amount of $know_what_you_are_doing
<klange>
by the time I get around to power management, we'll have solved the energy crisis and batteries will last seven years on a single charge
<geist>
hell if nothing else you're going to need some pretty good equipment to do accurate power mesaurements that'll be many times the value of the board you're worrying about
<klange>
you know how i do power management? 'wfi'
<geist>
and anyway for the most part modern ARM and x86 scheduler centric (ie, many times a second) cpu power management is 'idle the cpu'
<geist>
it's a lot simpler now than 10-20 years ago
<geist>
there's more gross level power management, like when you put the whole systeminto different power states (on battery, screen is off, user hasn't touched the device) sort of things
<geist>
but that's very much system driven
<geist>
and that usually involves turning off or declocking lots of peripheral parts
<geist>
or parking blocks of dram
<heat>
actually modern linux power management requires a boat load of MSRs or ACPI, idling the cpu through hlt is the last idle method they use
<clever>
geist: youve done it again, now i'm wondering how i can adjust my dram refresh settings, to disable half the banks
<geist>
yah thinking more of the ARM side
<heat>
but that's besides the point
<geist>
on ARM there's PSCI (except raspberry pi which is highly nonstandard) to do a bunch of that for you
<geist>
but even that, i think it's fairly rare to want to park a core. that's more of a system reboot or shutdown case
<clever>
geist: ive also read one of the arm docs, the arm cluster has a status signal, to signal that every core is in wfi/wfe state, then the implementor can use that to just turn the arm clocks off entirely
<geist>
yep. also the code that's doing that needs to flush caches, power clock them, etc
<geist>
that's what PSCI is supposed to do
<clever>
and you may then write to some control register, to signal what to do the next time that is true
<clever>
and PSCI is meant to hide all of that i believe
<heat>
geist, what happens to PSCI when ACPI is in the picture?
<geist>
right
<clever>
but the rpi is bending the rules, and lacks PSCI
<geist>
PSCI is the way to do it, ACPI on ARM i dont think takes on any power management roles
<geist>
it's more of a simpler configuration role
<geist>
the ACPI spec talks about a simpler ACPI format, for non x86s basically
<heat>
"but the rpi is bending the rules" is the clever's motto
_saltd has joined #osdev
<heat>
oh yeah re: device tree on ACPI
<heat>
what exactly are you doing to get it?
<geist>
you dont
_saltd has quit [Remote host closed the connection]
<heat>
so how does fuchsia have it?
<geist>
you get one or the other, period.
<geist>
even the the ARM SBSI says that. if EFI/ACPI is present, no device tree
<geist>
it was sobering to discover that, but alas that's how it is. probably MSFT had something to do with it
<heat>
so fuchsia isn't using dt currently?
<geist>
i think there *is* a way to get device tree from EFI, but it's nonstadard. basically you ask for a table by a specific table
<geist>
correct
<clever>
geist: remember that ugly pcie controller in the bcm2711 and other things? acpi doesnt have a magic# to describe it
<geist>
well, wait. no. i mean on machines where DT is present we use that
<geist>
on machines where UEFI is present we use ACPI
<geist>
which is what you're supposed to do
<clever>
geist: so the pi4 uefi firmware, just claims the pcie xhci is a platform device, and there is no pcie bus
<heat>
yes, I think I've read somewhere in the edk2-devel ML that some platform was passing both tables
<geist>
yes iirc the unoffocial way to get it is via a known GUID
<clever>
geist: but even that isnt correct, because acpi doesnt have a guid to describe an xhci that is non-coherent
<heat>
isn't fuchsia supposed to only boot in EFI? or did that change?
<geist>
no, that's 100% incorrect
<clever>
so windows still needs custom patches, to deal with cache management
<heat>
hrm
<geist>
like precisely the opposite. fuchsia boots in whatever it can. we aint callin no shots there
<clever>
i'm just going off memory from what i read in a discord log, there could be mistakes at either level
<rpnx_>
ok, would uart work similarly in virt and rpi?
<geist>
but in general hoping that we can stadardize on less boot methods over time
<geist>
rpnx_: no. different uarts
<rpnx_>
I am not sure how to get uart working in virt
<wxwisiasdf>
UEFI :>
<geist>
more specifically, raspberry pis uart is different than all other uarts
<clever>
geist: i thought virt was PL011?
<geist>
it is
<clever>
geist: and the rpi has one PL011
<geist>
the console one?
<clever>
depends on config
<geist>
i thought that was some stripped out mega stupid one
<clever>
the PL011 is better, and is routed to BT by default
<heat>
wxwisiasdf, waz the problem
<geist>
then answer rpnx_
<clever>
but config.txt can swap the uarts
<rpnx_>
Is there any information anywhere about how to get uart/serial output working on qemu-virt?
<clever>
rpnx_: what address are you using to reference the uart?
<clever>
rpnx_: when i ran your .elf in qemu earlier, it showed up in the serial0 tab in the GUI
<heat>
rpnx_, pl011
<heat>
take the address from qemu's sauce
<heat>
(linked before)
<geist>
heat: basically on PC (x86) we mostly boot via UEFI (using a secondary loader called gigaboot), but can boot either via a multiboot shim (qemu, or grub) or via a direct boot (some chromebooks)
<heat>
woah, direct boot?
<geist>
on ARM we boot mostly via either direct uboot or via a shim or via gigaboot (uefi), depending on what we're booting on
<bslsk05>
github.com: qemu/virt.c at master · qemu/qemu · GitHub
<geist>
annoying to be honest. i dont know why the chromebooks are so nonstandard, but the solyutio was to just get ZBI/fuchsia support added to the chromebook firmware, because we're google
<clever>
rpnx_: i believe those are 64bit addresses, but they are stored as a pair of 32bit ints
<heat>
geist, chromebooks boot with coreboot, coreboot is AFAIK kinda designed to boot linux or another payload
<geist>
that's base + length
<geist>
yep, fuchsia
<clever>
geist: ah, thanks for the correction
<clever>
that does make sense, on a second look
<heat>
at the end of the day the chromebooks could just boot EDK2, and EDK2 would boot fuchsia
<geist>
yeah
<heat>
that would make more sense IMO, but wtv
<geist>
yah same. i dunno what the raitionale is behond coreboot on those but whatever
<heat>
(because EDK2 is already a popular coreboot payload)
<geist>
probably breaks secure boot
<heat>
no
<clever>
rpnx_: so on -M virt, the PL011 is at 0x0900_0000, on rpi the PL011 is at 0x??20_1000, with ?? being 20 on rpi0/1, 3f on pi2/pi3
<geist>
(really i thin kthat's the main reason coreboot exists on chromebooks, secure boot)
<clever>
in every one of those cases, the PL011 itself, functions the same way, and can use one driver
<heat>
geist, I think google really likes coreboot because it's conceptually a lot simpler than tianocore
<heat>
it's also implicitly a lot more open, GPLv2 and all
<geist>
yes. i think it's a combination of the desire to do a 'good' secure boot, keeping it much simpler than EFI, decision was made 10+ years ago, and NIH
<geist>
but honestly i dont know, i do not officially speak for anyone
<geist>
i am certainly in no position to throw stones re: bootloaders, since i'm indirectly responsible for billions of them
<heat>
the big advantage of EFI is that it's super easy to integrate proprietary shit in a structured way
<heat>
:D
<rpnx_>
clever, setting "UART_BASE" to that didn't make the prints work, so I am guessing that they do not work the same way?
<rpnx_>
I think RPI used "mailbox" for uart
<geist>
rpnx_: this is where you're about to slam headfirst into 'osdev is not easy'
<clever>
rpnx_: can you pastebin the code that is using uart_base?
<clever>
rpnx_: and what value was uart_base before you changed it?
<geist>
anyway, gonna stop for the afternoon. ciao eveyrone!
<geist>
need to get a few thngs done
<heat>
byeeeeeee
<rpnx_>
UART0_BASE = 0x09000000,
<wxwisiasdf>
ciao
<clever>
rpnx_: there is no mailbox uart, but the mailbox can be used for the uart reference clock, thats a seperate factor, that ive been ignoring because qemu ignores baud
<heat>
oh yeah also have you looked at linuxboot?
<heat>
they're replacing a bunch of firmware with linux
Burgundy has joined #osdev
<heat>
apparently deployed by google in some servers
<clever>
klange: i believe qemu's PL011 properly does flow control across the border, if you stop reading the rx fifo, then qemu will stop reading whatever FD is plumbed to the uart, and thats where playing with rx irq's can become handy
<heat>
I need to find a better way to get a livecd
<heat>
regular builds have actual livecds but they don't have 2GB worth of packages to load
<wxwisiasdf>
also lol, the bottom of the readme
<heat>
:v
<heat>
klange, how are you doing live cds? do you just download packages from the interwebz
<heat>
for the few I know you had/have
<klange>
my CDs only come with first-party stuff
<klange>
everything else is from the repos
<klange>
well, first party stuff plus Deja Vu and libgcc.so
<heat>
yeah
<heat>
so downloaded?
<klange>
yeah, you want doom you get it from the interwebz
<heat>
how about ssl?
<klange>
interwebz, default repo link is unencrypted
<heat>
(and ca-certificates)
<heat>
oh no, http D:
<klange>
and yes I realize the bootstrap problem of getting the ca-certificates package and mbedTLS from an unencrypted source
<klange>
I'll do package signing with an algo I can spit out myself eventually and then it won't be so much of a problem
<heat>
my issue is that some packages are clearly too large (python, binutils, gcc, future clang, etc)
<heat>
i'm thinking about making my livecds have the base system + some basic packages (including openssl and ca-certificates)
<klange>
yeah, binutils+gcc take a bit of time to download and decompress with my babby's-first-DEFLATE
<heat>
:D
Burgundy has quit [Ping timeout: 260 seconds]
<klange>
~11s total, 7 of which is gcc, 6 of which is just the decompression
<wxwisiasdf>
add qemu :>
<heat>
great idea
<wxwisiasdf>
i rarely see qemu ported around
<heat>
are you tired enough of Onyx? use qemu!
<wxwisiasdf>
without kvm obv
<klange>
sortie has a qemu port
<heat>
yeah
<heat>
I suspect it isn't that hard
<heat>
(hopefully)
<klange>
I apparently have a ticket for it... it's a user request, so take it with a grain of salt, but it's technically there!
<wxwisiasdf>
what about mupen or dolphin
<wxwisiasdf>
which is basically mesa + qt/gtk
<heat>
fuck are those
<rpnx_>
ok, I am going to write a generic uart driver and then have a different address to pass to it for each board I think
<heat>
>mesa
<heat>
shit
<wxwisiasdf>
sorry
<wxwisiasdf>
i forgot mesa is shitfuck
<heat>
mesa is great but also shitfuck to port because most of it requires DRI
<heat>
and/or LLVM
<klange>
just port an old version with the original swrast
<heat>
and its crucial enough that I'd like to have it(or my own vulkan impl) in the base system
<klange>
i miss my swrast mesa port :(
<heat>
nah, that's cheating
<klange>
'twas a different era
<wxwisiasdf>
>own vulkan impl
<wxwisiasdf>
o_o
<heat>
also llvmpipe is way more performant
<heat>
there's a new patch upstream(or on the ml) for AVX-512 llvmpipe
<wxwisiasdf>
i once tried to rollout my own opengl impl
<heat>
vulkan is drastically simpler
<wxwisiasdf>
i got relatively far,..... in opengl 1.0
<heat>
fuchsia did it
<klange>
you know what else i miss, having a progress bar widget on the package installer instead of just running the command-line utility in a terminal... I should do that...
<wxwisiasdf>
i want to integrate opengl in my kernel
<heat>
say what
xenos1984 has joined #osdev
<wxwisiasdf>
but then i would make a gazillion syscalls
<heat>
thats a
<heat>
horrible
<heat>
idea
<heat>
like *****horrible*****
<clever>
heat: no worse then the broadcom opengl stack, lol
<clever>
where every gl function is an RPC to another cpu cluster!
<heat>
all the complexities of opengl put together with the niceness of kernel development
<wxwisiasdf>
yes
<heat>
plus a syscall per call
<wxwisiasdf>
its horrible, ugly, terrible - but for le meme
<heat>
you're really into cursed shit
<wxwisiasdf>
i also need a 3D widget for my UI shit
<heat>
have you seen: rpi, uefi, acpi
<wxwisiasdf>
yes
<clever>
wxwisiasdf: might i recommend you re-implement the broadcom gl? :D
<wxwisiasdf>
clever: does it has open specs
<clever>
wxwisiasdf: it has source for the client side, ....
<wxwisiasdf>
how cursed it is
<clever>
gl, audio, general commands, shared memory, all going over one protocol
<wxwisiasdf>
interesting, i might look into it
<clever>
codecs are also there, h264 accelerated encode/decode, and camera stuff
<clever>
vchiq is the codename for it
<clever>
video core host interface
<wxwisiasdf>
mmmm
<klange>
man, looking at these pre-NIH ToaruOS builds and being nostalgic about what I had back then... that Cairo-backed compositor was so much faster, pixman runs circles around me...
<bslsk05>
github.com: linux/bcm2835-v4l2-isp.c at rpi-5.15.y · raspberrypi/linux · GitHub
<geist>
klange: hmm, reminds me, how much trouble did you have with an atapi driver?
<clever>
wxwisiasdf: this opens an mmal session (a protocol tunneled within vchiq), and then a compoent (ril.isp, another protocol tunneled within mmal, within vchiq)
<geist>
haven't actually tried, except actually detect that there's a cd attached to SATA
<klange>
geist: I don't trust my atapi driver far enough to throw the CDs
<wxwisiasdf>
oh
<clever>
wxwisiasdf: that part, is all kernel-side, to talk to the videocore, but vchiq is also available to userland, https://github.com/raspberrypi/userland contains those libs
<bslsk05>
raspberrypi/userland - Source code for ARM side libraries for interfacing to Raspberry Pi GPU. (1088 forks/1957 stargazers/BSD-3-Clause)
<clever>
somewhere in userland, is the opengl libraries
<klange>
I don't even use it regularly, I use `-M q35` and have no support for the drives it gives me, so I only see the atapi CD drive when I spin up a VM from scratch or use VirtualBox.
<wxwisiasdf>
clever: so if i (had the skills/willingness) to implement qualcomm, wouldn't i be basically just emulating a Rpi GPU? :D
<geist>
klange: oh you said something about the cd installation and whatnot
<clever>
wxwisiasdf: the ultimate in cursed drivers!
* kof123
throws in ancient chromium multiheaded x over opengl some such for display walls (many monitors)...some portion was written in python IIRC. put that in your kernel
<bslsk05>
kuroko-lang/kuroko-efi - Kuroko built to run on UEFI. (0 forks/6 stargazers/MIT)
<geist>
ah CPPC it seems maybe how linux knows to treat the E cores
<zid>
is that new? last time I tried I ejected the cd and it hard locked
<geist>
on this alder lake it seems to assign the P cores the highest performance # of 61 or 63, and the E cores 36
<klange>
zid: toaru? always been running off a ramdisk, but will still attempt to talk to an ATAPI drive by default; my be hardlocking because an of unexpected interrupt, or something tried the ejected disk elsewhere and I have no error handler
<geist>
as read by /sys/devices/system/cpu/cpu*/acpi_cppc/highest_perf
isaacwoods has quit [Ping timeout: 264 seconds]
<heat>
geist, CPPC?
<geist>
acpi cppc
<geist>
gotta go, but it's some way of describing differences in cpus
<geist>
though it looks like it might be part of the DSDT so it's not going to be trivially parsable from the kernel
<heat>
ah
<geist>
loos like you call a _CPC method on some thing and it returns a structure describing a bunch of crap
<heat>
praise be the bytecode
zaquest has quit [Remote host closed the connection]
zaquest has joined #osdev
smach has quit [Ping timeout: 260 seconds]
lentement has joined #osdev
rpnx_ has quit [Ping timeout: 260 seconds]
mykernel has quit [Quit: leaving]
wxwisiasdf has quit [Ping timeout: 264 seconds]
lentement has quit []
lentement has joined #osdev
lentement has quit [Remote host closed the connection]
rpnx_ has joined #osdev
<rpnx_>
What are the steps to set up uart on a non-raspberry pi device?
<rpnx_>
& can the size of uart registers be different depending on the machine?
<clever>
rpnx_: for -M virt, nothing actually has to be setup, you can just write to the data-reg immediately
<clever>
rpnx_: the PL011 uart is a standard hw block, and has registers at the same offsets and always behaves the same way
<rpnx_>
ok, let me delete all the other init code
wxwisiasdf has joined #osdev
<clever>
id say comment it out, so you can bring it back later when you want to support real hw
<rpnx_>
that worked, interesting
<clever>
rpnx_: thats because qemu doesnt emulate the "serial" side of a serial port, it just takes the bytes from the DR reg, and prints them
<rpnx_>
well, in this case, I am just implementing std::puts with uart for qemu-virt, so when doing real hardware, the entire qemu-virt folder will be missing, should be easy to tell what needs to be reimplemented :)
<clever>
rpnx_: one option is to have a dedicated PL011 driver, that works on everything with a PL011
<clever>
move the mailbox stuff outside of the PL011 driver
<clever>
that driver should accept 2 mail inputs, the base addr of the uart, and the reference clock
<clever>
2 main inputs*
<rpnx_>
uart is asynchronous right?
<clever>
yeah
<clever>
the PL011 uart takes a reference clock in, lets say 1mhz for example
<clever>
the baud rate divisor (IBRD and FBRD regs i think), then says how many cycles to count, for each uart bit
<clever>
but its also x16 over-sampled
<clever>
so at 115200 baud, the PL011 will read the rx line 115200*16 times per second
<clever>
which is ~1.8 mhz, so ignore that 1mhz example
<clever>
lets say the reference clock was 54mhz, 54000000 / 115200 / 16, so the divisor is ~29.3
<clever>
rpnx_: so, internally, the PL011 will count out ~29 cycles of the ref-clock, then sample the RX pin, and repeat
<clever>
and after 16 samples, it generates 1 bit on the input fifo
<clever>
your PL011 driver needs to take that 54mhz refclk, and the 115200 baud goal, compute a divisor of ~29.3, and configure the PL011
<clever>
the mailbox on the rpi, is to query/set the refclk
<clever>
on other systems, the device-tree tells you how
<clever>
rpnx_: this part of the dts, says to grab the "clocks" device described elsewhere, and use the driver-specific api to get the "BCM2835_CLOCK_UART" clock
<clever>
86 clocks: cprman@7e101000 {
<clever>
87 compatible = "brcm,bcm2835-cprman";
<clever>
rpnx_: and this describes that clocks device, and says what driver to use, to translate BCM2835_CLOCK_UART into a hw clock
<rpnx_>
interesting..
<rpnx_>
so there are specific names for particular drivers they expect you to know about?
<clever>
the compatible= is a list of drivers that a given piece of hardware is compatible with
<rpnx_>
or is dtb just something linux came up with
<bslsk05>
nibblestew.blogspot.com: Nibble Stew: Questions to ask a prospective employer during a job interview
lentement has joined #osdev
<rpnx>
This hits home so hard at my current employer :)
<rpnx>
The questions I should have asked but didn't.
frkzoid has joined #osdev
frkzoid is now known as freakazoid333
<clever>
rpnx: heh, just 2 questions in, and i'm already seeing issues with my dads work laptop, full disk encryption, no admin access, windows, and it can lock him out if the pw expires
<clever>
and how he intentionally kept it off the internet when they fired him, so the laptop wouldnt know and brick itself before he could get files out
lentement has quit [Remote host closed the connection]
<clever>
incase your wondering, covid, lack of work, downsizing
<heat>
best question: monie how much
<clever>
my dad had been with the company for 30+ years
<rpnx>
I mean, one should not keep personal files on a work laptop
lentement has joined #osdev
<clever>
and half way thru a yearly performance review, they go "oh yeah, your fired"
lentement has quit [Remote host closed the connection]
<rpnx>
My company has most of these issues
<clever>
rpnx: yeah, it probably should have been cleaner
<rpnx>
fortunately I have an interview tuesday :)
<selene_>
good luck lol
<rpnx>
we'll see what'll happen
solenya has joined #osdev
<heat>
some of the questions look stupid though
<rpnx>
in a way where I work now is worse than microsoft was
<rpnx>
heat: which ones?
<heat>
"Are developers free to choose and install the operating system on their development machines?"
<heat>
"Could you explain the exact steps needed to get the code built?"
<clever>
heat: where i work, yeah, i can install any os i want
<heat>
particularly "Does this team or any of the related teams have a person who actively rejects proposals to improve the code base?"
<rpnx>
No, those are good questions.
<heat>
are they?
<selene_>
they sound rational enough to me
<rpnx>
When I worked at microsoft..
<heat>
hey mr interviewer, does my team have an asshole?
<rpnx>
the build was... not straightforward
<heat>
hey mr interviewer, can you take me through the steps to build the project?
<rpnx>
we had some weird thing, I forget the name of, to download and build stuff
<heat>
wtf kind of question is that?
<selene_>
i don't see what's wrong with asking that
<rpnx>
Hopefully you ask these questions to the hiring manager?
<heat>
also being free to install your own OS isn't particularly an advantage
<selene_>
why not?
<heat>
in fact, probably a disadvantage - means that your IT isn't up to speed with what you're doing with their shit
<heat>
i could install debian from 2004 and they'd be none the wiser
* klange
installs ToaruOS on dashboard box
<rpnx>
heh. i had to use windows here for almost a year until I convinced them to get me a mac, but they still wont let use linux on work desktop
<rpnx>
which is annoying because
<selene_>
gross
<clever>
heat: thats why we also use the nix package manager, its kinda like docker, and smooths out any such differences
<rpnx>
Our product only runs on red hat 7 and red hat 8
<rpnx>
So this is a *linux only* app
<heat>
that is indeed stupid
<rpnx>
And we are not allowed to use Linux workstations.
<heat>
vm?
<rpnx>
Instead, putty into server with x11 forwarding over vpn
<rpnx>
or use clion remote mode
<selene_>
wtf
<selene_>
that sonuds needlessly complicated
<rpnx>
or jetbrains gateway into server
<rpnx>
The server also has no internet
<rpnx>
because "security"
solenya has quit [Ping timeout: 244 seconds]
<heat>
who are you working for?
<heat>
just so I don't end up working for them :P
<rpnx>
Intercontinental Exchange
<heat>
oh, finance?
<rpnx>
ya
<rpnx>
But microsoft had way better security and apart from the insanely bad CI the security was much less annoying.
<bslsk05>
github.com: edk2-platforms/Inode.c at master · tianocore/edk2-platforms · GitHub
<heat>
musl code is a fucking piece of cake compared to this
<heat>
and no I didn't write this
<rpnx>
Uggh
<rpnx>
In out
<heat>
yes
<heat>
this is uefi
<rpnx>
reminds me of working for microsoft
<heat>
I don't actually dislike it
<heat>
this is very much microsoft-style code
vdamewood has joined #osdev
<heat>
if you ever get to EFI or ACPI you'll see it again
<heat>
my big problem with it really is the CONST and STATIC, ...
<rpnx>
that's not the assembly we can reimplement it right?
<heat>
hm?
<rpnx>
in/out were just
<rpnx>
#define IN
<rpnx>
#define OUT
<heat>
yes
<rpnx>
used by some static analysis tool
<rpnx>
I hated that thing
<rpnx>
fxcop?
<rpnx>
Or something like that
<rpnx>
If you tried using modern C/C++ it would flag you all the time
<heat>
fxcop seems to be for .NET
<rpnx>
nah they had it working in C++ code for the hypervisor in azure
<rpnx>
oh it may have been .net C++
<heat>
probably
<rpnx>
that could explain some weirdness
<heat>
C++/CLI or whatever that's called?
<rpnx>
Yeah
<rpnx>
Hopefully I can go work for Google
<rpnx>
I heard it's better
<rpnx>
People say they have good build systems.
<rpnx>
They also made ninja
selene_ has quit [Quit: WeeChat 3.6]
selene_ has joined #osdev
selene_ has quit [Client Quit]
selene_ has joined #osdev
<heat>
they didn't make ninja
<rpnx>
They didn't?
<heat>
someone at google made ninja
<heat>
but not as google
<rpnx>
well... heh
<heat>
i think
<heat>
they did however make three ninja generators
<heat>
gn, gyp, kati
<rpnx>
My workplace uses a bunch of open source code
<heat>
also b a z e l
<rpnx>
but does not allow us to contribute back
<heat>
and android blueprints
<rpnx>
Even when I fix bugs in open source packages
<rpnx>
we don't upstream it
<rpnx>
we aren't *allowed* to upstream it
<heat>
whats the reasoning?
<rpnx>
I don't understand it tbh... but another example
<rpnx>
they wanted me to submit conflict of interest approval forms for my crappy open source code on gitlab
<rpnx>
Apparently. I think in order to submit a message using our work email to a mailing list, we would have to get approval from HR
carbonfiber has quit [Quit: Connection closed for inactivity]
<rpnx>
and that if I wanted to submit fixes to OSS projects I'd have to do it on my personal time.
<rpnx>
even if it would literally save us time because then we wouldn't need to keep patching stuff
<rpnx>
I think the answer is that they don't care.
<rpnx>
They want people to keep a low profile to keep attention away from the company i think.
selene_ has quit [Quit: WeeChat 3.6]
<rpnx>
same reason our badges don't show the company
<rpnx>
just our name and photo
selene_ has joined #osdev
<rpnx>
and the building is unbranded
<rpnx>
it's a little weird
<clever>
rpnx: i have seen one guy posting on the rpi forums, explaining how to patch some OSS software, but providing no patch file
<clever>
because his employeer owns the rights to any code he writes, even to oss projects
<clever>
so he cant contribute to an oss project, it would somehow be a copyright violation, lol
<rpnx>
not sure that is legally enforceable but who wants to fight that battle while employed? lol
<clever>
yep
selene_ has quit [Client Quit]
<rpnx>
Work for hire is defined in copyright act so I'm pretty sure you'd be able to win against those claims
<rpnx>
but then what? they fire you.
<CompanionCube>
clever: also wow a mobo serial module is cheaper than i thought
<CompanionCube>
maybe i might go for that after all...
<rpnx>
But it could be a strategy to just keep contributing without telling them, hope they don't notice, and only litigate it if they notice.
<clever>
code-wise, the only thing my current employer really limits, is a non-compete clause in my contract
<clever>
if i should ever quit or be fired, i cant work in a similar field for X months
<clever>
i think thats more so i dont jump-ship to a competitor
<rpnx>
Fortunately my company doesn't do that they just say basically I cannot write finance software in my free time
<rpnx>
which is super boring so I'm okay with that
selene_ has joined #osdev
selene_ has quit [Client Quit]
selene_ has joined #osdev
<rpnx>
clever: I think such things should be illegal but unfortunately the courts are very lax about antitrust enforcement
heat has quit [Ping timeout: 240 seconds]
<rpnx>
instead the government is going after stupid s***
<rpnx>
like play store
<rpnx>
really?
selene_ has quit [Client Quit]
<rpnx>
you can sideload.
<clever>
rpnx: apple cant
selene_ has joined #osdev
<rpnx>
Well I agree with going after Apple tbh
<rpnx>
Google Play I do not understand as much
<rpnx>
Clearly one is quite different than the other?
<rpnx>
Android is open source too
<rpnx>
mostly anyway
<clever>
rpnx: ive also seen a few videos about how crazy apple is with in-app purchases
<clever>
apple wants to take a cut on every purchase
<rpnx>
Apple should allow competing app stores.
<clever>
but your not allowed to charge less on your website, and link the site from the app
<rpnx>
Google doesn't prevent that
<clever>
so if your profit is razor thin, and apple takes a cut, you loose money
<rpnx>
I have one on my phone right now
<rpnx>
F Droid
<clever>
yep, i also have f-droid
<clever>
f-droid is also special, in that it only hosts open source software
<clever>
which is nice, ive found an app on there before, and wanted to know HOW it worked, boom, theres the source
<rpnx>
I am a little annoyed that I can't uninstall Google apps I don't use though
<rpnx>
But the bootloader is unlockable.
<clever>
i did also run into that recent
<rpnx>
so I could... just not because DRM video.
<clever>
google hangouts has been deprecated, replaced by google chat
<clever>
but, hangouts is installed on the /system partition by the OEM
<clever>
so i can never uninstall it
<CompanionCube>
termux is special too
<rpnx>
Google apps installed on 64GiB emmc
<rpnx>
no sdcard slot
<clever>
but, there is another nail in the coffin for my phone...
<rpnx>
it can be annoying in that regard
<clever>
slack updated, and is no longer compatible with my android version
<clever>
the play store now lies, and claims there is no (compatible) slack app
<rpnx>
I would mind less if the phones had e.g. 2TiB emmc
<clever>
the already installed copy crashes when attempting to login
<rpnx>
or sdcard slot
<rpnx>
but 64GiB?
<clever>
my phone does have a uSD card slot
<clever>
samsung S5 neo
<CompanionCube>
rpnx: you know what's annoying
<rpnx>
Hum?
<CompanionCube>
advertising 8g storage but like 3ishg of it is taken up by the system
<CompanionCube>
fuck you very much
<rpnx>
Haha,
<rpnx>
yeah...
<clever>
let me check my phone...
<rpnx>
TBH I am debating getting an iphone at this point. Android keeps breaking.
<rpnx>
my main phone got stuck in emergency firmware download mode
<rpnx>
all it did was reboot
<rpnx>
still bricked, using an older one now
<clever>
my samsung has never really gotten bricked or anything like that
<rpnx>
I looked into getting my data off
<clever>
the only weirdness ive really had at the hardware level, is the charging status is a bit wonky
<clever>
sometimes it will claim to be charging when nothing is connected
<rpnx>
but I couldn't figure how to get the UFS key off the titan M chip
<clever>
once, it refused to claim charging, despite being plugged in, and the battery% didnt update
<rpnx>
even though I knew the pin
<clever>
after a full reboot, the battery was suddenly more charged then before
<clever>
so it was just a software issue, not updating the UI
<wxwisiasdf>
to ACPICA or to not ACPICA
<clever>
ok, the du scan has finished, my phone claims to have 11gig of storage
<clever>
the /system is using 971mb, my media 844mb, and my apps 8.75gig
<clever>
with 522mb of free space
<rpnx>
Trying to recover my data did teach me how secure the pixel is though... I just wish there was a tool to pull the ufs key off the titan M chip if you know the pin code
<rpnx>
Because.. well
<rpnx>
the phone wont boot to the recovery
<CompanionCube>
clever: my tablet is often at 0.0k free for complex and also lazy reasons.
<clever>
rpnx: that reminds me, my dads cellphone, due to a company policy script, auto-encrypts any SD card you insert into it, without any confirmation
<rpnx>
That sounds a bit hazardous
<CompanionCube>
works well...most of the time, lol
<clever>
rpnx: and one day, it wound up eating a zip file with some recovery things for rooting
<clever>
rpnx: so, by accident, i wound up running the wrong recovery, didnt read a confirmation, and recovery factory-reset things for me, lol
<rpnx>
I wish I could factory reset my phone
<clever>
rpnx: but, i had previously used root to image every partition, so i re-rooted it, copied the images back, and it somehow even restored the SD card encryption keys
<rpnx>
it wont load the bootloader
<clever>
which implies that the encryption keys are in the backup i had made
<rpnx>
I am using one with a cracked screen because I pre ordered a 5G phone
selene_ has quit [Quit: WeeChat 3.6]
<rpnx>
so no point buying a new one while waiting for that to ship
selene_ has joined #osdev
the_lanetly_052_ has joined #osdev
<CompanionCube>
if i could backup my data and wipe some things i very much would, but you can't root or unlock without wiping
<clever>
rpnx: i still need to decide what to buy as a replacement
<rpnx>
Google phones bricking themselves is a regular occurrence for me.
<clever>
CompanionCube: there was an exploit ive used before in android, to root without wiping
<rpnx>
This one is by another manufacturer
<clever>
probably long since patched
<CompanionCube>
and i have apps that stupidly disabled 'adb backup' which sucks hard.
<rpnx>
hopefully it will not brick itself
<CompanionCube>
clever: which android
<clever>
CompanionCube: whatever came on the original kindle fire 7"
<clever>
basically, when you use adb to restore a backup, it sets the chmod on a dir, before restoring the contents of the dir
selene_ has quit [Client Quit]
<clever>
so, you create a fake backup, with a 777 directory, 9999 dummy files to slow it down, and a 10000'th file with a known name
<rpnx>
all my phones... died. Or became almost unusable. The astro has been my only real voluntary upgrade... for 5G. Still waiting for it.
<clever>
then, in a non-root `adb shell`, you setup an infinite loop, to symlink that 10000th file to something in /etc/
<clever>
CompanionCube: the restore thread, then makes the dir 777 as the backup said, spins for a bit, then opens the 10000th file, and writes the contents out
<clever>
blindly following the symlink :P
<CompanionCube>
>2011
<CompanionCube>
lol, way too far back
<clever>
that symlink overwrites a config file, that tells android its running under a VM
selene_ has joined #osdev
<clever>
and since its under a VM, there is no point in restricting `adb root`
<clever>
but, that also breaks hw accel
<clever>
so, you have to use that to make a setuid root `su` binary, then undo it
selene_ has quit [Client Quit]
<CompanionCube>
clever: once i had a sketchy tablet which didn't even block 'adb root' but i rooted it like normal
selene_ has joined #osdev
<clever>
CompanionCube: oh, my biggest complaint with android, is linux namespaces
<rpnx>
I want to root the walmart tablet and port some linux distro to it eventually.
<clever>
i had some apps, that would show the top cpu users, in the notification tray
<rpnx>
imagine how popular that could be.. heh
<clever>
pid namespaces say no to that :P
<clever>
an app can only ever see itself now
<clever>
but `adb shell` doesnt follow that rule
<clever>
so if i want to see why the phone is slow, i have to plug it into another computer
<CompanionCube>
rpnx: postmarketos?
<CompanionCube>
clever: luckily i am 5.1.1 on the tablet so i just flipped 'show cpu usage' to true.
<clever>
CompanionCube: in the debug options?
<CompanionCube>
when using later android i found myself missing the feature.
<CompanionCube>
clever: yes, works up until android 7 i think/
<clever>
yeah, i have it on my phone, but its missing on my tablet
<clever>
what version do i even have...
<clever>
there it is, android 7 on my phone
<CompanionCube>
i wouldn't mind a newer android on the tablet, but my data lol
<bslsk05>
easyperf.net: Performance Benefits of Using Huge Pages for Code. | Easyperf
<mjg_>
you know, articles like this are kind of sad
<mjg_>
the real impact would be higher if the more bottlenecky lolwtf problems got fixed first
<mjg_>
or to put it differently, the article underestimates impact of using huge pages, provided the code is not doing things which are outright stupid
woky_ is now known as woky
solenya has joined #osdev
lentement has joined #osdev
lentement has quit [Remote host closed the connection]
lentement has joined #osdev
lentement has quit [Remote host closed the connection]
<mykernel>
trying to jump to userspace but i am getting page fault on addr fffffffffffffff8
troseman has quit [Read error: Connection reset by peer]
rpnx_ has joined #osdev
<rpnx_>
has anyone attempted implementing stuff like __cxa_begin_catch?
troseman has joined #osdev
heat has joined #osdev
<heat>
hello professional operating system developers
<heat>
kqueue vs epoll??????
<heat>
srsly though
<heat>
I found someone arguing that epoll is better because it uses fds which work like handles (so like WaitOnMultipleObjects, etc) and that kqueue is technical debt
<zid>
select pls
<heat>
OTOH, certain events will probably never be represented as handles, like futexes
<heat>
and waiting on a file OR a futex sounds super useful
<heat>
(even though no BSD seems to have that)
<heat>
zid, the world isnt ready for >= 1024-fd select
<rpnx_>
I think both are annoying to work with but kqueue is probably better
<heat>
we could cover the whole fd range with 256MB bitmaps
<heat>
rpnx_, why annoying?
<heat>
i'm all ears
<mykernel>
hey heat, can you help me? i get this page fault on the first instruction in usermode (https://pastebin.com/VqfjWkkv)
[itchyjunk] has quit [Read error: Connection reset by peer]
<rpnx_>
Macros are annoying
<heat>
mykernel, i'm assuming everything is mapped correctly?
<heat>
i.e user permissions
<heat>
... why are you at 3GB in x86_64?
<mykernel>
i just initialized blank tss (but other things should be ok)
<mykernel>
higher half?
<zid>
that's nowhere near half
<zid>
half is at 128TB
<heat>
that would be higher half in x86-32
<heat>
in x86-64 you generally place your kernel at -2GB
<zid>
(128TB + 128TB -2B)
<heat>
so 0xffffffff80000000
<zid>
GB*
<heat>
mykernel, anyway, that explains things
<heat>
blank tss = rsp of 0 -> tries to push something -> fffffffffffffff8 -> page fault
<heat>
tries to push something because of an interrupt I assume
<zid>
yea I knew rsp was fucked but I couldn't remember where rsp came from
<mykernel>
idk how to setup tss because all the info i could find is about 32bit tss, not about 64bit tss which has entirely different structure
<heat>
intel SDM and chill
<zid>
there's always the manual?
<heat>
or amd's manual which is more readable
<zid>
but, the only thing you need to do for long mode tss is fill out rsp0
<zid>
with a valid stack pointer you can use to take IRQs in
<heat>
hey baeeeeee come home intel SDM and chill
<heat>
lets discuss x86 architectural details all night
<heat>
all you weirdos sniffing bike seats and drawing layouts upside down
<zid>
but 0 is still at the bottom
<zid>
err below the base*
<zid>
or if 0 is at the top
<zid>
Like I said, it's normal, the taxo in your brain have just eaten a connection or two. This has deregulated your limbic system and made you angry at reality.
xenos1984 has quit [Read error: Connection reset by peer]
freakazoid333 has quit [Ping timeout: 244 seconds]
xenos1984 has joined #osdev
<wxwisias1f>
hmm
SpikeHeron has quit [Ping timeout: 268 seconds]
eck has quit [Quit: PIRCH98:WIN 95/98/WIN NT:1.0 (build 1.0.1.1190)]
eck has joined #osdev
frkzoid has joined #osdev
wootehfoot has quit [Read error: Connection reset by peer]
Lumia has quit [Ping timeout: 244 seconds]
epony has quit [Ping timeout: 252 seconds]
epony has joined #osdev
Gooberpatrol_66 has quit [Remote host closed the connection]
Gooberpatrol_66 has joined #osdev
SpikeHeron has joined #osdev
Gooberpatrol66 has joined #osdev
<moon-child>
zid: 1 is just control a, ez
Gooberpatrol_66 has quit [Ping timeout: 244 seconds]
[itchyjunk] has joined #osdev
<zid>
if that wasn't bound to "select all" instead maybe, smh windows keybinds :(
* geist
yawns
<geist>
good afternoon folks
<clever>
afternoon
<geist>
question, for folks that deal with pull requests on github
lentement has joined #osdev
<mjg_>
geist: don't
<geist>
do folks usually flatten the changes into one or two CLs when committing into mainline, or do they usually do a branch?
<mjg_>
you are we lcome
<geist>
well, i mean merge
<zid>
rebase --ff-only
<zid>
linux kernel does merges
<geist>
i deal with gerrit most of the time, which generally enforces the rebase/fixup strategy
<mjg_>
people like to merge and make spaghetti out of the history
<geist>
but i wonder if it's reasonable to request that PR owners rebase when they fix their changes (ie, fixup vs a new chane in their branch)
<geist>
or if that's fine because i just flatten it when i take the commit
<geist>
for gerrit, that's implied, but then gerrit is somewhat more advanced than githubs ui
<geist>
so i wonder if i'm basiclaly off base
<geist>
the fact that githubs PR ui seems to try to show you essentialy all of the commits combined implies that it's built around the idea that the PR branch is basically a single commit
<geist>
either via it being flattened or a merge request
<mjg_>
it is easy to ask people to rebase, i do it every time i do a PR
<mjg_>
erm, it is easy for people to rebase, so not a problem to ask them
<mjg_>
and it is up to the project to come up with a policy
<mjg_>
you should only get pushback from people with commit messages "Update crap.c"
<geist>
yah trouble is 90% of the time i do they just ghost me
<mjg_>
fuck em
lentement has quit [Ping timeout: 260 seconds]
<mjg_>
are they patches anyg ood?
<geist>
well, yes except it's a patch that is okay
<mjg_>
now *that* is definitely news to me ;)
<geist>
well, i'm just saying that writing code and driving git are two different skill sets
Gooberpatrol66 has quit [Ping timeout: 244 seconds]
<mjg_>
git did not take over yesterday
<geist>
well, hang on. the problem is what i'm trying to solve here
<geist>
i dont have any up front written rules about submitting CLs to my project
<geist>
so it's not like i've made it super clear what they're supposed to do, how they're supposed to format things, etc
Lumia has joined #osdev
<geist>
but what i'm trying to really get at is what is a reasonable level of care for github, because github's ui has a specific strategy to it
<dh`>
just merge and push for fixes to git and github so merges actually work as the gods intended
* dh`
not helpful
<geist>
well, that's kinda what i'm trying to read, what folks's opinions are
<geist>
*usually* I'm a rebase and commit to head kinda guy, though i use merges sometimes for long topic branches
<geist>
but a PR thats like 'add foo; fixup foo; another fix for foo;' etc
<geist>
i can squash and commit that *myself* but is it reasonable to request the commiter squash it themselves, when the github ui doesn't seem to help you here (vs gerrit which does)
<mjg_>
tell people to git rebase -i, most notably squash fixups to noew patches into one commit
Lumia has quit [Client Quit]
<dh`>
I strongly dislike rebasing ever, except perhaps for very simple changes with a short time horizon
<mjg_>
they can literally do this in the console and force push
<geist>
dh`: interesting! so you're more of a merge kinda person?
<geist>
i'm not judging at all, just curious why?
<dh`>
the github UI has a button somewhere to squash when merging a pull request
<dh`>
yes
<dh`>
rebasing lies about history
<geist>
and yeah i just saw the squash and merge button
<geist>
ah yeah, i know what you mean re: lying about history
<dh`>
this severely decreases the value of keeping it around
<mjg_>
what history
<geist>
flip side is the fact that you can rebase means you can throw commits around like candy because you can just squash them, which is what makes it more useful tome
<dh`>
having been stuck using git and github for paying work recently I have come to understand why the git ecosystem is committed to rebasing
<geist>
vs needing to be very very careful about it
<mjg_>
of a temp branch which is supposed to disappear/
<dh`>
if you start merging heavily git just loses the plot
<geist>
yah there are some tools that enforce a merge per 'PR' and those histories turn into a huge mess
<mjg_>
one functional change at a time with no known bugs in any of them is the way
<geist>
most projects i work on at work are gerrit, and tip-of-tree rebase only
<geist>
and yeah that rule
<dh`>
hg doesn't have most of these problems; e.g. you can read the hg log of a repository with lots of merges without becoming hopelessly confused
<dh`>
git doesn't provide a coherent enough log report to follow
<mjg_>
but nobody should give a shit about 'lol fixup previous' commits, if said fixups are to commits which only arrive with the branch
rpnx_ has quit [Ping timeout: 252 seconds]
<geist>
well, rebased stuff is at least better than a pile of merge bubbles, even if the date isn't exactly right
<mjg_>
or rather, these commits should not survive to the main branch
<dh`>
not really
<geist>
mjg_: yeah execpt they do in the sense that f you look at it with a tree based viewer, you still seem em
<mjg_>
not if you don't push them
<dh`>
as soon as you rebase you lose the ability to revert to the prior state
<geist>
oh i'm ssaying if you did a merge that is
<geist>
dh`: you mean forever, or just locally?
<geist>
locally uou have the ref-log, but in the long history, yes
<mjg_>
what prior state, with the known bug?
<mjg_>
fwiw i checkpoint work by creating another branch with + 1
<dh`>
you lose the ability forever
<mjg_>
crap1, crap2....
<mjg_>
what
<dh`>
suppose for example you're working on something nontrivial
<mjg_>
say i'm doing something very trivial instead, but forgot to free() in the error path
<geist>
what i tend to do nowadays since i know i can just squash it (vs the days i worked with mostly perforce or svn) is create tons of WIP and debugging CLs
<dh`>
you see a crash, it's not related to what you're working on, you press on
<mjg_>
i don't want to push 2 commits intsead of 1
<geist>
then once i'm done with the whole thing, add some CLs to clean it up and squash it into a set of feature commits
<dh`>
then you merge that work and two months later the crash reappears only this time it's happening in production
<geist>
but then i care about the results, so i spend the time doing it. i *could* just merge that all in
<dh`>
can you flip back to the state where you saw it? Nope.
<geist>
bt then the WIP commits are all laid bare
<geist>
dh`: tis true
<mjg_>
that much i can with my stale branches
<mjg_>
which i only clean up *way* later
<dh`>
this happened to me last spring, it was intensely aggravating
<mjg_>
my general pint is that you don't want lol commits in the master branch if it can be helped
<geist>
yah keep in mind i generally mean rebase at the end of a develoopment sprint, before you push into mainline
<geist>
and part of the art is figuring out how to break it up into meaningful sub commits
<mjg_>
for one, someone may be bisecting and your "oops fixup" just fucks with their work
<geist>
yah we have the rule at work that every CL must build for sure, and we have CQ and CI stuff that enforces that
<dh`>
conversely, your "oops fixup" might be the problem they're looking for, though
<geist>
tis true, but in my experinece that's pretty rare
<mjg_>
then they find the actual commit which did not have known bug
<mjg_>
s
<geist>
so it's a tradeoff, a compromise
<mjg_>
well one funny note about bisects is that you bisect far enough and so many other problems keep piling up
<mjg_>
that the ln scale does not help
<dh`>
my experience is that the history is valuable only for two things: (a) checking out older revisions for testing, and (b) inspecting how something came to be the way it is
<dh`>
in both cases, lying about history significantly decreases the value
<mjg_>
how is that lying
<mjg_>
it would be lying if someone force pushed to master to cover up a mistake which was alreday published
<mjg_>
i don't care how many typos and brainfarts someone made while writing a patch, i care about the end result
<dh`>
if I have sixteen (or six hundred) WIP commits while building feature X, and then a year later after I've left the company someone has questions about why something's the way it is
<mjg_>
as in what they submit for inclusion
<dh`>
if my WIP commits are still available there is much more information
<geist>
and more importantly it gives you the license to make lots of little commits, which is a great experience
<dh`>
both in the incremental diffs and in the change messages
<geist>
dh`: yeah i dont disagree, but not sure i think that's a big enough value
<geist>
but somethign to think about
<mjg_>
you can keep your wip stuff outside of the master branch
<geist>
and actually kinda the reason i sometimes use merges on my repos for large topics (write some new driver, redo the net stack, etc)
<dh`>
well
<dh`>
being able to inspect the history is about the only reason for keeping it at all
<dh`>
why bother using version control otherwise?
<dh`>
(yeah it's also useful for distributing the tree, I know)
<geist>
well, i agree that you lose a bit of history, i just dont think it's nearly as dire as you say
<geist>
yes *some* information is lost on rebase, but in general my experience is the info you lose is not high value, and the history most certainly still has mostly value in it
<geist>
i remember when i first switched from svn i have very much these concerns, it seems *wrong*
<dh`>
my experience with rebase and squash has been almost exclusively that it destroys what I was looking for
<dh`>
or in the case of other people's squashed commits, maybe so
<dh`>
it's hard to tell if what you were trying to figure out could have been discoverable if the information that *was* tossed hadn't been
<dh`>
some people when they're working just commit at arbitrary points with messages like "asdf"
<dh`>
which provides 0 information
<zid>
I do that just to checkpoint
<zid>
but I never keep that in the rebase
<zid>
(I also don't squash, I edit/pick)
<dh`>
well, even "checkpoint in case what I'm about to try doesn't work" provides info
<zid>
you'll never see my asdf
<zid>
because it never ends up on a branch outside of my local machine
frkzoid has quit [Ping timeout: 244 seconds]
<dh`>
I'll grant that there's nothing helpful about keeping asdf commits, but I think the existence of asdf commits is a sign of other problems
<dh`>
(in most cases anyway)
<zid>
yea, that I am too lazy to use named stashes
<zid>
and that I trust them less
<dh`>
no, that kind of thing is fine
<zid>
that's what an asdf commit replaces, a named stash called asdf
<dh`>
yes and no
<geist>
yah that's one of my points: git allows you to make daily (or even more than that) asdf commits to snapshot your progress, because you can just squash em later
<zid>
sometimes I need to build the 'real' tree so that I can remember what the hell the correct behavior is or something
<dh`>
there are people who code for a while, commit -m asdf, code for a while longer, commit -m qwerty, etc until they think they're ready to merge
<geist>
whereas with svn or perforce or something you have to be more careful about i
<zid>
so I will just quickly asdf
<dh`>
rather than actually committing in meaningful steps
<geist>
i do that most of the time, yes
<dh`>
I never do because I like keeping the history
<zid>
I generally merge/edit them into real commits, then fast-forward the commits onto master
<zid>
squash/edit*
<geist>
i found that to be incredibly helpful, especially if you're working on multiple things at the same time, you can put it down, switch branches, work on that, put it down, switch back
<zid>
so from 9 asdfs I get 3 full commits
<geist>
it's like a journal of things you're doing
<dh`>
sure
<geist>
to me that's a total breakthrough in development (vs the older models where you couldn't do this, or needed to do it outside of the SCM)
<zid>
what ends up being pushed is highly curated
<zid>
what my tree looks like when I work on it? ehhh..
<dh`>
that's what stash is for, if the stash UI worked better
<geist>
i usually leave myself little notes at least: 'WIP add foo, bar is still broken, start of baz'
<geist>
bit yeah
<dh`>
but apart from that, that's not what I mean either
<zid>
stash is identical except it's harder to get back with cherry-pick or reflog or whatever
<wxwisias1f>
i don't have consistent commit styling
<geist>
yah but stash is local, i like to push WIP branches to the repo so i dont lose em
<zid>
so I just use real commits
<wxwisias1f>
it's mostly: "fix bugs", "remove [name] class from file and move it to [file]", etc
<geist>
yah
<zid>
I do name of file or feature, depending on scope, colon, short desc
<zid>
so networking: TCP and UDP split into two files
<geist>
the other thing i like sometimes is as i'm working on a topic, i find a bug in some other code, so i slam down a CL that fixes the other bug, then WIP snapshot what i'm on, then use rebase -i to move the other fix to the head of my local patch list and commit that to mainline
<zid>
or udp.c: Fix checksum offload calculation
<dh`>
I tend to write books in commit messages :-)
<geist>
oh i do too, but WIP ones i tend to leave at one line so i can quickly scan them with tig or gitk
<geist>
huuuuuuuge pet peeve of mine: official commits with one liners like 'fix PR #34252'
<zid>
\n\n, The checksum calculation was accidentally using a signed shift left, which wasn't checking for negative results in bad-cases and could cause an UB shift by -1. Switch 'chsk' to unsigned long and add a macro blah blah.
<geist>
the one liner needs to sum up whats going on, then write a paragraph, and please leave context as to what the bug fix was
<dh`>
I guess I mostly just don't believe in making wip commits in the sense you mean
<zid>
That's before I do rebase -i, and change it to 'e' from 'pick' ofc, it previously having been called udp asdf
<zid>
:P
<zid>
and it gets merged with the commit after it, which fixed the typo in the previous commit
<geist>
dh`: yeah it's a easy of working. i tend to work on features across multiple days, usually interspersed with other CLs, so to be able to break things up is key
<zid>
that then all gets fastforwarded onto master later
<dh`>
squashing things like typo fixes in your own chains of unmerged commits isn't really a problem
<geist>
and i dont trust my computer not to shit the bed on any given day. i've lost weeks of work because i didn't push upstream
<geist>
*especially* computers i dont own or control fully
<dh`>
rebasing a long chain of commits onto a new base revision is bad though
<dh`>
that is where it is far better to merge (and to merge often enough that there aren't huge gaps to cross)
<geist>
well, to be clear i think rearranging the commits, and whether or not you rebase it on master/main are two different things
<dh`>
yes
<geist>
you can still curate a list of CLs and then decide to merge at the end
<geist>
it's the curation that i'm mostly into
<dh`>
that they're both "rebase" in git-speak is somewhat problematic
<zid>
A merge just generates a merge commit or not
<zid>
I don't personally like them I think they're ugly, but I also don't have to go around tracking down who merged what and shout at them etc so it's less of an issue
<geist>
but at least at work, since gerrit enforces 'must FF' then rebasing your stuff on main is actually a fairly common occurrence
<zid>
linux kernel has multiple tiers of mergers so it's important to them
<geist>
at lea tone you do before committing
<zid>
it isn't just linus ffing onto master, it's linus ffing (or merging) the maintainer's trees of merges
<geist>
zid: yeah i dont mind em for topic stuff, almost nice if a topic was developed over a period of time
<zid>
maintainers'
<geist>
but i'd rather the branch not be full of asdf or WIP
<dh`>
that is the point at which you lose the ability to recreate your development history
<mjg_>
as someone who spent tons of time looking at foreign codebases, i 100% appreciate concise history
<dh`>
(rebasing onto a new base)
<zid>
If it has asdf commits in what they ask me to merge, it ain't getting merged
<mjg_>
and find lolfixups annoying
<zid>
rebasing onto main just means doing your own merges basically
<dh`>
one of the problems is that it ought to be possible to preserve the detailed change history and _also_ treat it as a single change
<zid>
rather than the person doing the merge
<zid>
the history is the same either way
<dh`>
which is something neither git nor hg can manage
frkzoid has joined #osdev
<geist>
also flattening things does give you the ability to put in stuff that you shouldn't commit
<geist>
since yo ucan remove it later
<dh`>
no, rebasing onto main means that each one of your versions changes
<zid>
no?
<dh`>
are we talking about the same things?
<dh`>
suppose main is at commit aaaaa
<zid>
I work against it and develop cccc, someone sneaks bbb into the middle
<dh`>
you hack away, meanwhile somebody commits/merges some other stuff so main is at bbbbb
<zid>
so I rebase against bbbb
<zid>
and now it merges onto main as a fast-forward again
<dh`>
you rebase so your branch moves from aaaaa to bbbbb
<dh`>
and now you've lost all your history
<geist>
yes and no. yes the merge is just a multi headed commit, but if the merge is the head of the tree (with say 20 changes on the 'right side') and you do a `git reset --hard HEAD^` it'll step over the merge
<geist>
not one commit into the merge
<zid>
what history did I lose?
<geist>
so it's treated as a single commit from main's point of view, from an atomicity point of view
<dh`>
zid: the history you lost is the state of the tree when you wrote each of your changes
LostFrog has quit [Quit: ZNC 1.8.2+deb2 - https://znc.in]
<dh`>
you wrote them without bbbbb, now they have bbbbb in them, and now maybe they all don't compile
<zid>
so the aaaa -> cccc case?
PapaFrog has joined #osdev
<dh`>
or more likely, they all don't run but you don't know it yet
<zid>
I'm not sure testing for a branch that nobody but me ever ran is important, it just means there's a bug with my rebase onto bbbb, I'm still supposed to test that
<dh`>
you're missing the point
<mjg_>
i think everyone here is happy they don't have to cooperate with other channel members
<zid>
did you say it?
<mjg_>
:>>
<dh`>
you can test it all you like, that avoids the common failure mode
<zid>
I just kept seeing "it loses history"
<dh`>
but it still means you can no longer check out the tree you had when you wrote the first patch of your series
<zid>
my response was that it's irrelevent history because we're never going to swap the order of bbb and ccc, I *need* it to work on top of bbbb, so either I fix it, or I fix it
<zid>
and I can recreate aaaa -> cccc myself trivially
<dh`>
yes, sure, the _current_ version is the same either way
<zid>
or just rebase into a different branch, which is actually more likely
<dh`>
but supposing your branch had the commits 11 22 33 44 55, and after rebasing they're 11b 22b 33b 44b 55b
<dh`>
you've lost 11
<dh`>
you can get it back by doing the rebase in reverse, maybe
<zid>
because I have aaaa -> ccccc already, I pull current master, then rebase (aaaa->cccc) onto (aaaa->bbbb) that I just got
<zid>
aaaa->cccc still exists, in that case
<zid>
until I delete it
<dh`>
you have aaaa -> 11 -> 22 -> 33 -> 44 -> 55 -> cccc
<geist>
dh`: yeah i get what you're saying. and that's a thing, indeed
<dh`>
you rebase, you now have aaaa -> bbbb -> 11b -> 22b -> 33b -> 44b -> 55b -> ccccb
<dh`>
you've lost 11 (not to mention 22, 33, etc.)
<geist>
that was my general complaint about gerrits 'force FF' mode that we run it at at work
<dh`>
and it's not necessarily the case that 11b and 11 are equivalent
<geist>
it's mitigated somewhat by gerrit always doing a full suite of tests (which is why commits take hours to go through)
<dh`>
in fact, unless the change in bbbb was absolutely trivial they are usually not equivalent and if you're trying to find a problem you probably want 11
<geist>
since what you push to the server isn't precisely what is committed, since gerrit locally rebases + tests + commits
<zid>
but like I said, I still have aaa->ccc if I need it, and I *need* this to work, and I am never not responsible for making it work
<zid>
if I deleted the aaaa->ccc branch that's my fault too
<dh`>
git always deletes the aaaa -> cccc branch when you rebase it
<dh`>
you have to take explicit steps to preserve it
<geist>
OTOH once a git repo gets busy enough you either have to hve some sort of server mediated thinglike this or everything has to be an atomic merge, because the repo will move out from underneath you too fast
<geist>
the fuchsia repo at this point is pushing 200-300 CLs at a day easily, so wouldn't be able to locally race to get a tip of tree commit in
<zid>
dh`: I have aaaa->cccc, I pull aaa->bbb from the remote, I work on *this* branch, cherry-picking, merging, editing, rebasing commits onto it, then rename it for-geist
<dh`>
that is part of why merging is a better approach, it scales
<zid>
aaaa->cccc still exists in that case
<geist>
dh`: indeed, or you need something to mediate it, and establish some sort of server based queue, which is what gerrit does for us
<geist>
no idea what github does with conflicting requests
<dh`>
tells you to merge
<dh`>
zid: the standard approach is "git rebase main" and that throws away the old version of the cccc branch
<zid>
I mean, that's a solution to the problem you have with your own local history at least
<geist>
though to be fair, merging has the same scaling issue: if everyone is working against a raw git repo with no mediation, then a merge or a rebase still requires that you win the race: between you pulling, rebasing/merging, and pushing it has to have not moved
<zid>
I'll carry on just doing the former because I have never needed aaa->ccc
<dh`>
if you don't do that because you want to preserve the original version, great, but now (like me) you're fighting against the way Everyone Does Things (TM)
<geist>
it's just merging is perhaps a bit simpler
<geist>
once you get more than a few CLs per hour or so (depending on how big your codebase is and how long it takes to make a local change) it starts to have problems
<dh`>
I have never worked on a tree that busy with modern tools
<dh`>
so idk
<zid>
I prefer putting the onus on the people doing the pushing, by making it ff only
<zid>
rather than overloading one guy doing a bunch of merging
<dh`>
who said anything about one guy doing merging?
* dh`
confoosed
<geist>
yah the last project at google i was on was using gerrit but with 'must cleanly apply to head' bit set (one of the config options) and it started off as a small project that quickly hit this scaling issue
<geist>
we had to switch it to 'gerrit will auto rebase + test and commit' mode, which is how most projects i've worked on that are reasonably large work
<geist>
which means what you push is *not* precisely the same tree that it gets committed, which seems really dangerous, and you mitigate it by having a crapton of tests
<dh`>
current $WORK averages 5-10 commits to main a day
<dh`>
keeping up is not a big problem
<geist>
flip side is since the unit tests in fuchsia take a long time to run, it's usually something like 2 or 3 hours minimum from push the button to it getting actually merged
<geist>
but alas, it's The Way
<geist>
it's all pipelined
<dh`>
nothing else that I've worked on in the past ten years or so has been even that busy (or was just me) except for netbsd and netbsd is still stuck using cvs :-/
<geist>
yeah, i thought they both moved to git? or is that they just both have a git mirror
<mjg_>
everyone has a git mirror
<dh`>
netbsd is partway through moving to hg, there's a git mirror
<geist>
at some point i switched my local mirror of their source to the git versions, because the novelty of doing a cvs up ran out
<mjg_>
:))
<dh`>
cvs is absolutely awful
<mjg_>
fwiw i found out cvs is sightly less that utterly bad
<dh`>
btw, never ktrace cvs, it'll give you a hernia
<mjg_>
in that you can cache local history
bauen1 has joined #osdev
<geist>
yah surprisingly when i worked at apple in 2005ish they were still using cvs, but just starting a longass replay of their history into svn at the time
<geist>
i think it took months
<mjg_>
repocopy
<geist>
i think this was before anything standard existed, so someone had to do it themselves, or maybe they developed repocopy
<mjg_>
i don't know if there was one blessed script or everyone had their own variant
<dh`>
anyway with these things I worry less about whether automerging breaks head than whether the scheme compromises the value of the history
<dh`>
if something breaks the tree you can always fix it
<geist>
yah the tough part apparently is reconstructing what is a CL from the disparate cvs rcs files
<dh`>
yes
<dh`>
it's highly nontrivial :-(
<geist>
i remember shelling into the cvs box and the README (or LICENSE) file in the root of osfmk was like 10MB iirc
<geist>
because of the bazillion tags on it, iirc
<geist>
and then there's all the stuff in /attic, which i dunno how to get that back into RCS. but they really wansted to do the right thing and not just start over
<dh`>
tagging a large cvs repository takes hours because you have to write the tag into every ,v file
<geist>
alas, i think us oldies value history a lot more than folks nowadays. even in fuchsia we've reset various repos more than once because of some reason
<geist>
and folks didn't bat an eye: 'the old repo is still there, can just look in history there'
<geist>
and the blame is completely blown away
<geist>
sigh.
<dh`>
when we did vino, we'd back up and then semi-reset the cvs repo periodically to allow moving files around
<mjg_>
i force pushed a shared branch once, because of a new hire
<mjg_>
several commits looked like this: his unreviwed change, someone reverting it
<dh`>
with the result that if I wanted to salvage the history into a real repo it would be a very big manual job
<mjg_>
i mean several cycles
<mjg_>
i talked to the team, we agreed i'm gonnna clean it up and force push without any of this shit
<mjg_>
then the new hire force pushed it back to the state i just finished fixing
<geist>
that being said with git it's so much easier to disconnect history and start over
<dh`>
then there's the OS/161 history
<geist>
so even what linux does where they make a split and then keep the old one around if you want for a extra remote makes sense
<dh`>
which is the version control equivalent of walking into a room of people talking TCP and shouting "multicast!"
<geist>
haha
<dh`>
os161 has both its own history and the additional history-like sequence of there being a series of solution-set patches
<dh`>
(each of which also has history)
<dh`>
there is no such thing as a tool that can handle this cleanly
<zid>
Leelo Dallas Moolticast.
<mjg_>
the biggest problem with git is that people new to it make a total mess out of the repo
<geist>
right
<mjg_>
and if you that goes unchecked for few days, you are stuck with it
<geist>
which i guess is why something like gerrit is a great solution, evne if you leave it in a pretty 'open' mode
* mjg_
mumbles something about a new project at work
<dh`>
the os161 main is currently 31 hg repositories and a lot of scripts
<geist>
at the minimum it keeps folks from being able to just force push willy nilly
<mjg_>
git commit -am "fixups for the evening", git push --force
<mjg_>
echo "done for the day, cheers suckers" | mail -s lol team@initech.com
FreeFull has joined #osdev
<dh`>
the existence of force-pushes is itself a git bug
<mjg_>
mostly agreed
<mjg_>
good news is that they can be blocked
<dh`>
in hg you can push -f, but all that does is force someone else to do your merge for you
solenya has joined #osdev
<mjg_>
how long can you keep postponing it like that
<dh`>
but seriously, I would sack anyone who force-pushed to main without permission
<mjg_>
agreed
<dh`>
forever if you want to
<mjg_>
albeit you could argue only 1-2 people should have server perms to force push to begin with
<dh`>
basically all that's in play is some nag messages/restrictions that encourage you to have only one head per branch
<dh`>
you can't create more heads by pushing unless you push -f
<dh`>
and if you pull and get new heads it'll suggest that you merge (iirc)
<dh`>
the fact that all heads have to be explicitly named and manually managed in git is a fundamental design flaw
<geist>
i thik the original reason is it used file perms for it, inside the raw git repo there's a refs/heads/<bunch of directories> that you can set individual file perms on
<geist>
so you can create a refs/heads/users/dh and then set it so only you have perms on it
<geist>
so it's very path based, for better or worse
<dh`>
hmm interesting
<dh`>
but anyway this design flaw is at the root of a large fraction of git's problems
<geist>
if yo uhave a shared raw git repo, with everyone accessing it via a ssh account with the shell set to git you can use file perms for most of this stuff, and it works okay
<geist>
a startup about 10 years ago i set up and managed the git server in this fashion and it worked wlel
<geist>
for stuff like force pushes i think you can use server side hooks to nak it
<geist>
and have an admin group that only admins can edit the hook, etc
<dh`>
I've only ever used hg in that type of environment
<geist>
it's quite powerful, just fairly arcane and i dont know if folks manually admin it thatmuch anymore when there are all these fancy web front ends
<dh`>
(and never had a situation where one set of perms for the whole repo wasn't sufficient)
<dh`>
(including when were providing hg space for students for a class)
<geist>
when you grok the file layout of the git repo, the `git push` stuff makes even more sense. in the raw form `git push repo ref:path/on/server` it's quite simple
<geist>
it's really just literally trying to push a file called path/on/server to the server and the file tiself is text with the ref to the hash
<geist>
and that's why refs/heads is this sort of magic path, because it's literally the layout of the repository
<dh`>
seems like a classic case of exposing too many of the gears to the user
<dh`>
kind of like sysvinit
<geist>
yah
<mjg_>
ok mofos, how about this flame
<mjg_>
i have a routine crfree which expects a non-null argument
<mjg_>
now i want to add a variant which tolerates that, how to name it
<dh`>
crfree'
<dh`>
except I suppose you're using a language that doesn't allow that :-)
<mjg_>
no comment
justache- is now known as justache
solenya has quit [Ping timeout: 252 seconds]
Lumia has joined #osdev
<heat>
oh shizzle
<heat>
now that the bsd people are here, what do you think about kqueue vs epoll
<heat>
mainly the thing where epoll forces a fd for everything you want to wait on (and how that's "good design")
<bslsk05>
ariadne.space: actually, BSD kqueue is a mountain of technical debt | Ariadne's Space
<dh`>
kqueue is also a fd for everything you want to wait on, isn't it?
<dh`>
I've never particularly understood epoll or why one would want to care about it
<mjg_>
no, kqueue has support for general events
<mjg_>
see EVFILT_PROC etc.
<dh`>
oh, I see
<mjg_>
heat: i don't know if kqueue is any good in concept, i can tell you internals are crap
<dh`>
there is no *good* way to implement select/poll/queue/whatnot
<mjg_>
heat: as in i don't know if it really wins user-api-wise againts epoll. as for internals, if general linux state is to be used here, i have to expect they are similarly bad
<mjg_>
and i'm not just randomly ranting here, spent years debugging problems in linux and they more often than not were of "how did this happen" nature
<mjg_>
as in how the code doing this went in
<dh`>
from the link it sounds like epoll is bad because it requires making up arbitrarily many new kinds of magic fds, and kqueue is bad because it has a separate enumeration of waitable things that it accidentally made too explicit
<dh`>
whereas in practice the real problems with both are deeper...
<wxwisias1f>
ah, we implement ehci
<dh`>
there's some kind of problem with kqueue where if you want to monitor a whole subtree of the fs for changes it becomes really awkward but I forget why or how
<mjg_>
afair that bit does not really work
<mjg_>
in contrast to inotify you don't get names et al when things pop up
<mjg_>
probably a leftover from general bsd aversion to storing names
<dh`>
that or attaching at the wrong level
<dh`>
probably it just hooks to the vnode and for these purposes it ought to be hooked to the name in the directory as well
<mjg_>
fwiw it probably does not reliably work for that either, most notably for renames
<heat>
1) why are the internals crap 2) what are the real problems with epoll/kqueue?
<mykernel>
first thing in usermode: i get gpf for executing iretq from timer interrupt
<mjg_>
heat: the part which managed to bite me is locking, which ensures things don't actually scale
<mjg_>
heat: it kind of works for single-threaded cases where there is literally nobody else waiting for events on the given underlying object
<mjg_>
heat: ... but shit hits the fan when there is more than one consumer, and yes it is a real usecase
<zid>
mykernel: well, post another old=0xfff new=0e thingy for us to look at
<zid>
d I guess
<mykernel>
lol
<mykernel>
<3
<mjg_>
heat: it is also slow as shit if you have to change anything
<mjg_>
heat: in the monitored dataset
<mjg_>
heat: funny bit is that several systems tried to reimplement poll and select on top of kqueue, only to find it demolishes perf
<heat>
mjg_, but isn't kqueue supposed to be super fast considering most servers in the past 20 years are using it?
<mjg_>
it is super fast aka does not SUCK TERRIBLY like poll and select do thanks to being able to retain state across calls
<mjg_>
so when you have $BIGNUM fds to monitor, and the set is stable, you only do hard work when there is any
<mjg_>
but if you have a high churn accept/close rate, it still sucks
<heat>
but is the suckyness enforced by shit api or is it just another case of the BSDs being hilariously/depressingly underfunded
<mjg_>
underfunding is not the factor
<mjg_>
linux is not underfunded and lolz keep ensuing
frkzoid has quit [Ping timeout: 244 seconds]
<mjg_>
the badness i'm noting here is predominantly just internals being bad
<heat>
linux has a lot less lolz
<mjg_>
linux has less user visible lolz
<CompanionCube>
where's the rant i once read about epoll (dunno if it's accurate)...
<heat>
there has been no BKL in linux for 15 years
<heat>
(no significant BKL I mean)
<mjg_>
i pasted some linux code samples few weeks back, did not i
<mjg_>
and not from staging/ either
<mjg_>
basically as long as something does not outright break functionality, it remains in lulz state for years
<mjg_>
quire frankly embarassing, you would think people would have some pride
Lumia has quit [Ping timeout: 244 seconds]
<mykernel>
hey zid how its going
<mjg_>
heat: you may want to read up on solaris solution!
<mjg_>
heat: ... which is roughly like kqueue
<CompanionCube>
huh, the epoll rant is from 2017 so at least one of the problems it mentions isn't as much of an issue because on 4.5+ it's workaroundable
<heat>
mykernel, listen, you're fucking up the stack. you need to debug that yourself
<heat>
nerdsniping us isn't a helpful learning experience
<bslsk05>
idea.popcount.org: Epoll is fundamentally broken 1/2 — Idea of the day
<mjg_>
i'm in the mood so here is some more linux lulz
<mjg_>
the mount system call *ignores* all flags when used with MS_BIND option
<mjg_>
most notably ignores requests for the mount to be read-only
xenos1984 has quit [Read error: Connection reset by peer]
<heat>
yall are just nerds, all your OSes suck, use windows
<mjg_>
so: mount --bind -o ro foo bar
<mjg_>
would give youa *read-write* bar
<dh`>
that link gets into what I was going to mention, which is that the wakeup semantics of inotify and wait and select/poll and whatnot are not all quite the same and jamming them all together under one roof without some thinking is going to end badly
<mjg_>
you know they fixed it after years of this being a problem?
<klange>
The only good bars are the one you order alcohol at.
<mjg_>
mount does this transparently for you, except it of course comes iwth a race
<mjg_>
good job linux
<heat>
linux is raceist
<mjg_>
i can't stress enough how idiotic the mount point situation is in linux
<mjg_>
do a mount --bind
<mjg_>
then run just "mount" to list everything
<mjg_>
/dev/mapper/ubuntu--vg-root on /tmp/bar type ext4 (ro,relatime,errors=remount-ro)
<dh`>
I remember years and years ago having to apply a good deal of pressure to get mount to handle bind at all
<mjg_>
and that's because linux mount reads /etc/mtab
<moon-child>
why is this a thing
<moon-child>
._.
<mjg_>
which got symlinekd into /proc/self/mounts
<mjg_>
which has a different format than actual mtab
<dh`>
mtab is a thing because in 1994 or something the kernel didn't have a mount list inside it
<mjg_>
this used to be even more stupid when mount maintained it's own copy of mtab
<dh`>
and in the way of linux it's never been cleaned up
<mjg_>
i tried to talk to the linux person about providing mtab-compatible file in the kernel and symlinking to that instead
<mjg_>
but got beyond idiotic response, so i just dropped it
<mjg_>
the same twat pulled of the double mount pasted above
<dh`>
I got them to add the original /proc/mounts or /proc/self/mounts or whatnot a long time ago but was unable to persuade the mount guy(s) to use it
<dh`>
like, probably ~25 years ago
hussein has quit [Remote host closed the connection]
<dh`>
there was some scenario in which /etc/mtab would end up with more information in it, I forget what, and therefore of course it had to be kept
<mjg_>
the bind mount
<mjg_>
it remembers the source dir
<mjg_>
while /proc/self/mounts does not, shows block dev instea
<mjg_>
d
<dh`>
I think this was before that
<dh`>
and involved nfs?
<dh`>
but it was a long time ago and I forget
<mjg_>
huh
<dh`>
anyway that seems like a bug in /proc/self/mounts
<CompanionCube>
didn't linux get a new mounting api recently?
<dh`>
(shocking, I know)
mykernel has quit [Quit: Lost terminal]
<heat>
CompanionCube, internally
<heat>
I'm not aware of a new syscall
<mjg_>
CompanionCube: i dn't know if they did, but viro has been talking about it for some time
<mjg_>
afair the idea was to split mount process into stages
<mjg_>
with one final syscallt o actually plop the the new fs in a directory
<heat>
oh yeah no
<mjg_>
mostly to accomodate things like network filesystems and other crappers with extensive setup
<heat>
that's not a thing
<mjg_>
still did not happen?
<mjg_>
bummer
<heat>
nop
<mjg_>
[to be clear, the twhat is not viro :P]
<dh`>
sounds like a good idea if you ask me
<mjg_>
dh`: ye i like it
<dh`>
syscall-level mount has always been gross everywhere
<heat>
HOT NEWS: FREEBSD BIGSHOT DISSES AL VIRO
<dh`>
proliferations of flags and magic strings and special structs
<mjg_>
linux is not doing itself any favors with usermodehelper
<heat>
whats usermodehelper got to do with this
<dh`>
in bsd we get not only a mess of a mount(2) but every filesystem has its own /sbin/mount_foofs just in case there wasn't enough duplication of logic
<heat>
also feel free to say im wrong but nfs and filesystems like those should stay in userspace
<dh`>
userspace filesystems aren't performant enough and deadlock too easily
<heat>
slow userspace filesystems sounds like a solveable problem
<heat>
in fact linux is doing userspace block devices now
<dh`>
only up to a point unfortunately
<dh`>
that point may be a point where mostly you don't have to care, though
<heat>
doing performant userspace filesystems through io_uring sounds doable
* dh`
notes how nobody cares how slow macos is
<dh`>
the deadlock problems are more serious
<heat>
macos bad, linux bad, plan9 good?
<heat>
plan9: the weirdo's UNIX
xenos1984 has joined #osdev
<mjg_>
heat: maybe i'm misremembering, but i thought funzies like smbfs spawn may end up spawning something for mount
<bslsk05>
github.com: Split zstd into OS dependant files (use FreeBSD UMA) by allanjude · Pull Request #10975 · openzfs/zfs · GitHub
<mjg_>
or rather, responses to it
<mjg_>
fucking hell is my summary
mykernel has joined #osdev
solenya has joined #osdev
<heat>
mjg_, oh yeah are you aware of the reasons behind fbsd's new eventfd shit? considering kqueue and all
mykernel has quit [Client Quit]
<mjg_>
heat: compat with linux
<heat>
cheers I hate it
<mjg_>
well fuck you too
<heat>
i thought freebsd was supposed to be cool and different
<CompanionCube>
heat: another thing to date: epoll is also a thing on illumos
<mjg_>
there is no pleasing you
<CompanionCube>
s/d/h/
<mjg_>
CompanionCube: again or linux compat :>
<mjg_>
but i don't know to what extent it works
<heat>
illumos is cringy solaris
<mjg_>
you may argue illumos is the real solaris
<heat>
if it doesn't have doors on it, i dont like it
<mjg_>
given the exodus after solaris got closed
<CompanionCube>
mjg_: linux compat, except iirc it's slightly different because they thought the behaviour on fork or something was crap and they changed it
<mjg_>
ok, i suspect what it actually boils down to it being hard/impossible to implement with devpoll
<mjg_>
but instead of saying that they claim the functionality is retarded (which it is)
<netbsduser>
yeah, epoll has curious and interesting semantics
<CompanionCube>
would you need devpoll though? they've got their own different epoll equivalent, can't you implement it on that?
<mjg_>
i'm confident they ipmlemented epoll on top of devpoll
<mjg_>
which is precisely where some limitations will be coming from
* CompanionCube
learned this from cantrill's rant about epoll, he's a memorable speaker
<heat>
have u kissed a girl
<CompanionCube>
lol
<mjg_>
cantrill is fun to listen to, but not necessarily good as source of info
<mjg_>
check out his talk about running linux binaries on illumos
<mjg_>
for example trashes vdso as a concept
<mjg_>
cherry on top is that illumos eventually added it
<mjg_>
:>
<CompanionCube>
lol
<netbsduser>
vdso is the mach emulators-like functionality that linux acquired, isn't it?
<CompanionCube>
vdso is the answer to 'syscall overhead is a thing and there are syscalls we can do entirely in userspace' i think?
<mjg_>
netbsduser: uh? it's mostly about exposing crap to usersapce so that it does not have to do syscalls
<mjg_>
canonical example is gettimeofday
<CompanionCube>
maybe you mean the newer 'syscall user dispatch' thing apparently used by wine?
<netbsduser>
utah figured out that the best way to enhance mach4/bsd's performance was to embed extra functionality in the "emulation library", a virtual shared object of sorts loaded into each process' address space and used to handle traps
<geist>
also about exposing multiple mechanisms to syscalls on architectures that have it (like x86)
* mjg_
speed up getpid....
<mjg_>
just kidding
<heat>
vdso wasn't even the first mechanism
<heat>
you had vsyscall
<netbsduser>
initially the emulation library simply forwarded system calls on to the bsd server, but later it began doing all sorts of things to eliminate unnecessary messaging
<heat>
random page at the top of the address space
<mjg_>
netbsduser: good grief
<geist>
also note that some BSDs have been doing this for years too, so it's not that linux invented it
<geist>
it's just their particular implementation
<mjg_>
you reminded me of a funny solaris syscall: uucopy
solenya has quit [Ping timeout: 260 seconds]
<heat>
top 10 funny solaris mechanisms by mjg_
<mjg_>
user to user copy, used by libs to catch faults and return EFAULT
<mjg_>
for bad args
<heat>
why
<mjg_>
solaris, that's why motherfucker
<dh`>
<mjg_> <dh`> solaris put uucico in the kernel?
<geist>
what's wrong with that? that seems like not a bad idea
<mjg_>
geist: poorformance
<heat>
get 10x slower memcpies to catch user errors
<mjg_>
geist: even prior to the meltdown era
<geist>
a) sure, but the point is to catch errors and b) is it 10x? what are the numbers?
<mjg_>
also bonus: uucopy had a security bug which allowed you to whack the kernel
<geist>
okay fine can we have a real conversation about it instead of just solaris bashing?
<dh`>
wait, catch SIGSEGV and turn it into EFAULT?
<mjg_>
sure
<heat>
i dont know im making up figures - the best kind of statistics
<mjg_>
dh`: prevent sigsegv to begin with
<geist>
as in 'okay assume you want a safe mechanism to copy in user space'
<geist>
then it seems like having a kernel mediated mechanism isn't half bad
<geist>
some sort of way to register 'i'm inside a copy mechanism, if i fail here then back me out of it without going through the signal chain'
<heat>
SEH?
<heat>
:P
<mjg_>
if you really need that kind of feature (why?), you should set something in tls
<geist>
actually been thinking about something like that in fuchsia, for a project now
<mjg_>
to let the kernel know
<geist>
right, but then the kernel has to know what TLS is, etc
<mjg_>
instead of blindly making a syscall to do the copy, which modulo bad corner cases, will never return an error
<netbsduser>
i do sometimes find myself wanting to have the ability to in a userland program set up my own arbitrary translation tables for some region and handle faults myself
<geist>
if you wanted to do it in a highly generic way then maybe there's a cleaner/more abstract solution
<mjg_>
but by making a syscall mandatory you are making it prohibitively expensive
<mjg_>
and that's only to handle programmer error
<geist>
okay, fine. again lets separate the why from whatever bad time you had with solaris when it touched you in the bad place
<heat>
hahahahaha
<geist>
yes would probably do something that doesn't volve a syscall, but it is an intriguing problem to solve
<heat>
again, I think SEH would be a good solution
<geist>
isn't there some sort of register this range of text in linux thing?
<geist>
i thought someone here was talking about it
Lumia has joined #osdev
<heat>
yeah for percpu code
<mjg_>
linux has hackery for its own user-accessing code
<heat>
rseq
<netbsduser>
solaris remains the queen of unix even 12 years after oracle shitcanned it, i'm not surprised to learn they had yet another cool feature
<heat>
dors
<geist>
netbsduser: uh oh, you just poked the fire
<mjg_>
netbsduser: is this a troll statement?
<geist>
netbsduser: mjg_ hates solaris, so i think you're about to hear an earful
<netbsduser>
mjg_: what approach does it take? in netbsd a new interrupt handler is installed for page faults which knows how to tell the caller of a copyout/copyin operation, "it failed, now continue in that knowledge"
<geist>
i'm pretty ambivalent. i thought it was pretty good at the time, but time has moved on
<mjg_>
go fop_lookup -> vn_updatepath and tell me what you think :>
<netbsduser>
mjg_: it's a truthful one
<geist>
it wans't the fastest thing but it had a lot of eatures and was pretty solid, *for the time*
<mjg_>
netbsduser: i'm pretty sure netbsd copied the solution from linux
<geist>
but like most of the big unices eventually became more and more irrelevant
<mjg_>
netbsduser: if you are talking about recognising faults from copyin/out
<netbsduser>
zfs alone would be a massive merit; add zones, dtrace, smf, fmd.... these improve on most of their successors
<mjg_>
zfs is great, dtrace is ok
* mjg_
uses both all the time
<dh`>
if we're talking about who came up with pcb_onfault I am reasonably certain it was in bsd before linux existed
<CompanionCube>
netbsduser: i mean, the openzfs upstream isn't solaris anymore
<netbsduser>
bonwick must rank as one of the finest engineers to have walked the earth
<mjg_>
dh`: the hack is to *not* set pcb_onfault
<CompanionCube>
....i wonder how illumos has done openzfs 2.0, now :p
<mjg_>
dh`: instead recognize on fault that you are in the area which would be enclosed by it
<netbsduser>
i was mindblown when i realised that he applied his experience in memory management (slabs, vmem, magazines etc) to the world of filesystems, it was an incredible cross-fertilisation
<geist>
i met him once, it's a nice guy
<geist>
he's
matt__ has quit [Ping timeout: 244 seconds]
<dh`>
oh that, right
<netbsduser>
CompanionCube: yes, unfortunately i think reason has to restrain the raised hand of adulation here
frkzoid has joined #osdev
* dh`
looks at netbsduser
<netbsduser>
i am glad illumos is around but over 12 years of existence it has not majorly innovated as far as i am aware
Lumia has quit [Ping timeout: 244 seconds]
<geist>
right, and even in the later days of it still being sun (say late 90s, early 2000s) it was alread starting to get lamented as 'slowaris' etc. linux at the time didn't have most of the features, but being newer and whatnot was faster on cheaper hardware, etc
<heat>
so how is solaris the queen of unix again?
<mjg_>
sollaris most definitely innovated claiming scalability while having next to none
<dh`>
the term "slowaris" or "slowlaris" dates to the early 90s
<geist>
and so the world moved on, cheap hardware (low end x86 servers) became the norm, and big unix boxes became old hat, etc
<netbsduser>
the sheer quality of the 5 things i enumerated i've yet to see matched
<geist>
dh`: yeah, was slowlaris when i was first exposed to it in the mid 90s
<CompanionCube>
netbsduser: the init system part of systemd is decent, i think?
<netbsduser>
as for 'slowlaris' that's just, like, your opinion man
<dh`>
because in say 1994 or 1995 solaris was slow, clumsy, badly broken, and sun was scrambling hard to try to make up for their poor performance with more cpus
<geist>
though i also remember being mega impressed with the first solaris box i ever used: sun enterprise 20 i think, somethingl ike 8 cores, many GB ram, and had like 300 logins at the same time
<netbsduser>
i would need quantitative data to speak to that
<netbsduser>
CompanionCube: it is a great opportunity wasted
<geist>
but the equivalent x86 box was not really there, so it was the thing you ran on the bigger box you could get form sun
<geist>
same with AIX, Irix, etc. you ran the native unix on the big box you bought
<heat>
EBR
<geist>
but by the late 90s, early 2000s that was starting to erode becaus eyou had cheapo x86 boxes
<CompanionCube>
>ext4 >innovative
<CompanionCube>
lol
<geist>
and those were getting pretty powerful
<heat>
yeah that one is kinda a stretch
<netbsduser>
SMF was working on encapsulating various sorts of resources into manageable logical entities; there was work for example to put network interfaces into that network
<dh`>
yes, but if you wanted a working system you were far better off going with DEC or even SGI than Sun
<netbsduser>
this was possible because SMF abstracts the concept of a 'restarter'. this is similar to systemd's Unit abstraction, except more generic + not an internal detail
<heat>
mach vm
<geist>
dh`: perhaps, though by late 90s DEC was already circling the bowl, got bought and gutted by compaq around then
solenya has joined #osdev
<heat>
also most/all the innovation in POSIX in the last 15 years at least
<geist>
99 i think? I was interning at compaq the day they bought DEC and you could go to the DEC intranet page
<dh`>
s/innovation/blunders/
<dh`>
:-)
<CompanionCube>
netbsduser: iirc there have been attempts to do a SMF-alike in linux but never particularly well. I think it was called SystemXVI and no longer exists.
<heat>
posix was a blunder anyway
<netbsduser>
last i checked the systemd solution to e.g. say "service a needs interface eth0 to be up" is to call some script or utility to wait on eth0 externally! so systemd is both too encompassing and yet not nearly enough
<heat>
posix bad, use windows, end of conversation
<dh`>
by 1999 your best bet was x86 boxes and linux or freebsd or netbsd
<geist>
also by 99 itanium was killing off most of the alternative cpus, and thus most things were collapsing onto x86 and ia64 (which then never really materialized)
<CompanionCube>
netbsduser: iirc you van wait on devices or network services?
<geist>
dh`: yeah but was a few years until you got 64bit, so it was still a toy if you needed something 'big'
<netbsduser>
CompanionCube: devices, i think you can wait on the existence of a device node. but not network interfaces, that's a whole different object model
<geist>
but i think the idea of running lots of inexpensive machines was taking more hold
<dh`>
but 1999 was also pretty much the absolute low point from an OS point of view
<bslsk05>
unix.stackexchange.com: linux - Systemd wait for network interface to be up before running service - Unix & Linux Stack Exchange
<geist>
dh`: somewhat. yeah i graduated college in 2000, right into the nadir of it.
<geist>
OTOH i worked at Be from 98-2000 so we had a little spark there!
<CompanionCube>
i can't test, this box uses openrc because that was the default and i don't care which is in use
<netbsduser>
i have the beginnings of an SMF port to NetBSD and i have a quite functional systemd port to all the BSDs
<mjg_>
what?
<heat>
delete it
<CompanionCube>
wait, isn't systemd widely known to be unportable?
<heat>
yes
<heat>
it doesn't even run on !glibc
<netbsduser>
CompanionCube: it's either wait for the device node to appear (not terribly useful) or wait for whatever networkmanager or systemd-networkd call "networking is up"
<netbsduser>
CompanionCube: it was not overly difficult
<mjg_>
i'm pretty sure you would need cgroups for systemd to work
<netbsduser>
systemd is unimaginative and doesn't much exploit its dependency on gnu/linux
<dh`>
as I recall in 1999 1GB of ram was still hella expensive
<dh`>
so the actual need for 64-bit boxes was limited
<heat>
>doesn't much exploit its dependency on gnu/linux
<heat>
yes it does, it 100% does
solenya has quit [Ping timeout: 260 seconds]
<netbsduser>
mjg_: i took two different approaches: one was to implement no-controller cgroups as a FUSE/PUFFS filesystem on the back of kqueue's EVFILT_PROC; the other was to replace the cgroup logic with direct use of EVFILT_PROC
<geist>
dh`: yeah that's about right. my 'big expensive' desktop i bought in 1999 was a dual p3 500 + 128MB ram
<netbsduser>
heat: not fundamentally. in systemd-the-pid1 it is almost entirely confined to unitfile options you can specify to e.g. namespace this, or install ebpf filter that
<geist>
BeOS didn't even boot on 1GB ram (physmap style issue) that never got fixed because we literally didn't have anymachines around with that much memory
<mjg_>
netbsduser: quite frankly sounds like an exercise in making things work when they should not, as opposed to a viable approach
<netbsduser>
perhaps the new ancillary daemons take more advantage, but i am not overly familiar with them
<geist>
probably only super expensive compaq proliant servers would have 1GB+
carbonfiber has joined #osdev
<dh`>
in say 1994 though if you needed a mailserver for 10000 undergrads x86 wasn't a viable choice, but then neither was solaris
<geist>
but my point re:64bit in 1999 is if you thought you needed it or whatnot the only solutio was a 64bit box from sun or sgi or ibm or whatnot
<netbsduser>
mjg_: it was a great diversion during the covid times
vdamewood has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
<geist>
dh`: disagree, my school ran on solaris just fine
<geist>
seemed to spool mail just fine
<geist>
and it was 45k undergrads
<dh`>
my school tried to switch to solaris for some reason and it was a huge failure
<netbsduser>
i volunteered at an old folk's home during that and was struck down by the virus and had to stay at home for a long time recuperating
<mjg_>
my college ran mostly on solaris until about 2005
<mjg_>
it was fine afaics
<geist>
that's hwat i mean, i was highly impressed with the shared mail/shell solaris box at the time
<mjg_>
(until they tried zfs.. :>
<netbsduser>
the greatest amusement i could think of was to get systemd to run on openbsd and then silently post it to #openbsd and see what they said
<geist>
a 8 or 10 core machine with GBs of ram was like mind blown back then
<geist>
and it ran well
<dh`>
though that was more trying to use one four-way login server for everyone; the mailserver was an outdated alpha
<CompanionCube>
netbsduser: that's the most high-effort troll ever
<geist>
but then that was a very expensive machine at the time i'm sure
<mjg_>
my troll idea was to make binary compat with openbsd under freebsd
<mjg_>
and offer to build releases much faster
<netbsduser>
CompanionCube: to me it's an art
<mjg_>
it's probably not very hard if you nop-out all the security features like pledge
<CompanionCube>
netbsduser: i would agree with you on that one.
<bslsk05>
github.com: InitKit/README.md at main · InitKit/InitKit · GitHub
<geist>
huh wonder how that handled 5GB ram, since those were 32bit sparcs. i wonder if sparcv8 at the time had some sort of PAE style larger phyiscal aspace thing
<netbsduser>
CompanionCube: that is a troll project on my part
<mjg_>
sparc had software tlb
<netbsduser>
i was "inspired" by a CVE around PolicyKit's javascript interpreter
<mjg_>
i can only imagine what fuckery was needed to make that to work, if it was supported
<dh`>
anyway, the university's stupid choice of trying to replace 4-5 chronically overloaded mips decstations with one multiprocessor solaris box was not solaris's fault
<geist>
mjg_: yeah so would have had to extend the TLBs to h ave 33 - 12 bits for the paddr and you're good up to 8GB i guess
<netbsduser>
it seemed like it would be funny to sketch out the most absurd architecture i could think of for a service manager
<geist>
mjg_: though i'm not so sure sparcv8 was a software tlb. i know v9 is, but v7 and v8 maybe was page table based?
<CompanionCube>
i remember cloning one of the older repo names
<CompanionCube>
let's see what version it is
<dh`>
what was solaris's fault was various things that just plain didn't work, like logging out didn't remove entries from utmp so in addition to finger lying utmp would grow rapidly without bound until / filled
<mjg_>
geist: ye i'm not checking that. i used to read old sparc support out of curiosity, but made myself a promise to drop it
<bslsk05>
temlib.org: MCU : MMU TLBs and Pages | TEMLIB
<CompanionCube>
....huh, i can't actually find it
<dh`>
as well as sun's well established habit of never updating or fixing userland so all kinds of historical bugs came back to life
<heat>
huh?
<netbsduser>
but certainly in terms of systemd on non-unix, i think it is much more viable than people think, indeed initware already has the core functionality of systemd, systemd-logind, and systemd-journald, but i would say that radical changes would also be wise
<geist>
that's what i remember, though i guess that does mean it's possible to have a software fill mechanism, or other implementations change the mmu
<mjg_>
geist: so i think i got performant uucopy
<mjg_>
geist: you merge vdso + typical copyin/copyout
<geist>
oh yeah?
<CompanionCube>
netbsduser: as a matter of fact, those are the components of systemd that i've not had anything against conceptually...
<mjg_>
uucopy is exported by the kernel, with well known user access points
<geist>
oh fair. you could say 'if the cpu faults in this range it was a uucopy failure'
<geist>
good idea
<mjg_>
should any of them fault, userspce RIP gets redirected to code which returns EFAULT
<mjg_>
and there you go
<mjg_>
right
<geist>
question is what security issues that exposes. would have to run it past security folks. ie, can you build some sort of exploit out of that
<dh`>
why would it?
<geist>
well, if i knew that i wouldn't have just stated the previous sentence
<geist>
it just smells like something you have to consider someone hostile abusing
<mjg_>
famous last words, but i don't see any problems past what you normally get for adding code
<dh`>
they can write their own code that returns EFAULT
<geist>
at first glance though probably nothing they can do with it, because they'd worst case just get their PC set to the exit of the routine
<mjg_>
:)
* geist
nods
<mjg_>
hm.. i guess it would help to make sure userspace cannot mess with the page
<geist>
i like this, i think i might use it
<mjg_>
can't unmap or mmap something else on top
<dh`>
the only problem might be that if you don't limit the number of ranges a process can request special handling for it might be able to run through a lot of kernel memory
<mjg_>
but it is userspace doing the copy
<geist>
dh`: yeah but if it's done in user space it'd still trap the same way
<mjg_>
so no special permissions given
<mjg_>
it's just not giving you sigsegv but instead of magically returning efault
<geist>
i think the funny security bits would be in the kernel side: the code that traps the page fault and then rewinds the PC would need to be super solid
<geist>
and make sure it only triggers for that narrow path (not just any exception) otherwise you could try to trigger other faults inside that range and maybe fuck something up
<dh`>
that code already exists, you're talking about a ten-line patch to it
<mjg_>
so can i claim now solaris is crap for making a syscall? :)
<geist>
mjg_: well, see i actually think i *need* something like this for a feature in fuchsia. and fuchsia has no such sigsegv
<geist>
since there are no posix signals at all
<dh`>
what do you do to support generational garbage collectors?
<geist>
the feature ic an't really talk about but it involves basically doing the equivalent of copyin/copyout, except in user space
<geist>
but need the same semantics: try to copy something and if you get slapped, it's okay
<netbsduser>
dh`: fuchsia has special support for generational gcs?
<geist>
and since fuchsia already has a very complete vdso, adding routines like this are no sweat
<geist>
and the kernel knows alreayd where the vdso is and there are already checks to make sure the vdso is mapped precisely once, cannot be modified, etc
<mjg_>
nice
<mjg_>
reminds me to put string ops in vdso
<heat>
why
<geist>
heh you say that, might not be a bad strategy. i know that darwin-arm did, for example
<netbsduser>
i am planning for scal/ux to offer a nicer interface than mprotect+sigsegv for write barriers so that i can get my smalltalk vm working nicely on it (i use the Ravenbrook MPS, which does use mprotect+SIGSEGV trapping to that effect)
<mjg_>
geist: i know
<geist>
i think darwin ppc had it too
<mjg_>
heat: so that i have one place to update
<mjg_>
and older containers automatically benefit
<heat>
but now your kernel supplies the vdso...
<mjg_>
also setup should be cheaper since you don't have to ifunc on each exec
<heat>
sorry, the string ops
<mjg_>
ye, with simd
<geist>
mjg_: i haven't broached that topic but since you bought it up the other day i have been sniffing around for exactly the state of the stringops in and out of kernel in fuchsia. got someone working on updating the kernel versions for the <256 byte stuff that you mentioned
<mjg_>
it's not the same used internally if that's what you mean
<dh`>
if you want to supply a string ops set that's customized for the processor currently in use, there are many worse ways
<geist>
but user space is in worst case
<mjg_>
geist: well i would say consult the go guys, they clearly know osmething about it
<geist>
well it's not really having user space copies, its' that it's more political, etc
<geist>
for reasons i dont want to go into right now
<geist>
but kernel space, we actually do already have a good combination of copies, but without the <256 slow startup path
<raggi>
i am hopeful for llvm-libc
<mjg_>
so do you have any tooling to chekc what kind of copies you are doing in practice?
<geist>
raggi: i am too, but i think too many folks are, and it's kinda holding things up
<mjg_>
a'la ebpf, systemtap or dtrace
<raggi>
geist: :(
<geist>
since thats 'what's gonna solve all the problems' while current problems exist
<geist>
it's the 'looming solution over the horizon' problem
<raggi>
yeah, i looked at the status today and i wasn't entirely surprised at the things that seem like stalling points, such as printf
<raggi>
llvm-libc is a project to make a portable easily crossable libc
<dh`>
those who will not learn from history^W glibc are doomed to repeat it
<netbsduser>
i have had a great time with mlibc
<geist>
mjg_: what is that preicsely measuring? the sample size of calls in the kernel?
<mjg_>
geist: no i mean it is almost certain your kernel is doing small memcpy/memset a lot and it would pay off even to write a toy instrumentation
<klange>
I just continue to use my own pos, at least I have more flexibility than I had with newlib
<netbsduser>
the managram project decoupled it from their microkernel so that others can use it
<geist>
mjg_: absolultely
<mjg_>
geist: it counts all sizes passed in
<geist>
got it
<klange>
netbsduser: mlibc was never coupled to managarm
<mjg_>
geist: i think the above is from buildling the kernel
<heat>
i dont trust mlibc
<raggi>
llvm-libc will also i expect have strong performance and toolchain integration
<heat>
it's not mature enough
<netbsduser>
it felt almost like cheating using it
<mjg_>
geist: i also checked linux at some point and no surprise, it also keeps passing small sizes evyerwhere
<klange>
cheating is using musl
<heat>
raggi, if toolchain integration means it will only compile reliably on llvm, no thanks
<heat>
klange, no u
<mjg_>
geist: albeit it requires patching to allow ebpf to trace it
<geist>
mjg_: yah in geera fuchsia should have small memcopies/moves and user_copies spread over a larger size, but probably never above about 2k since that's the largest in-kernel buffer in one segment
<dh`>
one of these years netbsd will finish moving to hg, and then it will become markedly easier to use netbsd's libc
<raggi>
no, toolchain integration means trying to avoid falling out to asm every time there's a performance challenge
<geist>
and then a special copy routine for PAGE_SIZE
<netbsduser>
what, i just implement a few system calls, and then i get not only gnu gcc, bash, coreutils, but also all sorts of other stuff which uses funny features like TLS, and it just works? and i had to spend almost no time binding it together?
<geist>
and most small static copies are gneerally inlined by the compiler
<dh`>
(why? because cleaning up the kernel headers mess needs mass renaming)
<heat>
raggi, I don't think the compiler will ever get to "super-hand-optimized AVX routine levels"
<netbsduser>
heat: they are bringing Google Chromium to managarm
<heat>
it would be ridiculously hard
<dh`>
why shouldn't it?
<dh`>
it's the compiler's job to know how to make a simple loop run as fast as possible
<netbsduser>
a libc which can reliably run that is one i can call at least relatively mature, because i imagine such a program is a great test of its reliability
<geist>
problem there is the compiler doesn't know what you're targetting, so any highly specialzed compiler codegen only works for situations where you're compiling for a specific hardware target
<heat>
netbsduser, cool. doesn't matter. musl has a lot more mileage and even that runs into issues from time to time
<geist>
ie, maybe good for server side stuff where you know the hardware you're on, or a consumer device
<moon-child>
hence: jit :)
<netbsduser>
heat: from what i gather musl has an intransigent opposition to "GNUisms"
<dh`>
you can build multiple different versions with multiple target versions
<netbsduser>
even when the GNUisms are BSDisms that GLibc adopted at some point
<heat>
dh`, do you want every loop in your program to become SIMD-fest 2022?
<dh`>
er, multiple different builds with multiple target cpu versions
<raggi>
you have march, mtune and mcpu to provide some ranges
<heat>
most things don't need to be as optimized as memcpy, and memcpy is already rolled as asm
<dh`>
no, but it would nice to be able to tag the ones that matter that aren't memcpy
<heat>
also intrinsics, etc
<geist>
dh`: one can do that, but then your testing requirements go through the roof, and for generic solutions (like most x86 world) you dont have that luxury of knowing ahead of time what hardware its on
<raggi>
heat: already depends on where.
<geist>
except for maybe server stuff
<netbsduser>
in any case mlibc is enjoying a lot of interest, and has a very kindly team behind it who have been of assistance to me a few times
<heat>
raggi, most production-grade implementations
<heat>
netbsduser, the problem isn't "lack of features". it's bugs
<heat>
musl has everything, it's just still lacking some maturity
<heat>
apart from alpine and some containers, no one really uses it
<dh`>
geist: well, you can either have one version tuned to be adequate on a range of cpus (and again, the compiler is in the best position to do this) or multiple versions
<netbsduser>
mlibc has something musl has not
<netbsduser>
a focus on portability and encouraging people to port it to their OS and full encouragement and support thereof
<geist>
huh. interesting the sparcv8 page tables are (though 32bit) 3 levels, with 6 bits of shift between levels, which implies (6 + 3 = 9) 512 byte page tables
<geist>
that's annoying
<heat>
netbsduser, ok. but it's not used anywhere of importance, just hobby OSes and hobby developers
<heat>
no backing
<geist>
oh even weirder than that. 8 + 6 + 6 + 12
<heat>
also "focus on portability" doesnt mean good; see glibc et al
<geist>
so the top level table is 2k?
<klange>
glibc doesn't have a focus on portability?
<heat>
it does
<klange>
i mean, good joke claiming it does, but it absoluteley does not
<heat>
that's why is damn near unreadable
<mjg_>
portability to what?
<raggi>
this is an argument of generalities, it's not going to resolve
<netbsduser>
they have a formal portability at least
<netbsduser>
there is a system of 'sysdeps'
<heat>
yes
<heat>
everything is stupidly indirected and fragmented
<geist>
yeah it's reasonably portable, in that it attempts to be somewhat
<geist>
much more so than not portable at all
<geist>
it's just a mess and hard
<netbsduser>
mlibc has it in a seemingly simpler form but glibc appears to have been designed to run on everything from 4.2BSD to SVR3 at some point (it almost certainly did)
<dh`>
geist: given how old v8 is, probably they figured wasting a minimum of 8k on every process for page tables was excessive
<geist>
dh`: yeah that's fair. you could i guess fit a 2k top level, and 4 second levles in a single 4K physical page
<dh`>
before it became the linux libc, glibc was theoretically a drop-in replacement for libc on "any vendor OS"
<geist>
i mean, for example, we used glibc in BeOS
<heat>
just like gcc
<dh`>
except that you had to be insane to actually use it
<mjg_>
wut
<mjg_>
so who killed it? drepper?
<dh`>
and it was pretty much a dead project until someone decided it ought to replace linux libc5
<netbsduser>
drepper's "hellish fascism" was certainly famous everywhere
<raggi>
the thing that "killed it" is people starting to need to do proper cross builds all the time
<geist>
looks like drepper got ahold of it in 1995, and by 97 more or less ran it (according to wikipedia)
<dh`>
geist: I have thought about this some from time to time in the pursuit of making fast OS/161-derived kernels :-)
<netbsduser>
i heard his gunshots ring out even from the BSD camp
<geist>
which was about when BeOS was using it too
<dh`>
mips is software-only so you can use whatever structures you want
<dh`>
and in an environment where the default ram size is 1M extra pages add up
<geist>
yah
<geist>
kinda makes me want to fire up my old sparcstation again
<dh`>
and since for reasons historically most of the performance benchmarks are out-of-core saving a few pages makes a noticeable difference
<netbsduser>
i had relatively little knowledge of beos until recently but then i opened Giampaolo's excellent book on the be filesystem
<geist>
there's a bit of a resurgence in it i think because some guy on youtube tried to get t running on his ppc
<netbsduser>
it inspired me to have a go at designing an fs of my own (granted it will never be btrfs or zfs, but it's going to be an excellent learnign experience. i had only implemented a toy FS during my undergrad prior to it)
<netbsduser>
geist: there seems to be a certain lag period after which the old is cool again
<geist>
yah generally i'd start with a fairly simple ffs style thing, but without all the fixed region stuff
<geist>
netbsduser: indeed
<netbsduser>
vinyls became a sensation once again, but i remember people scoffing at me when i joked about cassette tapes being next
<dh`>
we're due for a resurgence of interest in incoherent caches
<netbsduser>
now there are new releases on cassette tape again
<mjg_>
that's some perversion
<mjg_>
what's next, vhs? :)
<geist>
dh`: yeah years ago i remember interviewing for Convex (bought by HP a few years later) and most of the interviews about how to deal with incoherent caches
<klange>
Tape doesn't have any of the good qualities that vinyl has. It degrades faster, you can't "see the music", and it was never expensive so it lacks hipster cred.
<mjg_>
a format nobody misses, or so people thought
<geist>
they were at the time i think building huge incoherent MIPS machines, moving to itanium
<qookie>
dh`: we already have that to some extent on aarch64, the icache is not required to be coherent w.r.t. the dcache
<heat>
i t a n i u m
<mjg_>
heat: solaris on itanium would be the shit, would not it
<netbsduser>
geist: that's broadly the ballpark i'm aiming at, while i have plans also on interesting things i can do after i have the basics
<dh`>
not just aargh64
<geist>
ah no PA-RISC i think
<qookie>
aarch64's the one i'm familiar with that's like that
<heat>
mjg_, holy fucking shit would it not, TCP/IP over STREAMS on doors on solaris on itanium
<heat>
actually, sorry
<netbsduser>
transparent compression should be relatively easy to churn out, snapshots will require some thought, and probably also some kind of dynamic allocation of inodes to make it work
<heat>
on itanium on a dual itanium/ia-32 mobo
<dh`>
but I/D flushes are a minor manifestation (that's why they have remained tolerated)
<dh`>
basically imagine that everywhere you have a memory barrier, you need an explicit cache flush, except that instead of just saying "flush" you need to know which lines to flush
<geist>
yah for the most part anything that's not x86 (or designed back when there was either no cache or one unified one) has coherent i/d caches
<heat>
netbsduser, compression is not that easy, given that ext4 still doesn't have that
<heat>
and never will most likely
<dh`>
and if you get it wrong, things mostly work except when they don't
<geist>
s/coherent/incoherent
<dh`>
on the other hand, with the rise of managed code this becomes a lot easier to handle
<netbsduser>
heat: my thinking was to group some number of blocks together into a compressible unit, then if i can compress each into fewer blocks, do so
<heat>
yes, I think thats how the old e2compr patchset does it
<netbsduser>
it's simplistic and it doesn't really play well with extents if i ever want those, but it ticks the box
<qookie>
oh yeah another annoying thing (compared to x86 afaik) is that coherence is not maintained between contexts with caches on and off, since afaict the cpu just stops looking at caches with SCTLR.C=0
<qookie>
so you either need to flush everything or create a new set of page tables since you can't use the caches with mmu off
<qookie>
or well, this is specific to the uarch but the architecture permits it
wxwisias1f has quit [Ping timeout: 260 seconds]
opal has quit [Ping timeout: 258 seconds]
<geist>
yeah you really shouldn't be turning the mmu on and off on ARM
<geist>
it's clearly designed such that you turn it on and leave it on
<geist>
if you need identity map it helpfully has huge large pages for you (1GB and 512GB on some implementations) so you can pretty easily use them without any overhead
<qookie>
yeah making an identity mapping is the better option, but this caught me off guard when i was working on some uefi stuff and wanted to reclaim the uefi reclaimable memory (after ExitBootServices) that is
<qookie>
since the page tables are in there, so you have to switch off of them before you can safely use the reclaimed memory
<heat>
you should be with the mmu switched before doing that
<heat>
reclaiming UEFI stuff is super late game compared to basic bootstrapping
<qookie>
it was in a bootloader so i wanted to avoid making new pts
<qookie>
or well, the bootloader makes new pts for the kernel, but we don't use them in the bootloader itself
opal has joined #osdev
<qookie>
and in the end we solved this by reclaiming the memory (by adding it to the memory map), and then not allocating anything anymoree (which only happened due to a bug)
<qookie>
since the kernel is entered right after the reclaiming
<heat>
you're mangling the UEFI memory map?
<heat>
that's $bad_idea
<CompanionCube>
netbsduser: curious and didn't find an obvious answer in the docs: does initware do systemd user service management?
<qookie>
nah, it's a memory map generated by the bootloader
isaacwoods has quit [Quit: WeeChat 3.6]
<heat>
qookie, i would just not put boot services memory in "free memory" until i was firmly in the kernel, where I would reclaim it