<Ermine>
Anyway, using uefi is much more pleasant to use than finding interrupts you need in RBIL. But it is a complex thing and a large attack surface, unfortunately
<zid>
RBIL?
<zid>
red black is life
<Ermine>
Ralf Brown's Interrupt List
<GeDaMo>
Rabbit Burrows Into Lettuce :|
<zid>
why on earth would you bother with bios routines in 2022
<zid>
even without uefi, we've had very capable bootloaders for 30
<Ermine>
I followed "OS from 0 to 1"
<zid>
There's gigabytes of code you *won't* be writing yourself in terms of desktop software
<zid>
I *highly* recommend bootloaders being in those gigabytes
<gog>
that's the really annoying part about UEFI is that it's such a large attack surface because they decided they needed to support all the things
<gog>
i wish they had pared it down to boot time stuff
<gog>
and PXE and other stuff was totally different ROMs
<gog>
i mean local boot time stuff
<gog>
most minimal set possible
<Ermine>
Well, uefi is done by intel and microsoft, and 'most minimal set possible' is not about them
<clever>
one way in which ive noticed efi being more minimal, is that most implementations only support fat
<clever>
but apple's uefi supports HFS+
mzxtuelkl has joined #osdev
<Ermine>
can they support anything else?
<clever>
so the ESP can be on a more modern fs
<clever>
Ermine: the fs drivers are just a .efi binary that translates between fs and block-io, and its purely a matter of including the right drivers in the main flash
<Ermine>
ah well
<clever>
Ermine: i have seen one project, that wraps ALL of the grub fs drivers into a .efi, which you can then load from a fat32 at bootup
<clever>
and if you can modify the flash (as the rootkit does), you could embed that in there
<Ermine>
on the other hand, I think fat is sufficient for esp
blockhead has quit [Remote host closed the connection]
<clever>
Ermine: what if power failure occurs while your writing to the FAT table?
<clever>
that could potentially corrupt every file in that sector of the FAT
<Ermine>
good question.
<clever>
your only options are to either use a journaled fs, with proper journal support in the efi
<clever>
or to just never modify the ESP, put a grub.efi on ESP, and put your kernel on an ext4 part
<Ermine>
But I think this is an event that is quite rare
<clever>
apple went the route of using a journaled FS, HFS+
<clever>
but most OS's cant do that, because it requires modifying the firmware on the motherboard
<clever>
and only because apple owns everything, can they freely modify anything
<Ermine>
I modify esp only once, when I install bootloader. After that, there are no writes to esp (except bootloader config updates)
<zid>
I'd rather the fs be ro until I say otherwise ngl
<zid>
I might be trying to train my ram or something
<Ermine>
If something happens during installation, I can take recovery flash drive and recover
<clever>
Ermine: yep, thats what i meant with grub.efi + ext4 /boot
<Ermine>
Also one can protect computer against power failures with UPS
<clever>
power failure isnt the only way it could die
<clever>
my system has been having a lot of issues locking up solid lately
<Ermine>
SSD failed?
<clever>
i suspect its either the cpu or ram
<clever>
even bloody memtest locks up :P
<Ermine>
My computer also locks up ocassionally. I previously thought this is GPU, but replacing it didn't help :(
<clever>
i have once even observed a bitflip on disk, but its zfs, which checksums up the wazoo
<clever>
the flip must have happened before the write() syscall
<Ermine>
is this bitflip occured due to cosmic rays?
<clever>
given how rapidly memtest crashes, i dont think so
blockhead has joined #osdev
<Ermine>
well
<clever>
yeah, it could always be 2 different problems
<zid>
not suspecting the psu?
<clever>
zid: hadnt considered that option
<zid>
My cras issue is my psu
<clever>
of note, the machine has 4 sticks of ram, 8gig each, 2 different brands/models
<zid>
nice, my memory is in numa because the capacities don't even match
<zid>
I have 18GB of quad and 8GB of dual channel
<clever>
if i test each stick of ram seperately, 2 of them will show up as bad memory, then crash memtest
<clever>
the other 2 will just immediatly crash memtest
<clever>
with a mix of sticks, it just crashes immediately
<zid>
16GB*
<clever>
the fact that i can change the error by swapping ram around, implies the psu isnt to blame
<zid>
I had a dead FSB once that was fun
<zid>
certain patterns caused noise that would just always give the wrong results
<zid>
so in memtest it'd pass every single test up to 5 ish, then test 6ish would fail on every single access
<clever>
in the past, ive had memtest driving me to tinfoil hat levels of insanity :P
<clever>
my desktop was reporting bad ram at about 256mb in
<clever>
i swapped all of the hw with my nas, and oh, more bad ram
<clever>
then i began to spiral into insanity, i tested the ram on my laptop, its also bad, at the same addr
<clever>
then i had a friend over the internet test his ram, its also bad, at the same addr
<clever>
zid: would you be loosing your mind by this point? :P
<zid>
no but I might be losing it
<clever>
one option we briefly considered, was a bios rootkit
<zid>
heh yea that'd do it
<clever>
but there was such a wide variety of hw ages
<zid>
or just SMI overwriting bytes and memtest noticing it
<clever>
but no, it was something far simpler
<zid>
obviously
<zid>
but I'm saying it is actually plausible
<clever>
gcc hardening flags :P
<clever>
the build of memtest was defective, because the distro we used made hardening the default
<zid>
so you compiled your own memtest and broke it?
<clever>
the distro we used builds everything from source
<zid>
yea don't do that for memtest imo
<clever>
like gentoo, but with caches for shared build products
<bslsk05>
xorl.wordpress.com: The forgotten SUAVEEYEFUL FreeBSD software implant of the EQUATION GROUP | xorl %eax, %eax
<mrvn>
hehe, user complains 'fun1' isn't faster than 'fun2'. When you compile the test case in both cases the while test loop is optimized away. :)
blockhead has quit [Ping timeout: 240 seconds]
<mjg>
so technically they are correct
<mrvn>
but it's 3 times slower.
foudfou has quit [Remote host closed the connection]
foudfou has joined #osdev
Vercas has quit [Remote host closed the connection]
gildasio has quit [Remote host closed the connection]
Vercas has joined #osdev
mzxtuelkl has quit [Quit: Leaving]
foudfou has quit [Ping timeout: 268 seconds]
Vercas has quit [Ping timeout: 268 seconds]
foudfou has joined #osdev
Vercas has joined #osdev
ajr has joined #osdev
kof123 has quit [Ping timeout: 268 seconds]
gildasio has joined #osdev
chartreuse has quit [Ping timeout: 240 seconds]
foudfou has quit [Remote host closed the connection]
foudfou has joined #osdev
gildasio has quit [Remote host closed the connection]
gildasio has joined #osdev
gildasio has quit [Remote host closed the connection]
gildasio has joined #osdev
kingoffrance has joined #osdev
Dreg has joined #osdev
WaxCPU is now known as Andrew
gildasio has quit [Remote host closed the connection]
bauen1 has quit [Ping timeout: 245 seconds]
Andrew is now known as WaxCPU
foudfou has quit [Remote host closed the connection]
foudfou has joined #osdev
gildasio has joined #osdev
Vercas has quit [Remote host closed the connection]
gildasio has quit [Remote host closed the connection]
gildasio has joined #osdev
Vercas has joined #osdev
foudfou_ has joined #osdev
foudfou has quit [Remote host closed the connection]
bauen1 has joined #osdev
dude12312414 has quit [Quit: THE RAM IS TOO DAMN HIGH]
ajr has quit [Ping timeout: 244 seconds]
elderK has joined #osdev
GeDaMo has quit [Quit: There is as yet insufficient data for a meaningful answer.]
<heat>
slow ass day huh
ajr has joined #osdev
ajr has quit [Client Quit]
ajr has joined #osdev
<zid>
I could annoy you if you like
<clever>
heat: i could fill your chat logs with more rpi details!
jafarlihi has joined #osdev
<jafarlihi>
Hey guys. I'm trying to read CLRS book but I don't understand 95% of the maths. What book do I need to read first to understand those? I lost my maths somewhere in high school.
<jafarlihi>
I basically don't know shit about logarthims and all that
<zid>
*scrolling through it*
<jafarlihi>
What? It's full of maths notation
<zid>
Yea this is definitely a CS book
<zid>
found some logs, and all the questions are math
tsraoien has joined #osdev
<zid>
If you can get through it well done, khan academy or something can teach you the math I'd imagine
<zid>
khan*
<jafarlihi>
I need a book, I hate videos
<zid>
so skim a calc book
<zid>
(Personally I'd just not give a shit about the math)
nanovad has quit [Quit: ZNC 1.7.5+deb4 - https://znc.in]
nanovad has joined #osdev
<friedy>
The stuff in that book looks like a bunch of math, but it's actually not that bad. You could easily review all these things in a weekend. The proofs are a bit harder though, but you could just ignore those.
<zid>
I'd personally just use it as a reference to start implementing (and knowing the names of, so you can look up exteranl info on) various data structures
gildasio has quit [Remote host closed the connection]
<zid>
Once you've implemented them all you should have internalized a whole butt load about programming data structures regardless
gildasio has joined #osdev
<heat>
clever, talk dirty to me
<clever>
one min, lol
<jjuran>
There's nothing so dirty as underestimating latency ;-)
<mjg>
tail end latency == gg
<mjg>
often time
<mjg>
s
<mjg>
which reminds me i wanted to implement that stuff from gil tene for latency histograms
<mjg>
in the kernel
<mjg>
jafarlihi: if you want ot learn algos i recommend 'algorithm design manual' by skiena
<mjg>
cormen is notoriously shitty to read
<mjg>
i even got a cs prof admitting their pseudo code syntax causes eye bleeds
<heat>
my man finally ran out of rpi facts
<heat>
zid, annoy me pls
<zid>
heat: That makes a change, normally you're annoying everybody else by breathing
<heat>
shit now im sad
<heat>
no u
<zid>
Hope that was annoying
<heat>
kinda weak
<zid>
That's fine, I'll do it every day for a year, it'll add up
gildasio has quit [Remote host closed the connection]
gildasio has joined #osdev
nyah has quit [Ping timeout: 245 seconds]
<mrvn>
jafarlihi: what kind of math is in that book? O() notation?
<mrvn>
mjg: any recommendation for a book for algorithm for multi-core? And I mean multi-core, not multi-threaded.
<mrvn>
shitty to read? But it's "Clearer, more personal, and gender-neutral writing style"
<mrvn>
and now in color[tm]
<mrvn>
Don't you all hate it when your algorithms are male?
<mrvn>
Alice is now Alex, Bob is Bibbi and Mallory is Mata Hari.
<bslsk05>
mirrors.edge.kernel.org: Is Parallel Programming Hard, And, If So, What Can You Do About It?
<zid>
Give up
<heat>
use a single core
<gog`>
don't program
dude12312414 has joined #osdev
<heat>
+1
<heat>
become monke
<zid>
It's one of those problems that's a problem because it's a problem, if that makes sense.
<zid>
The reason people fuck it up is because it's fuck-uppable
<zid>
As in, *describing what you want to do at all* is the hard part
<mrvn>
mjg: Just from the Contents that looks awfully multi-threaded.
<zid>
once you're capable of doing that, it's just writing it down.
<zid>
Same for most other 'hard' parts of programming.
<heat>
slap a good old lock around everything
<heat>
tried and true
<zid>
I know the language, it's just a question of can I describe it succinctly enough for it to function as a program.
<zid>
Like how there was a broken proof for the four-colour theorem for 80 years or whatever
<zid>
It might seem obvious what you're trying to write down, but the devil is in the details, and non-functional math proofs are akin to locking bugs
<heat>
aw the author of that pdf is the rcu guy
<heat>
downvote
<mjg>
wut
<heat>
Paul E. McKenney
<Griwes>
Paul's fine
<heat>
i don't care if its his fault or not but rcu is a patent mess and the non-linux world is 20 years behind on concurrency for that reason
<zid>
rcu = read, cry, update
<heat>
sorry, non-GPLv2 world
<heat>
only freebsd got something comparable in 2020
<mjg>
that's a half truth
<j`ey>
heat: wait wut, rcu is patented?
<Griwes>
was
<heat>
j`ey, yes, heavily so
<mjg>
there have been other ideas like epoch based reclamation
<heat>
there are still a lot of patents on RCU
<mjg>
j`ey: rcu was patented into oblivion
<j`ey>
heat: oh
<Griwes>
haven't all the major ones expired?
<zid>
so how did that work, linux had a free licence?
<mjg>
i don't know what is not expired yet, i'm guessing most of the work past 128 cores
<mjg>
zid: ye, afair ibm handed it
<heat>
mjg, there have, but they only reached mainline OSes with freebsd in 2020 AFAIK
<mjg>
note mckenney worked for ibm at the time
<mjg>
well ebr happened to be used in 2018 or 2019
<mjg>
2020 is when a homegrown code got implemented
<heat>
RCU allegedly was licensed to GPL and LGPL projects
<mjg>
even then netbsd(!) had provisional rcu-like solution for at least 10 years by now afair
<heat>
which is why liburcu
<mjg>
as in sticking to by that time expired patents
<mjg>
it sucked hard though as you can imagine
<zid>
I don't believe in software patents though so it's fine
<Griwes>
RCU started being proposed for an ISO standard in 2017 I believe, which, given the rights given by publishing things for ISO, I think would've made enforcing the patents hard at that point already?
<zid>
I found someone in the wild defending TTC the other day, I was blown away
<mrvn>
zid: Would you sue Linux? Apart from the public relations nightmare who would you even sue? The web mirrors hosting it?
<mjg>
Griwes: i don't know anything about that
[itchyjunk] has joined #osdev
<Griwes>
there's WG21 papers from Paul that trace back to at least 2017
<mjg>
zid: personally i don't like patents either, but that's te world we live in
<zid>
It's not the world I live in
<mjg>
zid: or at least you do, afaik in europe they are not recognized:)
<mjg>
i thought you are in the us?
<heat>
Griwes, I don't know
<mrvn>
patents are fine. It's the (abuse of the) patent system that's problem
<zid>
god no
<heat>
plenty of RCU patents have *not* expired though
<Griwes>
patents stifle innovation
<Griwes>
that much is clear
<mjg>
oh wait, you were talking shit about scotland
<mjg>
so UK i presume
<zid>
That makes me english
<zid>
de facto
<mrvn>
Griwes: they also finance innovation
<heat>
like everything that makes linux's RCU fast
<zid>
americans *love* scotland
<Griwes>
mrvn, they are primarily used to stifle innovation
<zid>
they're 1/128th scottish don't you know, they practically piss haggis
<mrvn>
Griwes: in IT, yes.
<Griwes>
outside of IT too
<mrvn>
medicin, yes
<Griwes>
pharma patents are the same kind of hell
<zid>
SO for those that don't know about TTC
<Griwes>
tbh I can't think of an industry where patents have a positive impact
<mrvn>
The problem with pharma patents is that they re-patent things over and over with slightly different use cases. So they never expire.
<zid>
They have a dubious software patent on the mechanics of tetris
<mrvn>
well, the biggest problem.
<zid>
and their entire business model is to sue people who implement tetris
<Griwes>
mrvn, nah, that's at best the second biggest problem. the first biggest problem is that it's decreasing the supply of new, improved drugs that are needed by people
<mrvn>
zid: should get thrown out of pretty much every court because sueing just one of 10000 tetris implementations is vindictive. They would have to sue all.
<zid>
pharma patents are at least semi useful, given how much drug development actually costs. but naturally because you're allowed 'pharma patents' what they end up patenting is 'everything'
<mrvn>
Griwes: how so? The only reason you get the new drugs is because firms spend money on research in the hope of getting it pack from licensing and high prices.
<zid>
and not just some specific high-end cancer drug they want production exclusivity to in exchange for having researched
<Griwes>
mrvn, right, which is why we need to dismantle not just patents but also capitalism
<Griwes>
but we're going off on a tangent ;p
<mrvn>
The problem I see is more that year old drigs that have been paid for over and over are still patented and nobody can produce them for cheap.
jafarlihi has quit [Quit: WeeChat 3.0]
<mrvn>
Patents should also expire when the product isn't produced anymore.
<mrvn>
or when obsoleted by the next generation. Think about what that would mean for e.g. RCU. Everyone could use last generation RCUs.
<mjg>
to be fair though, even today most contemporary computing can get away ok performance wise without rcu
<mjg>
i mean people don't have super high core counts
<mjg>
in literal hundreds
<mjg>
apart from fringe cases really
<mjg>
and for that scale other trickery tends to be sufficient (but still slower of course)
<gog`>
read-cinematic-universe
<mrvn>
mjg: I always have a problem with user defined time limit before things expire in the RCU.
<mrvn>
You should be setting the upper bounds to the maximum time an object can be referenced. So take that operation that takes the longest and now assume every single of your 1024 cores operates on the same object. How many seconds is that going to take with the cachelines bouncing around 1024 cores like crazy?
<mrvn>
And then 1023 will fail and have to redo it all over again?
<zid>
never a frown with golden brown
<mjg>
i don't what's going on in userspace rcu
<mjg>
i only ever did kernel stuff with it and as you can imagine that one does not have the problem :-P
<mrvn>
mjg: in user space you can't really do RCU like intended. You don't have an upper bound. A thread might be frozen for a long long time.
<mjg>
although i have to note, with how NOT scalable pthread mutexes are even on linux, i can only feel sad for people who have to deal with this crap
<mjg>
ye, that's my point
<mjg>
although there is preemptible rcu in the kernel, i don't know what kind of upper bound that places on stuff
<mrvn>
mjg: how does that help? The problem is one thread deleting an object. How long do you have to wait before you can free it? Every thread has to run at least once before you can think about it,.
<mjg>
well if *preemptible* rcu is employed, it still has the problem of making sure all threads gtfo'd
<mjg>
similar to userspace
<zid>
ya'll are boring
<zid>
can we talk about how amazing tomboys are or something
<mjg>
the regular rcu can of course only make sure all cpus gtfod, which is easy
<mrvn>
You need a wait-free algorithm to use RCU
<mjg>
zid: i knew a tomboy, she liked bsd
<zid>
I don't think you should just randomly tell people about your friends sexual habits
<mrvn>
mjg: how do you make sure of that?
<mjg>
mrvn: i'm saying that preemptible rcu allows for the thread in a rcu section to go off cpu any time, just as if it was userspace at least in this regard
<mjg>
mrvn: it is plausible though there is a deadline for preemption which makes it for a different problem at the end of the day though
<mrvn>
mjg: yeah, preemtible RCU seems to make things worse, not better
<mrvn>
But even in the classic RCU how do you make sure all readers are done with an object?
<mjg>
with classic that's conceptually easy
<mjg>
by api contract you can't go off cpu while in a rcu section
<mrvn>
and?
<mjg>
so there is a generation count of osme sort and you observe all cpus gtfo'd before actually freeing the object
<mrvn>
that does so not work. The generation count would make a cachline bounce between all cores and be the bottleneck.
<mjg>
that is fixed from perf standpoint
<mjg>
rcu uses a tree
<mjg>
where the count propagates
<mjg>
that's apparently good enough for the 1024 core boxen
<mjg>
i don't know the details
<mrvn>
The design described on wikipedia just waits a fixed time. All operations have an upper bounds so after N ticks you know all other cores have progressed past the object.
gildasio has quit [Ping timeout: 268 seconds]
<mjg>
that sounds like another mechanism which used rdsctp to mark in an out times
<mrvn>
I don't know of any other way for the RCU to work that doesn't create contentions, in which case you can just use locks / refcount on the objects.
<mjg>
oHow can the updater tell when a grace period has completed if the RCU readers give no indication when they are done?
<mjg>
> if we remove an item from a linked list, and then wait until all CPUs have switched context, executed in user mode, or executed in the idle loop, we can safely free up that item.
<mjg>
as noted above, there is a shared state propagated in form of a tree. you can look for mckenney's talks or articles for more details
<mjg>
last time i briefly looked into it was 5 years ago
<mjg>
and apparently the above is fast enough for boxes bigger than vast majority ofp eople have
<mjg>
of what people have*
<mjg>
but i'm also pretty sure that's patented
<mrvn>
mjg: sound slike you do: for (int i = 1; i < num_cpus; ++i) { while (cpu[(MY_ID + i) % num_cpus].ring != RING_USER) { pause(); }}
<mjg>
i was starting to play with my own approach, the basic idea was to delegate a dedicated line for each cpu *and* a dedicated area in each numa node
<mjg>
then for example every 4th cpu thread would collect it's own state + 3 others it wsa responsible for and plop it into the staged per-node area
<mjg>
once a tick or maybe every other tick
<mrvn>
mjg: I'm not sure an RCU is that usefull at all. The premise is that you have many readers all working in parallel and few writes that should atomically update objects, right?
<mjg>
not ideal by any means, but given all the atomic ops saved by code using this to avoid locking to begin with i would expect a great win
<mjg>
yea
<heat>
it's very useful
<mjg>
it is very useful and i can show you example numbers
<mjg>
lemme find it
<heat>
where's that freebsd example of the epoch speedup?
<mjg>
here is freebsd building the linux kernel, incremental -j 104
<mjg>
make -s -j 104 bzImage 142.96s user 1025.63s system 4924% cpu 23.731 total
<bslsk05>
freebsdfoundation.org: 5x if_bridge Performance Improvement | FreeBSD Foundation
<mjg>
and here it is after i applied smr (rcu-like) for path lookup
<heat>
by using epoch, which is an rcu-like mechanism
<mjg>
157.37s user 172.31s system 2457% cpu 13.414 total
<mjg>
note system time collapsed
<mrvn>
heat: how much faster would it be if you bind a bridge to a core, or a small set of cores?
chartreuse has joined #osdev
<heat>
linux completely avoids contention on the fd table by using rcu there as well
<heat>
you can have 5000 threads banging the fd table, doesn't matter
<heat>
it Just Works(tm)
<mjg>
it really does not
<mrvn>
heat: as long as they only read
<mjg>
apart from fd lookup
<mjg>
that's a really bad example man
<mjg>
see path lookup for a good one
<mjg>
numbers provided above
<heat>
no it's not
<heat>
it works fine
<heat>
also yeah, path lookups in linux are also entirely RCU'd in the best case
<mjg>
fd alloc + free are TERRIBLE
<mrvn>
mjg: path lookup uses reference counting last I checked
<heat>
mjg, talked about lookups, not allocate + free
<heat>
it falls back to ref counting and locks in the worst case
<mjg>
heat: i know, real world code is doing enough alloc + free that there is a huge problem
<heat>
best case it does a fully lockless, refcless RCU lookup
<mjg>
no
<heat>
yes
<heat>
i've looked at dentry code enough to know this
<mjg>
fast path lookup is doing write-free all the way up to the last component
<mjg>
so lookups of foo/bar/baz and foo/bar/qux indeed scale perfectly
<mrvn>
heat: why does an FD lookup needs an RCU? isn't that a straight filep going directly to the file structure?
<mjg>
but 2 lookups of foo/bar/baz don't
<mjg>
as they ping pong baz
<mjg>
is what i mean
<heat>
mrvn, the file descriptor table can change, the file descriptors can be freed
<heat>
mjg, ok you win
<heat>
i'm not that intimately familiar with that lookup :)
<mrvn>
heat: only if nobody has the FD open anymore, which the refcount will tell you
<mjg>
so some freebsd numbers, before rcu-like and after
<mjg>
incremental -j 104 bzImage on tmpfs:
<mjg>
before: 142.96s user 1025.63s system 4924% cpu 23.731 total
<mjg>
after: 147.36s user 313.40s system 3216% cpu 14.326 total
<mjg>
<mjg>
Sample microbenchmark: access calls to separate files in /tmpfs, 104 workers, ops/s:
<mjg>
before: 2165816
<mjg>
after: 151216530
<mjg>
the last one is the real killer here
<mjg>
note while the last one is a microbenchmark, the first set of numbers comes from a real workload
<mjg>
and the improvement is quite drastic i would say
<mrvn>
mjg: That only tells you that your rcu was bad though
<mjg>
what?
<heat>
yeah what
<heat>
before = no rcu
<heat>
after = rcu-like
<mrvn>
mjg: you replaced the RCU with SMR, right?
<heat>
no
<mrvn>
what was it before SMR?
<heat>
nothing
<mjg>
locked lookup
<mjg>
read-write locks for hash chains and for each traversed vnode
<heat>
as I said, other operating systems were stopped from using RCU beacause of p a t e n t s
<mjg>
and of course ref counting on it
<mrvn>
fully locked or read-write lock? With or without atime updates?
<mrvn>
If you do atime updates then I think you can throw away your RCU or SMR. every access is a write, that's death for copy-modify-update.
<mjg>
no atime updates during lookup
<mjg>
do the numbers above suggest to you there is tons of contention during lookup in the 'after' caes?
<mrvn>
mjg: I have no idea how your filesystem is mounted. Default nowadays is usually to only update atimes the first time and then once in a blue moon.
<mjg>
it's all fully scalable as long as the terminal path component is different
* zid
prods the channel to do something interesting
<zid>
It keeps hilighting itself and I keep tabbing back to it
<zid>
but it's not getting better
<zid>
like when you're hungry and you check the fridge, decide it's all terrible, then check again in 5 minutes
<zid>
in case anything changed
<mjg>
is not that when you browse reddit?
<mrvn>
zid: hey, the mold could have grown into something edible.
<zid>
I'm actually reading a book I'm just incredibly distractable
<mjg>
you get bored of it so you close the tab, only to open immediatley after
<mrvn>
mjg: For path walks what do you think of having a per-process/thread path cache?
<mjg>
depends what do you mean by path cache
<mjg>
in freebsd i have a copy-on-write struct which refs the root vnode and current working directory
<mjg>
so i can always safely access that without dirtying squat
<mjg>
as for trying to cache actual lookups i don't think that's useful
<mjg>
what would be useful a full rcu lookup which returns an entry while still within a rcu section
<mjg>
while you can't service open() like that, you can sort out things like stat or access
<mjg>
in fact i had early code to do it
<mjg>
did not get around to productizing the thing
<mrvn>
mjg: during lookup build a copy of the path nodes in the process space with links to the kernel structures (so deletion of nodes tells the process) and then repeated lookups in the process can just use the cached values.
<mjg>
(and rest assured both stat and access get called A LOT)
<mjg>
ye that's what i got from your message, i don't think that buys much
<mjg>
from my experience processes don't tend to reopen the same stuff at high enough frequency to maek this work
<mjg>
what they do a lot is traversing the same dirs, but that already scales with rcu
<mrvn>
mjg: /usr/include/* /home/you/src/kernel/?
<mrvn>
I don't see gcc using openat(include_dir_fd, path)
<mrvn>
how often in that kernel compile does gcc open config.h?
Vercas has quit [Remote host closed the connection]
foudfou_ has quit [Remote host closed the connection]
<mjg>
i don't remember that stuff now, i can check tomorrow if you prod me around 20 CEST
foudfou has joined #osdev
<mjg>
what i do remember is tons of stat calls
<mjg>
which would be sorted out with the patch mentioned above
Vercas has joined #osdev
<mrvn>
mjg: The aim would be that "/usr/include/foo.h" caches the dir and then "/usr/include/bla.h" only walks the last step.
<mjg>
but with rcu-employing lookups traversal of /usr/include does not run into scalability issues
<mjg>
there is literally 0 writes to anything but the last path component, in your examples foo.h and bla.h
<mrvn>
still 2 directory lookups and permission checks. And many paths are longer.
<mrvn>
no write to foo.h or bla.h either.
<mjg>
this one is also mostly circumvented -- the hack is to note that most dirs have effectively +x for everyone
<mjg>
and then you can skip any machinery for perm checks
<mjg>
unless you have LSM modules which fuck with it