<heat>
it seems that the normal behavior is to just suspend ticking when idle
<heat>
there's also a switch to suspend ticking when there is only a single active task on the CPU (core)
<heat>
apart from that, the scheduler always keeps on ticking
<geist>
yeah that's not an awful strategy. idea is if the cpu is running, taking monotonic ticks isn't that big of a deal
<geist>
its continually waking up an idle core that's a waste of power
<geist>
re: LK and periodic timers. it's surprising. i've worked with one project with a cortex-m3 where we were trying to use the absolute least amount of power
<geist>
and i remember being genuinely surpriosed that the periodic tick used less power
<geist>
basically there were 3 states of the system: off (no cpu running at all, just some few hundred bytes in retention), and two substates of running (cpu in wfi, and cpu actually running) and the latter ran with periodic
<heat>
geist, isn't fuchsia fully tickless?
<heat>
or am I misremembering it?
<geist>
the idea was to get to the off state as soon as possibke, and we determined that the overhead of continually recomputing the dynamic timer was actually worse
<geist>
since on that particular cpu the total overhead of a periodic tick with nothing to do was like tens of cycles
<geist>
yah it's tickless
<heat>
how do you do this sort of cpu time measuring then?
<geist>
building fully tickless systems is easiler if you never baked into the core that there are ticks somewhere
<heat>
currently I'm paying the cost of a lfence;rdtsc each context switch
<geist>
re: cpu time measuring in the embedded case? a scope and a power meter
<geist>
were actually watching the instantaneous current draw with a scope
<heat>
and this doesn't work for the timer itself, aka getting a signal
<heat>
i mean in fuchsia, not embedded
<geist>
can you rephrase your question?
<geist>
i'm not sure i precisely understand what you're asking
<heat>
how do you measure cpu time in fuchsia
<geist>
TSC/arm counter
<heat>
and how do you keep cpu timers (as in a timer that measures cpu time and can fire an event/signal/wtv)
<geist>
local apic/arm
<geist>
wait i'm not sure i'm answering what you're asking
<geist>
i'm not sure what you mean precisley by 'cpu timers'
<heat>
not this last question, no
<heat>
ITIMER_PROF or ITIMER_VIRTUAL
<geist>
i dont now what those are
<heat>
they measure on-cpu time
<geist>
ah
<heat>
you can use setitimer(ITIMER_PROF, 2secs) and you get a signal after 2s of process cpu time
<geist>
ah. no we dont have anything like that
<geist>
seems like it'd be a driver triggering an event, doable, but i have no idea what the hardware deps are there
<heat>
i was also warned by mjg that doing rdtsc on every ctx switch is super expensive in a vm
<geist>
it is
<geist>
well, depends o if it trips a vmexit or not
<heat>
yeah
<geist>
but it's a huge complicated matrix of aww-fuggit
<heat>
i tested and qemu+kvm seems to
<geist>
as in, you can choose to get fixated on it, or just shrug and move on
Vercas6 has quit [Quit: Ping timeout (120 seconds)]
<geist>
i think we just generally try to avoid tsc as much as we can, but not getting too fixated
<geist>
since running optimally in qemu isn't a first priority
<heat>
i don't quite understand why they would want to vmexit on tsc reads
<geist>
i think it's highly dependent on the host hardware. notably does it have built in TSC scaling
<geist>
ie, if it ticks at host frequency, then it's 'giving away' a detail of the host hardware
<geist>
and disallows you to migrate it to another host that ticks at a different frequency
<geist>
usually 'migratable=off' or something is a switch you can set on the cpu arg to disable ab unch of this sort of stuff
<heat>
ohhhhhhhhhhhhhhhhhh
<heat>
thats what that does?
<geist>
some of thenewer/higher end intel and amd machines i think have TSC hardware scaling, where you can in the host set the tsc tick rate, effectively
<heat>
I didn't test the "does it vmexit" thing with it
<geist>
i think so yeah. we generally set that switch on our qemu scripts because of a bunch of basically forgotten reasons
<geist>
i think that's one of them
<pbx>
hmm. figured out why my job control and signals were not working
<geist>
may have something to do with whether or not qemu exposes invariant/constant TSC and/or if it exposes the tsc-deadline lapic timer
<geist>
generally migratable=off gets you more host bits
<pbx>
typoed my code and was handling per thread signals twice instead of handling both thread and process signals
Vercas6 has joined #osdev
<geist>
oops!
<heat>
i schedule process signals to a single thread
<heat>
chosen fairly by dice roll (read: i pick the first that doesn't have it masked at the time)
<heat>
i don't know if its fully POSIX compliant but idc
<pbx>
heat: that seems problematic for SIGSTOP etc
<geist>
reminds me does posix have a notion of a 'main thread' of a process?
<heat>
no
<geist>
ie, if this one exits the whole process goes down, etc?
<heat>
hrm
<heat>
i guess?
<geist>
i dunno, just curious
<heat>
i think posix just defines that exit() kills all the threads
<geist>
if there was a main thread concept it could be where signals go in some cases, etc
<pbx>
heat: you want ctrl-z to stop all your threads, right
<heat>
pbx, oh yeah, ctrl-z is handled completely separate
<heat>
STOP and CONT are special bois
<heat>
same for thread KILL too
<heat>
and it's how it seems to be done for other unixes
* pbx
isn't exposing threads to user code yet, but because of future proofing the kernel has them (and uses them for things like networking, idle task etc)
<heat>
your networking uses threads?
<heat>
for what?
<pbx>
deferred input handling. don't wanna go through the whole stack in interrupt level
<pbx>
wanted something more portable than BSD's VAX/68K IPL stuff
<heat>
softirqsssss
<heat>
geist, yeah posix doesn't seem to define "main thread"
<geist>
okay
<pbx>
heat: sure, but i'm not a linux dev, right :) let me stumble around and try to invent my own crappy way of doing things
<pbx>
whats the joy in building a 1:10 scale model of linux ?
<heat>
idk i would never do that haha
* heat
runs
* pbx
's mind might also have been poisoned by years of java backend dev work
<heat>
freebsd's spl seems to be analogous to softirqs, but vaxier
zaquest has quit [Remote host closed the connection]
zaquest has joined #osdev
<heat>
one day i'll say fuck it and implement lwp/thread groups
epony has quit [Ping timeout: 252 seconds]
<heat>
for fun and horrifying people
<heat>
what's a thread if not a weird process?
<heat>
now im curious, when did threads come up in unix?
<pbx>
heat: tricky q
nyah has quit [Ping timeout: 248 seconds]
<pbx>
anywhere between long ago and never could be a correct answer
<heat>
the syscall returned one word after the sys instruction
<heat>
for the old proc, that is
<zid>
call fork; jmp parent
<CompanionCube>
heat: re threads: 'Some Unix flavors (Solaris and UnixWare) also have an older API to their native threads. This API is called Unix International (UI) threads, and uses the thr_ prefix.'
<zid>
Some old, deprecated, slow, bad unix flavours, like solaris*
epony has joined #osdev
<pbx>
in today's offtopic wtf: how on earth is paypal, of all things, serving a revoked TLS cert
<zid>
your ISP is mitming you
<zid>
which is why it is revoked
<CompanionCube>
zid: if they're gonna bother mitming why bother obtaining a certificate that's gonna get revoked
<zid>
I don't understand, that isn't what's happening (not that it's actually happening)
<pbx>
idk, while it is possible that they leaked their keys and someone is mitming me, i think it's rather unlikely, especially as others are noticing this as well
<heat>
CompanionCube, interesting!
<CompanionCube>
there's 'deprecated deprecated' and 'removed from gcc deprecated'
<heat>
the oldest threads in unix seem to have come from sunos
Terlisimo has quit [Quit: Connection reset by beer]
Terlisimo has joined #osdev
<tsandstr>
I am dipping my toes into the world of osdev for the first time, and there is a general theme that I am having a hard time understanding. I just finished reading the Multiboot2 spec, and I threw together a Multiboot2 header with a few different tags--nothing fancy--and then used GRUB to boot it up in QEMU and used gdb to poke aound memory and confirmed that I am getting the information that I expect from the boot lo
<tsandstr>
ader. Great! Next steps should ar ethat I want to set up a GDT and an IDT and get paging going. But what I am confused is that I cannot seem to figure out. Where should I put these tables in memory? Am I supposed to be parsing the memory map provided by EFI, finding a random block of unused memory, and just throwing them there?
<tsandstr>
Are there any good resources explaining this aspect of system initialization?
Vercas6 has quit [Quit: Ping timeout (120 seconds)]
kindofwonderful has joined #osdev
Vercas6 has joined #osdev
<kof123>
"zid: new vegas seems exactly like living in pheonix or austin or something rn" i have a weather-related theory there was a reason ID made doom in texas
<Mutabah>
tsandstr: Thats the right way
<Mutabah>
But, you can also reserve space in your binary
<Mutabah>
For the early critical data structures (GDT/IDT and the initial page tables) this is the best/easiest
<Mutabah>
The bootloader puts you somewhere there's space, and you just use the loaded location
<tsandstr>
Mutabah: What does that look like? Could I, for example, just create a new section called .gdt and reserve a bunch of bytes there and then fill it in from my kernel?
<tsandstr>
Mutabah: And in general, how do I decide what should go in the binary like this? At what point do I start grabbing memory outside of the binary and using it?
<bslsk05>
github.com: rust_os/start.asm at master · thepowersgang/rust_os · GitHub
<Mutabah>
I wait until dynamic allocation is possible and needed
<Mutabah>
So anything that needs to be present for the dynamic allocation code to run is statically allocated
<tsandstr>
So, when I am creating page tables and stuff, wold you typically set up the first page tables this way too?
<tsandstr>
I'm sorry if these are stupid questions btw
<Mutabah>
Not stupid questions, it's a bit of a chicken&egg problem
<Mutabah>
Initial page tables are also statically allocated
<Mutabah>
(They're actually a few lines above in that same file)
<Mutabah>
Although - this is just one way - You could also have your allocator backed off a statically allocated set of data
<tsandstr>
But then once I have paging up, in theory I can try to set up some primtive dynamic memory access and then I can dynamically allocate space for future page tables. Does this sound right?
<Mutabah>
but - I prefer having all memory set up by the time I enter HLL code
<Mutabah>
Yep - once you have a basic setup, you can use the more complex logic that can allocate pages from the memory map
<Mutabah>
(and from its own internal strutures: e.g. a bitmap or free list)
<zid>
yea you just need a bootstrap set
<tsandstr>
Mutabah: Thank you so much for explaining this! The example code you sent is quite helpful. How about your TSS segment descriptor in your GDT? Does it have a hardcoded address?
<zid>
My bootstrap code just does "find a small region of contiguous memory, set free_page there" "new allocations do free_page += 4096 and return the old value" then "mmap all of the memory we have used so far" "switch to those page tables"
<Mutabah>
tsandstr: Dynamic address - set by HLL code later on
<Mutabah>
the TSS isn't needed for that early bringup, so can be deferred until just before user code starts running
<Mutabah>
Well... actually - the TSS has a static address - but that's mostly laziness
<tsandstr>
Mutabah: aha I see. The TSS was actually the first time that I ran into trouble with this, so it's good to know that I can delay setting it up until I am tring to get ring 3 stuff going
<zid>
nod, TSS isn't needed until you need to reload rsp to service an irq
<Mutabah>
The TSS is only ever used on ring transitions
<tsandstr>
Mutabah: That is what my reading was starting to suggest
<Mutabah>
(or an interrupt with IST set)
<zid>
so not until you have ring3 code and interrupts enabled
<tsandstr>
makes sense!
<zid>
neithr of which are technically mandatory
<zid>
are you doing x86 or amd64?
<kindofwonderful>
now
<kindofwonderful>
what is TSS
<kindofwonderful>
HLL
<kindofwonderful>
IST
<Mutabah>
TSS = Task State Segment (an x86/x86-64 thin)
<Mutabah>
HLL = High-level language
<zid>
should I get you a bucket, that spoon seems a little small
<Mutabah>
IST = Interrupt Stack Table (an x86-64 thing)
<kindofwonderful>
zid: mocking me, as always\
<Mutabah>
They should have been googlable, with a little bit of effort
<kindofwonderful>
at least i heared about IRQ before
<kindofwonderful>
Mutabah: they don
<kindofwonderful>
t
<Mutabah>
"TSS x86"
<kindofwonderful>
yeah i tried TSS code
<kindofwonderful>
and i couldnt' even infer that the topic is x86
<kindofwonderful>
im not a COMPLETE dumb, common
<kindofwonderful>
:)
<Mutabah>
It was mentioned right above your query
<Mutabah>
Although... it wasn't mentioned before, just inferred (grub is an x86 bootloader)
<kindofwonderful>
yeah at that time i was typing/frustrating from google result
<kindofwonderful>
Mutabah: anyway thanks for the info
<kindofwonderful>
did you noticed i didn't ask what is IRQ ? :)
<kindofwonderful>
Mutabah: i appreciate your help
<kindofwonderful>
IST = interrupt service table ?
<Mutabah>
Interrupt _Stack_ Table (... I think, working of memory)
<tsandstr>
Mutabah: osdev wiki suggests that Stack is correct
scoobydoob has joined #osdev
scoobydoo has quit [Ping timeout: 252 seconds]
scoobydoob is now known as scoobydoo
<zid>
tsandstr: are you doing x86 or amd64?
<tsandstr>
zid: amd64
<zid>
good good
<tsandstr>
zid: really just playing around for now though. I'm in the middle of my first semester of senior year at uni, so I really dont have time for any of this right now, but its great prorastinating material ;)
<Mutabah>
That it is :)
<kindofwonderful>
You guys and your universities ...
elderK has quit [Quit: Connection closed for inactivity]
<kindofwonderful>
no im not jelous
IRChatter has joined #osdev
scoobydoob has joined #osdev
scoobydoo has quit [Ping timeout: 260 seconds]
scoobydoob is now known as scoobydoo
scoobydoo has quit [Ping timeout: 264 seconds]
Burgundy has quit [Ping timeout: 268 seconds]
scoobydoo has joined #osdev
garineko has joined #osdev
scoobydoob has joined #osdev
xenos1984 has quit [Read error: Connection reset by peer]
scoobydoo has quit [Ping timeout: 268 seconds]
scoobydoob is now known as scoobydoo
garineko has quit [Changing host]
garineko has joined #osdev
scoobydoo has quit [Ping timeout: 268 seconds]
scoobydoo has joined #osdev
\Test_User has quit [Quit: \Test_User]
heat has quit [Ping timeout: 260 seconds]
garineko has quit []
garineko has joined #osdev
xenos1984 has joined #osdev
\Test_User has joined #osdev
Starfoxxes has quit [Ping timeout: 252 seconds]
Starfoxxes has joined #osdev
wand has quit [Ping timeout: 258 seconds]
wand has joined #osdev
wand_ has joined #osdev
wand has quit [Ping timeout: 258 seconds]
scoobydoob has joined #osdev
scoobydoo has quit [Ping timeout: 268 seconds]
scoobydoob is now known as scoobydoo
elastic_dog has quit [Killed (osmium.libera.chat (Nickname regained by services))]
elastic_dog has joined #osdev
scoobydoob has joined #osdev
scoobydoo has quit [Ping timeout: 268 seconds]
scoobydoob is now known as scoobydoo
sikkiladho has joined #osdev
wand_ has quit [Ping timeout: 258 seconds]
vinleod has joined #osdev
vdamewood has quit [Read error: Connection reset by peer]
wand has joined #osdev
genpaku has quit [Remote host closed the connection]
genpaku has joined #osdev
ptrc has quit [Ping timeout: 264 seconds]
__xor has joined #osdev
Vercas6 has quit [Quit: Ping timeout (120 seconds)]
ptrc has joined #osdev
nyah has joined #osdev
Vercas6 has joined #osdev
DonRichie has quit [Ping timeout: 246 seconds]
IRChatter0 has joined #osdev
Vercas6 has quit [Remote host closed the connection]
Vercas6 has joined #osdev
IRChatter has quit [Ping timeout: 248 seconds]
IRChatter0 is now known as IRChatter
Burgundy has joined #osdev
DonRichie has joined #osdev
DonRichie has quit [Remote host closed the connection]
DonRichie has joined #osdev
Yaku has joined #osdev
scoobydoob has joined #osdev
scoobydoo has quit [Ping timeout: 260 seconds]
scoobydoob is now known as scoobydoo
GeDaMo has joined #osdev
y0m0n has joined #osdev
Burgundy has quit [Ping timeout: 264 seconds]
sikkiladho has quit [Quit: Connection closed for inactivity]
xenos1984 has quit [Read error: Connection reset by peer]
freakazoid332 has quit [Ping timeout: 264 seconds]
<pitust>
why not udev?
<ddevault>
wut
<pitust>
udev for driver loading, for a hobby os
<ddevault>
my OS is not linux
<pitust>
udev is portableish
<ddevault>
nor is it unix
<ddevault>
it's a microkernel
<pitust>
managarm is a microkernel, and they use udev
<pitust>
afaik
<ddevault>
managarm is a unix
<ddevault>
good for them
Burgundy has joined #osdev
<zxrom>
What licensing or patent restrictions exist for independent commercial OS developers?
zxrom has quit [Ping timeout: 246 seconds]
vinleod is now known as vdamewood
dude12312414 has joined #osdev
<ddevault>
hrm, why am I not getting serial interrupts
<ddevault>
enable ETBEI on IER
<ddevault>
wait for IRQ 4
<ddevault>
nothing
<ddevault>
FIFO is enabled
zxrom has joined #osdev
dude12312414 has quit [Quit: THE RAM IS TOO DAMN HIGH]
<mjg>
can you use after shave if you don't shave? :thinkingface:
<ddevault>
got it, though I don't understand why
<ddevault>
sent an EOI before starting my driver
<ddevault>
¯\_(ツ)_/¯
<mjg>
ini?
<mjg>
not unix, it is however a microkernel
<mjg>
so you are windows?
<mjg>
:]
<mjg>
not judging!
<ddevault>
ini is nice and simple
<ddevault>
¯\_(ツ)_/¯
<mjg>
i'm in a trolley mood, ignore :)
nur has joined #osdev
srjek has joined #osdev
kindofwonderful has quit [Ping timeout: 260 seconds]
<mrvn>
ddevault: did you EOI it?
<ddevault>
I was EOI'ing it before
<ddevault>
but I added an EOI prior to any other initialization and it "fixed" it
<ddevault>
not sure why an interrupt was already pending
<mrvn>
ddevault: the device likely had an interrupt at boot or was in an undefined state. So you have to EOI to clear it.
<ddevault>
well
<ddevault>
aight
wand has quit [Remote host closed the connection]
wand has joined #osdev
IRChatter8 has joined #osdev
IRChatter has quit [Ping timeout: 268 seconds]
IRChatter8 is now known as IRChatter
axis9 has joined #osdev
k0valski1889 has joined #osdev
Ram-Z has quit [Ping timeout: 250 seconds]
Yaku has quit [Ping timeout: 248 seconds]
isaacwoods has joined #osdev
axis9 has quit [Remote host closed the connection]
axis9 has joined #osdev
frkzoid has joined #osdev
rurtty has joined #osdev
joe9 has quit [Ping timeout: 252 seconds]
Ram-Z has joined #osdev
xenos1984 has quit [Ping timeout: 264 seconds]
xenos1984 has joined #osdev
axis9 has quit [Remote host closed the connection]
puck has quit [Excess Flood]
puck has joined #osdev
netbsduser has joined #osdev
<netbsduser>
good evening to all osdevers. i have two questions this evening: 1. does anyone know good books on the internals of OS/400, aka IBM i? and 2. has anyone experimented with a single-level store in their projects? i am keen to introduce this to my bare-metal smalltalk-like environment, but there are hard problems
rurtty has quit [Quit: Leaving]
<netbsduser>
the problems include the fact that every object ought to be persisted, yet many objects are extremely transient, and it would be a certifiable offence to put them all to disk. so some kind of minor collection before checkpointing to disk is utterly imperative
<netbsduser>
i have some vague notions of the minor collection looking like this: have a 'is only referenced by incore objects' set in new objects; as soon as an object which references that object is put to disk, unset that bit. minor collections then scan only core, and treat any object without that bit set as a root
joe9 has joined #osdev
xenos1984 has quit [Ping timeout: 260 seconds]
vdamewood has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
joe9 has quit [Quit: leaving]
<sham1>
You could treat the persistent storage as yet another place to store your objects during a minor collection in, say, a generational collector
<sham1>
Some kind of a copying collector would probably also be of interest for storing any long-lived, later generation data for when a major collection happens. Generational hypothesis and all that
<sham1>
Actually, disregard any minor collections. You should only really persist on major collections any reachable objects
xenos1984 has joined #osdev
DoubleJ has joined #osdev
sonny has joined #osdev
<sonny>
Is a subsystem just a dll in windows?
<zid>
no
<sonny>
ok
<mrvn>
netbsduser: you really want a generational GC for hat.
srjek has quit [Ping timeout: 260 seconds]
axis9 has joined #osdev
dutch has quit [Quit: WeeChat 3.6]
SpikeHeron has joined #osdev
joe9 has joined #osdev
justHaunted is now known as ffffffradiofree
ffffffradiofree is now known as jusFffffffradiof
axis9 has quit [Remote host closed the connection]
gog has joined #osdev
nvmd has joined #osdev
<CompanionCube>
netbsduser: did you know ##ibmi exists
<CompanionCube>
someone from IBM did publish some books but i dunno if they have the internal details you'd want
wootehfoot has joined #osdev
Burgundy has left #osdev [#osdev]
Burgundy has joined #osdev
<CompanionCube>
i would also look at the bitsavers documentation for the preceding System/38.
jusFffffffradiof is now known as justThumbs
justThumbs is now known as justVain
* geist
yawns
rurtty has joined #osdev
<netbsduser>
sham1, mrvn: this is how my 'minor collections' of the entire incore heap look, as a sort of minor collection of a generation from the perspective of whole-disk-object-graph collections as a major
<netbsduser>
CompanionCube: i had no idea, thanks for bringing it to my attention
<netbsduser>
unfortunately i suspect that an efficient single-level-store for smalltalk is a PhD thesis' worth amount of work
<netbsduser>
i need to study PhantomOS and see how it does it for its own very OO language
heat has joined #osdev
isaacwoods has quit [Quit: WeeChat 3.7]
<heat>
helo
<GeDaMo>
EHLO
<heat>
smtp moment
nvmd has quit [Read error: Connection reset by peer]
<CompanionCube>
netbsduser: as a less complex alternative you could take a different leaf from the OS/400 book about its pervasive database functionality and apply that for an object database, with networking you could even do a 'root on nfs' equivalent?
ss4 has joined #osdev
wootehfoot has quit [Ping timeout: 260 seconds]
<mrvn>
doing persistent storage and networking becomes insane. You have to safe every packet to disk before ACK and imagine what that does to your throughput.
<sbalmos>
OS/400's single level store and persistent DB also worked because at a very low level they /never/ reuse a vaddr once it has been used
<geist>
interesting and probably t the time the amount of virtual address space was vast and infinite?
<CompanionCube>
geist: in the virtual ISA it's defined as 128-bit.
axis9 has joined #osdev
<geist>
yah if it's using the POWER style MMU at the core too, that can easily scale to an actual full 64bit vaddr too
<geist>
since there's no fundamental overhead to a mapping in any one given spot (ie, no page tables)
* CompanionCube
doesn't recall if it was 64-bit originally or if that was later
<CompanionCube>
perhaps they did that when moving from their custom CISC CPUs to extended POWER CPUs.
sonny has left #osdev [#osdev]
<geist>
yah i dont actually know if they ever did, but at least given the design of the mmu it's mostly how many bits you want to add to the TLB entries and match
<geist>
vs being limited by how many levels of page table you want to dedicate, etc
<geist>
i always was a bit fond of POWER style page tables, alas
<sbalmos>
it was 128-bit, but it's two 64-bit segments (wrong word, but couldn't think of another good one) to define it
<CompanionCube>
ah yes, the original CISC IMPI CPUs were 48-bit so the size changed to 64-bit with the POWER migration
<sbalmos>
geist: POWER was another odd bird
<CompanionCube>
sbalmos: there is also the bit that's used to mark 'teraspace' addresses used for PASE's AIX
<geist>
yah i do wonder how the 128 bit stuff work. is it somewhat like a segemnt override? ie, here's a 64bit address but add this extra tag to access something in an alternate aspace?
<sbalmos>
geist: there's a whole chapter on it in the OS/400 internals book I have. I'll have to see if there's a digest reference online somewhere
<CompanionCube>
i think it only uses some of the bits?
wand has joined #osdev
<heat>
mjg, wtf are rfork and rfork_thread for?
<heat>
i thought you guys did linux-style user threads but it turns out you don't, so i'm even more confused
<geist>
yeah i dont thik so at all
<geist>
rfork is similar but not the same thing
<heat>
rfork is very very similar to clone
<sbalmos>
geist: oh... wait that's right. it's a 64-bit raw pointer + a 64-bit capability handle of sorts
<heat>
you can almost definitely build a linux-ish thing
<geist>
oh interesting "The rfork() function call first appeared in Plan9."
<heat>
yeah
<mjg>
rfork is clone-ish
<heat>
something cute i figured out is that fork manpages seem to be confused as to where fork came up
<mjg>
that's the "pretend" thread support
<heat>
some old ones say V6, new ones say V1
<mjg>
you actually fork, except not divorce various parts of the process
<mjg>
depending on flags
<heat>
i know
<heat>
but WHY
<heat>
is this actually useful?
<mjg>
i find it plausible it was useful to get thrading off the ground at the time
<heat>
i found like one real usage of rfork_threads across fbsd and it was posix_spawn
<mjg>
however, knowing general geezer code, for all i know it was a lolo hack which should have never been implemented
<mjg>
which would be apparent if the offending geezer spend a day thinking about the problme instead of doing drugs
<netbsduser>
CompanionCube: that would indeed be a nice consolation prize, though i need to study os/400 in much greater depth - right know i only know some of its talking points, no deeper knowledge
<geist>
yah keep in mind if it showed up in 1996 it way predates linux's clone thing
<netbsduser>
and geist, funnily enough the idea crossed my mind to give each object at least one page in the virtual address space which would've simplifeid things a bit, but the cost would be incredible
<mjg>
most notably had kernel support for user locks
<netbsduser>
sunos and solaris were remarkably early to the game on a lot of things
<geist>
and yeah i remember in the late 90s at university some mention that solaris had the only functional threading support in the unices
<netbsduser>
i liked that they already had a modern tmpfs while csrg bsd had the cruder MFS
<geist>
at least the ones that we had floating around at skool
<geist>
maybe IRIX though
<heat>
given this, why go for super-pseudo processes?
<heat>
plan9 did, freebsd copied it, linux also copied it(?)
<geist>
my ;limited interactions with irix was that it was basically 'run the pretty graphics run fast as hell, everything else is secondary'
dude12312414 has joined #osdev
<mjg>
irix sources frell of the truck, i had a look
<mjg>
i don't know about graphics, but the "usual" stuff was defo slow
<geist>
yah it was jut some BSD derivative, iirc
<geist>
probalby nothing fancy except in the graphics, and the FS department (at the time). XFS
<mjg>
no i'm pretty sure it was sysv
<heat>
praise xfs
<geist>
i remember there was some fancy bits in the block device path, whereyou could get guaranteed bandwidth, tc
<geist>
oh okay, yeah guess sysv.
<heat>
"IRIX 3.x is based on UNIX System V Release 3 with 4.3BSD enhancements"
<heat>
whatever 4.3BSD enhancements means
<gog>
TCP/IP
zxrom has quit [K-Lined]
<mjg>
geist: looks like we both had a point :-P
<geist>
ah okay. i'm sure there are irix folks that will fight me but i got the vibe that they were mostly about selling hardware, and not trying to buid the best evar os
<heat>
gog, tcp/ip support is pre-4.3BSD
<geist>
which is why it just turned off as soon as the hardware stopped
<gog>
oh
<gog>
why don't we get together and write the best evar os
<mjg>
it is dangerous to call out an old system as crap
<mjg>
not that i'm afraid
<heat>
4.3BSD already started with the tcpip extensions
<geist>
i think i only ever fiddled with irix 5
<mjg>
i was meaning to do it on tuhs for some time
<geist>
oh i think it's okay to call old things lame, but i just disagree with generally comparing with modern standards
<heat>
"anyway, in conclusion, netbsd bad openbsd bad freebsd semi-bad linux bad solaris bad irix bad"
<geist>
it's always relative to the hardware and the expectations at the time
<mjg>
dude what
<geist>
but also there are generally good and interesting thigns in most old designs, and i find it much more fun to focus on the interesting/good bits than the negatie
<mjg>
rest assured the typical old geezer had an incredibly naive approach
<geist>
negative is just... negative over time
<mjg>
if you insist i can give you examples in private
<geist>
side note i just watched some old systemd talk a while ago
<geist>
was interesting
<mjg>
they either justify something by their Deep Conviction or a benchmark which does not test what they think
<geist>
probablys omeone in here posted the link, but it had been sitting in my queue for a long time
<heat>
i think they were naive because computers were never meant to be this big and complex
<mjg>
heat: my personal theory is that they got away with bullshit because single-threaded processing just kept speeding up for years
<geist>
well, fork() in general makes total sense when you look at the hardware at the time
<sham1>
Can you call it naïvité if they didn't have any reason to think that computers would grow to what they are now
<geist>
copy segment, assign new segment to process, run it
<mjg>
heat: as long as you did not make osmething O(n^2) or worse you were golden
<mjg>
heat: shit hit the fan when smp showed up
<CompanionCube>
huh, didn't know that second one was paywalled lol
<heat>
geist, yeah, because it was never designed for 50 years later
<mjg>
i'm not criticizing fork fwiw
<geist>
honstly i think the real issue is still using the same api 50 years later :)
<heat>
fork deserves criticism tbh
<heat>
exactly
<sham1>
I feel that people tend to be too harsh on fork
<mjg>
i'm saying i don't blame them for fork at the time
<geist>
oh 100%. i think it's a pretty clever idea to be honest
<heat>
of course
<heat>
sham1, in 2022? i don't think so
<geist>
a fairly elegant solution on the limited hardware at the time, building a limited kernel
<heat>
fork just doesn't work
<heat>
vfork/rfork/clone is wayy faster and more efficient
<sham1>
Well your child processes do get to do their own thing independently from the parent, something you don't get with threads
<heat>
and if you're not spawning, why are you forking?
<heat>
fork servers are objectively maybe not great
<sham1>
...because fork is for more than just doing exec?
<geist>
yah tell me about it. have had to push back on that sot of stuff for fuchsia
<heat>
sham1, like what? why not threads there?
<sham1>
As I said, better isolation
<sham1>
You don't share an address space after all
<heat>
well, if you want security maybe you can take a big perf hit when calling fork
<heat>
but fork-semantics-by-default is a bit odd
<mjg>
well how often do you do fork
<sham1>
Also with fork you get to do things like assign the various file descriptor of the child, even when doing exec afterwards
<sham1>
Can't do that with posix_spawn IIRC
<heat>
you can, 100%
<geist>
yah one of my main complaints with fork is it assumes that process assigned resourcs (file descriptors, etc) can be duplicated
<geist>
it actually seriously runs afoul of capability based handle systems
<geist>
since now you're on tap to duplicate a bunch of handles
<mjg>
sham1: no? maybe posix_spawn_file_actions_t can do it, i never lookede into it myself
<heat>
it can
<geist>
something like fuchsia is fundamentally incompatible with it because of that, and for good reason. you cannot simply duplicate handles
<sham1>
Well okay then
<heat>
that runs into vfork/rfork_thread fuckery
<heat>
but it works and doesn't dupe your whole address space for 5 syscalls
<heat>
cow is still pretty expensive
<mjg>
cow is a tragedy
<heat>
mooooooooo
<sham1>
An this is why capitalisation matters
<mjg>
there is a myth that fork cheap bro because cow
<sham1>
CoW and cow
<mjg>
ask any webdev
<mjg>
geist: so what do you do about what is supposed to be a shared resource
<sham1>
Although of course *puts on the hat of a lisp weenie* this entire idea of even doing this stuff with processes seems primitive and things like getting the ability to not have things tamper with your data by physically duplicating the address space seems primitive
<mjg>
geist: example: you have a webserver which forks off workers, they all have a fd for logging
<geist>
one does not fork is what you do
<heat>
emacs kernel
<mjg>
geist: that's on unix
<mjg>
geist: what is one supposed to on fuchsia
<geist>
oh you can start workers easily, you just dont do it in a fork like way
<geist>
you start a new process and you hand it as part of its startup packet the appropriate handles
<mjg>
this runs into duplciating the handle roblem
<mjg>
oh ok
<geist>
ah duplicating handles is not a problem, but duplicating handles blindly as part of the kernel is the issue
<heat>
also fork plays horribly with threads
<sham1>
That is does
<mjg>
heat: man
<geist>
ie, the kernel has no fundamental knowledge of what the handles are *for*, so it's not really safe to just dup them
<mjg>
heat: it's worse than you think
<sham1>
I mean, sometimes I'd say that not duplicating the threads is useful, but mm, it's rough
<mjg>
geist: ye ye that makes sense
<heat>
and for the latest/-next posix, fork() is going to lose any MT-safe properties it had for fork+exec
<mjg>
oh?
<heat>
you're supposed to use _Fork() now AFAIK
* sham1
squints at a ISO C reserved function name
<mjg>
i'm gonna argue if you fork in am t program, you are doing osmething wrong
<heat>
ah sorry, async-signal-safe
<geist>
for some other µkernel i wrote a whiel back that was more QNX style i had the notion of the kernel *did* duplicate the handles, but it basically informed the servers (the IPC channels were one directional so it was more straightforward) that the open channels were duped
<heat>
it's in the latest fbsd manpage
<geist>
so the services had the opportunity to know that children had been spawned
<CompanionCube>
the 2019 microsoft paper on fork sucking has a very memorable title: 'A fork() in the road'
<geist>
and thus had the time to build context for the new open channels
<sham1>
mjg: well you'd fork in an MT program for the purpose of execing most likely, but since you apparently can indeed do things like assigning file descriptors with posix_spawn, I suppose that reason is moot
<geist>
but it was a more traditional synchronous IPC based µkernel with unidirectional IPC channels
<geist>
where the semantics are much easier to define at the kernel level
<mjg>
sham1: again, why is a mt program execing
<heat>
it should've been a vfork in the road, or a syscall(SYS_clone, SIGCHLD, 0) in the road
<mjg>
sham1: i knew of one example doing it and it was stemming from the author not knowing how to not use threads :)
<geist>
mjg: i have an example. i wrote a forking MT program once. was a server that was talking to multiple devices synchronously (usb) so needed worker threads to do it
<geist>
and then on some events over usb it needed to fork/exec a shell
<sham1>
Well an MT program might just be execing to run an external command of some sort
<geist>
so... problem was you were forking a MT program. not because it *wanted* to
<sham1>
I don't know why that'd be a weird thing to do
<geist>
now, some solution for that would have bee to start a helper program that sits around and forks itself to keep it from being the main one, i suppose
<mjg>
this really wanted a separate proc, prior to threading, just for forking
<mjg>
the fork + exec dance for a mt prog is a lot of trouble
<geist>
yeah, that would have been the solution had it been untenable. but it worked well eough. but it's when i learned about some of the trickery glibc does to keep forking from being a problem
<geist>
ie, unrolling locks on the child side
<mjg>
sham1: the case had hand was dude reimplementing part of xargs
<mjg>
sham1: he did not know how to spawn and wait for multiple procs with just an event loop
<mjg>
sham1: so he created a thread for each one
<geist>
but it was basically a situation where something needed to start a process and it just happened to be from a process that needed threads
<mjg>
sham1: also there was no need to even write the program, which is extra funny
<geist>
also seems like it's easy now to include threads just as a thing you do. ie, some coroutine stuffin your program
<geist>
ie, programs dont need to be held bac just because some part of it may need to fork somewhere
<mjg>
well if you routinely frok from a program and it is multithreaded, that'sp retty shitty
<mjg>
makes one wonder -- does go avoid the problem somehow?
<geist>
i argue that's a implementation detail of the system
<mjg>
since you mention coroutines
<mjg>
i would argue realities of systems you are working on do matter :)
<geist>
as in, if your OS has problems with spawning new programs whiel MT that's a shitty OS
<mjg>
that's linux
<geist>
indeed.
<geist>
even posix_spawn, we were looking at it recently, just does a clone/exec under the hood, iirc
<mjg>
i can't stress enough how disappointing linux is
<mjg>
geist: yea
<sham1>
I mean, forking to exec in a MT program to me seems like it shouldn't be a problem since you'd be just forking the current running thread anyway, which would just go out
<geist>
so it's not like that avoids the problem, it's just an api to make it easier to create a new proc
<sham1>
Of course, I'd like to be enlightened on this
<geist>
sham1: yeah the problem is things like 'what are the other threads doing at the time'
<mjg>
geist: at least should someone finally fix it, consumers will automatically benefit, so that's something
<geist>
specifically things like holding locks that are now in some indeterminate state
<sham1>
Right
<heat>
geist, netbsd does posix_spawn in the kernel
<kazinsal>
yeah, the fork/vfork + exec mechanism is really not meant to work in a multithreaded environment
<geist>
functionally it's the equivalent of truly forking a process and synchronously killing all the other threads
<sham1>
The biggest problem I'd see is that for example if you fork, and then you have a GC running
<sham1>
You'd want to stop all the threads for collection in the child process, but suddenly all the other threads are gone
<geist>
or they're in the heap, and you try to do something on the other side that mallocs
<mjg>
i don't think that's a valid example
<geist>
though you can generally avoid the problem by not doing that
<heat>
the GC thing isn't valid
<sham1>
Yeah, a language with a GC probably shouldn't do bare forks
<mjg>
the surviving thread in forked proc is not doing gc
<heat>
you're either forking as the GC thread (weird?) and nothing else is there, or you're in the new process without a GC thread
<mjg>
and there is no problem in the parent
<geist>
well, no if the other threads are in the middle of a GC, it's basically the heap problem
<geist>
ie, they're in the middle of mucking with some data structure that's left in an indeterminate state
<mjg>
that is true
<mjg>
maybe summary this way: a single-thread proc has a well defined state at all times, so you fork no problem and still have said state
<geist>
now if you just blow it away with an exec without touching anything important, you generally avoid it, which is probably why posix_spawn, etc generally works
<heat>
don't you basically stop-the-world before GC'ing?
<mjg>
a multi-threaded proc has an arbitrary state
<geist>
ie, only do shit that is local to the stack
<sham1>
heat: not necessarily
<sham1>
For example you could have it such that the first generation of a GC would be "per-thread"
<heat>
geist, fork+exec in MT processes is 100% defined by POSIX
<mjg>
reminds me to check how go's gc is doing perf wise
<heat>
A process shall be created with a single thread. If a multi-threaded process calls fork(), the new process shall contain a replica of the calling thread and its entire address space, possibly including the states of mutexes and other resources. Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of the exec functions is called.
<geist>
yah makes sense
<geist>
for some reason that program i was talking about i wrote years ago with MT + USB required some stupid trick but i think it was to avoid a bug in glibc + ARM at the time
<geist>
ie, it tried to unroll some internal locks atfork() time, but had a bug there that would fuck up because ARM
<heat>
yay ARM fuckup
<mjg>
was that a 32-bit arm?
<sham1>
Indeed, the idea of having per-thread (or I suppose in context it'd be per-process (not OS processes ftr)) heaps is how Erlang manages to do very nicely
<mjg>
extra fuckery
<heat>
that reminds me, glibc abort() has a huge MT problem
<sham1>
Of course Erlang is also helped by immutability
<mjg>
heat: fuck man
<mjg>
heat: you reminded me of something
<sham1>
You reminded me of glibc
<heat>
sorry for ruining your day
<geist>
mjg: yeah this was in 2009ish where ARM SMP and linux was very new
<geist>
and there were bugs
<mjg>
heat: back at red hat there was a customer who ran into a problem: they would exit() in a mt prog and sometimes crash
<mjg>
heat: in atexit handlers
<heat>
LOL
<heat>
wait, exit in atexit handlers or crash in atexit handlers?
<mjg>
heat: so i told them exit is not mt safe, go write a fucking dedicated routine to exit kthx
<sham1>
Absolutely dreadful
axis9 has quit [Remote host closed the connection]
<Griwes>
easy fix, just call _Exit
<sham1>
Or quick_exit
<mjg>
Griwes: at exit handlers
<Griwes>
or was it _exit
<mjg>
Griwes: so no _Exit
<heat>
* sham1 squints at a ISO C reserved function name
<Griwes>
Can't remember
<mjg>
anyway they kept giving me shit
<heat>
you have 4 exits
<Griwes>
Yeah I'm saying, just call the one that doesn't call atexit handlers lol
<heat>
quick_exit, _Exit, _exit, exit
<mjg>
unless someone with Bigger Job Title showed up, repeated what i wrote in different wording
<sham1>
Hey, at least _Exit is actually an ISO C function
<mjg>
Griwes: but they needed them for clean up
<Griwes>
heat: that one actually is C lol
<mjg>
Griwes: wtf man
xenos1984 has quit [Read error: Connection reset by peer]
<mjg>
s/unless/until/
<mjg>
that's where it was "ooh i see bro"
<mjg>
fucking twats
<heat>
aren't _Names reserved for the "implementation"?
<sham1>
For ISO C
<sham1>
Let me look it up
<Griwes>
Yes they are reserved for the implementation
<Griwes>
But wg14 likes to infringe on that namespace
<Griwes>
Because they, frankly, have no other choice lol
* CompanionCube
remembers musl had interesting times with fork, malloc, and mt-safety
<geist>
yep, it's gnarly
<heat>
POSIX is the implementation
<heat>
it augments ISO C
<mjg>
who even cares about posix today
<mjg>
it's linux all the way down
<heat>
true
<geist>
the systemd talk had an interesting perspective on posix. he posits basically that it's dead. ie, there's linux, and then there's rounding error. posix exists to ensure compatibility between all the unices in the 90s, but that has mostly moved on.
<sham1>
Personally not the biggest fan of Linux as such, it's just inescapable
<bslsk05>
git.musl-libc.org: musl - musl - an implementation of the standard library for Linux-based systems
<geist>
and he's not necessarilly happy with it, but the argument is that's kinda the state of things
<mjg>
geist: that is defo the state of things
<heat>
anyway, funny abort() rant: I was adding abort() support to Cloudflare Workers(for context, highly MT server with 600+ concurrent threads at any time) because we wanted coredumps instead of _exit()'ing
<geist>
but the other way of looking at things is, you're free of posix becaus eit's moslty irrelvant
<heat>
we noticed weird signals popping up in our crash logs
<heat>
Debug traps, segfaults, etc
<geist>
or at least if you (ie, freebsd, etc) hold yourself to posix you hold yourself back
<heat>
glibc abort() has a global counter to track "progress"
<mjg>
not even freebsd takes posix seriously
<mjg>
see kqueue
<geist>
exactly
<heat>
as in, it tries 4 or 5 things before giving up and doing CPU-specific trap
<heat>
the problem is THAT ITS FUCKING GLOBAL
<geist>
and OSX still leans on tons of mach and whatnot
<sham1>
I think the people who might take POSIX most seriously might just be in this channel, or in general the hobbyist OSDev community since it does give *some* compatibility at least for like the most basic things
<heat>
if you have 6 thread aborting at once for any reason, you *easily* overflow and do the cpu trap without anything going wrong
srjek has joined #osdev
<geist>
right. i think for hobby osdev it makes sense to bootstrap you to basically some sort of simple unix thing
<netbsduser>
it depends on how one takes 'seriously' to mean
<geist>
but once you get into the corner areas it starts to get questionable
<netbsduser>
an OS limiting itself to what posix offers is not going to thrill anyone
<heat>
the "weird signals" corresponded to the cpu trap. x86 does ud2 IIRC which gives you a SIGILL, arm does brk #3 which gives you a debug trap
<heat>
it's inzane
<sham1>
Of course OSDev for better or worse is either UNIX clone number whatever or something very avant garde
<netbsduser>
who among us doesn't enjoy self-pipe tricks and suchlike in place of kqueue?
axis9 has joined #osdev
<gog>
i should do avant garde
<geist>
yeah it gets into the meta of hobby osdev, which has never been answered because it's unanswerable
* sham1
glares angrily at epoll
<geist>
depends on what you're trying to get out of it, etc
<heat>
geist, was the speaker benno rice?
<netbsduser>
while getting all the semantics right of pthread, signals, etc is tedious and makes for a bit of a mess
<geist>
yeah
<heat>
that sounds like one or two benno talks
<heat>
yeah that was probably me
<geist>
yah you probably pasted it here. it's a systemd talk from 2019
<heat>
the UNIX one is also really good
<geist>
it was nice. actually kinda energized me a bit, he seems reasonable
<geist>
as in that's the kind of reasonable osdev communication that i like to hear
<netbsduser>
the trouble with unix is that it stultified and fossilised
<heat>
sham1, epoll is arguably fine
<geist>
like look at things that are Other and learn from what they do well
<geist>
instead of throwing poo
<netbsduser>
solaris in particular took the time to try to improve things and produced fantastic innovations like ZFS, SMF, and FMD
<sham1>
I'd like avant garde. The problem is that the common tooling assumes UNIX to some extent. For example gcc requires at least some POSIX and the kind of model for things like `read` and such. So you either need to shim it by adding something like a POSIX server for microkernels or whatever OR you patch the compiler in such a way that it is compatible with whatever model you're working with
<heat>
yeah
<heat>
avant garde doesn't get shit done
<netbsduser>
but most people who self-profess as 'unix fans', i've found, have little interest in embracing the development of unix into a better unix than unix itself
<geist>
it can with sheer force of will (amd a bunch of engineers)
<geist>
or, if you limit yourself to smaller subsets where these things dont matter (ie, embedded)
<sham1>
I'm not Google and I cannot force a Fuchsia into existence
<geist>
exactly, that doesn't come along very often
<heat>
In theory I think fuchsia would be way further along if not for non-POSIX
<heat>
but it would lose all the interesting bits
<geist>
but i can tell you the latter kinda works. there are tons of places in the embedded side of things where fun, clever designs actually work
<CompanionCube>
netbsduser: who needs unix, we have linux \s
<heat>
or most of them, probably
<heat>
in traditional usability at least
<geist>
right. it's a tradeoff that is debated constantly
<netbsduser>
heat: i don't foresee it being an issue
<netbsduser>
if google will it, fuchsia will spread far and wide and devs will adapt to its requirements
<heat>
google wills a lot of things and they're very much not flop-proof
<sham1>
Unless it gets killed like many a Google project
<geist>
yep. believe me that is not un-noticed by google folks
<heat>
RIP stadia
<mjg>
well how many progs even rely on unixy-stuff
<sham1>
RIP Google+
<mjg>
look past your niche
<mjg>
most code is wirtten in very high level langs
<netbsduser>
yeah, the average mobile phone app is written to a much higher level of abtraction
<mjg>
which are quite detached from unix fuckery
<mjg>
it's a matter of converting the runtime
<geist>
exactly. also remember all of the programs that run on windows, mac
<netbsduser>
posix doesn't talk about these things which matter to a desktop or mobile app
<sham1>
Hell, I remember from like beginning of the last decade when Google had that whole "make your own Google startpage" -thing. Sadly it got axed as well
<geist>
those are not built on unix fuckery
<heat>
but maybe some runtimes don't fit on non-traditional designs well
<geist>
sham1: oh yeah! ire embmer that
<geist>
i forgot what it was called, but it was nice
<sham1>
It is comforting that most things don't depend on UNIX fuckery. It's also not really comforting when you're actively in need of something that explicitly needs it
<heat>
if your app is exec heavy it will struggle in perf in a lot of non-POSIX designs for instance
<mjg>
bro your app is bitcoin miner heavy
<heat>
or that whole rustup saga where they did things that Just Worked(tm) in UNIX but had a heavy cost in Windows
<sham1>
Emacs...
axis9 has quit [Remote host closed the connection]
<heat>
emacs needs unix?
<sham1>
People joke about the whole "great OS which needs a good editor", but if you really look at it, it does require a lot of handholding from UNIX stuff and tools
<heat>
it seems that windows emacs is a thing
<netbsduser>
something just occurred to me as a solution to my problem of object persistence, although the solution may be a little radical
<mjg>
if you are using emacs you are on your own
<heat>
damn right
<heat>
we're GNU nano users
<CompanionCube>
sham1: iGoogle?
<CompanionCube>
i remember using iGoogle way back in the day
<sham1>
For example performance on Windows is miserable. Might just be a Windows thing tho. And a lot of the Emacs ecosystem is built around the assumption of UNIX
<sham1>
CompanionCube: yes
<sham1>
That's the one
<mjg>
as for path ops vs windows, they should probably make htem faster....
<clever>
i was watching a guy working on ancident dos machines on YT yesterday, and then he pulled out bloody vim :P
<clever>
vim on dos, lol
<CompanionCube>
clever: emacs on dos is also a thing
<mjg>
like for realzies
<sham1>
Vim was originally developed for the Amiga. If anything, vim on DOS seems tame
<mjg>
unix progs can't bve the only victims
<netbsduser>
treat the entire object heap as a memory mapped file. when i want to checkpoint it, unmap everything that's read/write mapped (i.e. has been written to) and write all modified pages to disk
<mjg>
why unmap
<heat>
mjg, windows flushes the page cache on CloseFile (for some reason)
<netbsduser>
mjg: or unset the dirty bit i should say
<heat>
why would this stop being a thing?
<mjg>
netbsduser: write protect and halt anyone who faults
<netbsduser>
this of course still is no answer to the question of efficient garbage collection because i am now pretending that the entire thing is in core
<mjg>
heat: i don't know if close can be augmented. given other commentsi 'm pretty sure straight up lookup, not opening anything, is already terrible
<mjg>
someone posted powershell starting up some time ago
<heat>
it wasn't just path ops though
<sham1>
Yes, the file IO can be terrible
<mjg>
the number of cats which get murdered for that to happen...,.
<heat>
closefile was put in a worker thread for rustup
<sham1>
There's also the Microsoft special of "well, someone has opened this file. Guess what, you're screwed!"
<mjg>
ye that looks like a wtf is going on here workaround
<sham1>
"Cannot save!"
xenos1984 has joined #osdev
<mjg>
i would love a windows kernel dev commenting on it
<heat>
these are the kind of differences that make non-UNIX not work well, even on higher level languages
<heat>
if you consider rust a higher-level language, which I kinda do
<sham1>
Well it is
<sham1>
Just like C is
<heat>
rust is quite a bit higher level than C IMO
<sham1>
Oh absolutely
netbsduser has quit [Remote host closed the connection]
<sham1>
Although they still do similar patterns, like needing to do (expensive) copies of data when sharing it because of things like ownership issues
<heat>
mjg, i would love a zfs dev commenting on my "fsync doesn't sync" issue
<heat>
we all can't get what we want :P
Burgundy has left #osdev [#osdev]
<clever>
heat: head on over to #zfsonlinux or #openzfs
<mjg>
heat: i'm pretty sure it syncs\
<j`ey>
sham1: copying or locks, what else is there?
<sham1>
And *notices that he hasn't still taken off the lisp weenie hat* Rust also uses things like reference counting in many places *snorts*
<heat>
it doesn't sync, it syns to a special table that then gets synced
<mjg>
heat: there is some bad fuckery going on with posix layer support
Burgundy has joined #osdev
<mjg>
ye fsync avoids the actual sync, but data stability should be the same from your pov
<heat>
but metadata clearly isn't sync'd
<heat>
anyway, not ranting on fsync semantics today
<mjg>
it is synced, but posix layer support does not see it
<clever>
mjg: i think the bug heat is complaining about, is that while the journal can ensure data stability, the metadata like size-on-disk is still in flux
<heat>
I really, really, really don't understand what the point of optimizing fsync is
<bslsk05>
github.com: Creating + writing to a file + stat() in quick succession returns bad st_blocks · Issue #13991 · openzfs/zfs · GitHub
<sham1>
j`ey: Well part of the problem is that the data is mutable in the first place, requiring that you copy or lock, but I won't get into that. That's a thing you do even if you intend on just reading the data, because you have no assurances of the lifetimes in cases
<heat>
it seems like empty ass optimization
<mjg>
whicn, fsync or not, should have reported correct result
<j`ey>
sham1: well the data doesnt *have* to be mutable
<sham1>
For sharing that kind of data even for just reading, you need some kind of garbage collection, whether it be RC or actual GC stuff
<sham1>
Otherwise you'll muddy up the lifetimes of the data
<sham1>
And of course RCs are sad
<clever>
heat: if you write a single 128kb block of a file on zfs, then it has to create a new version of the indirect blocks (varable number, up to ~6 i think), it then has to modify the dnode, and re-create the indirect blocks in the objset, then it has to update the dataset dnode in the mos, and its indirect blocks, and then the spacemap
<heat>
mjg, i think what happens is that the bullshit fast-fsync table can't know i_blocks ahead of time because it didn't actually sync and COW, etc
<clever>
heat: so thats at least 20 blocks, possibly more, for every 1 block you write to the disk
<clever>
and the point of the ZIL/journal, is to reduce that down to just 1, while still maintaining the data integrity
<mjg>
heat: but there should be no issue even if you don't fsync
<heat>
i think that expecting fsync sounds fine
<heat>
but fsync'ing and not actually sync'ing does not sound fine
<sham1>
j`ey: and if you wish to consider even for mutable data, there's always CoW
<mjg>
if you check how getattr on zfs perates you will see it digs deep below the posix layer
<mjg>
and that has a long update delay
<heat>
all of this is why IMO ext4-like, ufs-like boomer filesystems will not lose their place for a long time
<sham1>
Yeah, the boomer FSes are great
<heat>
despite zfs users saying it cured their cancer or whatever