<klange>
mrvn: no, and I'm not sure it would gain much in a lightweight C codebase like this; maybe if I were doing C+...
<klange>
C++*
<mrvn>
klange: if you do "make clean" a lot it helps a lot
* heat
sees this as a prime oportunity to circlejerk
<heat>
why did you not write kuroko in rust?
<heat>
you should rewrite it in rust
<zid>
like anybody needs an excuse
<klange>
Kuroko is C because all of the OS is C, which is a boring answer, and I still like to fall back on the OS being C because rust didn't really exist at the start.
<heat>
everything not written in rust is insecure
<heat>
therefore, rewrite or replace
<klange>
gotta take up that one dude's request to have kuroko compile to native code, and then write Kuroko in Kuroko
Likorn has quit [Quit: WeeChat 3.4.1]
<heat>
oh right
<heat>
you could also write it in python
<heat>
but purposefully make it incompatible with kuroko
<klange>
any reasonably-sized Python application is incompatible with Kuroko unless specifically written to be compatibile
<klange>
you can do it, by having all of your function variables be arguments, and some of my test suite is written that way so I can compare results between the two
<klange>
particularly the complicated async stuff
<klange>
you can check out my ponysay port to see a python-to-kuroko conversion looks like; ponysay, despite being pretty, uh, bad and unpythonic... was a fairly straightforward port, as it doesn't have any library deps
<heat>
this is of course subjective, but I've never seen a reasonably sized Python program that I thought "yeah, this totally should be written in python"
<heat>
i basically use it as a nice scripting language
<klange>
Two of my prior employers did a lot of Python. One is a big, household-name web service - they were virtually all Python when I was there, but it was many years ago now and I'm sure they've picked up other things.
<klange>
And the other was a robotics startup that had Python for their UI layer and C++ doing all the real work behind the scenes.
<heat>
the thing with python is that it's painfully slow
<heat>
I also find it hard to organise a large python codebase (although that's probably my lack of python experience talking)
<klange>
If you look at PyPy benchmarks, it is quite clear that is purely a CPython problem - as much as the whole industry is, for the most part, still using CPython and not PyPy.
<heat>
yup
<heat>
the EDK2 build system is mostly a makefile generator written in a bunch of obtuse python
<heat>
and it's sooooooooo much slower than what I'd ask for in a build system
<heat>
a good chunk of the build time is it trying to regen makefiles
<heat>
one day I should try to port over a portion of it to a different build system as a POC
<heat>
I have a feeling that it would be much faster and easier to work with
<heat>
maybe using gn... I think that it would work pretty well as a very non-opinionated, generic build system
<mrvn>
python lacks lists. python lists are arrays and any sort of appending or removing is dead slow.
<heat>
appending should be pretty fast with a vector
robert_ has quit [Ping timeout: 246 seconds]
<mrvn>
assuming it ammortizes growth
<mrvn>
memory is ref counted so it also leaks
<heat>
removing is of course O(n) (?) but proper appending should ammortize to O(1)
<mrvn>
inserting is dead slow too
<heat>
Just Python Things
<psykose>
if it becomes fast it means there is a bug in the language
<heat>
nobody removes the GIL because it's hard
<heat>
you don't write python to do hard things!
<klange>
GIL removal is happening right now
<heat>
oh what
<mrvn>
won't GIL be removed next week for a decade now?
<sbalmos>
tuesday
<klange>
uh, no, this is all within the last couple of months
<klange>
there's been a decade of talk, though, that's definitely true
<sbalmos>
removing the GIL will enable commercial fusion power
<mrvn>
synergy?
<klange>
(the GIL is _also_ just a CPython thing, pypy has no GIL either...)
<mrvn>
enterprise level next generation prototyping?
srjek has joined #osdev
<klange>
I've tried to not have a GIL in Kuroko, though in its current state it quite obviously demonstrates why CPython has a GIL.
<heat>
"what if python didn't have a GIL" is software engineering's "what if we used 100% of our brain"
<klange>
< mrvn> python lacks lists ← deque is a suitable linked-list with true constant-time append, and insert from a preknown location
<bslsk05>
github.com: cpython/listobject.c at 3.10 · python/cpython · GitHub
<klange>
Most of why CPython remains slow is refusals to perform optimizations in the compiler stages beyond dead code elimination. There is a lot going on behind the scenes for method caching and various other things; the VM has seen a lot of speedups.
Vercas has quit [Remote host closed the connection]
Vercas has joined #osdev
gxt has quit [Ping timeout: 240 seconds]
gxt has joined #osdev
* geist
tries to look at the meteors
<geist>
Seen a few so far
<geist>
Clouds are moving in though
<Mutabah>
:( It's middle of the day here, and partly cloudy
<Griwes>
I caught a few
<Mutabah>
I could go check... but lazy
<Griwes>
the sky's very clear here (Gold Bluffs Beach in Redwoods) but it's also too cold outside of the tent to spend much time looking up
<geist>
Also saw a satellite i guess go right through the center of the field
<geist>
Yah
<geist>
It’s about 10C here right now so a bit chilly. I shoulda brought a sleeping bag, could easily camp out on top of a picnic table at the park I’m at
<Griwes>
yeah about 10C here too, though I am right next to the ocean so it's chillier than that
<Griwes>
highly recommend camping over here by the way, it's a really nice place and, by some miracle, the campground even has wifi :'D
yewscion has quit [Ping timeout: 240 seconds]
gxt has quit [Ping timeout: 240 seconds]
gxt has joined #osdev
lanodan has quit [Ping timeout: 260 seconds]
lanodan has joined #osdev
puck has quit [Excess Flood]
puck has joined #osdev
puck has quit [Excess Flood]
puck has joined #osdev
wand has quit [Ping timeout: 240 seconds]
vdamewood has joined #osdev
vinleod has joined #osdev
vdamewood has quit [Killed (zinc.libera.chat (Nickname regained by services))]
vinleod is now known as vdamewood
Burgundy has joined #osdev
eroux has joined #osdev
Vercas has quit [Remote host closed the connection]
<bslsk05>
wiki.osdev.org: GDT Tutorial - OSDev Wiki
<ckie>
(specifically the CS jump)
<mrvn>
How should that work in C? The C code before the asm is 32bit and after 64bit? How are you going to tell the compiler that?
<ckie>
no, the gdt is already 64bit before, i just want to clean up the old entries and put some fresh new ones in
dude12312414 has joined #osdev
<mrvn>
So you want to change from descriptor 0x08 to 0x08?
<ckie>
no, BOOTBOOT drops me with an unknown GDT, and i want to move things around so they are known
<mrvn>
Usualy you have a GDT in boot.s with the 7 required entries and you load the registers once and never touch that part again.
<ckie>
but if I don't want that?
<\Test_User>
why would you want to write inline assembly over plain assembly
<ckie>
\Test_User: well, i am trying to avoid the extra makefile shenanigans and thinking about that
<mrvn>
ckie: Still don't see a reason to change CS.
gog has joined #osdev
<ckie>
mrvn: idk if the order of my GDT is the same as BOOTBOOT's
<\Test_User>
wouldn't take makefile shenanigans, just a small added part for it
dude12312414 has quit [Client Quit]
<mrvn>
ckie: you have to set CS to go to long mode. What do you want to change later than you can't set initially?
<ckie>
mrvn: but then CS stores the GDT entry idx, correct?
<\Test_User>
mrvn: from the sounds of it they don't have control of the bootloader
<mrvn>
ckie: the GDT is irrelevant. it's just used to load CS. Once loaded you can delete the GDT entry.
<mrvn>
ckie: just load your own GDT for exceptions and interrupts and syscall to use.
<ckie>
I think I might just shove this off to the side with a TODO and hope bootboot behaves nicely
<ckie>
aren't exceptions/interrupts separate tables?
<mrvn>
ckie: check that bootboot is using 0x08 for CS
<\Test_User>
yes that's the IDT, not the GDT
<zid>
why are you wanting bits of assembly into C inline assembly exactly?
<mrvn>
ckie: the mode switch when an exception or interrupt happens and back will use the GDT.
<mrvn>
ckie: but seriously. Just install your own GDT before you jump to C code.
<zid>
C compiler gunna be very not happy if the cpu mode changes in the middle of its TU :P
<ckie>
zid: i am somewhat afraid of assembly and the auto register allocation makes me feel cozy
<j`ey>
ckie: but it's probably .. what 5 instructions?
<zid>
gcc -m32-but-then-m64-in-a-while
<\Test_User>
ckie: learn it and you'll have no need to fear it
<mrvn>
zid: That's not what he is talking about
<zid>
it's also five instructions *once*
<ckie>
mrvn: they, not he
<zid>
which means it's on the nanosecond scale
<ckie>
but yes, fair, will just write it in asm when i come back around to this
<ckie>
(since it seems bootboot is okay and i don't need to worry about this *right now*)
<mrvn>
ckie: Do you know where the 32bit CS is in the bootboot GDT?
<mrvn>
or where the GDT is at all so you don't overwrite it?
<ckie>
mrvn: bootboot supplies a memory map at boot, and it seems the GDT is just before the bootstrap stack starts
<Andrew>
You can't really avoid assembly
<mrvn>
does bootboot call directly into C code?
<ckie>
Andrew: so far i have, short for debugging gcc's register allocation when i forgot to specify inline asm io
<ckie>
mrvn: yes
<mrvn>
chekc the specs if the GDT is mentioned.
<ckie>
i have
<ckie>
i think i can go on from here though, thanks
<mrvn>
If it's calling C code then it must have CS/DS/ES/SS at least in some defined state. And FS/GS have MSRs.
puck has quit [Excess Flood]
<ckie>
i *could* just absorb bootboot in-tree
puck has joined #osdev
<ckie>
then the GDT is specified and all is well
SGautam has joined #osdev
bauen1 has quit [Ping timeout: 240 seconds]
<mrvn>
"The GDT is unspecified, but valid." I.e. install your own as soon as possible.
<mrvn>
Coyping that asm blob into some *.S file and adding a static GDT and lgdt call to it seems like a simple fix.
<ckie>
fiine i'll give it a shot 😛
<mrvn>
you have time till you want user space.
<ckie>
it's not that, it's that i'm trying to keep it short so the compiler-writing is doable
<ckie>
(trying to alternate steps so: write kernel a bit, write compiler a bit, jmp -2)
<\Test_User>
writing an assemler is much simpler than a C compiler
<\Test_User>
...and your C compiler will include an assembler anyways bc inline asm
<mrvn>
Have you thought about how to add inline asm to tcc?
<ckie>
\Test_User: no, the compiler will output C (for now)
<\Test_User>
eh? what kind of compiler is this
<\Test_User>
compile from what into C
<ckie>
\Test_User: that's the whole experiment! i have some vagues ideas as to what i want, but i want to just go along and optimize for the kernel code to be clean
<ckie>
so hopefully the language will naturally develop with the OS
<j`ey>
I think tcc supports inline asm
<ckie>
j`ey: hi!
<j`ey>
hi!
<ckie>
neat to see you around again
<ckie>
i like this chat
<ckie>
pretty cozy
Bonstra has quit [Quit: Pouf c'est tout !]
<mrvn>
ckie: modifying the language on the fly is hard. Will you break backward compatibility or support compiling multiple flavours of the language? A simple syntax or semantics change can mean a massive change across all of the kernel or even the algorithms you use.
<mrvn>
ckie: if you start with C then you will be pretty locked into a C-like language.
<ckie>
mrvn: well, right now kernel.c is just 350 lines + [iu]int{8-64} typedefs
<ckie>
so i can rewrite all of it in a hour or two
<mrvn>
plut the compiler, or not?
<mrvn>
plus
<ckie>
idk, i haven't written it yet, but i'll try to keep it small too
<ckie>
maybe in py or js for easy-to-refactor
<mrvn>
Didn't you want to start with tcc as base or was that someone else?
<ckie>
it can just run the tcc cli for prototyping purposes
<ckie>
(please don't kill me)
<mrvn>
#!/usr/bin/tcc your kernel
<ckie>
no, tcc is just the last step in the pipeline since register allocation seems scary and i am not ready to debug obscure crap like that
<ckie>
(but yes, me)
<mrvn>
register allocation is really easy (allocate till you run out) or very hard (spill registers optimally)
<ckie>
generally i'm much more interested in the language part of this whole endeavour for now
<j`ey>
then why not write a compiler instead of a kernel? :P
<mrvn>
me too. In camomile the compiler simply fails when it runs out of registers.
<ckie>
once the language is decent i'll probably rewrite the compiler in the language
<ckie>
i want to go: [typing]
<ckie>
barebones kernel + stdlib -> compiler -> betterer kernel -> rewrite the compiler in the OS||lang
<ckie>
god it's weird how english "or" is actually a xor
<\Test_User>
what's weirder is it can be used as either xor or or
<bslsk05>
zick/TempLisp - LISP implementation in C++ templates (0 forks/2 stargazers)
<GeDaMo>
Not the only one :P
heat has quit [Remote host closed the connection]
<mrvn>
That's a compiler of sorts, not Lisp in c++.
heat has joined #osdev
sonny has joined #osdev
sonny has quit [Remote host closed the connection]
sonny has joined #osdev
knusbaum has quit [Ping timeout: 246 seconds]
SGautam has quit [Quit: Connection closed for inactivity]
knusbaum has joined #osdev
vdamewood has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
shikhin has quit [Quit: Quittin'.]
shikhin has joined #osdev
sonny has quit [Ping timeout: 252 seconds]
<opios2>
anyone here tried Intel 12gen processors with P-Core and E-Core?
kingoffrance has quit [Ping timeout: 265 seconds]
the_lanetly_052_ has quit [Ping timeout: 240 seconds]
miezekatze has joined #osdev
miezekatze has left #osdev [WeeChat 3.5]
miezekatze has joined #osdev
miezekatze has left #osdev [WeeChat 3.5]
Likorn_ has quit [Quit: WeeChat 3.4.1]
ketan has quit [Quit: Leaving]
the_lanetly_052_ has joined #osdev
the_lanetly_052_ has quit [Ping timeout: 258 seconds]
elastic_dog has quit [Killed (mercury.libera.chat (Nickname regained by services))]
elastic_dog has joined #osdev
kingoffrance has joined #osdev
wootehfoot has quit [Quit: Leaving]
jafarlihi has joined #osdev
arch-angel has quit [Ping timeout: 244 seconds]
arch_angel has joined #osdev
arch_angel has quit [Read error: Connection reset by peer]
arch_angel has joined #osdev
the_lanetly_052_ has joined #osdev
Likorn has joined #osdev
jafarlihi has quit [Quit: WeeChat 3.5]
heat_ has joined #osdev
GeDaMo has quit [Quit: There is as yet insufficient data for a meaningful answer.]
heat has quit [Ping timeout: 258 seconds]
sonny has joined #osdev
sonny has quit [Remote host closed the connection]
heat_ has quit [Read error: Connection reset by peer]
heat has joined #osdev
the_lanetly_052_ has quit [Ping timeout: 258 seconds]
sortie has quit [Quit: Leaving]
sortie has joined #osdev
<mrvn>
"I saw you eat a living chicken and a whole goat." "Nobody's perfect."
sonny has joined #osdev
mahmutov has quit [Ping timeout: 260 seconds]
<moon-child>
opios2: I have not, why?
<opios2>
just curious about the experience and how it will affect multi thread programming etc..
divine has quit [Ping timeout: 248 seconds]
<zid>
doesn't really affect anything
<zid>
the OS sched has to be aware, but if it is it moves shit around to make it work
<zid>
otherwise it's just some threads randomly take slightly longer than others
* mrvn
is still happy not having threads.
dude12312414 has joined #osdev
<sonny>
what unit of concurrency do you have?
<mrvn>
IPC
<mrvn>
message passing
<klange>
con$s
<sonny>
mrvn: ok, is it all premptive?
<mrvn>
sonny: i don't think anyone builds anything not preemptive anymore
<sonny>
I see
<mrvn>
sonny: not preemptive would mean the process runs till it yields.
toluene has quit [Read error: Connection reset by peer]
<sonny>
yeah, that's what I mean
<mrvn>
The bigger question is how often and where you preempt, not that you preempt. :)
<sonny>
how could you know ahead of time?
toluene has joined #osdev
<mrvn>
My scheduler is pretty simple, a round-robin scheduler. But when a task gets woken up and hasn't used up it's time slice (or has sleept for a while) it gets put at the front of the list, otherwise at the end. So most times when you send a message and include the yield flag the recieving task runs next.
toluene has quit [Read error: Connection reset by peer]
<geist>
mrvn: yah that’s basically what i do with LK as well, and it’s a pretty good solution
<mrvn>
If your system is design to "race to sleep" it's damn near perfect. Any improvement would just waste time deciding what to do.
toluene has joined #osdev
<mrvn>
One thing I stll have in the back of my mind is a flag to lend the senders time slice to the reciever.
<geist>
It gets a little hairier with SMP and if you want multiple priorities and whatnot, which are also useful, but it’s nice to have simple O(1) selection algorithms at least
jack_rabbit has joined #osdev
<mrvn>
priorities go towards the length of time slice currently.
knusbaum has quit [Ping timeout: 276 seconds]
<mrvn>
well, more like a time slice enum class.
<mrvn>
geist: How do you pick a task in O(1) with priorities? Assuming you don't count having 20 queues and taking the first with a task as O(1).
<geist>
That’s still O(1) if you have a finite number of priorities
<geist>
But the simplest optimization there is keep a bitmap of which queue has at least one thing in it
<geist>
and then do a ffs on it
<mrvn>
sure. but I think it's cheating. You should evaluate it with infinite priorities.
<geist>
*shrug*
<geist>
It’s a finite thing, not something you just scale infinitely, so i think it’s fine to pick a N and then optimize for it
<mrvn>
It works in prctice. Up to 64 priorities it works very well.
<geist>
Alternatively I’ve seen an implementation (i think beos did this) that kept a single double linked list queue, and then a list of pointers off into the middle of the queue
<geist>
So selecting a thread was just pop the queue
<geist>
But inserting you’d start with the queue pointer for your priority which might point off in the middle of the list, etc
<geist>
But i think the trouble isn’t worth it, since you end up with some complexity there to just try to work around having a bitmap and ffs
<mrvn>
I can see that work if you have a dummy element per priority in the queue that the priority pointers cna point too. But then the scheduler has to skip over them all the time.
<geist>
Yah exactly, which then devolves to the same thing as having a bitmap, etc
<mrvn>
Wores, that's O(priorities)
<geist>
Well, that’s the same as ‘search all the queues’ but yeah
<geist>
If you want to get technical, it’s only a win if your arch has a fast ffs. Hypothetically the ffs search could be O(priorities) too
<geist>
But of course there are fast algorithms for that even if you dont have an instruction on your arch
<mrvn>
I have one problem left with multiple queues + bitmap. How do you do that in SMP? Do you migrate a task if it has a higher priority as the best local task?
<geist>
But it’s O(1)all the same, and not hard on the memory subsystem (ie, not bad for cache misses)
<geist>
That’s the rub. SMP really messes up a lot of the perfect universe you can build with a single cpu
<mrvn>
ffs is O(log n) in software.
<mrvn>
same as a heap of queues.
bauen1 has joined #osdev
<geist>
The naive SMP implementation (one that LK currently still does) is to just maintain a single run queue and then basically let each cpu pick from it
<geist>
It’s essentially perfect latency if you’re aggressive about forcing other cpus to reevaluate their queues
<geist>
But not very efficient
<geist>
Since threads bounce around unnecessarily, and cpus reschedule unnecessarily
<mrvn>
By the time I've locked all the scheduler queues, found the highest priority, migrated the task and released the lock my local task has probably already run and gone back to sleep.
<geist>
So basically waiking up a thread is to put i on the run queue and then wake up either *all* cpus, or find the one running the lowest priority thing than the thing that just got woken up,and interrupt it., etc
<geist>
But all serious systems evolve towards having a per cpu run queue, and now you have a second level scheduler, effectively
<geist>
Deciding which cpu to run a thread on first
<mrvn>
Do you have a per process sleep queue?
<geist>
No
<geist>
Sleep queues in LK (and fuchsia by proxy)are simply a queue somewhere. Usually at the bottom of a locking primitive, but could be anywhere where some sort of blocked thread queue is desired
<geist>
A mutex, a semaphore, an ipc primitive, a futex, etc
<mrvn>
I have wait queues for other structures so I already have to support removing a sleeping task from an arbitrary queue. So a per process sleep queue wouldn't be anything extra.
<geist>
Yah i just started that by creating a wait_queue object and then having other things have one as part of their implementation
<geist>
It’s functionally the only place a thread can block on
<geist>
So higher level constructs are built around it
<mrvn>
The sleep queue is actually more the timers heap. Having per cpu timers might be better.
<mrvn>
wake up the process on the cpu it went to sleep on by the cpu that it sleeps on.
<geist>
Ah for time based wakeups, it’s a function of the LK wait queue: wait_with_timeout() basically. The thread creates a timer structure, puts it in the timer queue, and then goes and blocks on the queue. The timer callback if it happens simply removes the thread from the queue it’s in and reschedules it. If it doesn’t happen by the time the thread wakes up for another reason the timer event gets cancelled
<geist>
Thread_sleep() just does the same thing, but never wakes up for another reason
<geist>
Thread_suspend() is similar. Just blocks on an implicit wait_queue that has no other purpose but to park threads in
sonny has quit [Quit: Ping timeout (120 seconds)]
<geist>
This wait_queue pattern I’ve settled on over the years in various kernel projects and have a really tough time finding a better solution. If nothing else its a strong fingerprint for geist’s kernels
sonny has joined #osdev
<mrvn>
I still have to make that change to have wait queues with timeout.
\Test_User has quit [Quit: e]
<mrvn>
Would actually make it simpler again, because then I don't have tasks queues and tasks on heaps as special cases. It's always a queue and an optional heap to timeout.
<geist>
Right
<geist>
Having a single wait queue at the bottom of the thread’s state also has the advantage that it’s easy to unblock a thread if you want to kill it or deliver a signal to whatnot
<geist>
Since it doesn’t matter what type of primitive or the reason the thread was blocked, you simply unblock it from the wait queue it was on and then it’s the responsibility of the blocking primitive that wrapped the queue to deal with that appropriately
<mrvn>
yeah, currently I have to check if task->next == task. If so then remove the task from the timer heap.
<geist>
Ie the thread_t itself only knows that it’s blocked on a wait queue, but doesn’t know how it got there, but that’s okay. That’s another layer
dude12312414 has quit [Remote host closed the connection]
<mrvn>
scratch that: task->next == nullptr.
gildasio has quit [Ping timeout: 240 seconds]
Likorn has quit [Quit: WeeChat 3.4.1]
<mrvn>
You task has a pointer to the wait_queue object it's blocked by and that has a wakeup() function?
<geist>
Yah
<geist>
I haven’t back ported it from zircon but basically you can if marked appropriately (it’s up tot he blocking primitive to mark the wait as interruptible) wake it with a custom error code
<dh`>
> its a strong fingerprint for geist’s kernels
<mrvn>
So far I just remove it from the queue directly. There is no abstraction for queues that need to do something more to wake up a task.
<dh`>
not sure how I follow it's different from anything else, unless you mean specifically the timeout handling
<geist>
So in the case of forcibly unblocking a thread you say ‘wake_up_thread_from_this_with_error_code(wait_queue)’
<geist>
And then the caller has something like
<geist>
Int ret = wait_on_this(my->wait_queue); if (ret == ERR_INTERRUPTED) un_roll_state;
<mrvn>
dh`: does your harddisk driver have a wait_queue for tasks trying to read data?
gildasio has joined #osdev
vdamewood has joined #osdev
<geist>
dh`:well there are some other things that folks have done here, but I’m not claiming i invented that strategy or whatnot
<geist>
I think it’s fairly standard in general though differing implementations
<dh`>
mrvn: no, it has a condition variable, the queue is inside that
<geist>
It’s one of those good patterns thats hard to not reuse for every solution
<geist>
Some things just make good sense
Likorn has joined #osdev
<mrvn>
Not sure how widespread it is to only have wait queues and nothing else. I bet many kernels have a queue for sleeping task and other subsystems wake up tasks from that instead of having specialized queues.
<geist>
But any kernel derived from one of my projects has a very specific fingerprint when it comes to those things is what i really meant. Like it has a very specific api
<dh`>
yeah I know what you mean, my kernels have very noticeable fingerprints too
<dh`>
just don't understand this particular bit since it seems very basic
<dh`>
I guess the traditional BSD sleep/wakeup interface doesn't have this property but it went out with uniprocessors
<geist>
I think the idea of having a *single* kind of wait queue and then constructing other things out of it is not really universal
<geist>
Especially among hobby OSes, since i think it requires a particular level of experience and design to notice and then unify
<mrvn>
well, it it really a wait queue? the running tasks are in a queue too :)
<dh`>
I'm not sure it ever would have occurred to me to do it any other way
<geist>
Well, good foryou
<dh`>
given that I learned the basics from amigaos and it's present there
<mrvn>
dh`: you have queues in the mutex, futex, vfs, block devices, timers, sockets, ...?
<geist>
You rolled a +3 in wait_queue foo on your character sheet
<dh`>
:-)
<geist>
Yah same. I learned the basics on beos, and though it’s not public source it was really a nice kernel to learn on
<dh`>
actually hmm
<dh`>
now that I think about it, I'm not so sure.
<geist>
It’s major downside in the blocking primitive part is it really only uses multi count semaphores and though it was kinda neat in the 90s, it i think is ultimately pretty limiting
<dh`>
when I did multiprocessor OS/161 I built it that way, but before then it was different and I'm hazy on how at this point
<mrvn>
Another problem I have yet to solve well is aborting IPC messages. Say a thread wants to read, so it asks the VFS, the VFS asks the FS, the FS asks the block device and then the process dies.
<geist>
I’ve always thought the classic BSD strategy of ‘wait_on_this(some pointer)’ was nice, but i guess the machinery behind that requires some though
<geist>
And I’m not sure it’s a great idea personally
<dh`>
mrvn: that's one of the downsides of message systems
<geist>
It’s convenient though, since it’s kinda like futexes in kernel space
<geist>
Some other layer deals with rolling a wait queue for you,e tc
<dh`>
yeah
<dh`>
the original uniprocessor OS/161 had that because it was supposed to have standard things in it
<dh`>
I forget how it worked under the covers though
<geist>
But i very much *do* like the idea if being able to roll a wait queue (or a primitive with one in it) on the stack, for each use
<geist>
Some of my wait_queue is carefully written to allow that: ie, when you wake up a thread it cannot assume the object itself is valid anymore
<dh`>
in vino I had all kinds of weird stuff but vino also let you send interrupts to specific processes while they were blocked without causing wakeup races
<geist>
Because it may have already been moved out of scope on the original scope
<dh`>
which was expensive and totally a solution in search of a problem
* geist
nods
<mrvn>
geist: how do you validate that?
<geist>
It just carefully doesn’t touch anything else in the queue
<dh`>
that is, you could be running one of these interrupt things when your wakeup came in, and it wouldn't get lost
<geist>
Ie, if you call wait_queue_wakeup_all(queue); then inside the wait_queue_sleep() call the threads waking up cannot touch anything from the queue on the way out
<geist>
Since it may have already been destructed at the point at which the woken threads get scheduled
<mrvn>
geist: but what if the thread hits a timeout?
<geist>
That doesn’t happen
<geist>
By design, it’s simply not possible for that to happen because that’s how it’s designed to work
<mrvn>
mid wait_queue_wakeup_all() another core can getg a timr interrupt and try to wake up a thread.
<geist>
Yah and that’s tricky
<dh`>
anyway, in OS/161 there's spinlocks and scheduler-level wait queues and everything else is built out of that; OS/161 doesn't have signals but there's at least one version of the same stuff that does
<geist>
At some point a spin lock is involved to make that atomic
<mrvn>
I would say the queue object has to be destroyed after wait_queue_wakeup_all()
<dh`>
basically you can just take the kernel-level thread off whatever queue it's on and move it to the run queue
<geist>
Yep
<geist>
Pretty solid design
<geist>
Okay, gotta get back to work for a bit
<geist>
Gotta get my 1 unit of work done today
<dh`>
and then if it did an interruptible sleep it's the caller's responsibility to notice that it was interrupted by a signal and return EINTR out
<dh`>
timeouts are dodgier, none of the systems I've built have ever really had good time handling
<mrvn>
dh`: I'm not sure I want EINTR
<dh`>
EINTR is necessary if you're going to forward interrupts or signals or anything like that to userspace
<dh`>
pretty much
<dh`>
but iirc your system is odd enough that it might not apply :-)
<mrvn>
interrupts or signals just gets turned into messages. All operations are async so they don't get interrupted by it.
<dh`>
geist: I've found that it's reasonable to never destroy anything that has active waiters; you need to make sure you can't get _new_ waiters before you hit destroy, and that usually means you have to remove whatever it is from the live operating path, and letting it drain also isn't a big problem
pretty_dumm_guy has quit [Quit: WeeChat 3.5]
<dh`>
mrvn: then they're just more messages
<mrvn>
dh`: I would even say that while you drain it the core should yield the lock every now and then to unblock other cores that want to access the same queue.
<dh`>
the second question then (which doesn't have to be the same question) is what you do when things get stuck
<mrvn>
dh`: that goes towards the abort of messages above.
<dh`>
e.g. you're waiting for a message or a reply or something and the user knows it's never coming, so they hit ^C or type kill -9 or whatever
<dh`>
in a traditional kernel there are wait points all over everywhere deep inside many layers of goo and you need to unwind it all before you can exit safely (even if what you're planning to do is exit rather than post a signal to userspace or whatnot)
<geist>
Indeed
<mrvn>
Then the kernel needs to terminate all message pipes. The processes listening on the other and then have to see the closing and terminate whatever operations are associated with that pipe and signal other processes to cancel certain operations too.
<dh`>
the pending messages you have are basically dual/comparable to that call stack state
<mrvn>
as said I don't have a good generic way for that yet.
<heat>
fwiw wait_on_this(addr) is implemented on top of a hashtable of wait queues in linux
netbsduser has quit [Ping timeout: 244 seconds]
<heat>
it's basically on there for tiny structs that don't need a permanent wait_queue (like struct page)
<dh`>
that's the traditional BSD implementation
<mrvn>
Yeah, messages that trigger operations should have a call history of some kind. Something where I can say: Hey block device, cancel all reads initialited by pid=1234
<dh`>
if I were building something like that I'd probably materialize some kind of work request object and pass it around along with the messages that actually do things
<mrvn>
currently I just do the work and at some point on the way back it runs into a closed pipe and dies.
<mrvn>
I'm not prepared for things that never finish.
<dh`>
then keep a list of those in the kernel projection of the user processes and mark them aborted before exiting
<dh`>
maybe cause them to be moved to the ready queue in whatever server process is currently handling them
<dh`>
so as to get an effect like "oh, one of my pieces of work is ready; oh, it's aborted itself, great, I can flush it"
<mrvn>
You also have to consider complex cases, like 2 processes reading the same file.
<dh`>
but that might get expensive depending on layers and whatnot
<dh`>
also I've thought about this for about ten minutes and you shouldn't take it too seriously :-)
Burgundy has quit [Ping timeout: 240 seconds]
<heat>
i can see how having everything cancellable may give you some grief when it comes to DoS attacks
<dh`>
basically, message send and reply is the same as call and return, except it's asynchronous
<dh`>
so if you've messaged down into some disk driver via a raid driver and the buffer cache and a filesystem and the vfs layer, it's basically the same state as having called down there
<dh`>
and you need to do pretty much the same things to clear it out
<heat>
yes but most ops in a UNIX-like kernel aren't cancellable for instance
<heat>
if they were you could have like two processes trying to read stuff and then cancel and try to absolutely smash the system's performance by just hogging a lock
<mrvn>
yeah, so my stuid design solution isn't worse than unix.
<heat>
unix is the best worst stupid design solution ever :)
netbsduser has joined #osdev
<dh`>
uninterruptible sleeps are a bug :-)
<mrvn>
tell that to NFS
<mrvn>
or any IO on disk error
<dh`>
...that was a design decision made by Sun based on 80s standards of error handling
<dh`>
and it's only the default, not a requirement
<dh`>
anyway preventing local DoS is a hard problem
<mrvn>
I also want to have subscription based IO. Say you open a socket the process would subscribe to the socket. Then whatever data comes in gets put in a message and send to the process. But if the process doesn't process the data fast enough there is no back pressure to stop the socket from ACKing.
<mrvn>
Have to implement some limit on message queue sizes.