<nikolapdp>
heat there's is a massive difference between zfs/btrfs and journaled ext4
<heat>
why?
<nikolapdp>
i've tried to explain it to you multiple times but you just didn't care
<heat>
besides the block checksums?
<geist>
but yeah as nikolapdp is saying the *design* is completely fundamentally different
<geist>
like it's coming at it from a different universe
<heat>
i'm not talking about the design, i'm talking about the end result
<nikolapdp>
again, different
<nikolapdp>
snapshots, shallow copies, etc
<nikolapdp>
it's not even close
[Kalisto] has joined #osdev
<heat>
i'm not talking about those, i'm talking about data safety
<geist>
yah, btrfs doesn't expose as much of the internals as zfs does, but they have a very distinct layering between stuff on disks, and stuff on top
<nikolapdp>
there's a difference there too heat
<geist>
which is why you can do raid, etc natively in both FSes
<nikolapdp>
journaled FSs generally don't journal data
<nikolapdp>
and you can easily lose data on powerloss
<heat>
>and data-journalled ext4/xfs
<geist>
because they dont really raid the same way, they just arrange for data to be present in multiple locations, etc
<nikolapdp>
ah my bad, disregard that
<geist>
well then a big difference is fully data journally ext4 is pretty slow
<nikolapdp>
true that
<heat>
is it? i've never tried it
<heat>
i know it disables delalloc and O_DIRECT
<geist>
yah i think it blows out the journal really fast, so it ends up havig to go through tons of cycles abd fushes
<geist>
since the journal is only some finite size. it can at the most allow up to a journals worth of transactions before syncing
<heat>
yeah
<geist>
i suppose you could increase the siez of the journal maybe
CryptoDavid has quit [Quit: Connection closed for inactivity]
<heat>
ext4's jbd2 journal can be up to 4G
<geist>
but think of btrfs as running more or less the same level of safety as ext4 with ordered journal (which is the default)
<nikolar>
The idea behind log based fss is "what if the journal was the whole disk,"
<geist>
ie, if you pull te plug you may get some intermediate state, but never data that isnt written yet, just sort of some intermediate transaction point
<heat>
depending on the amount of memory you have your system will throttle writes and writeback way before 4G anyway (usually)
<nikolar>
Don't know about btrfs, but with ZFS you always get one or the other, never the intermediate state
<geist>
yah the amount of metadata to track that much journal stuff in flight may slow things to a crawl as the amount of outstanding journal increases
<heat>
there's a big incentive now to pre-clean pages so you don't have to do IO in memory pressure
<heat>
basically control the dirty pages to a reasonable amount
<geist>
nikolar: well, it's basically the same, it's just where the fs decides to commit a new volume tree
<geist>
if you're say just happily writing to a file and pull the plug, it'll presumably reset back to the point at which it committed the tree last
<nikolar>
Right yeah
<geist>
which is what i mean, it'll be some point as you were filling in the file
<nikolar>
The on disk format and all of data until then will be there
<geist>
exatly
<geist>
that's what ext4 ordered mode somewhat promises too
<geist>
ie, it'll only commit metadata that covers fully written data data
<heat>
extended filesystem four!!!!!!!
blockhead has joined #osdev
<nikolar>
ZFS
<heat>
i like ext2 because it keeps you on your toes
<heat>
don't turn off your computer too quickly or you'll lose your childhood photos
<heat>
that, to me, is OPTIMAL
<nikolar>
PDP11
<heat>
listening to mjg's advice is leading me to creating a horrendously complicated slab cache shrink path with two memory barriers
<heat>
instead of le locky bois
troseman has joined #osdev
<childlikempress>
referring to the locks as 'bois' infantilises them, stripping them of agency and responsibility for the harm they've caused
<heat>
the lock lads
<childlikempress>
heat please stop trying to normalise problematic behaviour
* kof673
points childlikempress at corporations . not trying to start anything, but that point was long past centuries ago :D
Turn_Left has joined #osdev
Left_Turn has quit [Ping timeout: 255 seconds]
blockhead has quit []
Matt|home has quit [Quit: Leaving]
Turn_Left has quit [Read error: Connection reset by peer]
voidah has joined #osdev
<heat>
i thought i had a bug but it turns out it was just working too quickly in a loop
netbsduser has quit [Ping timeout: 272 seconds]
Gooberpatrol66 has joined #osdev
vdamewood has joined #osdev
srjek has quit [Ping timeout: 255 seconds]
Arthuria has joined #osdev
gog has quit [Quit: byee]
Arthuria has quit [Ping timeout: 260 seconds]
Arthuria has joined #osdev
heat has quit [Ping timeout: 268 seconds]
goliath has quit [Quit: SIGSEGV]
goliath has joined #osdev
xFCFFDFFFFEFFFAF has joined #osdev
masoudd has joined #osdev
pebble has joined #osdev
MrCryo has joined #osdev
MrCryo has quit [Remote host closed the connection]
vdamewood has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
Arthuria has quit [Ping timeout: 268 seconds]
masoudd has quit [Quit: Leaving]
pebble has quit []
sauce has quit [Ping timeout: 260 seconds]
pebble has joined #osdev
Mach has joined #osdev
Mach` has joined #osdev
Mach has quit [Ping timeout: 268 seconds]
kfv has joined #osdev
kfv has quit [Client Quit]
navi has joined #osdev
sauce has joined #osdev
m5zs7k has quit [Ping timeout: 246 seconds]
m5zs7k has joined #osdev
Left_Turn has joined #osdev
<ddevault>
signals /o\
<ddevault>
today's problem is EINTR
<mjg>
signals are such a shite hack
<ddevault>
they're so bad
<mjg>
product of its time though
<nikolar>
What are the solutions
<ddevault>
imho you should have an uncatchable kill signal and a cooperative interrupt signal(s)
<ddevault>
no preemptive signals
<nikolar>
So you have to explicitly wait on those?
<ddevault>
yeah
<ddevault>
or incorporate it into your event loop
<mjg>
ye all progs should be event loops
<mjg>
they partially already are, except done in a more error-prone fashion
<ddevault>
or spawn a thread to wait on signals
<mjg>
dude that's webdev
<ddevault>
(if you _must_)
<ddevault>
(just use an event loop)
<mjg>
a kqueue-esque loop would do the trick
<ddevault>
only reason to use threads is if you have a compute workload
<mjg>
the difference to epoll & friends is explicit support for event types
<mjg>
as opposed to being fdonly
<nikolar>
Doesn't epoll do that too
<mjg>
no
<mjg>
for examle you can EVFILT_SIGNAL in kqueue, no fuckery required
<mjg>
but since epoll only takes fds, they had to implement signalfd
<mjg>
they also had to implement timerfd
<mjg>
(see EVFILT_TIMER)
scaleww has joined #osdev
gog has joined #osdev
gog has quit [Remote host closed the connection]
gog has joined #osdev
gog has quit [Remote host closed the connection]
gog has joined #osdev
flash has quit [Quit: leaving]
gog has quit [Remote host closed the connection]
gog has joined #osdev
xFCFFDFFFFEFFFAF has quit [Ping timeout: 268 seconds]
gog has quit [Remote host closed the connection]
gog has joined #osdev
GeDaMo has joined #osdev
<ddevault>
fixed signal return when signal interrupts a syscall \o/
<ddevault>
broke signal return when signal preempts userspace /o\
<ddevault>
signal handlers hose the kernel stack for the process, how tf do I deal with that
<sortie>
ddevault: I never ever deliver signals to threads when they are in the kernel. It uses a cooperative model where it sets a flag on the thread that a signal is pending, and then wakes up any kthread_mutex_wait_signal and kthread_cond_wait_signal calls that explicitly want to abort if a signal happens, and then those syscalls return EINTR appropriately and the signal is dispatched on syscall exit
<ddevault>
yeah, I've been consulting your implementation, and toaruos as well
<ddevault>
so you essentially wake up the blocking syscall, tell it to EINTR, and then... at the sysret you deliver the signal?
<sortie>
What is your threading model? I have basically unified threads that happen to run in user-spacer or in the kernel
<sortie>
And switch back and forth. All threads are always preemptible at all times. (Except for very short regions in the kernel)
<ddevault>
no SMP, setjmp/longjmp across kernel stacks to context switch
netbsduser has joined #osdev
<ddevault>
kernel is non-preemptable
<sortie>
My experience was that a non-preemptible kernel became really difficult to do properly
<sortie>
Having system calls block became a real headache
<ddevault>
well the kernel can cooperaetively switch out
<ddevault>
so like read on pipe(2) will setjmp and then longjmp to the next task, essentially
<ddevault>
but then when a signal comes in, longjmp'ing back to it is problematic because the kernel stack is hosed
<sortie>
You also run into problems if you ever were to add SMP
<ddevault>
I have a solution which I haven't exhaustively tested but which appears to work in which we remember if we were blocked in a syscall when jumping into the signal handler
<sortie>
And you still pay fora whole kernel stack per thread with this model
<ddevault>
if so, we just go straight to returning EINTR and discard the wait queue
<ddevault>
upon returning from the handler
<sortie>
So you have the cost of preemptible thread in stack space, you have SMP issues, but you do have the simplicity of the kernel thread not having data race conditions
<sortie>
Plus any long running system call that does a big and legitimate operation will starve other threads
<ddevault>
well, it can add cooperative preemption points
<ddevault>
but yeah
<ddevault>
I do want to add SMP and kernel preemption eventually, and I know that signals are going to hurt me again
<sortie>
It's a trade off and an interesting design and I did want to do it initially myself, but eventually decided to go 100% preemptible at all times and it became much cleaner and powerful in my experience
<ddevault>
walk me through some of this state machine, if you have the patience
<ddevault>
what does your system do on the following scenario:
<sortie>
I don't have SMP yet, but basically 99.9% of my code already is multi threaded in the kernel and should work just fine in a SMP context, modulo bugs
<ddevault>
1. pid A calls waitpid w/o WNOHANG and it blocks
<sortie>
Sure ask away :L)
spare has joined #osdev
<ddevault>
actually let's not use waitpid, let's call it a blocking read on a socket or something
<sortie>
* :)
<ddevault>
2. pid B signals pid A
<ddevault>
3. pid A wakes up and runs the signal handler(?), and in this signal handler runs a bunch of other syscalls, hosing the kernel stack for that thread
<ddevault>
4. pid A returns from signal handler, and read(2) returns EINTR
<ddevault>
my assumptions holding up so far?
<sortie>
OK here's how Sortix handles that
<sortie>
Thread A runs blocking read(2) which ends up somewhere in a driver in a waiting loop for more data:
<sortie>
if ( !kthread_cond_wait(&new_data_cond, &mutex) )
<sortie>
return errno = EINTR, -1;
<sortie>
Err
<sortie>
if ( !kthread_cond_wait_signal(&new_data_cond, &mutex) )
<sortie>
return errno = EINTR, -1;
<ddevault>
do you yield the processor or do you loop until the PIT preempts the kernel
<sortie>
That _signal aspect is important because it wakes it up when a signal happens, otherwise it doesn't wake on signal.
<sortie>
The kernel thread can be preempted at any point
<ddevault>
is the stack coherent at this point? will you return up the stack when it unblocks or is interrupted?
Mach` has quit [Quit: Leaving]
<sortie>
When kthread_cond_wait_signal blocks, it ends up doing essentially a kutex (think futex(2)) call, and does an interrupt into the scheduler, asking it to put the thread to sleep until another thread signals that this memory location needs to be awakened (by kthread_cond_signal) (kutexes/futexes are out of scope for this conversation, just standard threading primitives done efficiently)
<ddevault>
(yeah)
<sortie>
While the thread sleeps, the kernel stack for the thread (8 KiB) is just what it was in memory, and the registers are saved in the thread structure
sauce has quit [Ping timeout: 260 seconds]
sauce has joined #osdev
<sortie>
Now along comes thread B which sends a signal using kill(2) to thread A's process. sys_kill(2) decides what thread to route the signal do and decides on thread A. It runs UpdatePendingSignals to figure out what signals needs to be dispatched based on the signal mask and signal handlers. If a signal does need to be delivered and it's not the current thread, Scheduler::SetSignalPending is called to inform the scheduler thread A needs to wake.
<sortie>
The scheduler is that 0.1% of the kernel that is non-preemptible and turns off interrupts for a moment to do the scheduling as an atomic operation
<sortie>
Thread B continues to run, presumably, after it scheduled thread A to run again. After a bit of time, thread A is done executing or is preempted.
<sortie>
*, thread B is done
<sortie>
Thread A then is scheduled to run. The kutex sleep inside kthread_cond_wait_signal continues and it notices the condition has not been met and it sees that the thread's signal bit is now set. The function returns false to let the caller know the operation was not completed because a signal arrived.
<sortie>
(Thread A just continues to execute normally here with its stack unaltered and the same old registers)
<kof673>
mjg: https://ariadne.space/2021/06/06/actually-bsd-kqueue-is-a-mountain-of-technical-debt/ explicit support for event types <mjg> as opposed to being fdonly > but since epoll only takes fds, they had to implement signalfd <mjg> they also had to implement timerfd # that is exactly what i thought (see link) -- someone still had to implement those, they did not magically show up lol
<bslsk05>
ariadne.space: actually, BSD kqueue is a mountain of technical debt | Ariadne's Space
<kof673>
so sure, add new event thing, or add new syscall ...either way, code does not magically show up lol
sauce has quit [Ping timeout: 256 seconds]
<sortie>
The system call fails with EINTR as I showed above in the driver and it basically bubbles up the call stack from the driver, through the inode layer, through the virtual fiilesystem layer, through the file descriptor layer, through the sys_read system call layer, and then the system call returns
<ddevault>
the syscall returns to userspace before the signal handler runs?
<netbsduser>
kof673: yes she has put the cart before the horse
<sortie>
ddevault: Yep, well almost:
<ddevault>
or does the system call return handler say "there's a signal pending, remember I was going to leave this with EINTR, run that, and then we'll EINTR on the sigreturn"
<netbsduser>
the kernel resources had to be invented late. moreover procdescs were invented for capsicum not for kqueue, and are good in their own right (i do generally support a unified abstraction, i.e. file descriptors, for kernel objects)
<mjg>
kof673: that post is total crap
<kof673>
well i am sitting in c89 land. what does "signal" give you? not a lot. "threads" lol only if you add them
<sortie>
At the very last moment before the system call returns, the system call dispatch checks if the signal bit is set, and if so, it asks for the signal to be dispatched to the thread in user-space. Basically it does a special interrupt into the scheduler. The scheduler then looks at the registers in user-space the system call was supposed to return to, and uses that instead of the actual thread registers, and saves those on the user-space stack
<mjg>
kof673: the author fails to realize having epoll support in anything already rquires explicit code in there
<netbsduser>
oh, and the "kernel accelerated form of WaitForMultipleObjects" is misleading - that API *is* a kernel API of Windows
<sortie>
Basically it sets up the same sigreturn trampoline that would have gone onto the user-space stack if the signal was delivered while the thread was running in user space
<netbsduser>
well it's a userspace wrapper of a kernel API but that's still "kernel accelerated", i am not sure where the intuition to the contrary came from
<sortie>
It saves all the registers that is supposed to be returned to on the stack, and pushes a return pointer to sigreturn, which I happen to store in a vdso like kernel page that's readable by user-space, which invokes the sigreturn system call. That way the kernel does not need to know where libc put sigreturn
<ddevault>
yeah, I saw you put the magic return thing in the aux vector
<netbsduser>
and i broadly agree with sortie, all real unixes do signals by having some mechanism of interruptible waits for use in code paths that should be signal-interruptible, and then percolating that interruption right up to the system call implementation which returns the needful and only on return to user space is a signal frame set up
<sortie>
The scheduler then changes all the registers in the thread to invoke the signal handler appropriately. Once it finishes, it returns to sigreturn, which makes the kernel pop the old registers from the stack, and restores what thread A was doing, which it read() failing with EINTR
<ddevault>
strictly speaking what thread A was doing is returning to userspace with EINTR after read() failed
<ddevault>
right?
<sortie>
Yeæ
<ddevault>
so the code path from there is very short, it does not resume read() per-se
<sortie>
Yeah
<ddevault>
okay, this makes sense
<ddevault>
thank you for explaining :)
<netbsduser>
mjg: just wait until the author finds out about how epoll needs code to support epolling itself
<ddevault>
oh, one more question
<sortie>
I don't have resumable signals just yet -- will need to figure out exactly how to do them
<ddevault>
what happens if a signal is delivered to a process while it's in the middle of execve(2)
<sortie>
The easiest way is just to configure the code so upon EINTR it just goes one instruction back and reinvokes the system call
<sortie>
But some system calls are defined to have special semantics with respect to EINTR and not be resumable
<ddevault>
that wouldn't work, the ABI registers are hosed
<sortie>
And others like read(2) will return a smaller buffer read upon signals
<ddevault>
you'd probably have to store registers at syscall handler entry point and repeat it, maybe if there was some kind of resumable flag stored somewhere that a resumable syscall handler would set
<sortie>
ddevault, the system call dispatch handler can save and restore those registers
<netbsduser>
ddevault: the signal return system call will put them back to what they ought
<ddevault>
the signal return will put them back to what they were *at the end* of the syscall, not the start
<netbsduser>
it will put them back to what they were saved as being at the time of the trap
<sortie>
execve(2) does not actually need to be special in any way, it is a standard system call with respect to this, it can either 1) use the non-_signal variants to keep going and eventually complete its task and the signal is delivered to the thread when the new program starts running 2) use the _signal variants and abort midway, or check the signal bit midway, and abort early
<ddevault>
ack
<sortie>
The most interesting thing about execve is that once you reach the point of no return, you want to go single threaded, which basically means SIGKILLING the other threads in the process
<sortie>
And since they're running in the kernel, it is *unsafe* to just stop running them (you'll deadlock & corrupt kernel state), so you need to ensure they always handle signals in *all blocking paths*
gog has quit [Quit: Konversation terminated!]
<sortie>
The nice thing about being unemployed is being able to again deliver lectures here at the University of Osdev :D
gog has joined #osdev
<sortie>
ddevault, yeah it took me a long while to figure all of this out and design it, basically I read *all the POSIX requirements on signals* and thought long and hard about how it could be done in a nice way which good semantics and security and quality, which also fit well into my system
<sortie>
It's not even that much code like you've seen, but it is very carefully written code, and it has consequences for the whole kernel with the requirement that any thread in the kernel must be abortable by signal in all loops that could block forever
<ddevault>
that's better than my approach
<ddevault>
which was to read about signals and weep
<sortie>
I mean I did worse stuff earlier
<sortie>
Sortix has been very poor lol
<ddevault>
mostly I read implementations rather than specs
<kof673>
and finally, "a mountain of technical debt" i get if your goal is to do everything in terms of fds, sure, that is "debt". but if that is not your goal, then that seems a strange projection onto other design
<sortie>
But you know, I gradually got there.
<sortie>
I mean, don't bite off more than you can chew.
<kof673>
*that is "debt" if such is missing
<ddevault>
in terms of reference
<ddevault>
linux for drivers, toaruos for "how can this be done simply", and sortix for "how can this be done right"
<sortie>
At one point I was ready to do it for real, so I did the big study and think, because I needed to know all the requirements, if not by heart, but so I would know if any weird requirement meant a given approach was a dead end
<sortie>
idk I honestly think of Sortix as simple
<sortie>
simple, but not too simple
<ddevault>
I have also gone through numerous pages of my journal, mostly diagraming out the scheduler and signals
<ddevault>
it's C++, so :^)
<sortie>
Sortix kernel C++ is basically C with classes
<ddevault>
if you're open to feedback
<sortie>
Only real thing is scoped locks with you can pry from my cold dead hands
<ddevault>
can I note that your magic return via an executable auxv leaves a bad taste in my mouth
<ddevault>
I'd sooner map auxv NX and use the page fault approach of toaruos (which I ended up doing after reading both)
<sortie>
ddevault, unclear what you mean, can you say what file and line?
<ddevault>
lemme find it again
<ddevault>
kernel/process.cpp:1048
<sortie>
Sortix's biggest strength is usually that it does things right in a simple manner
sauce has joined #osdev
<sortie>
Not too simple that it causes problems,and usually goes for some extra credit in quality
<ddevault>
how toaruos (and my OS, now) handle returning from signals is setting the return address to a never-mapped page and picking it up the magic address in the page fault handler
<sortie>
Though there is a bunch of old stuff in there that is incomplete and a bit problematic, fixing those in due time
<ddevault>
I haven't figured out sa_restorer yet so not sure if this works for that case
<netbsduser>
i have apps call a sigreturn syscall instead (in libc also a set_sigentry() syscall let the kernel know where the signal entry routine is)
<sortie>
ddevault: I mean that's the sigreturn trampoline I mentioned. It's basically like a vdso in Linux which is heavily used to optimize system calls without actually exiting into the kernel
<ddevault>
yeah
<sortie>
This also goes straight to the appropriate sigreturn handler rather than going through the more expensive general purpose signal handler
<sortie>
A magic value also needs to be reserved, you know? What better way to reserve a magical address than to, you know, just map it
pebble has quit []
<sortie>
It seems much dirtier to me to have the page fault handler use logic like this to identity if it's actually another handler, rather than just having two handlers called directly
sauce has quit [Ping timeout: 272 seconds]
<sortie>
ddevault, while we're talking surprising things in Sortix execve, did you notice the user-space stack and argv+envp allocations are separate?
<ddevault>
I would object less if you set up a vdso space separate from the aux vector
<ddevault>
oh, I didn't notice
<sortie>
That is the vdso space
<ddevault>
ack
<sortie>
I just used the term aux for it
<sortie>
So unclear what you mean
<ddevault>
I misunderstood
<sortie>
In Linux and basically any other Unix argv goes on the main stack. But here's a fun one. Did you know the main thread can run pthread_exit? The program only exits when the last thread exits.
<ddevault>
sure
<ddevault>
this isn't too surprising
<sortie>
So what happens to the main stack and argv when the main thread exits?`Well argv is process lifetime and needs to be kept alive. Therefore the main thread's stack is not unmapped on glibc and friends.
<ddevault>
oh, that's interesting
<sortie>
In Sortix, those are separate, and the libc thread implementation will just unmap the main thread's stack, and argv remains alive
<sortie>
You might also notice the thread local storage is mapped in execve
<sortie>
That does not happen on Linux. libc has to reopen the executable and locate it and map it and set it up for the main thread. Lots of logic before main() is even called.
<sortie>
On Sortix, I literally just define _thread int errno = 0; and it's online the moment _start is executed.
<ddevault>
yeah, I read your TLS implementation a long time ago
<ddevault>
much much better approach
<sortie>
It will cause me some grief when I do dynamic linking but let's cross that bridge later
<ddevault>
used the same approach for helios
<sortie>
Those are some of the interesting aspects where I aim for some higher qualities that also reduce complexity in other parts of the system
<sortie>
Linux has a big philosophy of just solving kernel problems from a kernel point of view, even though libc has to pick up the slack and do additional work to work around the kernel to implement POSIX requirements
sauce has joined #osdev
<sortie>
In Sortix, I designed the libc and kernel interfaces together so they solve the requirements holistically and simply, even if I have to diverge a bit from Unix tradition and move stuff a bit across the kernel boundary
<sortie>
This is also how I managed to golf true(1) down to idk some 2048 bytes or so when I statically link and compile -Os -s
<kof673>
^^^ yeah, IMO unless you are trying to be "portable" (to pre-existing systems) "designing together" should *always* reduce complexity, because you know up front more of what is desired, rather than x years piecemeal added separately
<kof673>
or...it is a matter of time :D
<ddevault>
I think the holistic approach is better
<ddevault>
though the balance lies somewhere in the middle
<ddevault>
NetBSD gets close
<ddevault>
speaking of which, I'm thinking about trying to port pkgsrc to my system if it becomes self hosting
<kof673>
just call it "design specialization" -- you have more meta information, and possibly existing implementations to study (even if they are not exactly what you want or you wish to extend) etc. all other things being equal, should be less complex
<sham1>
Wait, it was dropped very quickly into the conversion, but sortie, when did you stop working at Google?
<sortie>
sham1, bout three weeks ago they had a mild case of the 'hey let's layoff everyone working on engprod infrastructure and send their jobs to india and mexico without having hired replacements or any sort of continuity plan let's see how that works out'
<sortie>
Good news is that I have 6 months of paid severance with no responsibilities
<sortie>
So now I just go to concerts and have fun with friends and do some osdev in coffee shops from time to time
<sortie>
I like how you have /etc/rc whereas I have a whole systemd like init
<sortie>
Btw it's standard as /sbin/init rather than /bin/init, the daemons conventionally go in /sbin
<sortie>
ddevault: I'm very happy I built a new service(8) on my OS that can start and stop daemons now and enable/disable them permanently, and also fixed a bunch of input keyboard bugs related to alt-tabbing between windows in display(1), and also fixed a serious pty(4) deadlock that could happen when doing a lot of activity in an editor
<sortie>
Tonight's Sortix is definitely one of the best so far
<sortie>
Oh and I ported php'
<sortie>
And sqlite3
<ddevault>
sortie: yeah, regarding binary organization I intend to fix that up soonish
<ddevault>
right now I just chucked all of the bins into /bin
<sortie>
hier(7) yo
<ddevault>
also /etc/rc does not preclude having a service manager
<sortie>
Fun fact for a long time in Sortix /bin was a single directory read-only initrd
<ddevault>
imho /bin/init and /etc/rc and the supervisor are all separate problems best solved by separate software
<sortie>
My philosophy is that the init & service management code needs to be really, really carefully written to handle errors, especially out of memory errors, gracefully
<sortie>
Any shell script is full of a ton of implicit errors due to the fork, execve, and even just memory allocation in the shell
<sortie>
So shell scripts cannot be fail safe
<sortie>
Also the supervised daemons leak and needs to have the right parent process that can monitor them
<sortie>
So that's why I went with an init written in C that carefully manages all of this
<sortie>
Allocates stuff ahead of time where possible, gracefully handles data loss, handles any failure in the daemons, full dependency tracking, readiness, logging also built in (and handles out of disk space errors gracefully too)
<sortie>
That's how I build something rock solid and high quality
<gog>
but muh sysvinit
<sham1>
Runit!
<gog>
seriously though i was glad when systemd started to be the primary service manager
<sortie>
Sortix init is in a lot of ways a much simpler reimplementation of systemd concepts
<gog>
systemd probably creeps a little too much in scope
<sortie>
Kept it simple, plain C, well designed, limited scope
<gog>
i think that's a consequence of Linux just having 6 different ways of doing everything, even basic things
<ddevault>
note that in my view /etc/rc's job is to just do some limited system initialization tasks (e.g. mounting filesystems) and then exec into the service manager
<ddevault>
so it's not persistent and generally ought not to have to worry about robustness against ENOMEM etc
<sortie>
qemu-system-x86_64 -enable-kvm -m 1024 -vga std -boot d -cdrom vh978x.iso ← Error opening root device: No such device
<kof673>
if i ever get there...i think "es" shell and just drop all bourne etc. from the base system lol
<netbsduser>
i note that on solaris when they added what is generally considered to be the best service management system ever designed, they did not have it fully supplant the role held by init and the inittab
<netbsduser>
there was no real reason to do so after all
<sortie>
ddevault: How should the cdrom appear on the qemu line?
goliath has quit [Quit: SIGSEGV]
<kof673>
you could install "third party" but "system" stuff will assume "es" shell perhaps
<sortie>
I can never remember how to AHCI on the qemu command line, not my fault the syntax is so awful
<netbsduser>
the only real reason why historically you could "need" a service manager to be pid 1 is to do with tracking orphans as they reparent to you, but 1) this is obsoleted by subreapers 2) this was obsoleted on solaris by process contracts 3) this is essentially a hack, resolvable either by a) better daemons that don't leave detritus processes around that live on or b) a proper job/grouping mechanism that the service manager can use
<ddevault>
ah, I can reproduce your error
<ddevault>
let's see
<netbsduser>
so for my kernel without a doubt i am going to implement an inescapable hierarchical grouping mechanism for processes, hierarchical so that a chain of control can be established, and providing functionality like "signal every process in this subtree"
<ddevault>
it's a memfs (extracted from the initrd)
<sortie>
Ah like Sortix
<ddevault>
you can also mount ext4 r/w if you prepare a suitable disk
<sortie>
I do have a iso9660fs now that I am playing with
<ddevault>
qemu-img create whatever.qcow2, then mkfs.ext4 it with -O^metadata_csum
<ddevault>
needs to be available via ahci
<ddevault>
I can give you the appropriate qemu incantation if you wish to play with it
<sortie>
You need a uname
<sortie>
For screenshots
<ddevault>
mkdir, rm, uname, etc, are next up on the todo list
<sortie>
Looks like you wrote your own utilities
<ddevault>
the syscalls mostly exist (not uname, though), just need to add them to libc so the core utils build
<ddevault>
nah, they're sbase
<sortie>
I am stuck in scancodes :(
<sortie>
Somebody needs to learn how to SIGINT lol
<ddevault>
hah
<ddevault>
RIP
<ddevault>
this OS is 22 days old, give me a break
<sortie>
I was trying to go for 100% completion
<sortie>
Honestly for 22 days this is incredible
<ddevault>
thank you
<sortie>
how the heck did you even ext4 up in the time lol
<ddevault>
I vendored lwext4
<sortie>
Ah :D
<sortie>
Probably how you managed to do it :D
<ddevault>
I also pulled and adapted bits from my microkernel project
<sortie>
With sbase and such
<sortie>
What libc?
<ddevault>
AHCI driver mostly comes from there
<ddevault>
fork-ish of musl libc
<ddevault>
anyway the game is to speedrun writing a Unix clone
<ddevault>
the source code is here if you'd rather browse from the comfort of a more complete unix system rather than /src: https://git.sr.ht/~sircmpwn/bunnix
<ddevault>
lwext4 does the ext4 driver, libc is a musl derivative, dash is the shell, sbase is the coreutils, and some bits of helios were adapted for some parts of the kernel, rest is original
<bslsk05>
git.sr.ht: ~sircmpwn/dash: WIP Bunnix port - sourcehut git
<ddevault>
I could probably re-enable dash's kill builtin now that I have signals mostly working
<ddevault>
I also have fcntl(F_DUPFD) now, could put that back in
<sortie>
Alright played around with it
<sortie>
Any other stuff I should check out?
<ddevault>
want to play with it at runtime or do you want to be directed to interesting bits of the code
<ddevault>
has pretty good block device support via AHCI if you set that up, MBR and GPT partitions and they're available in /dev, I might add mkfs.ext4 and fdisk soon(TM)
<ddevault>
and it should work fine on real hardware too
<ddevault>
I mounted my laptop's linux ext4 partition and poked around in my files, weird experience
<sortie>
I will look at the code some other time
<sortie>
But happy to play around with runtime stuff if you got any
<ddevault>
if you've poked through /bin and ran the stuff that interests you, you got most of it
<sortie>
No source in /var/src here
<sortie>
No editor I can see
<ddevault>
it's in /src on the initrd, not /var/src
<ddevault>
no editor, but you have sed and cat :^)
<sortie>
Oh cool
<sortie>
SED
<sortie>
This is still way better than like Sortix ~0.7 era
<ddevault>
pipelines should work fine so if you can do something interesting with the coreutils you have it'll probably work
stolen has joined #osdev
<sortie>
$ cat co
<sortie>
That got stuck
<sortie>
On a non-existent file
<ddevault>
ack
<ddevault>
I got file or directory not found fwiw
<ddevault>
but you've presumably been running it for longer and there's some state tangled up somewhere
<sortie>
Yeah
<sortie>
It works properly after a reboot
<sortie>
Only instability I found so far
<ddevault>
I'll do sigint (and rig up ^D in the TTY to EOF) to make the terminal more useful
<sortie>
The rest was just missing features
<ddevault>
and port doom
<ddevault>
plus flesh out the coreutils and add an editor
<ddevault>
all of that ideally done this week
<ddevault>
I also have to convert the early boot serial console to a proper TTY so I can go back to running headless qemu for development
<ddevault>
oh, here's a fun one: echo hello >/dev/tty0 does what you expect
<sortie>
You seem to have a copy of the GPL in there
<ddevault>
indeed
Matt|home has joined #osdev
<sortie>
If you add some terminal colors, your system will seem much more powerful without being it
<sortie>
Fun to play around with
<ddevault>
I want to avoid termios, ansi, etc
<ddevault>
though since I resigned myself to making a unix and all that entails (e.g. signals, ugh) I might as well
<ddevault>
if I implement enough bullshit to get vi working that'd be nice
<sortie>
ddevault, try out my Sortix build above if you haven't tried it in a while :D
<ddevault>
where above? got an iso?
<ddevault>
ah I see it
<sortie>
Yup linked an iso and instructions
<sortie>
It's my unstable build but it should be solid today
<ddevault>
I'll be generous and give you two gigs of RAM :)
<sortie>
service(8) exists now though it isn't documented just yet, didn't write the man page
<sortie>
ooooh
<sortie>
1 GB is like minimum if you want to load all of the ports and even that is starting to be at the limit
<ddevault>
are you unpacking all of this stuff from the ISO or an initrd kind of deal
<ddevault>
oh GUI nice
<sortie>
Basically loading the whole ISO into a ramfs on boot
<sortie>
You can optionally select which ports to load
<ddevault>
I was thinking about doing ISO9660
<ddevault>
but all of my real-world test hardware boots over USB
<sortie>
That build even has such a driver
<ddevault>
so I'd have to do XHCI and a bunch of other shit first to use it in practice
<sortie>
Yeah I don't use it for that reason too, no USB Driver
<sortie>
But you can iso9660fs /dev/ata0 /mnt
<ddevault>
it's been a while since I tried sortix out
<ddevault>
this is looking really nice
<ddevault>
networking?
<sortie>
Yep network
<sortie>
ifconfig, ping, curl, ssh, git
<sortie>
Source code in /src
<ddevault>
what hardware, virtio?
<ddevault>
e1000?
<sortie>
e1000 yea
<sortie>
man 4 em
<ddevault>
do you have the qemu incantation for that handy
<sortie>
Works by default
<ddevault>
oh great
<sortie>
See also installation(7)
<ddevault>
ack
<sortie>
There's emacs, nano, ed, my own editor(1) too
<sortie>
man 1 display to read the keyboard shortcuts available
<sortie>
You can install it onto a harddisk using sysinstall(8) per installation, it boots basically instantly
<sortie>
service sshd enable ← to start sshd
<ddevault>
TLS stack also
<sortie>
Yup libressl
<ddevault>
can I change the video mode? want it to be beeg
<sortie>
chvideomode
<ddevault>
thx
<ddevault>
mesa?
<sortie>
cd /src && make ← All that's needed to rebuild the system & ALL PORTS out of the box
<ddevault>
very nice
<sortie>
I deleted the mesa port a few years ago, was too old
<ddevault>
fair nuff
<ddevault>
is this x11 or something else
<sortie>
You can play games like asteroids(1) and aquatinspitz(1)
<sortie>
This is display(1)
<sortie>
My own thing
<ddevault>
cool
<ddevault>
very nice! this is great
<sortie>
service nginx enable ← To start a webserver
<sortie>
service php-fpm enable ← To start a php daemon
<sortie>
You'll need to configure nginx appropriately of course
<ddevault>
sure
<sortie>
curl localhost
<ddevault>
what're the odds that this'll run nicely on my thinkpad?
<sortie>
Pretty high I'd say
<sortie>
This particular build also has some experimental EFI support
<ddevault>
no networking I presume
<ddevault>
do you have EFI runtime support or no
<sortie>
Yeah I did have networking working on older e1000 models but more modern intel chipsets don't work in my testing
<sortie>
sortix.org and irc.sortix.org literally is running on Sortix
<ddevault>
hell yeah
<ddevault>
there is an ethernet cable sitting here on my desk
<ddevault>
dunno where the other end is but I'll plug it in and see what happens
<sortie>
No EFI runtime support, but it does have FAT support and all of that, including bootloader support
<sortie>
You can run ifconfig and see if an em0 device exists. If not, a cable will do nothing
goliath has joined #osdev
<ddevault>
I kind of want to do an efivars kind of thing and write/port efibootmgr so that I can write an EFI-capable installer
<sortie>
I didn't need it because EFI will boot EFI/BOOT/BOOTX64.EFI by default
<ddevault>
right
<sortie>
gcc, g++, python, perl, awk included too
<sortie>
You can git clone some stuff and run it
<ddevault>
I bet awk would just werk on bunnix if I found a suitably simple version to build
<ddevault>
I was tempted to do some research to see how much work it would be to port some other stuff like that, so far my porting experience has been... pretty easy
<ddevault>
if I can get binutils working I can probably have it build hare programs
<sortie>
Ports get a bit harder
<sortie>
Helps if you're using musl
<ddevault>
yeah
<sortie>
gcc and binutils are not that hard ports all things considered
<ddevault>
libiberty though
<sortie>
Only annoyance with that is that you have to port it a couple times
<ddevault>
shit, binutils configure ran against my bunnix sysroot no problem
* ddevault
runs make
gog has quit [Remote host closed the connection]
<ddevault>
it got pretty far!
<sortie>
YES BUILD YOUR OWN ON MINE
gog has joined #osdev
<ddevault>
ah, sortix all booted up on my thinkpad but the keyboard does not work
<ddevault>
mouse works fine (both PS/2)
<ddevault>
my OS won't build on yours
gog has quit [Read error: Connection reset by peer]
<ddevault>
but you'd still need a hare port to build bunnix
<ddevault>
very nice :)
<ddevault>
I've thought about adding a generic-target-posix-and-link-with-libc build target for hare, but it's not there yet (though the openbsd port is close to that)
<bslsk05>
dimkr/lok - A Linux port of OpenBSD's awk (3 forks/6 stargazers)
<nikolapdp>
ddevault ^
<ddevault>
thanks!
<ddevault>
almost builds for bunnix OOTB
<ddevault>
missing popen, pclose, system
<nikolapdp>
which one
<ddevault>
lok
<nikolapdp>
very nice
<ddevault>
currently I am looking at what it'd take to port binutils... and being pleasantly surprised
<nikolapdp>
really
<ddevault>
I need to implement access, mktemp, and utime
<nikolapdp>
only those 3
<ddevault>
to get all of the bfd tools, yes, i.e. objdump, strip, etc
<ddevault>
to get as and ld, not sure yet
Turn_Left has joined #osdev
<nikolapdp>
still, i find that pretty surprising
<nikolapdp>
gnu do like their software portable, to be fair
<ddevault>
same
<ddevault>
oh and I need sysconf but I patched it out
<ddevault>
and I need rmdir
<ddevault>
make -k only errors on these four symbols
<nikolapdp>
pretty cool
<nikolapdp>
i imagine rmdir is mostly trivial
<ddevault>
yeah, more or less
<nikolapdp>
because of lwext4
<ddevault>
well, you'd think
<nikolapdp>
heh
<ddevault>
but actually lwext4 gets in the way as often as it helps
<ddevault>
to be sure on the balance of things it saves a ton of time
<ddevault>
but you should read through my ext4 driver glue code if you want to get a taste of what it does poorly
<nikolapdp>
yeah definitely, i don't see you writing a full ext4 driver in 22 days, no offence :)
<ddevault>
nah of course not
<nikolapdp>
i mean as a part of the project
Left_Turn has quit [Ping timeout: 256 seconds]
<nikolapdp>
do you have a set time for how long you are actively working on this
<ddevault>
¯\_(ツ)_/¯
<ddevault>
I've set a few goals
<ddevault>
one is to have a Unix, which to me means a reasonably complete set of coreutils and an editor
<ddevault>
another goal could be doom
<ddevault>
or self hosting
<ddevault>
in increasing order of difficulty/time investment
<nikolapdp>
fair enough
<nikolapdp>
it's the most advanced os writting in hare at the moment heh :)
<nikolapdp>
s/writting/written
leg7 has joined #osdev
MrCryo has joined #osdev
MrCryo has quit [Remote host closed the connection]
MrCryo has joined #osdev
netbsduser has quit [Ping timeout: 256 seconds]
netbsduser has joined #osdev
<ddevault>
I also considered writing an installer as another "goal"
<ddevault>
which would involve writing fdisk and mkfs.ext4
<nikolapdp>
eh you probably shouldn't vendor autotool generated files
<ddevault>
meh
<ddevault>
I don't want autotools in my dependency tree
<ddevault>
I was tempted to swap the whole dash build system for a makefile
<nikolapdp>
you probably could
<ddevault>
yeah
<ddevault>
on the todo list
MrCryo has quit [Quit: MrCryo]
<nikolapdp>
what mkisofs are you using again
MrCryo has joined #osdev
<ddevault>
xorriso
<nikolapdp>
heh my cdrtools craps out
<ddevault>
cdrtools probably works if you disable EFI support in config.mk
<nikolapdp>
i did
* ddevault
shrugs
<nikolapdp>
Bad Option '-isohybrid-mbr' (error -1 BADFLAG).
<ddevault>
ah
* ddevault
shrugs
<nikolapdp>
is there way to build a disk image
<ddevault>
the build process makes some for you
<ddevault>
target/disk.gpt.img and target/disk.mbr.img
<ddevault>
both have the root filesystem installed in partition 2 as ext4
<netbsduser>
nikolapdp: there is another in that language i have heard of
MrCryo has quit [Client Quit]
<nikolapdp>
netbsduser: could it be helios
MrCryo has joined #osdev
<netbsduser>
yes, that's the one
<nikolapdp>
guess who wrote it
<netbsduser>
i think it was renamed to ARES though (maybe to avoid confusion with HeliOS)
<netbsduser>
nikolapdp: who but yourself?
<ddevault>
the kernel is called helios, the OS as a whole is called ares
<nikolapdp>
ddevault do you have an incantation handy to boot off of the disk image
<ddevault>
the os as a whole is also... not in great shape, which is part of why I'm writing bunnix, since it's more fun than fixing my mess
<ddevault>
nikolapdp: make run
<ddevault>
or see the qemu flags in mk/x86_64.mk
<nikolapdp>
make run depends on iso
<ddevault>
ah
MrCryo has quit [Read error: Connection reset by peer]
<ddevault>
I am not certain that the disk images themselves are bootable
MrCryo has joined #osdev
<ddevault>
but just drop bunnix.iso from the command line in mk/x86_64.mk and see what happens
<nikolapdp>
yeah kind of seems like it
<ddevault>
disk.mbr.img might be bootable
<ddevault>
disk.gpt.img probably not
<ddevault>
I'd drop bunnix.iso, disk.mbr.img, and set the syslinux kernel command line to include root=sata0b
<ddevault>
see if that works
<ddevault>
drop bunnix.iso and disk.gpt.img *
<netbsduser>
ddevault: good to meet the author behind it
<netbsduser>
i had lofty goals for my first kernel and i failed to achieve them, when i went back and did a less sophisticated unix i learnt a lot along the way
<ddevault>
o/
<netbsduser>
i found even with what i thought was a concrete understanding of operating systems gotten from reading books about real kernels, either i was missing some sacred knowledge passed down by kernel elders, or i was just a bit too slow, and that middle area of figuring out the shape of things, between the general architecture of the system and the individual line of code, i wasn't much use at it at first
<nikolar>
isn't that the same process you go through whenever you get out of your comfort zone
<netbsduser>
nikolar: well i found that it was a more difficult process of adaptation
<nikolapdp>
yeah fair enough
<netbsduser>
i think because of the scale of it and the fact that there are so many things which every real kernel dev seems to know but which i have found entirely nonobvious
Arthuria has joined #osdev
<nikolapdp>
kernel development is a pretty esoteric topic, even amongst programmers
<GeDaMo>
Is that the file system trying to get the time?
<ddevault>
no, it's the programs being executed
<dostoyevsky2>
I was thinking, how I could get something booting in qemu but then I realized if I make my OS a simple C program, I don't need any qemu
<ddevault>
clock_gettime is shimmed in libc, anywhere I put a shim that returns EOPNOTSUPP I also put a little printf to tell me what function the program is expecting me to implement
<GeDaMo>
I'm just wondering why ed is asking for the time
<GeDaMo>
Maybe it has a dentist appointment to get to :|
<ddevault>
¯\_(ツ)_/¯
<ddevault>
gcc builds are so slow with make -j1
MrCryo has quit [Quit: MrCryo]
MrCryo has joined #osdev
<nikolapdp>
why is -j1 necessary
<ddevault>
so that if/when it fails I don't have to scroll through a thousand lines of bullshit to find the error
<ddevault>
but I killed it and switched to -j$(nproc)
<nikolapdp>
you can just rerun make -j1 after it fails
<ddevault>
yeah, will do that
<ddevault>
though it's not ideal
<nikolapdp>
better than waiting for literal hours :)
<ddevault>
-j >1 may rush ahead to some targets that fail before the earlier targets are done, so -j1 will run a bunch of earlier targets before getting to the bad one
<ddevault>
need to add -k or run make several times until it stops successfully building other bits of gcc
<heat>
ddevault, do you mean -ENOSYS instead of -EOPNOTSUPP?
<ddevault>
ah, yes
<ddevault>
but honestly w/e
<netbsduser>
just implemented the beginnings of 9p so that i can mount filesystems exported by qemu's builtin 9p-over-virtio-transport server
<netbsduser>
seems to work on m68k (only vnode ops implemented lookup and read yet)
<dostoyevsky2>
netbsduser: oh, that sounds interesting
<netbsduser>
dostoyevsky2: it's very convenient
<netbsduser>
i can avoid wasting time generating images and i can better localise problems in my code when i can say, "it's not my filesystem code being junk"
<dostoyevsky2>
netbsduser: is implementing 9p similar to implementing an nfs daemon? (I wrote one last week)
<netbsduser>
dostoyevsky2: i can't speak to daemons i only have a client here
<dostoyevsky2>
ah... yeah, the server side is in qemu...
<netbsduser>
but as nfs design reflects the sun vfs, 9p design clearly reflects something very foreign and alien to unix
<netbsduser>
less so with the 9p2000.L which is 9p with the oddities (which i can only presume are plan 9's idiosyncrasies) stripped out and replaced with linux's oddities instead
<dostoyevsky2>
that API looks a bit similar to NFS though: https://pkg.go.dev/io/fs (but not sure if it's 9p inspired)
<bslsk05>
pkg.go.dev: fs package - io/fs - Go Packages
<netbsduser>
i don't see much similarity
<netbsduser>
it's all pathnames instead of file handles
Arthuria has quit [Ping timeout: 260 seconds]
Arthuria has joined #osdev
<dostoyevsky2>
is there a standard port that qemu maps the 9p server to?
<dostoyevsky2>
seems like 564 it is
<netbsduser>
it provides it either through a virtio transport or through some xen-specific transport, i'm not aware of it providing any internet transport
<heat>
i mean, not sure if it'll work against mlibc, but it works against convention libcs
<zid>
THE SPELL IS CAST
<netbsduser>
heat: cheers
<heat>
it is weird that m68k (apparently) defaults to crt1 and i'm trying to look for that
<netbsduser>
mlibc leaves crt up to you to define in the sysdeps
<heat>
>sysdeps
<heat>
love the glibc vibes, keep it up
leg7 has joined #osdev
xenos1984 has quit [Ping timeout: 272 seconds]
xenos1984 has joined #osdev
<netbsduser>
i thought it was an mlibc idiosyncrasy at first but when i cross referenced glibc to figure out TLS on m68k that revealed the truth
<zid>
heat your summoning spell hasn't had an effect yet
<netbsduser>
the mlibc idiosyncrasy is probably naming files "-stubs.cpp" even when they are wholly complete implementations
<heat>
mlibc is in C++?
<netbsduser>
completely
<heat>
now that is a warcrime
<MrBonkers>
I was summoned?
<MrBonkers>
and yes, it is
<zid>
There we go, delayed action
<heat>
do you actually use, uhhh, the C++-y bits
<zid>
mustn't have put enough mana into it
<heat>
or is it C in a C++ file
m257 has quit [Ping timeout: 250 seconds]
<heat>
netbsduser, hmm no disregard what i said, even though it /may/ work
xal has quit []
<heat>
you need to unfuck your m68k target out of the gnu stuff
<heat>
gnu-user.h specifies crt1.o
xal has joined #osdev
xal has quit [Client Quit]
<heat>
i'm surprised your other targets pick crt0.o at all?
xal has joined #osdev
<Ermine>
hi
<nikolapdp>
hello
xal has quit [Client Quit]
xal has joined #osdev
xal has quit [Client Quit]
voidah has joined #osdev
Jari-- has quit [Remote host closed the connection]
vikn has quit [Excess Flood]
vikn has joined #osdev
gog has quit [Quit: Konversation terminated!]
xal has joined #osdev
<leg7>
does `iret` pop the error code for exceptions which push an error code onto the stack?
<leg7>
The manual doesn't say it does so I assume it doesn't
<sortie>
ddevault: Just use make -k -j to get through it all, then use make -j1 to get the actual failures
<leg7>
however I think something is wrong with the error codes and the stack in my code because If I pop in manually the program returns at the wrong address because my stack is wrong
<sortie>
ddevault: Congrats on gcc port, told ya it wasn't that hard :)
<zid>
I'm hungry but the asda man won't be here until 8pm :(
<oldgalileo>
Is it setting up space in the Kernel's ELF headers for EFI debug symbols?
gog has joined #osdev
<heat>
leg7, no
<heat>
oldgalileo, yep
<oldgalileo>
I guess more specifically I'm curious how I might go either loading EFI symbols for the Kernel to use when unwinding during an EFI crash **or** figuring out the mapping so that I can solve for the addresses myself
<heat>
not the kernel's ELF headers, but the efi stub's PE headers
<MrBonkers>
heat: Some C++ items are used, but it obviously needs no C++ runtime support (the things it does use are mostly STL and provided through a header only library made by us)
<MrBonkers>
qookie can probably be more exact here
<heat>
MrBonkers, yeah, personally i wouldn't do so, despite the stylistic choices, it's a bit risky to use something and accidentally find yourself in a bad dependency
<heat>
e.g system() using std::string which is not standard compliant (as you'll pull in malloc, string functions, etc)
<MrBonkers>
oh C library functions are standard compliant if that's your complaint
<MrBonkers>
(if I understand you correctly)
<oldgalileo>
heat: Thanks!
<MrBonkers>
std::string is for example not found in the entire repo
<heat>
i'm not complaining
<heat>
just saying i'd find it risky to use any sort of standard library because you can accidentally find yourself with nasty namespace violations
<MrBonkers>
it is, which is why we don't
<heat>
(luckily C++ name mangling is okay because of _ZN3uppercase)
<MrBonkers>
frigg is fully freestanding made by us
<MrBonkers>
it has to be, as we use it in the kernel too
m257 has quit [Ping timeout: 250 seconds]
xenos1984 has quit [Ping timeout: 268 seconds]
<oldgalileo>
Apologies in advance as I'm not super familiar with much of this (I've been learning over the last few days)
<netbsduser>
looking at the system() example i can see it's doing nothing C++ish
<ddevault>
just the build driver (our hare-aware equivalent of make), but it's a start
<nikolapdp>
very nice
<nikolapdp>
so no qbe/harec
<ddevault>
I have decided that my definition of done includes a working Hare and C toolchain, though I'll probably fall short of full self hosting unless GNU make is particularly easy to port
<ddevault>
nikolapdp: yeah
<ddevault>
though qbe builds already
<ddevault>
oh harec builds also
<ddevault>
so...
<nikolapdp>
nice
<nikolapdp>
give it ago
<nikolapdp>
a go
<ddevault>
derp
<ddevault>
I built harec and qbe for bunnix
<ddevault>
then tried to build bunnix (on linux) with a hare toolchain built for bunnix
<nikolapdp>
kek
divine has joined #osdev
MrCryo has quit [Remote host closed the connection]
<bslsk05>
udo-munk/s - A tiny vi like screen editor (16 forks/61 stargazers/Unlicense)
<ddevault>
if I can get C and Hare toolchains I will be about as capable as early Unices in a similar timeframe :)
<nikolapdp>
it build on unix v7 :)
<ddevault>
nikolapdp: nice
<ddevault>
doesn't seem to work on linux though (builds, does not seem to do much of interest when run)
<nikolar>
Eh I haven't tried on linux
<nikolar>
Probably isn't too hard to make it work
<ddevault>
"works" in xterm
<ddevault>
meh
<ddevault>
I'd sooner just stick with ed
<ddevault>
it is the standard editor
<netbsduser>
heat: i call upon your expertise
<netbsduser>
i want to improve my futexes by not pinning the physical page
<netbsduser>
i am thinking of instead separating futexes in MAP_PRIVATE from MAP_SHARED mappings and having it so that MAP_PRIVATE futexes are process-local and keyed by virtual address, while MAP_SHARED futexes are global and keyed by {vm_object, offset into vmo}
<netbsduser>
but i wonder how fork() interacts with this
<netbsduser>
can some mad bastard just allocate some private memory, fork(), and share a futex in that private area between parent and child?
<netbsduser>
my interpretation of linux man 2 futex is that you've got to use shmat or mmap() with MAP_SHARED to be able to share a futex between processes
<nikolar>
ddevault: why the quotes
<nikolar>
How bad is it
<netbsduser>
and indeed the example in that manpage which shares a futex between forker and forked calls mmap with MAP_ANONYMOUS | MAP_SHARED (a combo i don't support yet, probably imlpement it by allocating a vm object to do the sharing)
<ddevault>
it's bad
<childlikempress>
netbsduser: if it's a private mapping then wouldn't it be cow and not shared when you fork
<childlikempress>
so then the futex shouldn't be shared
<bslsk05>
github.com: Onyx/kernel/kernel/futex.cpp at master · heatd/Onyx · GitHub
<heat>
it's very simple stuff
<heat>
and yes, your interpretation is correct, MAP_PRIVATE is MAP_PRIVATE and futexes cannot do anything wrt CoW (in fact, that would make little sense, because CoW is technically not exposed to userspace)
<netbsduser>
heat: and if someone truncates a mapped file that someone was waiting for a futex on? just leave the futex alone (since you're not actually dereferencing the vm object pointer, just comparing if it matches) and let the moron who did that eat a very long wait?
<netbsduser>
i have truncation on the brain lately
<heat>
hmm that's a weird situation but i guess?
<heat>
play stupid games win stupid prizes
teardown has quit [Ping timeout: 260 seconds]
teardown has joined #osdev
<geist>
hmm, thinking about that. i guess if you block on a futex on an anonymous page, it can't be swapped out
<geist>
depends on how the indexing works, really, but its my understanding in linux it indexes on basically the physical page that backs it
navi has quit [Read error: Connection reset by peer]
<childlikempress>
why wouldn't you be able to swap it out
<geist>
because then the physical address goes away
<geist>
and when it swaps back in it might be a different page
<geist>
but it all depends on how they index it, and how swapped out pages are represented i guesws
<childlikempress>
well. you need some kind of persistent id. but you need that anyway for aliasing
<geist>
right, that's why traditionally i think the physical backing page is the id
navi has joined #osdev
<geist>
which is also why COW interacts with futex
<geist>
and a COWed copy becomes its own futex basically
<childlikempress>
i'm saying, you need an id that persists when you swap because otherwise aliasing doesn't work
<childlikempress>
so you can also use that for futex
<childlikempress>
no?
<geist>
yah, presumably they replaced a swapped page with effectively a pointer to a slot in the swap file, so then you end up using that as your id
gbowne1 has joined #osdev
gbowne1 has quit [Remote host closed the connection]
gbowne1 has joined #osdev
<heat>
geist, on linux anonymous memory is always private
<heat>
shared anon is actually secretly file backed, and those are represented in futexes as vm_object (address_space) + offset
<heat>
private is just address_space + address
<heat>
tmpfs is actually (AFAIK) the one filesystem that supports regular swap, and they just store special entries in the radix tree, instead of the folio*
<heat>
worth noting that the lifetime of the struct inode is tied to struct address_space, so address_space + offset is a stable reference while the vfs inode is alive
<Ermine>
futexen
<heat>
futexen patchen
<geist>
ah that makes sense. we only support futexes in zircon on a single address space, so we do the same thing and key it off of aspace + offset
<geist>
we could key it off of VMO + offset, but since we dont need to share it it's simpler to just go with functionally the address of it
<childlikempress>
how do you interprocess sync
<childlikempress>
do you need to use pipes or sockets if you wanna block
<childlikempress>
😿
xFCFFDFFFFEFFFAF has quit [Read error: Connection reset by peer]
foudfou has quit [Remote host closed the connection]
foudfou has joined #osdev
<heat>
*futexen
<heat>
geist, not just simpler but you get palpable WINS by doing so
<geist>
WINS
<geist>
childlikempress: zircon has a bunch of ways to interprocess sync, futexes is just not one of them
<heat>
seriously going down the page table chain with locks is bad mkay
<heat>
i think linux has a special version of their page table traversal that does not lock but does RCU instead
<heat>
and they can only use it in special occasions
<geist>
yah i was digging through KVM code last night and they have some code there that walks the S2 tables using RCU as well
<heat>
rcuuuuuuu
<geist>
lots of 'look up what this fault was at' when doing PFs via vmexit
<heat>
yeh
<heat>
yesterday i was writing a lockless slab shrink thingy and i had to do a bunch of ATOMICS and MEMORY MODEL THINKING STUFF
<nikolapdp>
did you THINK HARD heat
<heat>
yes actually
<zid>
nikolapdp keeps offering to show me his private futex what do I do
leg7 has quit [Remote host closed the connection]
voidah has joined #osdev
<heat>
futex is serbian for balkan warcrimes
<nikolar>
no, you're thinking of ратни злочини
<nikolapdp>
or as we say on pdp11 `QP0QP=P8 P7P;P>QP8P=P8`
<heat>
well said
<nikolapdp>
thank you
<heat>
you're very welcome
<zid>
I love a bit of battle mayhem
<heat>
zid, do you know the popular serbian expression "није ратни злочин ако је непријатељ нељудски"?
<nikolar>
*нечовек
<nikolapdp>
google translate does indeed suck
<zid>
It's not a warcrime if the enemy is portugal? Agreed.
<nikolapdp>
that works
<heat>
woah that's not what our trusted oldest ally in the world uk should say
<nikolapdp>
don't worry heat, it's nothing personal
<nikolapdp>
they warcrime everyone
<heat>
uk never warcrimed portugal
<zid>
The US likes to warcrime us, we're just passing it along
<heat>
the US also never warcrimed the UK
<zid>
does friendly fire not count
<heat>
no
<zid>
they do it constantly
<heat>
that's not a war crime
<heat>
also RIP pat tillman
<nikolapdp>
that's just a skill issue
<Ermine>
Oh I can interpret those
<nikolapdp>
interpret what
<heat>
cyrillic moment
<Ermine>
what was written in serbian
<Ermine>
so yeah, cyrillic moment
<nikolapdp>
kek
<heat>
кэк
<Ermine>
кек actually
<nikolapdp>
i am no russia
* kof673
points at chrono trigger frog
<nikolapdp>
*russian
<kof673>
white belly perhaps :D
<heat>
Ermine, how do you say linux in russian
<Ermine>
leenuks
<Ermine>
линукс
<kof673>
and must be cooked gradually, nature makes no leaps and bounds lol
<zid>
don't trust nikolar, he's on russia's side in ukraine
<nikolapdp>
i am not in ukraine
<zid>
good for us, cus you'd be on russia's side
<nikolapdp>
why would i
<heat>
zelenskyy uses rust
<nikolapdp>
ok now i am on russia's side
<zid>
Because ukraine splintered off from a former empire, like kosovo, so they support kosovo's right to independence
<heat>
i do not feel comfortable calling serbia an empire
<Ermine>
Putin is protected of traditional C values
<nikolapdp>
serbia in the 90s an empire?
<nikolapdp>
lol
<zid>
not in the 90s
<nikolapdp>
oh right, we're talking about the 14th century then
<zid>
yes
<nikolapdp>
SERBIAN EMPIRE
<heat>
i wake up every day in fear of the albanian empire
<heat>
skanderbeg's coming back one of these days
<Ermine>
olbanian
<zid>
catalonia also supports kosovo, to be relevent to heat
<nikolapdp>
don't forget, skanderbeg was a serb :)
<heat>
we're all serbs deep down
<nikolapdp>
i mean sure, but skenderbeg was of a serbian noble family
<zid>
me too
<zid>
It's why I live in povery
<zid>
poverty
<nikolapdp>
keep dreaming zid
<zid>
I wish I was a brit civilian instead of a serbian noble :(
<nikolapdp>
you'll never be glorious as the serbian empire
<nikolapdp>
*as glorious
op has quit [Remote host closed the connection]
gog has quit [Quit: byee]
gog has joined #osdev
Left_Turn has joined #osdev
Turn_Left has quit [Ping timeout: 256 seconds]
Left_Turn has quit [Read error: Connection reset by peer]
foudfou has quit [Remote host closed the connection]