youcai has quit [Read error: Connection reset by peer]
youcai has joined #osdev
hwpplayer1 has joined #osdev
X-Scale has quit [Ping timeout: 256 seconds]
X-Scale has joined #osdev
X-Scale has quit [Ping timeout: 256 seconds]
X-Scale has joined #osdev
X-Scale has quit [Ping timeout: 256 seconds]
slow99 has quit [Quit: slow99]
slow99 has joined #osdev
theruran has quit [Quit: Connection closed for inactivity]
<spiderella>
i have an idea for a thing and i would like some help with brainstorming it. it's very linux specific but i think this would be the best channel to ask
xenos1984 has quit [Read error: Connection reset by peer]
<spiderella>
suppose there was a system that acted as some kind of a weird runtime that did magic under the hood
<spiderella>
amongst the magic things, this system interacts with thread creation and memory maps, specifically when you call clone to create a thread it creates a process instead
<spiderella>
and also all private maps are magically shared maps instead. they're prefaulted in and copied from the underlying file into a shared map when they're created
<spiderella>
and all calls to getpid would magically return the original pid
<spiderella>
and the right clone flags are passed so that the file descriptors are shared, the cwd is shared
<spiderella>
and calling exit_group in one is magically replicated in all the others
<spiderella>
so, lots of magic
<kof673>
1) nice nick 2) sounds like you want interceptor pattern :)
<kof673>
that is, transparent to the program
<kof673>
"magic"
<spiderella>
well i ideally want to run normal binaries
<spiderella>
what i'm wondering is: if a program is not *actively* trying to figure out that they're in my weirdo env, would something break?
<zid`>
lots of things?
<spiderella>
please list some
<spiderella>
like assume that they're not looking in /proc/self or something
<Ermine>
you can LD_PRELOAD and intercept syscalls
<spiderella>
and uhhh let's say no direct interaction with block devices
<Ermine>
replicating exit_group would be tricky imo
<zid`>
literally anything with threads that uses shared memory?
<spiderella>
shared memory is still shared
<spiderella>
all memory is shared :)
<spiderella>
so those work
<zid`>
err right, literally anything with threads at all* :P
rlittl01 has joined #osdev
<zid`>
On the assumption they're not just starting threads 'for fun'
<spiderella>
ok but all the processes see the same memory map
<spiderella>
they map the same physical pages at the same addresses, just like they would in a normal process. they just happen to be shared maps so the writes in one process are visible to the other processes
<spiderella>
so i'm not immediately sure what would break
<spiderella>
there is a reason for doing this i promise
<spiderella>
but like it's quite a bit of work and i might be missing some obvious reason why the final goal doesn't work
<spiderella>
but i think threads would work and not notice any difference
<sortie>
spiderella: So you want to run each thread in its own process? But why?
<zid`>
yea I guess various simple things would work fine
<zid`>
as long as you got the split absolutely correct
xenos1984 has joined #osdev
<sortie>
spiderella: As a rule of thumb, there's always a way to detect your environment, even information leaks like timing attacks might even defeat full on virtualization
<Ermine>
you'll definitely find out when you implement your thing and see something break, or not break
<sortie>
So the question is rather whether the set of software you want to run in your env has all the things they require
<spiderella>
i get it sortie but i don't plan to run adversarial software
<spiderella>
i want normal things to not break too much
<zid`>
it'll break as much as you can hide it
<sortie>
So for things like mmap you want to propagate the changes to the address space to all the processes?
<zid`>
or rather, as much as you can't hide it
<spiderella>
yes sortie new mmaps will magically be propagated
<sortie>
But again, why? This seems like a lot of work for .. no benefit and a lot of breakage to just reinvent the process?
<zid`>
what's the opposite of a green thread? :P
<zid`>
purple threading?
<sortie>
In a sense, this is kinda what Linux already does since threads == processes for it to some extent
<kof673>
yes, red surely..
<sortie>
Except it pretends it's not the case
<spiderella>
the benefit is that now i can change the protection of some memory for a "thread"
<sortie>
Then you unpretend that
<sortie>
And then repretend it
<sortie>
Aha. That's a little bit interesting. Per-thread address space protections.
<sortie>
I imagine a better way of doing that is building true kernel support for it
<zid`>
yea that's inverting linux's model a little
<zid`>
you'd want yea, a per thread page table that you swapped into/out of realistically in the kernel
<sortie>
Although it is weak since a hostile thread can probably trick another thread into doing its bidding
<spiderella>
hostile/adversarial programs are phase 2 of the project :)
<sortie>
Splitting your program into multiple processes is a more simple and secure way of giving different trust levels different address spaces
<spiderella>
phase 1 is getting a hello threading world working
<zid`>
and at that point yea I agree, they're basically just processes, but that comes with *different* baggage too
<sortie>
Well don't split your project into the first phase which is possible and the second phase which is impossible. Your design gotta handle hostile workflows from the get go
<spiderella>
that just makes phase 1 impossible too :)
<zid`>
At *some* level, you're going to need to replicate mmaps between address spaces (i.e if I have 12 threads, regardless of if I have 12 processes, or 12 threads but with unique page tables, I need to do 12x the work)
<spiderella>
yes true
<zid`>
but sticking to threads rather than processes means you at least get the fd sharing and stuff automagically
<sortie>
Per-thread address spaces have some interesting properties like being unable to mess with foreign stacks (although a thread can legitimately send a buffer on its stack to another thread)
<zid`>
so keeping it as threads not processes makes more sense to me at least
<sortie>
Linux clone might actually let you keep it the same process but fork the address space
<sortie>
idk if supported
<spiderella>
no but it can do the opposite
hwpplayer1 has quit [Ping timeout: 252 seconds]
<spiderella>
different processes sharing the address space
hwpplayer1 has joined #osdev
hwpplayer1 has quit [Remote host closed the connection]
<spiderella>
making this a linux feature would be the right way to do it but then i can't run my thing on an unmodified kernel
<zid`>
I don't think you'll get far with just LD_PRELOAD regardless
<spiderella>
correct
<zid`>
but you're talking as though it could run on an unmodified kernel somehow though?
<spiderella>
:) hopefully
<zid`>
how?
<spiderella>
mix of ptrace and stuff
<zid`>
oh wow
<spiderella>
ideally as little ptrace as i can get away with!
<spiderella>
i have a phase 0.5 tho
<spiderella>
and this is the part where you're going to be like "wow this person doesn't understand how computers work"
<zid`>
I mean, it sort of already is the case :P
<zid`>
We already have exactly this system it's called "just use processes to start with"
<zid`>
people decided the effort slash security slash whatever tradeoffs made threads worth using, sadly
<zid`>
(monsters)
<nikolar>
zid`: that's why you don't have to implement threads in your own os :P
<spiderella>
readable maps stay the same, but writable maps are only writable by one thread at a time, and every other thread gets prot_none. accessing a prot_none map that's supposed to be writable segfaults, the fault handler marks the page as prot_none in every other thread, then marks it as writable for the faulting thread
<zid`>
"make everything really slow so that that guy who really loves threads can stay using them" is not my ideal universe
<spiderella>
now, i think if you squint, this still behaves in a way that's possible on a normal system
<zid`>
I'd rather just beat that guy with a hammer
<spiderella>
zid`: i get that synchronisation will be slow
<zid`>
people will just look at it and go "guess we're doing green threads now" and undo all your work imo :p
<spiderella>
oh yeah that's ok!
<spiderella>
i don't mind that! 👍
q3lont has joined #osdev
<kof673>
just translate to a vm/bytecode, then you can override every single instruction like story of mel. see you in 30 years :D
<spiderella>
who says i'm not doing that 👀
<zid`>
This is why I haven't become supreme overlord of computing
<spiderella>
(i'm not doing that)
<zid`>
because I'm too lazy to hunt all the dissidents
<spiderella>
my bar for acceptance is "the result is still a valid behaviour for the system had i not done any of that"
<spiderella>
there are reasons for doing this i promise
<zid`>
My gut feeling is that you will rapidly get partial compatibility
<zid`>
then have to chase edges forever to approach 100%
<spiderella>
ah yeah classic
<zid`>
logistic curve or whatnot
scaleww has quit [Quit: Leaving]
q3lont has quit [Quit: Leaving]
emntn has quit [Ping timeout: 255 seconds]
emntn has joined #osdev
<immibis>
spiderella: the linux kernel doesn't distinguish between processes and threads. processes are just threads with nothing shared; threads are just processes with almost everything shared.
<immibis>
glibc already emulates syscalls like getpid so they return the same value in all threads
<immibis>
from the kernel's perspective you can use a mix of sharing flags. you can make a thread with its own memory map. it will probably confuse glibc and crash glibc.
<immibis>
the thread with its own memory map can still be in the same thread-group (the closest thing the kernel has to a process), can still share its file descriptor table, etc
<immibis>
see: man 2 clone
<spiderella>
immibis: you cannot make a thread with its own memory map
<spiderella>
Clone thread without clone VM doesn't work
emntn has quit [Ping timeout: 246 seconds]
emntn has joined #osdev
<spiderella>
I actually don't believe getpid works like that anymore in glibc because too many things broke when they did cached the value
<spiderella>
Other things like setuid absolutely need libc to be dispatched to all threads
Dead_Bush_Sanpai has quit [Remote host closed the connection]
Dead_Bush_Sanpai has joined #osdev
<spiderella>
I'm not nearly as smart as the people behind glibc so I'm not letting the fact that they decided it wasn't worth caching the pid dissuade me from doing the much more intrusive stuff
<spiderella>
Also my bar for acceptance is way lower
orccoin has quit [Ping timeout: 252 seconds]
bauen1 has quit [Quit: applying updates or something like that]
hwpplayer1 has joined #osdev
edr has joined #osdev
heat has joined #osdev
<heat>
<immibis> glibc already emulates syscalls like getpid so they return the same value in all threads
<heat>
not true, getpid(2) deals with threads itself
<heat>
linux has the notion of threads as being processes that share thread groups. the thread group's leader's pid is what's returned by getpid
<heat>
just like getppid() returns the thread group leader's parent's pid
Gooberpatrol66 has quit [Ping timeout: 245 seconds]
Turn_Left has joined #osdev
Left_Turn has quit [Ping timeout: 265 seconds]
goliath has joined #osdev
netbsduser has joined #osdev
aethlas has quit [Quit: bye]
aethlas has joined #osdev
the_oz has quit [Ping timeout: 276 seconds]
emntn has joined #osdev
the_oz has joined #osdev
<immibis>
spiderella: you expect it to be fast so you don't cache it in your own code
jedesa has quit [Remote host closed the connection]
<heat>
maybe the real getpid workload was the friends we made along the way
<Ermine>
today in "ermine whines at setuptools"
<Ermine>
setuptools vendors another package, packaging, two times and those are different versions
<nikolar>
is that the python thing
<Ermine>
yes
<heat>
fix all your python issues by disaling the GIL
<Ermine>
import multiprocessing
<immibis>
dependencies were a mistake
<gog>
what if we injected them
<immibis>
they should be painful so you'll only use them when really useful
<nikolar>
heat: fix all your python issues by not having python
<heat>
and make them perl issues? fuck no
<heat>
perl was outlawed in the geneva convention
<Ermine>
more like geneva suggestion
Dead_Bush_Sanpai has quit [Read error: Connection reset by peer]
Dead_Bush_Sanpai has joined #osdev
<Ermine>
kbuild apparently doesn't use perl, right?
<kof673>
i never really used perl, although there was a "wizard book" i picked up at a surplus sale once lol but always remember: this is what happens when a linguist who knows language designs one
<Ermine>
high linguist*
<heat>
linux kbuild uses perl
<Ermine>
what about your?
<heat>
onyx uses python in one or two tidbits of the build system
<kof673>
when i was cobbling together an old qemu but new enough to have riscv64 user, i found qemu needs ninja :/ it was not too bad, libffi (already had built for something else) glib and then python for ninja
<kof673>
> distutils was deprecated in 3.10 and removed in 3.12. and then a tad older python to avoid whatever that was about
<Ermine>
there's samurai
Turn_Left has quit [Read error: Connection reset by peer]
melba has quit [Ping timeout: 252 seconds]
goliath has quit [Quit: SIGSEGV]
<netbsduser>
i have ordered the AIX book
<netbsduser>
let's see if i can add another source to the list i've pilfered ideas from