vdamewood has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
sprock has joined #osdev
froggey has quit [Ping timeout: 258 seconds]
<riverdc>
can anyone recommend some x86-64 learning resources, as a target for a compiler I'm writing?
sprock has quit [Read error: Connection reset by peer]
<riverdc>
the only instruction set I've implemented so far is aarch64
sprock has joined #osdev
<Mutabah>
Hmm... well, the intel/amd manuals are a first step
<Mutabah>
(... first priority)
<Mutabah>
And maybe an x86 assembler tutorial
mctpyt has quit [Ping timeout: 272 seconds]
mctpyt has joined #osdev
<doug16k>
ZetItUp, that code is ridiculous. there is no need to screw with segment registers
<ZetItUp>
doug16k yeah i thought so and also, can't do that in 64 bit mode so :D
<ZetItUp>
need a break now thou, off to bed :D
<doug16k>
insl will work already without touching es, because es already has to be the same as ds, because memset/memcpy string instructions already use it
sm2n_ has joined #osdev
sm2n has quit [Ping timeout: 252 seconds]
ElectronApps has joined #osdev
tacco has quit []
gog has quit [Ping timeout: 265 seconds]
isaacwoods has quit [Quit: WeeChat 3.2]
<doug16k>
if it wasn't such a pain to sign into the wiki, I'd fix that crap right now
<doug16k>
whoever wrote that looked up ins instruction, saw that it stores to es:[di], and panicked
<doug16k>
es!!!!!11111
<doug16k>
in flat memory model, ds and es and ss are the same
<doug16k>
gcc already requires flat memory model, so it's flat
<klange>
well, for _most_ code, gcc requires a flat memory model :)
sprock has quit [Read error: Connection reset by peer]
sprocklem has joined #osdev
<doug16k>
it is already counting on ds and es being the same so it can inline rep movs
<doug16k>
and it is counting on ds and ss being the same because it can use ebp as a pointer
<klange>
Yeah, the non-flat memory support in gcc is... kinda specific to tls and explicit address space attributes
<doug16k>
and it is counting on ds and cs being the same because it assumes it can point at code
<doug16k>
yeah fs and gs are special. nothing implicitly uses them in the ISA
<doug16k>
for data references anyway
<doug16k>
TLS uses them yeah
<doug16k>
ABI level
<klange>
Getting GCC to spit out raw GS-relative stuff in Misaka for the core-local stuff was fun.
<klange>
Was reminded of that from the conversation a few days ago about hard-coded absolute addresses, was going to suggest a `static const` pointer - that and the gs-relative address space attribute are how I managed to not get it to not produce a bunch of pointer dereferences.
sprock has joined #osdev
sprocklem has quit [Ping timeout: 272 seconds]
<klange>
static struct PorcessorLocal __seg_gs * const this_core = 0; /* then this_core->whatever is always a direct gs:memory reference, just as good as a global struct */
<doug16k>
if you make the struct begin with a pointer to itself, you can declare it as a ** and fetch the pointer and use it normally without segment override
<doug16k>
careful you don
<doug16k>
t get migrated to another cpu after
<doug16k>
that's what the TLS ABI does
<klange>
The TLS ABI is whack, but also it has to deal with multiple arbitrary allocations in "TLS space"
<doug16k>
using it completely directly is actually a trick that many architectures can't do
<geist>
i seem to remember looking into the __seg_gs stuff but we decided we couldn't use it for some reason
<geist>
possibly compiler bug related
<doug16k>
most architectures have to fetch a pointer, then read the struct
<klange>
The nice thing about this is that, syntactically, it doesn't matter what needs to be done behind the scenes.
<klange>
It's entirely gcc realizing a constant pointer value can be done that way and x86-64 supporting the gs-relative instructions.
<doug16k>
I did it manually
<doug16k>
but at the end of the day, it does what __seg_gs does
<klange>
I'm not sure how I'd do kernel "TLS" on, eg., ARM, where gcc doesn't provide address space attributes?
<doug16k>
you read a special register into a general register, then use that
<geist>
right, that's one of the reasons why on a portable system it's not great to bake that sort of notion into your API
<geist>
vs something like 'get_current_cpu()' or 'this_core()'
<klange>
In which case `this_core` would probably be a macro that does that anyway.
<geist>
right
<doug16k>
yeah, but it could just read that special register on non-x86
<doug16k>
and be non-atomic
<geist>
so for risc architectures it tends to either be a special control register taht you move into a regular one and then deref (ie, TPIDR_EL1 for arm64)
<geist>
or you burn a cpu with something like -ffixed-x18
<geist>
and then arrange for x18 to just hold your pointer
<geist>
or use say 'GP' on riscv
<geist>
the latter gets you basically atomic access a-la x86
<geist>
at least for reading or writing somehting (that can be done in a single instruction)
flx has joined #osdev
<doug16k>
consider disabling interrupts while holding a pointer to this cpu, in case you get preempted and migrated
<geist>
yah. even for the gs: case suggest pointing a pointer to itself inthe first slot, so you can get something liek get_current_cpu_struct(), but be aware that any preemption that moves the cpu is dangerous
<doug16k>
or do whatever thing you have to prevent context switch (enter critical section)
<klange>
I've been avoiding kernel pre-emption.
<doug16k>
no preemption in syscalls either?
sts-q has quit [Ping timeout: 268 seconds]
<klange>
Nope, all cooperative yields where relevant.
nur has quit [Remote host closed the connection]
nur has joined #osdev
sts-q has joined #osdev
<doug16k>
so do you check at the last moment when returning from syscall whether they need preemption?
<doug16k>
it doesn't disable interrupts for the syscall right?
<doug16k>
I am just curious
<doug16k>
I have only done psycho threaded style :P
<doug16k>
where it can be preempted almost anytime, unless it is in a critical section for a reason
<geist>
same
<geist>
though there are times when i think the voluntary preemption mechanism would be nice
<geist>
could even allow interrupts in the kernel, just no forced preemption in that case
<geist>
would be fairly responsive, but a lot simpler
<klange>
I could probably enable interrupts and just block the pre-emption signals, as I have made sure all the other interrupt handlers do not perform scheduling.
<klange>
But at the moment system calls are still just trap gates.
<klange>
I think with the work I'm doing to make SMP stable, there's no reason to be blocking interrupts, just ignore the pre-emption signal.
<klange>
Or even mark the thread as requiring pre-emptive switch on return.
<doug16k>
I have basically that. I can enter critical sections (nested) and, if a reschedule is attempted while the count is nonzero, it increments a deferred reschedule counter and continues the thread
<doug16k>
each critical section exit, it checks if it is last, and if it is, it checks the deferred cswitch flag, and if it is not zero, it clears it and volunteers a context switch right then
<doug16k>
and it can tell if it is an interrupt handler, and instead tells it to do a reschedule on the way out
<geist>
can have the irq handler just implicitly bump it
<geist>
then the same logic also handles preemption on the way ot of irq handlers
<geist>
if it nests, no sweat, but if it bumped/debumped with no implicit existing one then you have killed two birds with one stone
<doug16k>
yeah, there is a very high probability that the whole point of an IRQ was an I/O completion, and it probably awakened something
<geist>
right, but if it diodn
<geist>
if it did't then there's no queued up preemption and no sweat
<doug16k>
yes
<doug16k>
it might be very very cheap to peek at the scheduler data and see if the current thread would still be 1st place
<doug16k>
that part is fun because there are many ways to do something to reschedule there
<klange>
In a manner of speaking, Misaka is essentially a cooperatively scheduled system. There are kernel threads, and a kernel thread only ever explicitly yields or exits. The preemption timing source is just another "system call" that happens on its own, and it just yields like any other function in the kernel would.
<doug16k>
the kernel thread jumps into user mode and runs the user code?
<klange>
Page faults? User traps? Just more "system calls" that possibly yield, possibly exit. Hardware interrupts are... special, but they're not allowed to yield. Kinda like they exist outside of the realm of threads.
<geist>
that's preemptive. if user space code can be rescheduled whenever an irq fires, that's preemptive
<geist>
it's just not preemptive in the kernel
<geist>
ah. so in this case you *dont* preempt based on a hw irq (including timers)
<doug16k>
what if a user program does while (1) lol=1;
<klange>
Nah, you're right on the first thing - it _is_ preemptive, just not in the kernel.
<doug16k>
do you have a bunch of different kernel stacks, one for each kernel thread? and more threads than cpus?
<klange>
For scheduling purposes, the kernel only knows kernel tasks. User task context only exists because of interrupts saving it. There's fundamentally no difference between the int call for a system call and the hardware irq from the pre-emptive timing source.
<geist>
sure, that's pretty standard
<geist>
when you get into killing processes and whatnot that tends to leak at least some sort of notion of user space and not into deeper stuff, but only because you get certain additional powers if a thread is functionally sitting in user space
sprock has quit [Read error: Connection reset by peer]
sprock has joined #osdev
sprock has quit [Client Quit]
<doug16k>
there aren't kernel threads then?
<klange>
Yeah I don't think I'm accidentally doing anything esoteric?
<klange>
Every thread has a kernel stack, because every thread is a kernel thread. Is a thread a user thread? idk, only if it "returns" to userspace somewhere [Okay I do know because I have a flag on the thread that says what it is because just letting this happen naturally is terrible for debugging]
smeso has quit [Quit: smeso]
<geist>
sure
<doug16k>
that's how I do it. there are only kernel threads, and they may or may not be running the user code
<doug16k>
you can pretty much context switch any time
<doug16k>
deadlock avoidance permitting
<doug16k>
in the middle of a syscall, who cares
<klange>
The only thing exceptional to the model is the idle task, which would be just another kernel thread, but there's some cheating going on there; it enables interrupts, doesn't save context, and restarts from the top whenever it's scheduled anew.
<doug16k>
the idle thread actually does the most work on a gui desktop
<klange>
yep
<doug16k>
it handles almost all the irqs
<klange>
indeed it does
<klange>
and yet it spends all of its own time doing whatever it can to do nothing
<doug16k>
it's a philanthropist
<klange>
it just wants to sleep!
<klange>
but the phone keeps ringing
smeso has joined #osdev
<doug16k>
it's the environmentalist thread
<doug16k>
it tries to reduce CO2 emissions as much as it can
<klange>
It's the retired soldier hero in an action movie: It just wants to settle down, but the government has a job for it. Repeat for n sequels.
mctpyt has quit [Ping timeout: 252 seconds]
<doug16k>
my code says screw it and saves and restores everything when the idle thread takes an interrupt
<doug16k>
it would be pretty cheap to skip over them
<doug16k>
the context save and restore
<doug16k>
everybody loves low irq latency
<doug16k>
I wonder though, would the mispredict and restart cost less than just saving it
<doug16k>
oh
<doug16k>
actually the predictor would start to assume it is skipping and would skip it really well
<doug16k>
if idling
<danieldg>
deeper idle states kill the predictor state anyway
<doug16k>
yeah if deep idling then the idle thread context save is far from a concern
<doug16k>
screw c-state
<doug16k>
istate is more like it. instability
<doug16k>
g state, for "good luck with your vrm staying stable"
<doug16k>
I turn that crap off
<doug16k>
idling 38C on just regular mwait
<doug16k>
on a laptop, I say go for it, they made absolutely sure it works perfect there
mctpyt has joined #osdev
<doug16k>
imagine being asked to make a switching power supply that has microsecond response times and a range up to 100 amps and down to 15 amps? you can't. the cpu limits the ramp up/down. better if you start from well within operating range than from hardly any load
<doug16k>
I dislike cstate because I know how unreasonable it makes the vrm's job
<doug16k>
it has to be able to balance one pin on the head of another pin, and lift 20 tons
<klange>
Haven't really been testing this network stuff with SMP and it's highly susceptible to the typical race conditions some of my 'blocking' mechanisms are prone to, but hey, it managed to install gcc and build stuff with an extra core: https://klange.dev/s/Screenshot%20from%202021-06-29%2012-59-51.png
srjek_ has quit [Ping timeout: 250 seconds]
<doug16k>
man I wish I could hardware breakpoint a physical address
<doug16k>
it's shocking how many times a memory corruption happens through an unknown vaddr
<doug16k>
it's laughably easy if you know the vaddr
<doug16k>
you can make the cpu check every pointer access for you
<doug16k>
for almost free
<doug16k>
no way to tell it to watch a physical address
<doug16k>
no way to tell gdb or qemu
<doug16k>
no such thing
<doug16k>
bochs can
<doug16k>
watch w|write addr - insert a write watch point at physical address addr
<doug16k>
hmmm
<doug16k>
I wonder if that special packet might work
<bslsk05>
github.com: qemu/gdb.rst at 9692c7b0373677badcf8bb299ab8e2597244d436 · qemu/qemu · GitHub
<doug16k>
I can't get bochs to printf 64 bit numbers on glibc!
<doug16k>
I can't stop it sign extending from 32 bit signed and I can't see anywhere that is 32 bit at all
<geist>
hmm! sounds like maybe some subtle bug in sign extension in one of the implicuit modes?
<doug16k>
yeah I have been staring at it a while and I can't see how. I even made some stuff permanently 64 bit that could be 32 bit in 32 bit only build
<doug16k>
plenty is 64 bit in 32 bit anyway, due to pae
<doug16k>
oh I see. disastrous mismatches in compare-sections at entry
sprock has joined #osdev
tenshi has joined #osdev
AlwaysLivid has joined #osdev
froggey has joined #osdev
tacco has joined #osdev
mkopriva has joined #osdev
tacco has quit []
paulusASol has quit [Quit: Bridge terminating on SIGTERM]
medvid has quit [Quit: Bridge terminating on SIGTERM]
hgoel[m] has quit [Quit: Bridge terminating on SIGTERM]
GeDaMo has joined #osdev
soulisson has joined #osdev
<soulisson>
Hello. Sorry to bother you. I have a stupid question. I understand that when a dll is mapped to several processes a copy-on-write occurs. My question is what happens when the dll is mapped to a single process and this process needs to modify the ".data" or the ".bss" section. Does the OS perform a copy of the original data as well?
dennis95 has joined #osdev
sortie has joined #osdev
<mjg>
soulisson: consider what happens when another process would like to map the same dll
gareppa has joined #osdev
<moon-child>
soulisson: I don't think you generally do copy-on-write. You share text and rodata between all processes, and they're read-only so there's nothing to write to; and you make a copy of data/bss/tss
<moon-child>
I guess you could do cow for data, but I don't think there's a hugely compelling argument for that. You're just gonna pay the fault tax later
gareppa has quit [Remote host closed the connection]
<klange>
well, relocations [and that's why we centralize them into things like offset tables]
soulisson has quit [Killed (molybdenum.libera.chat (Nickname regained by services))]
soulisson has joined #osdev
<soulisson>
Apologies. My internet connection is not reliable.
<klange>
No worries. There's a log.
<soulisson>
oh ok, my last questions were not even sent, lol :)
<soulisson>
I don't know how the OS handles COW. If an area of virtual memory is marked as writable how does the OS know it must copy the physical pages when there is a write operation. Sorry for the dumb question
<moon-child>
the memory is marked as read-only
<moon-child>
when process tries to write, the os makes a copy of the page, maps the copy into the process's address space as read-write, and tells the process to try again
<moon-child>
(maps--at the same address, of course, removing the previous mapping)
<soulisson>
moon-child, did you mean virtual memory is marked as read-only or something else?
<moon-child>
I mean the mapping is read-only. Copy-on-write memory is read only from the MMU's perspective. Copy-on-write is an abstraction the kernel implements on top of that
<geist>
right, the mechanism is to map the page read only such that a write can be detected and handled
<geist>
as moon-child said
<moon-child>
distinction between mechanism and policy gets kinda fuzzy sometimes
<geist>
moon-child: re: data/bss COW in shared libs, that is definitely done on linux and whatnot. no idea what windows does
SGautam_ has joined #osdev
ElectronApps has quit [Read error: Connection reset by peer]
<geist>
but may as well. and data segments can be fairly large so it's potentially a win
<geist>
easy enough to do, just create a RO mmap() of the portion of the SO that is read only, and a COW mapping of the parts that are RW and let the OS deal with it from there on out
<klange>
Speaking of... I should probably actually implement this stuff for my ld.so...
ElectronApps has joined #osdev
<klange>
and get rid of the awful set of special system calls it has been using~
<geist>
also the .text segment may also need to be patched up for relocation, which is the main reason you'd compile a binary to be relocatable: change the codegen such taht the number of patches in the .text segment are minimized
<geist>
or perhaps clustered in one spot (GOT table, etc)
<klange>
< klange> well, relocations [and that's why we centralize them into things like offset tables]
<geist>
but the local relocation is probably private, so COW that too
<geist>
didn't read the whole backlog
<klange>
lol
<geist>
but i'll go to bed
<klange>
what's it, 2am or something?
<klange>
shoo, shoo
<moon-child>
geist: my assumption is sanely written apps won't put stuff in data if they're not going to change it. But I guess that's muddied by stuff like c++ constructors which don't get constant folded
<geist>
it is indeed
<moon-child>
klange: yea 2am
<geist>
moon-child: yah. good question as to in the end how much if any given bianry is dirtied
<soulisson>
deeply sorry, I'm struggling. Are there two sets of permissions? One set for virtual memory and another for the physical pages? I don't understand how the operating system can at the same mark an area of virtual memory as writable because it needs to be modified and at the same time read-only to be able to do the COW?
<geist>
i suppose the OS could keep some statistics about it and decide that this SO or that SO is more or less dirtied by local writes, so go ahead and pre-COW it, etc
<moon-child>
would be interesting to measure that. I assume linux has a reason for doing it
<geist>
anyway to bed i go
<j`ey>
soulisson: no permissions on physical pages
<j`ey>
soulisson: theres two different virtual addresses
<j`ey>
one for each process
<soulisson>
j`ey, when I start a new process in suspended mode for instance. The memory area of the ".data" and ".bss" areas are displayed as writable in the debugger but the COW features requires them to be read-only. Are those informations separate?
<j`ey>
I dont know how that debugger is getting that info
<moon-child>
conflict of mechanism and policy again :)
<moon-child>
cow is an implementation detail. From the _process_'s perspective, it's allowed to write those pages. It doesn't have access to the raw page tables, but to an abstraction the kernel sets up for it
SGautam_ has quit [Read error: Connection reset by peer]
<soulisson>
moon-child, ok. So the pages are marked as read-only in the page tables but there is something in the process data structures that indicates them as writable. Is that correct?
<moon-child>
yes
<soulisson>
moon-child, ah ok. Thanks a lot. Sorry for the stupid questions. I'm not very bright. I struggle even with simple things.
<j`ey>
soulisson: lol no need to be sorry!
<moon-child>
you're learning. Nothing stupid about that
<soulisson>
:) I aim at becoming better in computer security. I went to college but those topics were not taught.
opios2 has joined #osdev
Arthuria has joined #osdev
<klange>
oh great... now my surface is running windows 11 >_>
<kingoffrance>
it seems like an interceptor pattern to me :) process doesnt know anything is cow per se, nor does it need to know or care, it can be perfectly oblivious and none the wiser
gog has joined #osdev
dormito has quit [Ping timeout: 268 seconds]
<kingoffrance>
or, you know, it is a type of encapsulation. just different terms for different crowds
<kingoffrance>
such topics are probably taught, just not in such a context
<kingoffrance>
is what i mean
<moon-child>
klange: I installed windows 8 on my laptop. It's surprisingly tolerable (at least, when compared with windows 10)
<moon-child>
and it doesn't do major version upgrades without asking!
<klange>
I was fine with 10 and WSL2, and to be fair I had opted in to this - though I don't think anyone really expected to see a major release in insider previews without an additional toggle.
<klange>
Not that 11 is actually a major version upgrade, it's all marketing baloney.
<soulisson>
kingoffrance, I went to a school where thoses topics where not taught. The training was not well designed, the teachers didn't care much, and a tremendous amount of time was allocated to topics with no connection to Telecommunications or CS :)
<pony>
wait, what? does 10 upgrade to 11 without asking?
<klange>
If you are on the "dev" channel for insider previews, apparently.
<pony>
oh, ok
<klange>
This is apparently the price I pay to get nested virtualization in WSL.
<pony>
I heard lots of weird things about 11 like it requires a TPM, includes Android OS, will require a camera in 2023 for all laptops, etc...
<soulisson>
pony, it will run some of the android apps available on the amazon store.
<pony>
not that there are laptops without cams these days, but sitll creepy
<klange>
TPM 2; requiring a TPM at all would have been minor, but they're explicitly saying TPM 2.
<soulisson>
pony, yes, you're right.
<klange>
I get the feeling these were intended to be OEM certification requirements and someone missed the memo.
<pony>
what does TPM mean for hobby os devs, does it mean one day we cannot run our own OS?
<klange>
No, TPM has nothing to do with you being able to run anything.
<pony>
oh, ok
<klange>
You want to aim your scorn at Secure Boot for that.
<pony>
ahh
<pony>
yeah
<pony>
I tried to boot OpenBSD the other day, had to turn it off
<pony>
secure boot that is
<klange>
The TPM is just a piece of hardware that can sign things based on boot environment. Useful little thing if you know how to use it.
<pony>
I see
<soulisson>
pony, I think it's a device used for storing keys, encrypting and signing files. You can use it to check if your kernel,modules, etc, haven't been tampered, ... I never tried it my self, despite wanting to
<pony>
yeah
<Affliction>
Even then, secure boot should allow you to add your own keys (or keys used to sign any OS)
<Affliction>
But then, when 'minimum viable product' == 'can boot Windows'
<Affliction>
Has anyone here signed their loader for secure boot?
dormito has joined #osdev
<soulisson>
Affliction, I don't know about here but I know some people of #security did it
freakazoid333 has quit [Read error: Connection reset by peer]
kingoffrance has quit [Ping timeout: 250 seconds]
kingoffrance has joined #osdev
sprock has quit [Ping timeout: 268 seconds]
sprock has joined #osdev
kspalaiologos has joined #osdev
decartes has quit [Quit: Connection closed for inactivity]
xenos1984 has quit [Remote host closed the connection]
Arthuria has quit [Remote host closed the connection]
xenos1984 has joined #osdev
kspalaiologos has quit [Quit: Leaving]
Arthuria has joined #osdev
Arthuria has quit [Read error: Connection reset by peer]
Arthuria has joined #osdev
Arthuria has quit [Read error: Connection reset by peer]
Arthuria has joined #osdev
Asmodean has joined #osdev
Arthuria has quit [Read error: Connection reset by peer]
Arthuria has joined #osdev
Arthuria has quit [Read error: Connection reset by peer]