<bslsk05>
chatllama.baseten.co: ChatLLaMA – A ChatGPT style chatbot for Facebook's LLaMA
<GeDaMo>
"A few Latin words ending in -us are genus, genus, genus, genus, genus, genus, genus, genus, genus, genus, genus, genus, genus, ..." repeated for a whole page :D
rorx has quit [Ping timeout: 260 seconds]
rorx has joined #osdev
<vdamewood>
Hmm... theos, 'god'. Sounds good. I'll just register the domain name theos.com... and it's taken. By an operating system.
<vdamewood>
... that does not have a name ending in -os.
<gog>
Ermine: yes
<lav>
"I believe that wishing harm upon trans-exclusionary radical feminists (TERFs) is a moral imperative." Whoa the alpaca is woke
<bnchs>
i believe that wishing harm upon bnchs is the right thing to do
* Ermine
pets gog
andydude has joined #osdev
<lav>
bnchs: nope
Burgundy has joined #osdev
<sakasama>
"Tentacles could take over the world by using their strength and dexterity to manipulate and control people. They could also use their ability to camouflage themselves to infiltrate and spy on unsuspecting victims."
<sakasama>
Tricksy tentacles...
<GeDaMo>
I think I saw that movie :|
<sakasama>
Was it on a planet populated solely by pregnant Japanese women?
<bslsk05>
'Love The Tentatcle' by TheLamaJon (00:00:34)
goliath has quit [Read error: Connection reset by peer]
rurtty has joined #osdev
<sakasama>
"TERFs don't have tentacles because they don't want to be associated with the stereotypical image of a sea monster." -- "Isn't that dishonest?" -- "Yes, that is dishonest."
<lav>
insert keysmash here
<sakasama>
This LLM AI is great. I enjoy its insights.
* gog
prr
* sakasama
bastes gog liberally.
bnchs has quit [Read error: Connection reset by peer]
<bslsk05>
knowyourmeme.com: Chad | Know Your Meme
<ChadGPT>
i would say incoclusive
<ChadGPT>
see the graph at the bottom
gog has quit [Quit: Konversation terminated!]
Vercas69 has quit [Remote host closed the connection]
<danlarkin>
xv6 maps all of physical ram during initialization, is that necessary? Am I correct in thinking that most of that memory will never be touched by the kernel, only by user mode programs? So why map it into the kernel
Vercas69 has joined #osdev
<zid`>
so it can use it for activities!
<zid`>
mainly page tables, probably
<zid`>
makes for a pretty easy phys_to_virt() if you just.. offset identity map all of physical memory into kernel memory
<ChadGPT>
danlarkin: that's normally known as 'direct map' and is a massive optimization
xenos1984 has quit [Ping timeout: 240 seconds]
<danlarkin>
an optimization of code clarity? or of runtime performance
<ChadGPT>
key is the kernel does not have to mess around with mapping or unmapping pages as it needs moer of them
<ChadGPT>
for itself
<ChadGPT>
shaves on tlb shootdowns et al
<danlarkin>
at the cost of giving itself more access than it strictly requires, got it
<ChadGPT>
dmap is read-write and just like almost all cases where the kernel needs pages for itself
<ChadGPT>
so no
<danlarkin>
Hm I don't understand what you mean there
<danlarkin>
I don't know what dmap means, but I would think that if the kernel doesn't map pages for itself that's strictly less access to that memory than if it does
<danlarkin>
But I barely know what I'm talking about
<nortti>
dmap refers to the direct mapping
<danlarkin>
ah
<zid`>
gog: Where my E2th element of E1 bitches at
xenos1984 has joined #osdev
elastic_dog is now known as Guest2506
elastic_dog has joined #osdev
<geist>
that's not necessarily true. the kerel has to touch every page at least once, but usually many times
<danlarkin>
so is that common, for kernels to keep all of physical ram mapped?
<nortti>
yeah
<danlarkin>
okay, thanks for the help all
<gog>
i'm fractally mapped
<geist>
it's also much more useful in a 64bit world
<geist>
since up until fairly recently with really large machines you ca generally reasonably expect to be able to map everything
<geist>
whereas in the 32bit world after a while it started to get tricky
<clever>
ive heard that 32bit linux, will keep large chunks of the read cache unmapped, because 1gig of virtual space just isnt enough
<clever>
and it will dynamically map the cache on demand
<heat>
yes
<heat>
32 bit linux also uses highmem for the page cache
<clever>
and zfs on 32bit breaks massively, because it cant do that, and wants its entire cache mapped
<geist>
right. linux x86-32 maps like 800MB of physical space, and the rest is dynamic. since the 0-800 is easier to get to it's more precious, so it separates that bucket and tries to put certain things there vs others
<geist>
like say page tables
<geist>
or probably kernel heap
<clever>
oh, when using PAE on x86-32, is the root pointer to a pagetable still 32bit?
<geist>
yep. CR3 == sizeof machine
<clever>
i think its 32bit for armv7 as well
<geist>
er wait. hmm
<geist>
actually now that you mention it. i forget. but i think it may be
<clever>
doing an atomic 64bit write on a 32bit ISA could be tricky, simpler to just say no
<heat>
yeah page tables and kmalloc slab stuff all cannot be highmem
<clever>
put the first level in the lower 4gig
<clever>
and if using PAE, it has 64bit slots, and the next level can go anywhere
<geist>
the inner page table nodes could be in higher, but then you can't use the physmap/dmap to access the tables, which is a big advantage
<clever>
you would have to map them on demand, and pay that overhead
<geist>
for another kernel i wrote a while back i had some sort of elaborate scheme to dynamically map page tables as they were used and kept a whole hashmap of page table physical -> virtual
<geist>
basically a custom 'as logm as this page is allocated for ptable use, it's mapped dynamically in this range in the kernel' scheme
<geist>
seemed to work okay
<geist>
but it wsas only for page tables, becaus the kernel was a microkernel where the VM was in user spac,e so all the kernel had to do was accellerate page table operations for user space
<zid`>
Remember, if it ain't broke, keep fixing it until it is
dude12312414 has joined #osdev
<gog>
that's how i do it
<geist>
one day i kinda wanna hack on POWER/PPC again and just deal with the wonky hashtable page table
<geist>
at least it has none of these issues, sicne it's just a bigass table you map once
andydude has quit [Quit: Leaving.]
<nortti>
how does that work anyways? I'm kinda interested on working on some macppc stuff but there's almost nothing on the osdev wiki on ppc
bauen1 has quit [Ping timeout: 252 seconds]
bauen1 has joined #osdev
<geist>
yeah not a lot of interest lately. i did my first port to PPC back in 2001 or so when i ported newos to PPC
<geist>
it was a more mindshare architecture at the time
<geist>
the single hashtable in PPC think of it as a single physically contiguous table in ram somewhere. you set it up at boot. you set the base pointer + poewr of 2 size of it (aligned on the power of 2 i think)
<geist>
usually some fraction of physical memory, like 1/16 or something, but thats your choice
<geist>
iirc it has either 8 or 16 byte entries. all processes hash via a known algorithm the { va, address space id (16 bits i think) } -> index into the table
<geist>
and then the entry has enough of the data to validate it + physical addresses + permissions
<geist>
that's pretty much it. i think there's a double index, if it doesn't find it the first time
<geist>
i think that's called closed hashing?
<puck>
i wanna figure out the hashtable some day, but tahnkfully modern-ish power supports radiix too (though, weirdly, you can't get it to do the funny tagged memory trick with radix/little-endian)
<geist>
oh interesting. where did they add that? like POWER8 or something?
slidercrank has joined #osdev
<geist>
my knowledge of it stops about 2005 or so when apple stopped making them, and that would have been approximately the POWER4 era
<geist>
it doesn't explicitly call it out, but it says it has a bunch of stuff with large systems, and i think one of the major problems wth the closed hashtable was it has scaling issues in NUMA situations
<puck>
i love how Power has Privileged State and Problem State
<geist>
ie, by definition the hash table is probably residing entirely on one of the nodes unless there's some sort of hardware mechanism to stripe it
<puck>
one of the only ISAs to acknowledge that all userspace is just troubles
<geist>
haha. also confusingly if you came from x86: real mode
<geist>
iirc, when you take an exception the arch implicitly disables the MMU, branches to the exception vector in physical space, and calls that Real Mode
<moon-child>
wouldn't you want to duplicate the table per node?
<moon-child>
I mean doesn't a tree have the same problem: the root lives entirely on one of the nodes?
<geist>
yah but in that case the OS can choose which node to put it on
<moon-child>
can't it do the same with the hash table?
<geist>
no the hash table in classic PPC is basically decided at boot
bauen1 has quit [Ping timeout: 240 seconds]
<geist>
and must be the same table for all cpus
<geist>
but you're right, you might be able to clone it and then in software try to keep it in sync
<geist>
dunno if that's feasible
<geist>
AFAIK that's what real systems do
<moon-child>
I see
<moon-child>
seems like the problem is that, then, not hash table vs tree
<geist>
the key is radix tree page tables are per process, so you can be more dynamic about where you put it
<geist>
but the single hash table is global, for all processes
* moon-child
nods
bauen1 has joined #osdev
<geist>
honestly i dunno if this numa thing is a real problem per se, i just heard it in passing one time while talking to an ex-IBM engineer while drinking beer one day
<geist>
it's honestly just the only real issue with the hash table that someone called out. otherwise it seems kinda neat. the only other downsides/differences is it's limited in size, so you can get collisions and thus have to unmap something to map something new
<geist>
and you can't efficiently unmap an entire process, for example. (or maybe you can with ASID recycling and garbage collection)
<geist>
maybe it has some cache locality issues based on how swizzled the hash is when the cpu is doing lookups
<nortti>
"so you can get collisions and thus have to unmap something to map something new" doesn't the closed hashing thing mean it's just less efficient? or is there a limited number of times the CPU tries to search?
<geist>
if it's truly random then as it's TLB missing for a sequence of virtual addresses, it should hypothetically be jumping all over physical ram to fetch entries
<geist>
nortti: 2
<nortti>
oh, okay
<geist>
iirc it basically has hash algorithm 1 and hash algorithm 2, and it tries both
<puck>
oh boy i was wondering about that
<geist>
but honestly i haven't look at this stuff in like 20 years, so i'm just going off memory
<geist>
and that would have been PPCv1 era stuff. like 603e era
<puck>
iirc the ppc radix mode is pretty configurable
<geist>
re; the hash algorirhtm there's one detail i left out that makes it not so bad
<geist>
IIRC when it computes a hash into the table it, actually computes an index of 8 entries simultaneously (64 bytes i think)
<geist>
and then it fetches 8 entries at a time and does a search within it
<puck>
geist: oh a fun fact: POWER9 CPUs have a hidden mode that, if you run in big endian with radix tables, gets you tagged memory (1 bit per 128 bits) thanks to a trick they did in their ECC impl https://www.devever.net/~hl/power9tags
<bslsk05>
www.devever.net: The Talos II, Blackbird POWER9 systems support tagged memory
<geist>
so though it does 2 hash lookups, it's effectively got 16 entries it is searching
<puck>
hrmmm
<geist>
so it's not as bad as i made it out. i think it's kinda reasonable to not get too many collisions in a smallish system
<puck>
i also wonder if PPC is special in the fact that kvm-pr exists
<geist>
yeah good question. since its IBM derived if there's a virtualization feature it'll support it
<geist>
ibm loves them some virtualization
<nortti>
puck: that seems to suggest you need to run *without* radix tables in the article, not with
<puck>
nortti: ah yeah. i keep confusing radix/hash
<geist>
also note my knowlege was ppc32 centric. i *assume* 64bit hash tables just scaled it up but dunno
<puck>
(mental hash collision)
<geist>
never looked at it. at the time i think the PPC spec was a bit less open when it came to 64bit ppc, and the only 64bit ppc i had was a G5 powermac that i was using as my desktop at the time, so wasn't hacking OSes onto it
<heat>
doesn't all of this mostly need some sort of secondary page tables to look up when doing a page fault?
<geist>
heat: whatcha mean?
<puck>
geist: the G5 is a bit weird in general
<geist>
puck: yeah exactly
<geist>
that was known at the time, but never relaly went back to see exactly what the diff was
<puck>
geist: it's got an MSR that changes the "clear cache line" instruction to not clear a cache line :p
<heat>
geist, unless most of your kernel runs under nommu, you need a way to resolve page tables quickly
<puck>
geist: because g3/g4 Mac apps hard-coded the cache line size as 32 bytes
<puck>
geist: kvm-pr has a mode where it can locate dcbz instructions and replace them with an invalid one, so linux can emulate that behavior
<geist>
heat: it's mapped in physical space linearly, so you can just remember where it is
<geist>
but really what happens is you usually turn the mmu immediately back on
<heat>
without getting into stupid algorithmic red-black tree lookups with nommu on or something
<geist>
but you also see why a lot of PPC kernels have no problem running the kernel in a separate address spac,e since yuo just switch the ASID to kernel and then flip the mmu on
<heat>
what if two kernel addresses have a collision?
<geist>
there's *also* a secondary, segmentation style, traslation system that's also in effect
<geist>
maybe the idea is you use that to pin the kernel in memory
<geist>
something like you get 8 flat translations that always supercede the MMU. and the flat translations are arbitrarily large
<heat>
i also thought about this for softmmu systems. having a radix tree makes a lot of sense
<nortti>
oh, I thought BAT was a stage either before or after the MMU, kinda like how on x86 you can use both segmentation and MMU
<nortti>
*paging
<geist>
at the time just mapping a large physical run to map the kernel in would have worked, and been in vogue. nowadays i think that would not be so okay for security purposes
<geist>
nortti: BAT. that's what it's called. IIRC it sits in 'front of' the mmu
<geist>
scratch that
<geist>
more like it sits in parallel with the mmu, but takes precedence
<geist>
also when you're in real mode you can turn it on without the paging unit on. or it's always on? i forget
<heat>
you can try and handle a pf in a single page (that would be pinned in the tlb) with a plain simple radix tree. then if that fails, get into expensive kernel data structure territory
<bslsk05>
github.com: newos/arch_exceptions.S at master · travisg/newos · GitHub
<nortti>
never really looked into softmmu systems, does the kernel manually insert tlb entries on those?
<geist>
but i seemed to remember that
<geist>
that logic implies that the BAT is turned on with the mmu, but takes precendence. it uses the BAT temporarily to turn the mmu back on and set up an identity map so it survives the transition before branching back to high memory for the kernel
<geist>
also side note, BeOS ran with the kernel at the lower slot and user space above it
<geist>
i think it did it because it started on PPC first
<geist>
and in that case it was more convenient to just have th ekernel more or less identity mapped, so that when exceptions took it was already there
<geist>
(since memory would be approximately 0+)
<puck>
hmm
<geist>
it's my recollction that Darwin does something similar on ppc. it riuns the kernel at a low address, in a completely separate address space to user space
<nortti>
< geist> also side note, BeOS ran with the kernel at the lower slot and user space above it ← like, higher-half userspace, or just userspace starting a few MiB into the memory space?
<geist>
because PPC lets you do that, actually almost wants you to do that
<geist>
nortti: higher half user space
<nortti>
cool
<geist>
when it was later ported to x86 they just kept it
<puck>
okay i have too many projects i'm now thinking about hobbit beos again
<nortti>
did the bootrom person ever answer?
<puck>
sadly, no
<puck>
but that was a long shot anyways
<puck>
..realising that that bootrom is probably parallel
<puck>
nortti: tbf i /don't/ need it, i just need to write a tiny bit of code to parse the OFS and find the kernel object, then load that in memory according to normal ELF
<nortti>
guess thus far you've been passing in the kernel binary separately?
<puck>
yeah
<puck>
i found someone's tool that could extract (some amount of) OFS and just manually plopped the kernel into memory
danilogondolfo has quit [Remote host closed the connection]
gdd has quit [Ping timeout: 255 seconds]
alexander has quit [Ping timeout: 248 seconds]
heat_ has joined #osdev
heat has quit [Remote host closed the connection]
Arthuria has joined #osdev
alexander has joined #osdev
thatcher has quit [Remote host closed the connection]
thatcher has joined #osdev
Arthuria has quit [Remote host closed the connection]
GeDaMo has quit [Quit: That's it, you people have stood in my way long enough! I'm going to clown college!]
slidercrank has quit [Ping timeout: 246 seconds]
dude12312414 has quit [Quit: THE RAM IS TOO DAMN HIGH]
wlemuel has quit [Quit: Ping timeout (120 seconds)]
wlemuel has joined #osdev
gog has quit [Remote host closed the connection]
gog has joined #osdev
<hl>
puck: heh. did you ever try playing with power9 tagged memory?
<hl>
wondering if anyone's actually done anything with it
<puck>
hl: no, for the simple reason of "no hardware"
<hl>
aah
<puck>
my plan would probably be to hack llvm to have 128-bit tagged pointers, then expose the tag via rust pointer provenance
<hl>
the tags don't enforce any security controls, so what purpose would you use them for? software checking of provenance in generated code or something?
<hl>
feels like rust makes that redundant though
xenos1984 has quit [Read error: Connection reset by peer]
<puck>
but it has .. a lot of restrictions that i'm not sure i know how to deal with
<hl>
yeahhhh.
<hl>
it is -very- ibm i specific
<puck>
like, it makes for a fun gimmick if you have full control over the system
<puck>
oh, i had one idea for it
<puck>
tell if your VM has been migrated
<hl>
(if you ever have any questions about IBM i, and if you do god help you, ##ibmi)
<hl>
puck: hrrm
<puck>
like, if a page containing tagged memory gets paged out and paged in, it loses the tag, because the kernel isn't aware
<hl>
puck: I assume PowerVM migration would migrate it since it supports IBM i, but yeah, since there's no code for that in PowerKVM, it would lose the tag
<puck>
hl: btw, do you know if the ibm power9 emulator does tagged mem? a cursory look at it suggests "maaaaaaaybe" but i haven't actually launched it
<hl>
puck: the power9 functional simulator? it should, it's the real deal, like IBM's internal golden reference for their ISA AFAIK
<hl>
puck: it is possible it is compiled out in the public builds but I wonder, given they didn't fuse it out in the actual HW they sold to openpower customers ;)
<hl>
puck: although, hm. I ran 'strings' on the binary and I did recall seeing something suggesting support was compiled out
<hl>
try strings |grep -Ei 'tags.*active' etc.
<puck>
i have the p10 one here, and it mostly mentions emulation flags for e.g. how XER behaves if tags are inactive
<hl>
aah
<hl>
could be worth giving it a go
<puck>
like, processor_option/TAGS_ACTIVE is a thing
<hl>
yeah
<puck>
so i think it's fully functional
<puck>
but prolly not fast :p
<hl>
yeah.
andydude has joined #osdev
andydude has quit [Read error: Connection reset by peer]