<mjg>
have to be live overwrite and only reboot needed
<sortie>
mjg, hmm? Just return to single user mode or something, mount / read-only, then probably it will survive rewriting the harddisk with a Sortix image
<mjg>
no man
<mjg>
you just don't get it
* mjg
is staying on mirbsd then
<sortie>
You mean replacing OpenBSD in place with Sortix, keeping all your filesystem files?
<mjg>
ya man you migrate the system
<mjg>
seamlessly
<mjg>
i grant reboot to swap the kernel
<klange>
don't need to reboot, just load a kernel module that rewrites everything
<sortie>
OK cus I'm sure Theo would not like me hot hackin' the kernel to become Sortix
<mjg>
klange: giving him an easy way out
<sortie>
I have feeling OpenBSD doesn't have dynamic modules any more
<mjg>
sortie: it does not
<heat>
openbsd doesn't even have tmpfs
<mjg>
deemed insecure
<mjg>
heat: it does, but they don't build it
<mjg>
by default
<sortie>
mjg, heretic yes indeed
<sortie>
mjg, the biggest problem with that is really the filesystem type
<klange>
that's disturbing, how does openbsd deal with... plugging things in?
<sortie>
Sortix doesn't have ufs support, just ext2
<heat>
buttpl
<klange>
"guess you'll have to recompile and reboot"?
<sortie>
Extracting a Sortix base system onto / is simple enough
<mjg>
klange: you have all the drivers loaded
<klange>
Though I would guess the things I have in mind don't have _any_ openbsd drivers anyway, so it probably doesn't matter
<sortie>
Making that filesystem migrate to ext2... yeah... no... -- but implementing a ufs driver might be doable
<mjg>
klange: you do realize it's not many :)
<heat>
LMAO
<sortie>
The bootloading would be tricky but maybe not so much. GRUB can probably be installed from OpenBSD
<klange>
:headtap: don't need loadable modules if you don'
<klange>
t have any drivers in the first polace
<klange>
kewagjekwahgekwa
<heat>
sortie, ah yes, a ufs driver for THE ufs
<sortie>
The easiest is probably just to tar up the root filesystem, replace the harddisk with a minimal installation disk image, which then does mkfs and extracts the saved files
<mjg>
wait are you the weird which has pledge
<mjg>
or was it haiku
<sortie>
mjg, am I allowed to send all your personal files to the central Sortix migration and surveillance server?
<sortie>
I don't have pledge at this time. Was considering implementing it for fun
<heat>
mjg, I think you're thinking about unveil and serenityos
<mjg>
well of course
<mjg>
heat: technically haiku has some pledge tho
<mjg>
data/system/data/fortunes/Art:His performance is so wooden you want to spray him with Liquid Pledge.
<mjg>
data/system/data/fortunes/Goedel:THIS IS PLEDGE WEEK FOR THE FORTUNE PROGRAM
<mjg>
src/apps/mail/words:pledge/DS
<mjg>
but otherwise fair point!
<sortie>
niht
<heat>
sortie, niht
<mjg>
not invented here, theo
gog has quit [Ping timeout: 260 seconds]
<heat>
game of trees moment
<mjg>
dude i tried using it few days ago
<klange>
I should implement a pledge-alike. It's such a simple idea.
<mjg>
kept complaining about a missing file in the repo
<mjg>
i mean under .git
<mjg>
git (not got!) fsck said 's all good man and had no issues
<mjg>
in true openbsd fashion their got log does not roll with a pager
<mjg>
:p
<heat>
lol
<heat>
klange, the problem with kernel-side pledge is that you couple yourself to a libc
<heat>
which sux
<heat>
i actually wanted to see if I could do libc-side pledge by dynamically creating bpf filters for seccomp
<klange>
Well, I'm already tied to a libc, but also if your pledges are based on groups of syscalls, I would think that would be a lot less of a problem?
gog has joined #osdev
<heat>
depends
<mjg>
have you seen the realities of pledge in the kernel
<mjg>
abhorrent
<heat>
if your pledge("stdio exit") bans everything except write/read/writev/readv/.../exit, this works... except if you use another syscall in stdio
<heat>
which is very possible as you may need locks and/or memory allocation
<heat>
so now you're already creating exceptions to strict syscall<->libc groups to make sure your libc works
<klange>
Perhaps the pledge interface could be in the kernel, but the thing that maps group names to syscall masks could be handled by the libc
<klange>
So the libc could know that if you ask for stdio it needs lock syscalls.
<heat>
yes
<heat>
so you're basically getting into pledge in the libc with some sort of syscall filter
<heat>
which was my idea, seccomp as the backend of pledge
<heat>
seccomp for its powerful and fast nature, pledge for usability
* klange
. o O (Static checker for pledge declarations that checks that code paths after it don't lead to forbidden system calls)
vai has quit [Remote host closed the connection]
vai has joined #osdev
<vai>
good morning everyone from Finland EU
vai is now known as Jari--
<heat>
what are you on about
<heat>
it's night in finland
<mjg>
ahead of his time
<klange>
2:30am, early bird gets the osdev worm
<mjg>
allright mofoz
<mjg>
my magic rust patch compiled
<mjg>
and i verified the compiler did not lose its shit
<bslsk05>
'cyriak - Home' - 'I'm a freelance animator based somewhere in the UK. Feel free to get in touch for professional commissions (or anything else) via twitter or my website. I've given up trying to figure out youtube's messaging system.
<geist>
but just did a new one a few days ago
<nur>
oh I thought it was a systems youtuber I hadn't heard of
<nur>
anyway I think I have a lot more time for osdev again
<nur>
that's the good news
<nur>
the bad news is I got laid off due to "global recession"
<nur>
:-|
danilogondolfo has joined #osdev
fedorafan has joined #osdev
pretty_dumm_guy has joined #osdev
pog has joined #osdev
invalidopcode has quit [Remote host closed the connection]
invalidopcode has joined #osdev
justReddy is now known as justache
fedorafan has quit [Ping timeout: 260 seconds]
fedorafan has joined #osdev
<epony>
cheers
<epony>
orange juice for everyone and vodka for nur he's paying
<bslsk05>
'Sir Nils Olav promoted to Brigadier by Norwegian King's Guard' by Edinburgh Zoo (00:03:31)
<epony>
it's more like the "clientelle" of a nut-house (mental "health" facility) with male nurses and totally coucou doctors who are former patients
<epony>
and everyone has a strange dialect that only they understand themselves, but try to work out the TV watching rules in the main hall after the TV exploded
<epony>
the nurses are taking the same medication as the patients, which is big pharma researching fake opiates for the masses and use the nurses as control groups, the doctors are advance experience but can't really provide any value to the pharmacoms, so the nurses are the drop in assistants that do majority of the experiments
unimplemented has joined #osdev
<epony>
and the patients are the walk ins who just like it and do their time inside since it's more appealing to their natural predisposition for communal shared experiences of the self and group fun and games
<Ermine>
kaichiuchi: it's funny, but I don't see how does correlate with linux community
<kaichiuchi>
i have no rational basis for it
<kaichiuchi>
all I know is when I see it, I immediately think linux community
<Ermine>
Because penguin?
<kaichiuchi>
not only because penguin
<kaichiuchi>
but.. i don't know
<epony>
have you seen the sterilised gorillas?
<epony>
they look pretty sad and pathetic
<epony>
like apple and google employees
fedorafan has quit [Ping timeout: 260 seconds]
fedorafan has joined #osdev
genpaku has quit [Read error: Connection reset by peer]
genpaku has joined #osdev
bauen1 has quit [Ping timeout: 255 seconds]
<kof123>
i'm ok with such characterizations, but surely ibm/ms/whoever is 300-pound gorilla
<kof123>
this has long precedence, before other things existed
<kof123>
no wonder they are in love lol
fedorafan has quit [Ping timeout: 260 seconds]
<epony>
havinh gun is the best part
<epony>
having fun
fedorafan has joined #osdev
<epony>
wrong key opened the wrong door weird unreal level found by accident
* epony
goes warp drive
Antoinette has quit [Remote host closed the connection]
elastic_dog has quit [Ping timeout: 265 seconds]
elastic_dog has joined #osdev
qubasa has joined #osdev
<fedorafan>
king of fighters
<fedorafan>
where is the wodka
<fedorafan>
who got it
<fedorafan>
gimme :D
<kof123>
hi fedorafan. what are you working on/plans? such talk doesn't bother me, i am not a mod of any shape or form, but IMO focus on coding and nevermind the bollocks :D
<fedorafan>
i see just joined this chan totday and saw talk like that im fine with that
<fedorafan>
oh your nick doesnt refer to the game
<fedorafan>
cool arcade
<kof123>
i get that all the time, np. i do c89 and filesystem at the moment
<fedorafan>
what is c89
<fedorafan>
i dont do os development
<kof123>
ancient c standard 1989 lol
<fedorafan>
oh ok
unimplemented has quit [Ping timeout: 268 seconds]
<fedorafan>
lol
bauen1 has joined #osdev
pog has joined #osdev
fedorafansuper has joined #osdev
fedorafan has quit [Ping timeout: 246 seconds]
fedorafan has joined #osdev
fedorafansuper has quit [Ping timeout: 252 seconds]
dude12312414 has joined #osdev
<netbsduser>
how do people tend to mix fine-grained locking of virtual memory control structures with the difficult reality that one needs to sometimes do "bottom-up" locking and sometimes "top-down" ordering?
<sham1>
In an ad-hoc manner I'd think
<netbsduser>
take the case of a page fault: you must lock the address space map, then the vm object, then perhaps an individual page description; while for e.g. "remap a page read-only everywhere because it is now to be CoW'd"; or page replacement "swap out a page/put a mapped file page back to disk" needs to be 'bottom-up'
<sham1>
What do you mean by bottom-up and top-to-bottom here
fedorafan has quit [Ping timeout: 248 seconds]
<netbsduser>
sham1: the different between "lock address space map -> lock vm object -> lock page struct" and "lock page struct -> lock its VM object -> lock all address spaces in which that VM object is mapped"
<netbsduser>
the former ordering is what happens in a page fault; the latter is what happens when e.g. it's time to swap out a page, or to write it back to backing store (for file mmaps) and release it
<sham1>
Why would you need to lock the address spaces for the latter?
<netbsduser>
sham1: because the page is being released, it must be unmapped everywhere
fedorafan has joined #osdev
dude12312414 has quit [Quit: THE RAM IS TOO DAMN HIGH]
heat has joined #osdev
<heat>
netbsduser, my vm objects have multiple locks
<heat>
actually I found a lock-order inversion in my wp code lol
<heat>
but for instance if you want to wp a page in your vmo, you have a mapping_list_lock and a general lock
<heat>
grab the mapping list lock, iterate through regions that map that, get its address_space, lock the page tables and wp
<heat>
you don't need to grab a whole "mmap_sem" for that
<heat>
this can actually be done all inside spinlocks, no need for sleepable locks even
<heat>
i think that the gist is that if you find yourself needing to invert lock orders, your code is wrong or the locks are too big
<netbsduser>
i fear the locks are too small, a big lock would simplify this considerably
<heat>
what's your exact problem, in this case?
<heat>
its possible that you're trying to do too much under the lock
<netbsduser>
page-out. pages are organised into queues. to which queue a page belongs is protected by the page_queues_lock. when it is time to page out some pages, the page has to be locked. then every mapping of the page has to be iterated over and unmapped. the page's owning object needs to be updated to indicate that the page has been swapped out (or put back to disk in the case of a file mapped page)
<netbsduser>
so for page-out ones goes lock page->lock its object->lock address spaces it's found in. but for page-fault handling, lock address space->lock object->lock page (so we can map it safely, otherwise it could be swapped out meantime)
<heat>
ok, so: vm_object mappings_list_lock -> region->address_space->page_tables_lock -> mmu_unmap()
<heat>
where page_tables_lock is a smallish spinlock you only hold when messing with page tables
<heat>
my point is that in this case you don't need to lock the whole address space
<heat>
your vm_region's aren't going anywhere while you hold the mappings_list_lock (as destruction implies removal, hopefully)
<heat>
the page_table_lock ensures stability of your page tables
<heat>
i'm not sure if you even need the page lock here
<netbsduser>
yes, you make a good point that the address space locking doesn't necessarily need to happen
<heat>
I assume your pages have refcounts?
<netbsduser>
not directly, to prevent the struct vm_page from growing too large i have a separate struct paged_page for pages which are part of anonymous memory objects
<netbsduser>
so i can avoid carrying around that data in the case of a page being used for wired memory only
<netbsduser>
i am trying to think if there is any obstacle to having pages locked by the object (or paged_page) to which they belong
<heat>
so it has a lock but not a refcount?
<heat>
that seems kind of backwards IMO
<netbsduser>
the lock logically belongs in paged_page and i think could be moved there safely; i have no idea why it's in vm_page, can't remember my rationale
<heat>
so, I don't have page out yet but in theory I avoid pages getting swept from under me by just having a general refcount
<heat>
can't swap if you have more than 1 ref or something
<heat>
I'm still not sure if or how that's to be done
<netbsduser>
what does the refcount count, number of places in which the page is mapped?
<heat>
no, just active references
<heat>
the vm object has one, your local struct page * variable has another, etc
<heat>
so the page fault code does vmo_get_page, which grabs a ref and hands it back, then it maps it, then releases its ref
<heat>
so a page that's mapped under 1 VMO on N mappings has a refcount = 1
invalidopcode has quit [Remote host closed the connection]
invalidopcode has joined #osdev
pog has quit [Quit: Client closed]
nvmd has joined #osdev
pog has joined #osdev
bauen1 has quit [Ping timeout: 252 seconds]
bauen1 has joined #osdev
CaptainIRS has joined #osdev
x8dcc has joined #osdev
<x8dcc>
hello, I am trying to add a simple gdt in assembly but I can't compile the file because nasm can't find the gdt labels (gdt_data.data_descriptor and gdt_descriptor)
<heat>
try simply referring to .code_descriptor and .data_decriptor
<heat>
(also typo in your code, *descriptor)
<x8dcc>
oh, true
<x8dcc>
you mean refering to .code_descriptor when declaring "CODE_SEG" and "DATA_SEG"?
<x8dcc>
because that gives the same error. Same with gdt_start
<heat>
yes, i mean that
<heat>
/shrug
<heat>
I don't use NASM
<x8dcc>
my guess is that I have to make them global or something like that, but I am not sure
<pog>
hello
<heat>
no, no need for global there
<heat>
wait
<heat>
where's CODE_SEG defined?
bleb_ is now known as bleb
<heat>
this is all the same file?
<x8dcc>
yes
<x8dcc>
the paste I sent is the same file
<x8dcc>
I just removed some comments from the middle to make it shorter lol
<heat>
no idea then
<heat>
you shouldn't need global
<heat>
hopefully someone familiar with NASM can chime in
<x8dcc>
yeah, I didn't want to try stuff I didn't understand until my code became a mess ^^'
<heat>
CODE_SEG equ 0x8 DATA_SEG equ 0x10
<heat>
if you want to be boring :))
Left_Turn has joined #osdev
<pog>
yeah nasm doesn't really have sublabels that work like that iirc
* pog
labels heat
<x8dcc>
but what about gdt_start? that shouldn't fail, right?
<x8dcc>
uhm, nevermind
<heat>
i don't see why it would
<x8dcc>
it was a typo...
<x8dcc>
(I honestly didn't know how I fixed the gdt_start part, but the other one was another typo)
<x8dcc>
now it compiles and runs, so I *guess* my gdt is loaded now. is there an easy way to check it?
<heat>
what emu are you using?
<x8dcc>
qemu
<heat>
info registers should have a GDTR reg
<heat>
s/GDTR/GDT/
<heat>
which will hold the address and limit
<x8dcc>
not sure what you mean by info registers
<heat>
on the qemu monitor
<heat>
ctrl-alt-2 probably
<heat>
you can also pass -monitor stdio to the qemu cmdline for easier access
<x8dcc>
yeah, I am having a look now, I didn't know this was a thing
<x8dcc>
very useful, thanks
<heat>
np
<pog>
will nasm do that absolute far jump right?
<x8dcc>
looks like it's loaded, indeed
<pog>
ok then
<pog>
don't worry about me i'm very stupid
* pog
meow
<x8dcc>
:)
<x8dcc>
thank you for your help, heat
<heat>
no problem
<x8dcc>
and thank you for having a look too, pog
CaptainIRS has quit [Quit: Reconnecting]
<kaichiuchi>
pog: may i pet you
<pog>
heat how stupid am i on a scale of 1-10
<pog>
kaichiuchi yes
* kaichiuchi
pets pog
<heat>
pog, pog
craigo has joined #osdev
<pog>
heat heat
CaptainIRS has joined #osdev
<heat>
kaichiuchi
<kaichiuchi>
heat:
<heat>
xylophone
<kaichiuchi>
spongebob
gog has joined #osdev
pog has quit [Quit: Client closed]
bgs has joined #osdev
<gog>
mew
sortie has quit [Ping timeout: 252 seconds]
<sham1>
wem
sortie has joined #osdev
<heat>
gog, is gog a transformation of ada?
<heat>
doesn't seem to be a trivial caesar cipher
invalidopcode has quit [Read error: Connection reset by peer]
invalidopcode has joined #osdev
<kaichiuchi>
work is easy today
<kaichiuchi>
considering that perforce is shot, and jira, and fisheye, and and and
joe9 has quit [Quit: leaving]
<kaichiuchi>
i still can’t believe people use atlassian products
<heat>
sounds like you want BITBUCKET
<kaichiuchi>
no please stop i thought we were friends
<sham1>
You must become Atlassan
<kof123>
without commenting on anything in particular, "vulture's gotta eat too" -- some clint eastwood movie
<kaichiuchi>
freeRTOS is actually really cool
<heat>
i like git and I like gerrit and I likeish bugzilla
<heat>
and github is tolerable
<heat>
the rest sux ballz
<kaichiuchi>
i’m really surprised we didn’t do something crazy like buy QNX
<kaichiuchi>
you know what’s scary?
<gog>
heat: no i was using gog before i was called ada
<kaichiuchi>
i use adaptive cruise control, which adjusts acceleration/braking based on the speed set point, following distance, and the car in front of me
<kaichiuchi>
the system has crashed 4 times since i bought this car brand new.
<gog>
kaichiuchi: i use jira and bitbucket every day and it's fine i guess
<gog>
idk
<gog>
i'd prefer something less fiddly
<gog>
too many moving parts that are irrelevant for our use cases
<gog>
anyhow i'm going home see yall in a bit
gog has quit [Quit: Konversation terminated!]
<kaichiuchi>
the fact that any software in a car crashes at all is distressing
<kaichiuchi>
the whole point of why i bought that car is for its high safety ratings
<heat>
isn't that supposed to be super rare
<heat>
you should complain
<heat>
are you using a tesla
<kaichiuchi>
SUPPOSED to be, yes
<heat>
that sounds like a tesla
<kaichiuchi>
no a subaru
<kof123>
hmm, i used to like projecting speed/etc. on the windshield, which is kind of gimmicky, but means not having to look down
<kaichiuchi>
point being, what if the system decides to panic and the brakes won’t disengage because they’re being stuck into an engagement command
<kof123>
yes yes, feel the power. you have come over to the unix sysadmin side. terse commands, if you ran as root it is your own problem
<kof123>
do as i say, don't ask questions!
<kaichiuchi>
and I think to myself if car people can’t get safety features right
<kaichiuchi>
how safe are we
<kaichiuchi>
for an abnormally long time the nuclear launch codes were all zeroes
<heat>
isn't the car industry super regulated when it comes to that?
<kaichiuchi>
i would hope it is
<kaichiuchi>
crashing software in a car, plane, navigation system, etc etc is unacceptable
<kaichiuchi>
probably even your phone
<kaichiuchi>
your phone is an emergency device these data
<kaichiuchi>
days
<kaichiuchi>
i’ve had the safety systems (eyesight) entirely cut out on me until I shut the car off and turned it back on
fedorafansuper has joined #osdev
<kaichiuchi>
i plug in my phone and if google maps and Music are active it lags out, becomes completely unresponsive, then reboots as I’m fucking driving
<Bitweasil>
Oh man, that PAL set to zeros story just keeps popping up. :D
fedorafan has quit [Ping timeout: 252 seconds]
<kaichiuchi>
i don’t know how our species has billions of members
<kaichiuchi>
we accidentally dropped a nuke in north carolina and the only thing that kept it from detonating was the shittiest low voltage switch
<heat>
should've nuked it anyway
<kaichiuchi>
we’ve come close to blowing ourselves up more than once
* kof123
holds up picture of mushroom cloud labelled "bazinga"
<heat>
i am aware
<kaichiuchi>
and people shrug their shoulders and say “shit happens”
<heat>
kof123, dropping a nuke on yourself is a classic zinger
<kaichiuchi>
also the only reason i’m going on this rant is because my car just did the thing i’m complaining about
<kaichiuchi>
and it’s osdev related because they’re probably using QNX/INTEGRITY
<kaichiuchi>
so sue me
terminalpusher has joined #osdev
xenos1984 has quit [Ping timeout: 252 seconds]
nvmd has quit [Quit: WeeChat 3.6]
xenos1984 has joined #osdev
<geist>
side note i did an upgrade of my subaru's entertainment software and disassembled the update, and it's QNX all the way down
<geist>
thought it was kinda curious
<geist>
wasn't even encrypted, just signed
<geist>
basically some sort of zip file with a few file system images in it, etc
gog has joined #osdev
<kaichiuchi>
did it ever crash for you?
<kaichiuchi>
because, i bought the car with 1 mile on it
fedorafan has joined #osdev
fedorafansuper has quit [Ping timeout: 252 seconds]
<netbsduser>
heat: i put my problem to an NT expert
<geist>
the entertainment part of the subaru? not really. it's refused to pair with my bluetooth phone once or twice, but i realized if you just hold in the volume button for 10 seconds it reboots
<geist>
but of course that's not the engine computer, so it just reboots the entertainment bits
<netbsduser>
he counselled me that my locking scheme is dubious, with some locks too big and some too small, and explained briefly the NT approach, which is to have a single spinlock (and extremely simple structures in the VMM so you don't waste time when you hold it)
<netbsduser>
when you need to page-in or page-out, he advises to set a bit in the page struct explaining that the page is in transition, then if you run into an in-transition page during a fault, you wait on an event associated with the page, and then re-process the fault (in case anything changed globally)
<geist>
tat's a strategy
<netbsduser>
i am going to try it
<netbsduser>
my vmm is modelled after NetBSD's UVM, which was the subject of a PhD thesis. so i think for my first attempt at a proper VMM with swapping i can allow myself a bit of leniency
<mjg>
it was a thesis 2 decades ago
<mjg>
today it is a bad idea
<mjg>
most likely anyway
<netbsduser>
i should be glad to hear why
<mjg>
i can't stress enough it all happened prior to multicore being a thing
<mjg>
so if it happens to work out with that setting, which i know it does not in netbsd, it would be a pure accident
<mjg>
or to put it differently, there is way more ram and cpu cores to deal with
<mjg>
i would not expect any ideas which were sensible in that period to be sensible today
<mjg>
apart from aforementioned accident
<netbsduser>
but it was expressly designed for multicore
<mjg>
where
<netbsduser>
the Mach VMM which was its antecedent and inspiration was too
<mjg>
i know it does not perform for shit, lemme find it
<mjg>
the mach vm is what landed in freebsd and it did not perform either
<mjg>
i'm pretty sure the paper talks about it
<netbsduser>
cranor's thesis discusses the fine-grained locking which was intended for SMP viability
<bslsk05>
marc.info: 'performance issues during build.sh -j 40 kernel' - MARC
<mjg>
you do understand fine-grainedl ocking does not scale?
<mjg>
past a handful of cores
<mjg>
what you need is lockless operation aided with some form of safe memory reclamation mechanism
<mjg>
rcu being an example
<mjg>
as in, you fine grained lock something, except if it is used by numerous cores at the same time, you have a de facto global lock there
<geist>
i kinda didn't want to go there, because i sense that netbsduser is not ready for that sort of thing
<mjg>
to expand on this point, the vm in illumos and freebsd both suffers greatly from "fine-grained" locking
<mjg>
and of course in netbsd
<mjg>
interestingly this is largely true even for linux, albeit to lesser extent
<geist>
fuchsia too, i should add
<mjg>
linux protects the address space with a rw lock, taken for writing for mmap/munmap
<mjg>
but fauls are handled with a read lock
<kaichiuchi>
i think today i’m going to start osdev
<mjg>
the end rseult is that this performs better than the previously mentioned systems, but remains a bottlenek
<geist>
yah basically same as fuchsia. we have a rw lock for theoverall aspace, and then each object has it's own mutex, etc
<mjg>
i'm very surprised frankly they did not sort it out yet
<geist>
the object locks are a big source of contention
<mjg>
are you using 'object' here in the same sense mach vm is?
<mjg>
kind of context dependent :)
pretty_dumm_guy has quit [Quit: WeeChat 3.5]
<geist>
probably? ie, a vnode, a file, or a container of pages
<mjg>
do you have object chains to handle cow et al?
<geist>
yes, which is also a source of contention
<mjg>
yep
<mjg>
that's the mach vm stuff right there, also a problem on freebsd
<geist>
works well enough for v1, but ends up being kinda a problem in the long run, etc
<mjg>
i have a wip patch for it :p
<geist>
yep
<geist>
trouble is we are also somewhat more flexible than posix which in that case hurts the overall design, since there are a few optimization paths we can't go down
<mjg>
it requires memory safety, which if i recall correctly you guys don't provide on purospe
<mjg>
purpose
<geist>
hmm? what do you mean by memory safety?
<mjg>
rcu-like
<geist>
oh just haven't invested in lockless stuff
<geist>
that's part of the whole v2 lets rethink some of this bits
<geist>
baby steps, as usual
<mjg>
the fundamental observation is that with rcu-like you can safely traverse the obj chain and validate correctness of said traversal
<mjg>
without locking squat
* geist
nods
<mjg>
well apart from the final page or whatever you were looking for
<mjg>
at least freebsd, while there is contention in the main vm obj of course, there is way more on the backing one
<mjg>
which will be libc, loader and other crappers
<geist>
exactly. libc is the bad one
<geist>
we get a gnarly chain on it in particular
<mjg>
i had an idea to implement a mechanism which lets you express what typically happens with these
<mjg>
so you can do it upfront
<mjg>
and hopefully avoid overhead later
<geist>
yeah we have some transforms we were thinking about doing on that whole obj chain. have some things thinking of
<mjg>
fucking computers man
<geist>
yah totally
<mjg>
i don't remember if i asked, do you have flamegraphs from a real workload?
<mjg>
running on fuchsia
<geist>
i have a crazy idea in the back of my head where you invert the whole thing and just have objects refer to page runs directly, a-la ZFS or BTRFS
<geist>
no obj chains, just simply COW the pages directly
<mjg>
i think that's the gist of what netbsd is doing?
<mjg>
i don't remember exactly what they did, but i do rmeember they whacked obj chains
<geist>
possibly. also i know that linux kinda does this, but they do it because they're posix and most pages are not shared, etc
<geist>
that's the optimization route we can't easily take, because the fuchsia VM design is somewhat more flexible
<geist>
and makes no real distinection between shared and not shared vm objects
<geist>
which means you dont get an easy optimizatino path
<geist>
ie, a more classic mach/pmap design
<netbsduser>
netbsd returns to the sunos approach where you replace object chains with reference-counting of pages
<geist>
yeah, i think that's the transformation we are kinda thinking about for v2
<geist>
thinking about how to ref count stuff, and more importantly if/when you have to ref count 'holes' in vm objects which is kinda expensive, etc
<geist>
well at least i think a v1.5 we've been mulling is to keep the chain but have some ref counting to short circuit walking up and down the chain
<geist>
but i haven't thought about it in a while, i'm not directly working on it so the details are a bit hazy
<mjg>
well the crux of my idea is to not ref any singular pages
<mjg>
but what is a typically used collection of pages
<mjg>
from the backing obj
<geist>
yah i was thinking of something kinda like that: some sort of huge btree of page runs that things have refs on. probably would be a nightmare to pull off though
<geist>
and no guarantees of speed
<mjg>
say you could blindly sort out any faults cming from libc for "normal" programs
<mjg>
ye this requires a lot of experimentation
<mjg>
's basically the most time consuming thing of the entire ordeal
<geist>
yah
<geist>
re flamegraphs not exactly. i dont think we have things instrumented that finely
<mjg>
feed for flamegraphs is trivial to generate
<geist>
what we do have is fairly fine grained logging of events, so we can get a pretty nice systemwide trace of all the threads in and out of the kernel
<mjg>
onyx got it :p
<geist>
so it's sort of a simiar thing
<geist>
can see where the time is going
<mjg>
i'll be happy with whatever profile of a workload you can share
<geist>
just not at the funcgion level or whatnot
<geist>
oh i dunno, i dont have that handy, as is usual with fuchsia it's complicated as fucking hell
<mjg>
worst caes it wont be comprehensible without knowledge of fuchsia
<geist>
we have a problem with generating the most complicated fucking tools on the planet, because google
<mjg>
welp it would just a mild curiosity
<mjg>
heh
<mjg>
ye i heard about that being a thing in the past
<geist>
more like ou need to check out a 50GB tree, build it, and only once you've built it can the tools make heads or tals of it, because the trace is highlyc ompressed and needs the built tree, etc
<mjg>
i guess cultire is hard to change, innit
<geist>
the trace is nice, but it's one of those has to go with some sort of db that comes out of the build
<mjg>
well the guerilla flamegraph support is easily few hundreds lines of simple code
<mjg>
one can hack it in an evening
<geist>
honestly all you'll tell me is 'yes this here is a disater, there is a disatser' and we're well aware of it
<mjg>
benefit:effort ratio is galactic on this one
<geist>
it's kinda depressing to be honest, i really dont want someone else to tell me the same thing
<mjg>
:))
<mjg>
look i was just curious what's happening, that things will be slow was already explained
<geist>
but there are a few people that are working on it. there are a few low hanging fruits that dominate everything, so you can't get to the next level
<mjg>
i'm pretty sure you menionted you are rolling with making it work
<geist>
sure
<mjg>
perf will be a concern at some point down the road which makes sense
<geist>
it's a concern now. we're at 1.0, works well enough for current things
<mjg>
i mean as long as the current work does not go directly against that effort ;)
<geist>
and now we're starting to get much more concerned about scalability and the next level of perf, etc
<mjg>
well then the first thing i would do is real adaptive spinning in mutexes
<mjg>
afair you just roll with some spinning and time out
<geist>
yeah we have that
<mjg>
i guarantee this is a big perf loss
<geist>
that helped a lot
<mjg>
you do *adaptive* spinning now?
<mjg>
that is, you detect lock owner is off cpu?
<geist>
we have shit spinlocks, and there's still a single scheduler lock, which is the major bottleneck
<geist>
yep
<mjg>
openbsd-vibes
<mjg>
lemme check that real quick
<geist>
again i didn't want to mention because you were just going to rag on it
<mjg>
fine
fedorafansuper has joined #osdev
<geist>
yes yes i know it's bad. i honestly feel bad about it every day
<mjg>
i wont say squat
<mjg>
friendly suggestion was above :)
<geist>
because that's my doing. missed the opportunity to update before things got big, and now it's harder to break the lock apart
<geist>
but someone is on it, they have a design, working through it now
<mjg>
with scheduling i'm honestly more conerned about big.LITTLE
<geist>
but not a day goes by that i dont kick myself for having not at least done a trivial breakup early on
<mjg>
fine-grained locking in the scheduler is ultimately some work and one will get there
<geist>
but that was when we were just trying to get it working before getting cancelled
<geist>
yah but it's a big bottleneck Right Now
<geist>
we're seeing it, and it tends to dominate everything contention wise
<raggi>
geist: you're too hard on yourself, we shipped
fedorafan has quit [Ping timeout: 260 seconds]
xenos1984 has quit [Ping timeout: 252 seconds]
<mjg>
so that's a spinlock?
<geist>
yes
<mjg>
this probably can be damage-controlloed until fixed
<geist>
yes. that's what we've been doing for the last few years
<mjg>
1. perhaps the spinlock code behaves worse than it can under contentin
<geist>
yes yes yes
<geist>
i know.
<mjg>
2. it is plausible there ar pessimal relock trips
<mjg>
which you can whack altogether
<mjg>
as in something locks and unlocks the scheduler sveral times while doing something common
eck has quit [Quit: PIRCH98:WIN 95/98/WIN NT:1.0 (build 1.0.1.1190)]
<mjg>
ok i'm getting off your back
<mjg>
i need to find some real time(tm) to finish my patchez
<mjg>
got several "survives benchmarking" projects which need finishing up, but who wants to do that
<geist>
and yeah one of the other paths is to finally rewrite the spinlock. when that day comes i do actually have some good questions for you
<geist>
ie, ticket vs the other spinlock (i forget the name)
<mjg>
mcs
<geist>
and where they start to scale differently
<mjg>
clh
<geist>
mcs yeah
<mjg>
clh might be a better choice if you don't do numa
<mjg>
it has cheaper unlock
<geist>
right now we have the fairly simple spin with halt/wfi one
fedorafan has joined #osdev
<geist>
right now the kernel only actually has a handful of spinlocks. like in the 10 category, though as we break up the scheduler lock there will be a proliferation of them, since probably each wait queue will get one, etc
<mjg>
do you have something for deadlock detection?
<geist>
(remember it's a ukernel so there aren't really any drivers or large subsystems that have intricate dat structures)
<geist>
yes
<geist>
simple thing: the spnlock writes the cpu number that holds it, etc
<mjg>
solaris doe snot :p
<mjg>
i mean do you detect lock orderingi ssues between different spinlocks
<geist>
and we have a lockup detector where each cpu periodically cross checks the other ones that they've made forward progress
<mjg>
say foo -> bar -> baz
<geist>
works fairly well for finding deadlocks
<mjg>
and someone tries to bar -> foo
<Ermine>
kaichiuchi: lol, now I'm having weird issues with pasting in vim+foot
<mjg>
i see
<geist>
an no, but again we have like literally 10 spinlocks. can actually manually prove that they dont nest
<mjg>
mutexes?
<kaichiuchi>
Ermine: :D
<geist>
mutexes do though
<geist>
also have a debug lock ordering detector, which is pretty neat
<geist>
it can determine *potential* ordering issues
<mjg>
that is what i asked about
<geist>
yah have that, just not on spinlocks
fedorafansuper has quit [Ping timeout: 246 seconds]
<mjg>
so arm64 is your primary dev platform?
<geist>
x86-64 too
<geist>
they're equal
<mjg>
but you ship only on arm?
<nur>
I misspelled the company name that I tried to apply to
<geist>
no comment
<mjg>
curernt products
<nur>
well scratch that job
<mjg>
:)
<mjg>
ok
<geist>
they're equal priority, lemme just leave it at that
<mjg>
not digging
<geist>
in other words, even if we ship more on this or that, it's unacceptable to leave one behind
<geist>
except maybe some feature like PCID or whatnot, if it's purely optimization
<mjg>
i'll only say the most pragmatic choice for spinlocks is a fair lock
<geist>
yeah fairness is the big problem, especially with big.LITTLE
<mjg>
it will leave some perf on the table in terms of througput, but will also limit tail end latency
<geist>
we have actually observed that big cores with a pure spinning lock wins like 90% of the time
<mjg>
i have ot tested big.little on that front but i can easily believe this is true
<geist>
yah it's a double whammy: we have too much contention on too few locks, and then you get double penalized because the big cores barge in and take the time
<mjg>
oh wait on that fucking arm the little cores don't do cas, do they
<mjg>
just ll/sc
<geist>
correct, though newer cores do cas, but we're still shipping and will probably continue to ship on v8.0 (ll/sc) cores
<mjg>
and they still win most of the time without employing cas?
<geist>
the cas stuff (v8.1) is literally under a feature called 'large system something' LSE i think
<mjg>
if so that's pretty bad
<mjg>
ye
<mjg>
large systems extension or so
<geist>
well it's because arm has a mwait/monitor like thing (WFE/SEV) that you use to park the cores when they are looping
<geist>
nice, but i think the big cores wake up and get back to spinning faster
<mjg>
one potential solution is to add a bit that little cores are trying
<mjg>
may or may not be good enough
<mjg>
anyhow you should be able to find someone at G who is versed in the area
<geist>
yah, honestly i haven't fully grokked how mcu/etc deals with WFE, since the ll/sc model works nicely for WFE
<geist>
but cas may not interlock properly
<geist>
surprisingly few, lots of folks at G are *very* versed in x86. godlike status, etc
<geist>
but arm is much less so
<geist>
as you can probably imagine
<mjg>
well i'm not surprised most people are amd64-only or similar
<mjg>
i do note there are crazies for ppc, arm or what have you
bauen1 has quit [Ping timeout: 264 seconds]
<mjg>
and i would be quite surprised if you did not have any
<geist>
but i'm not saying i or others can't figure it out, it's more like i haven't sat down to take that task on and figure it out
<mjg>
ye ye
<geist>
i tend to try to take on a few tasks at a time, and try to deeply grok it before implementing
<mjg>
i guess one could literally ask arm
<geist>
so i know it's on the table somewhere but haven't sat down to really do it yet
<mjg>
or some other vendor
<geist>
yep, we have arm experts around
<geist>
just less of them, and again i havne't sat down to really solve it.
<geist>
only so many tasks and hours in the day
<mjg>
well i can say that spinning behavior massively influences perf
<mjg>
you can have easily a factor of 3-4 of diff even at a small scale
<mjg>
compared to a naive spinlock
<mjg>
even while still leaving a lot on the table due to contention being there to begin with
<geist>
yah though like i said the arm spin at least is pretty nice, since it doesn't actually spin
xenos1984 has joined #osdev
<geist>
basically a pretty fine grained monitor that's cheap to break
eck has joined #osdev
<mjg>
but what happens when you wake up? you try the atomic again?
<geist>
sure. but i mean it doesn't spin any more than it has to
<geist>
basically if it works right the spinlock only spins on any cpu as much as someone releases it (since that breaks the WFE on all the other cores)
fedorafansuper has joined #osdev
<froggey>
mjg: that thing about linux protecting the address space with a rw lock is funny. exactly the same solution I came up with
<geist>
basically with ll/sc there's an intrinsic monitor the cpu puts on that particular cache line, and if you WFE aterwards (after detecting that the lock is held), it waits for that monitor to break by some other cpu storing to it
fedorafan has quit [Ping timeout: 246 seconds]
<geist>
a nice elegant solution
<geist>
what i haven't grokked is how that works if you're using the cas style atomics on arm
<geist>
since you do that in one instruction
<mjg>
key is that if they wake up at roughly the same time
<froggey>
I feel like ll/sc looks a lot better on paper than in practice
<mjg>
the repeated attempt bounces the cacheline more
<geist>
oh 100% and then the big cores tend to win
<mjg>
right
<geist>
faster cache/cpu/etc
<mjg>
look you can probably get an armista to make this into a ticket lock in few hours
<geist>
no doy
<mjg>
and this will perform dratsically better
<geist>
possibly
<geist>
i'd like to look at it this year indeed
<geist>
i just have 5 other more important things to do right now
<geist>
including not blowing half the day on irc! (but this is actually a good conversation, so i'll bill it to the company)
<mjg>
note as "meeting" in the timesheet!
<geist>
heh exactly
<geist>
reminds me i need to go reboot my ARM workstation box
<geist>
it sounds like a vacuum cleaner when you start it
<mjg>
sun vibes
<geist>
totes
<geist>
it's basically a 4U longass server box standing on end with some screwed on feet
<geist>
when it first boots up all the fans go 100% *backwards* until the firmware tells them to stop and spin properly
<mjg>
ll
<mjg>
i mean lol, not link load
<ddevault>
single stepping my interrupt handler assembly in gdb
<ddevault>
I have seen this place before
<mjg>
ddevault: you reminded me of a funny quote from early 00s or so
<mjg>
ddevault: kernel debugger manual suggested you print it, becaues it is hard to read it while single-stepping the kernel
jafarlihi has joined #osdev
<ddevault>
hah
fedorafansuper has quit [Read error: Connection reset by peer]
<jafarlihi>
Hey. What's the best and cheapest way of learning what the currently utilized disk I/O speeds are in Linux? I know you can cat /sys/block/sda/stat but it has count for read/written sectors, so you have to keep past state to know what the current speed is. Is there a way of getting raw speed without persisting past state to compare to?
fedorafan has joined #osdev
<netbsduser>
mjg: omnipresent smartphones have really simplified things for the "only have one computer" case
<jafarlihi>
I'd use my smartphone for everything if you could root it and it'd work like normal linux and you still could get updates
<jafarlihi>
otherwise you just use it and pray it's not hacked cuz you can't audit shit
<mjg>
netbsduser: btw one funny thing i wanted to look at is the benchmark mentioned in the uvm paper
<mjg>
netbsduser: but i don't have era-appropriate hardware and i don't know if a virtual machine will be good enough to test
<mjg>
i don't remembert what it was exactly, but according to the results in the paper freebsd sucked horribly at the workload
<mjg>
while netbsd+uvm did not
jafarlihi has quit [Quit: Client closed]
<geist>
oh random thing re: weird x86 topology
<geist>
was watching some reviews of the upcoming ryzen 7x003D chips and apparently they're doing something kinda odd
<Bitweasil>
The "half the chip gets more L3 than the other half" thing?
<geist>
basically they're both giving you a high boost mhz *and* 3x the L3 cache. this is strange, normally stacking cache is ..
<geist>
yep, as Bitweasil says
<Bitweasil>
Yeah, that's going to be a fun pain in the rear to benchmark and schedule for
<geist>
so the idea is that you have two dies of 8 cores (as ryzens are) but only one of them has a butt-ton of L3 but has a lower cap freq
<Bitweasil>
I don't think schedulers are "Task L3 use" aware.
<geist>
yah no
<Bitweasil>
More and more, proper use of 'taskset -c' seems required. :/
<mjg>
huh
<geist>
and in windows world i'm sure win11 is basically mandatory, since i'm sure win 10 wont get scheduling logic for it
<mjg>
is this the year of fucking with scheduler authors?
<geist>
pretty much
<geist>
i'm guessing in cpuid this just all shows up as unified L3
<geist>
since in the past they've never treated it like a split L3 since it's shared
<geist>
or humm, wait didn't the L3 go on the io die in the zen 3 era? i have to refresh my brain with all of this i guess
<Bitweasil>
Didn't some of the Xeons have a split L3 capability?
<Bitweasil>
Where you could segment off sections of it for various tasks?
<Bitweasil>
I don't know if that ever got reported out, though.
<geist>
yeah interesting, the L3 on Zen 4 is on the io die, so does that mean with these 7x003d chips there's a second bank of L3 sitting on top of one of the chiplets?
<geist>
is it 'closer' to the cores on that die?
<mjg>
ddevault: hello from irc!
<geist>
ddevault: yay!
<mjg>
fuck this, from now on i'm only doing risc-v
<geist>
do it!
<mjg>
still too mainstream
<mjg>
gonna port fuchsia to ultrasparc
<Bitweasil>
RISC-V now seems worth paying attention to, for sure.
fedorafan has joined #osdev
<mjg>
dude this is #osdev
<mjg>
you are already in trashcan area
<mjg>
- do you know jquery?
<mjg>
- no, but i know how to arm interrupts on x86
<mjg>
also if you did not brick your bare metal box, did you even boot your os?
<ddevault>
good question
<ddevault>
the fuck is writing to 0x7f0000
<Ermine>
hello helios!
<gog>
womp womp
<gog>
what's a FAR
<gog>
explain aarch64 to me now
<Ermine>
ESC (setup) ... is from u-boot?
<ddevault>
no, edk2
<Ermine>
ah
<ddevault>
FAR is the fault address
<ddevault>
...fault address register
<ddevault>
I should figure out how to unwind the stack
<geist>
note on arm64 it's super cheap to build with frame pointer, and then you can unwind with r29 (fp)
<ddevault>
yeah I have frame pointers
<Bitweasil>
gog, when you take a memory fault on ARM, the FAR (and FSR) report "what happened where."
<Bitweasil>
Interrupts on ARM don't build an interrupt frame like they do on x86, so the information as to why you're there is typically stored elsewhere.
<gog>
so FAR is CR2
<gog>
got it
<Bitweasil>
System control registers, mostly.
<ddevault>
tbh it's pretty nice
<ddevault>
though I don't like that syscalls are mixed in with other kinds of interrupts
<gog>
CR2 is a more clear and obvious name imo
<gog>
(haha joke)
<Bitweasil>
:p
<geist>
yah the ESR is a beast to dcode, but pretty nice that it's there
<geist>
syscalls are not *too* bad to test for, since you can just test for a particular bit pattern in the top of the ESR and then fork off in asm to the syscall path
<geist>
it's like a 2 or 3 instruction sequence
<Bitweasil>
I like how Intel drops some, but not all, vowels when they're naming things.
<ddevault>
yeah, at least once you know which registers you can fuck up
<Ermine>
I guess QBE is resonsible for stack structure in hare?
<ddevault>
atm I just save everything though
<ddevault>
Ermine: yeah
<gog>
what about all the E
<ddevault>
the E?
<geist>
note that riscv is more or less the same idea: one big entry point, a register describing what the exception was for, an at least one aux register with some additional details
<gog>
EAX EBX
<geist>
i just totally dig ESR: 'exception syndrome register'
<geist>
syndrome sounds so hard core
<ddevault>
feel free to teach your aarch64 assembler to call x0 eax
<Bitweasil>
I just wish someone would bolt a M series interrupt system on the A series cores. :/
<Bitweasil>
1024 vectors or something silly? Yes, please!
<gog>
dang
<geist>
ddevault: that's what rosetta 2 does!
* ddevault
shudders
<Bitweasil>
I half expected Apple to have done something like that with the M1, but I don't think they did.
<gog>
what would one even use 1024 interrupt vectors for
<Bitweasil>
ddevault, Rosetta 2 is *legitimately* fast.
<ddevault>
do I implement stack frame walking
<ddevault>
or do I add a bunch of printf debugging
<ddevault>
choices, choices...
<geist>
dunno if you read it but in the last month or so someone wrote up a blog post basically describing how rosetta 2 seems to work
<Bitweasil>
gog, faster response to different types of hardware interrupt. ARM only had IRQ/FIQ, and practically most ARMv8 systems route FIQ to the secure mode, so you really only have one interrupt for "Hardware needs something!"
<geist>
and it's quite straightforward. most of the speed is because M1 is a monster, and they added some instructions here or there that directly map to x86 things, like the EFLAGS layout
<Bitweasil>
geist, no, do you have a link to that?
<gog>
ohh ok
<geist>
otherwise it's mostly directly translate x86 to ARM64
<geist>
hmm
<ddevault>
oh I see what this does
<ddevault>
that's not gross
<ddevault>
I thought it was an assembler or something
<bslsk05>
dougallj.wordpress.com: Why is Rosetta 2 fast? | dougallj
<Bitweasil>
Sweet, thanks!
<geist>
also some obvious and clever things like mapping the ARM text segment just next to the binary so it can directly reach into the data segment, etc
<gog>
what was the cpu that did AoT bytecode translation, alpha?
<gog>
also for x86
<geist>
hmm, transmeta?
<gog>
yes
<gog>
well no i thought there was another
<geist>
would be interesting to see if someone has actually fully decoded and documented the raw ISA for transmeta
<ddevault>
so the answer is evidently a bunch of printfs
<geist>
i think it was VLIW so probably wonky
<gog>
printf debugging for the win
<Bitweasil>
The only one I know of that did "translate x86 to native" in the hardware was Transmeta. Itanium, at least initially, had a x86 core laying around the die for running x86 code, though I think later they had a software translater?
<Bitweasil>
I never did much with Itanium.
<geist>
yah iirc the first version of itanium had a x86 decoder, but it ran like ass. basically like one instruction at a time, so it didn't really utilize the pipeline
<gog>
i thought there was something earlier than those even
<gog>
ok the alpha one was software
<ddevault>
oh duh my userspace executable is loaded at 0x7f0000
<geist>
hmm, alphas thing was it had the PALcode, but that wasn't really bytecode
<ddevault>
I guess the page mapping is wrong
<gog>
yeah
<geist>
more like you switched to a more native ISA that had even more access to things
<ddevault>
L0 fault, hm
<ddevault>
maybe my cache maintenance is wrong
<ddevault>
let's throw in an invalidate everything and see what happens
<geist>
yeah that can definitely happen when you're on real hardware
<geist>
also remember the page tables on arm also fetch 'through' the cache, so you can get into trouble if you have stale shit in the cache when you turn the mmu on, etc
<geist>
even if the cpu's cache is still disabled (iv'e been burned about that)
<ddevault>
yes that was it
<raggi>
re "why is rosetta fast" one of the big markers is how fast it does compilation toolchains - and the answer there is that it has the fattest loader cache that there's ever been, so it can spawn functional toolchain processes faster than native can load & link
<ddevault>
is there some kind of cache maintenance I need to do when writing to TTBR0_EL1
<geist>
well no, but you should make sure your ducks are in a row
<geist>
you probably want to fully invalidate the data cache befure turning on the mmu
<ddevault>
currently I do tlbi vae1, x0 where x0 is the virtual address | (asid << 48) for the new mapping
<ddevault>
yeah I do that, this is post-MMU initialization runtime mappings
<geist>
if the cpu's cache is off, then as you're filling in the initial page tables, the tables are being written directly to ram
<geist>
there are a couple bits at the bottom of TTBR0 you might want to look at but i think 0 is okay as the default for those?
<geist>
which one is tlbi vae1?
<geist>
ie, decode vae1 for me
<ddevault>
well, I'll tell you what I think it should be, then dig out the manual and double check
<ddevault>
it should invalidate a given page by its virtual address in a particular ASID
<Bitweasil>
I miss realworldtech's articles. :(
<Bitweasil>
It seems that all the "deep CPU technical" writers are getting hired off with NDAs.
<ddevault>
I also do tlbi vale1, which does the same but only invalidates the bottom of the page tables
<Bitweasil>
Anandtech's gotten worse too about it.
<ddevault>
I do vae1 for L0/L1/L2 changes, and vale1 for L3 changes
<ddevault>
see ARMARM page 5201; D8.13.5
<geist>
double check that you dont need to shift the address by 12
<Bitweasil>
ddevault, there are sequences in the manual for "Thou Shalt" when doing certain things.
<Bitweasil>
I would suggest following those.
<ddevault>
it's really unclear regarding shifts
<ddevault>
Bitweasil: ack, will look for them
<Bitweasil>
You're in Aarch64?
<geist>
it ashouldn't be. it'll tell you what bits of the VA go in what bits of the register
<ddevault>
yes, aarch64
<ddevault>
correction: it is /abundantly/ clear
<ddevault>
there are like 50 cache invalidation instructions and the manual goes into information overload
<geist>
well, if you want to make it a little simpler: the tlbi and the cache instructions are different
<Bitweasil>
Ehm. Sorry. You're writing aarch64 system code without the manual open to the relevant pages? :p
<geist>
thoguh yes, tlb is really ac ache too, but they act fairly differently
<ddevault>
the manual is 12,000 pages long, Bitweasil
<Bitweasil>
And, yes, TLB is a separate thing from the cache ops.
<Bitweasil>
I know.
<Bitweasil>
I'm currently writing an ARMv8 emulator.
<ddevault>
having it open is just the start
<Bitweasil>
Believe me, I know.
<geist>
Bitweasil: yeah i was pointing iut out to dennis95
<ddevault>
finding what you actually need is the hard part
<geist>
ddevault:
<geist>
hi dennis95 ! sorry for tagging you
<Bitweasil>
The table of contents is pretty good.
<ddevault>
aye, it is
<ddevault>
I know I'm probably at least in the right subchapter
<Bitweasil>
I literally keep a Kobo Elipsa and koreader around for that sort of thing.
<Bitweasil>
And it lives plugged in during the day, because it's constantly on.
<ddevault>
I had a kobo elipsa until recently
<geist>
at least it's nicely hyperlinked. a pdf reader that lets you go 'back' to the last page you were at helps immensely
<ddevault>
if I had to read ARMARM in it I would jump in front of a train
<geist>
since you can follow hyperlinks and then back out
<Bitweasil>
ddevault, koreader.
<Bitweasil>
The stock PDF reader sucks, yes.
<Bitweasil>
It's fine for linear books.
<Bitweasil>
But the third party reader is great.
<ddevault>
ah fair point
<ddevault>
did not try koreader
<ddevault>
moot point anyway, it was stolen last week
<geist>
yah dunno what platform you're on but the apple pdf viewer > *, on windows i've had good luck with okular
<Bitweasil>
I've got a good expanding table of contents, back buttons, hyperlink following, the works.
<geist>
and on linux i think evince is not great, but it seems to work
<Bitweasil>
If you're on Apple, GoodReader is 100% worth the money.
<Bitweasil>
(for iPad/iPhone)
<ddevault>
>Arm recommends not invalidating entries that are not required to be invalidated to minimize the performance impact
<Bitweasil>
The MacOS PDF reader is solid.
<ddevault>
oh really
<ddevault>
thank you arm
<Bitweasil>
"If you flush all, don't come whining to us when your performance sucks."
<geist>
yah that's where it also gets complex on arm. there are lots of areas where you can really fudge it and get even more performance than the default one-thing-at-a-time
jjuran has quit [Ping timeout: 260 seconds]
<geist>
but you're then playing with fire so it's always a hard call
<geist>
on x86 a lot of this is just not on the table so you dont even have to think about it
<geist>
(firing off multiple tlbis and then a single DSB is the biggie)
terminalpusher has quit [Remote host closed the connection]
<Bitweasil>
Yeah, thar be "implementation defined behavior."
<Bitweasil>
It'll work, until it bites you in the rear in an insanely difficult to debug way. :/
<ddevault>
okay I am not feeling this manual right now
<ddevault>
let's invalidate everything on any mapping changes and come back later
terminalpusher has joined #osdev
<Bitweasil>
It'll work...
<Bitweasil>
The ARM manual is different from the Intel manual, though.
<ddevault>
I am good programmer
<Bitweasil>
Intel's stuff is "This is what we did, from your view."
<Bitweasil>
ARM is "This is how to implement a spec-compliant chip."
<Bitweasil>
So they're a different style of reading.
<Bitweasil>
But the ARM one is a lot more "This is *exactly* how this instruction works," if you care.
<Bitweasil>
Intel tends to gloss over corner cases that you shouldn't be hitting.
<geist>
yah half the time i prefer one or the other, context sensitive
<Bitweasil>
The Intel is better reading, tbh.
<ddevault>
agreed with geist
<geist>
kinda wish you had a treat me like i'm 5 version of the ARM manual for when you dont want to go into that much detail
<Bitweasil>
ARM's manuals are like reading a legal document, mostly because they *are.*
<ddevault>
the ARM manual is very very detailed, which is nice when you need to know a lot of detail
<geist>
but the other half of the time the detail is very nice
<ddevault>
but it /sucks/ when you don't need that much detail
<geist>
i can tell you it gets better, but it's still a TOME
<Bitweasil>
*shrug* You learn what you can skip.
<ddevault>
and for the love of god split it up into volumes
<Bitweasil>
Eh.
<geist>
i want one of those standing pedestals with an ARM manual on it
<Bitweasil>
It's no longer paper form.
<Bitweasil>
I'm fine with one whopper of a PDF file.
<geist>
with a little skull on the book bound in human flesh
<Bitweasil>
Oh man.
<Bitweasil>
Yeah.
<Bitweasil>
That would be epic.
<ddevault>
I can only open it firefox or xpdf, and only on a good day
<Bitweasil>
Gold leaf decoration, artwork for each page...
<ddevault>
all of the other readers I've tried choke and die
<Bitweasil>
Metal clasp to bind it closed.
<geist>
fWIW i've had no problem openign it with evince on linux, it's just not fast
<geist>
ie, it doesn't crash on me
<geist>
which some folks have reported
<Bitweasil>
Evince should handle it on Linux, GoodReader will deal with it just fine on iOS.
<kaichiuchi>
hi
<Bitweasil>
Or get another Elipsa and put koreader on it. :)
<ddevault>
evince is really slow with it at first
<ddevault>
then after a minute or two it just crashes
<geist>
yah i just can't imagine using it on an ipad or whatnot. may be nice for reading but what i do half the time is global searches, then hyperlink to it
<geist>
ie, find the first hit of whatever i'm looking for and then there's usually a hyperlink to the description of the register, etc
<geist>
seems like with an ipad or whatnot it's more good for linear or table of contents style searches
<geist>
which would be pretty bad for something this big
<Bitweasil>
geist, GoodReader lets you do exactly that.
<bslsk05>
git.sr.ht: ~sircmpwn/helios: aarch64: the TLB isn't /that/ important, right - sourcehut git
<ddevault>
fixed
<ddevault>
moving right along
<Bitweasil>
I do that constantly.
<Bitweasil>
I'm not a "fill an iPad with a lot of apps" sort, but GoodReader of the current version is one I'll pay for easily.
bauen1 has joined #osdev
<geist>
possible the problem is the asid stuff you're passing in
<geist>
also you have a DSB in those arch routines right?
<ddevault>
nope
<ddevault>
totally forgot it
<ddevault>
probably related
<kaichiuchi>
i have to ask a dumb question
<ddevault>
will investigate later
<geist>
yeah... that's like, super important
<geist>
you *must* dsb after a tlbi
<ddevault>
ack
<kaichiuchi>
architecture abstraction is obviously possible, but it seems almost impossible
<ddevault>
done for tonight but I'll see if that trivially solves my problem tomorrow
<geist>
tlbi basically is non blocking, it only starts the transation, dsb waits until it completes
<geist>
kaichiuchi: what do you mean?
<kaichiuchi>
hm, i’ll do my best to explain this
<geist>
depends at what level is what i think we'll get at. the trick is finding the particular layer to abstract at
<geist>
as you get more and more lower level you'll find differences, but if you abstract at where arches tend to align then it works reasonably well
<geist>
since clearly it's done all the time
<kaichiuchi>
if you’re on x86-64, and you want to query, say, SMP status or something, whatever outside of the arch layer
<kaichiuchi>
do people literally just have arch_smp_query or something
<geist>
sure
<geist>
stuff like 'arch_get_cpu_id' or whatnot is fairly abstractable, since it's generally assumed that there's a mechanism to get some sort of current cpu id
<ddevault>
pretty small kernel with x86_64 and aarch64 and pretty good separation of concerns
<kaichiuchi>
but what if you’re on a system that doesn’t have SMP?
<ddevault>
arch-specific stuff is in directories or files named +aarch64 or +x86_64
<geist>
kaichiuchi: then you can just say 'there's one cpu'
<ddevault>
well, you write an abstraction
<ddevault>
yeah like that
<kaichiuchi>
i assume it just returns NULL and you say “this architecture doesn’t support SMP so the” yes
<geist>
sure
<geist>
or SMP is active but there's only one core
<netbsduser>
kaichiuchi: i have e.g. `CURCPU()` which is an inline function or macro (i forgot) that returns e.g. the current CPU
<kaichiuchi>
the reason i’m being diligent is because i don’t want to get architecture abstraction totally ass backwards
<netbsduser>
meanwhile there is e.g. a CPU count and an array of CPU objects
<geist>
for a while it was really beneficial to have SMP and non SMP builds, but in general since most machines are SMP it's less of a concern now
<geist>
vs say 10-15 years ago
<ddevault>
I mean the basic approach is
<geist>
but in that case a non smp build would be to just ifdef out parts and hard code things like 'there's one cpu'
<ddevault>
start with one target platform, but be sensitive to what code you're writing is and is not arch-specific
<kaichiuchi>
that’s what i’m thinking
<netbsduser>
like many people do nowadays i have been experimenting with C++ which i find makes it harder to do this kind of abstraction if you follow the typical C++ approach
<ddevault>
kprintf? scheduler? syscall implementation? not arch specific
<ddevault>
syscall ABI? interrupt handlers? SMP? arch specific
<ddevault>
and be careful not to mix the two when possible
<bslsk05>
github.com: lk/arch at master · littlekernel/lk · GitHub
<kaichiuchi>
does kprintf try and detect if there’s a screen and print to that?
<geist>
and try to abstract via arch specific headers, and arch specific implementations of things
<ddevault>
no, it should work through an abstraction
<kaichiuchi>
or does it do arch_printf()?
<ddevault>
which might print to a screen, or to a serial port, or something else
<kaichiuchi>
and how does that work with video drivers?
<geist>
yeah or some sort of 'console_putc' or something
<ddevault>
or struct console { write: *fn(buf: []u8) size }
<geist>
that is then filled out at run time and/or statically with something that implements it for that machine
<geist>
yah
<ddevault>
video drivers would provide an implementation of that write function
<kaichiuchi>
ahhhhhhhhh
<kaichiuchi>
but then, for example
<geist>
for LK for example there's a platform_putc() that gets filled in per platform (ie, this particular board, or a PC or whatnot) that knows what to do with it
<ddevault>
(well, probably not exactly, but you get the idea)
<kaichiuchi>
in arch_printf(), would I then look for a “video driver” and talk to it in some abstract way as well?
<geist>
well, personally i wouldn't call it 'arch' because arch really has nothing to do with video or whatnot
<kaichiuchi>
are there direct “classes” of types of drivers?
<kaichiuchi>
right
<ddevault>
these questions are pretty naive
<ddevault>
just start writing code and you'll get the idea
<kaichiuchi>
yeah that’s kind of the point.
<netbsduser>
kaichiuchi: in my previous kernel i had 'classes' of device, quite literally
<geist>
i tend to be more hard line about it: arch is things like x86 or arm. i have another layer (platform) that deals with 'a PC or a raspberry pi or whatnot'
<geist>
personally linux is a bad example here, since they toss IMO far too much stuff under 'arch'
<geist>
whereas the bsds tend to be a bit more abstract about it
<kaichiuchi>
geist: i think that’s a reasonable abstraction
<netbsduser>
i decided to implement my drivers in objective-c to gain suitable benefits from the object-oriented approach and structured my drivers in an object-oriented fashion
<ddevault>
I don't like the "I'm a raspberry pi" code
<ddevault>
I like the "I have a device tree" code
<geist>
yes, it depends on how generic you like things
<kaichiuchi>
i see
<ddevault>
recommendation: ignore everything netbsduser just said
<geist>
and what the intent of the design is. keep in mind generic arm kernels are far more complicated than an x86-pc in a certain sense
<kaichiuchi>
right, which is the “problem”
<kaichiuchi>
i don’t know exactly how far I need to go
<ddevault>
if I may venture a guess
<kaichiuchi>
but i guess you cross that bridge when you get to it
<netbsduser>
ddevault: why?
<ddevault>
your kernel is not mature enough to be thinking about these questions right now
<geist>
so for now i'd not get too worried about it. just where something seems to be arch specific, put it behind at least one layer of abstraction (arch_foo, etc)
<ddevault>
if you are not really sure how printk links to your video driver, for instance
<geist>
and things that seem to be specific to that device (the screen, the uart, etc) put it behind another layer
<geist>
and then build from there
<ddevault>
you'll probably paint yourself into a corner and add unnecessary complexity by going multiarch at this stage
<geist>
if you at least dont slam everything together you'll have like 80% of your abstraction
<kaichiuchi>
it’s not wrong to ask these questions out of curiosity
<gog>
abstract your abstractions
<geist>
might not be perfect but it's in the right question
<ddevault>
sure, no problem
<gog>
concrete your concretions
<kaichiuchi>
but yes your point is well taken all the same
<ddevault>
just letting you know that you may not be well-equipped to apply the answers right now
<geist>
like even an abstraction like 'console_write' that kprintf goes do is a good starting point
<geist>
ie, dont put code that directly fiddles with your VGA screen in the body of kprintf
<ddevault>
that much is true :)
<kaichiuchi>
geist: precisely what I want to avoid
<ddevault>
like I said before, try to notice when you're writing arch-specific or portable code and organize them into separate places
<geist>
and dont fiddle directly with ARM registers in your main(), etc
<geist>
yep
<geist>
that sort of things is usually what i go for directly if someone says 'take a look at this os written for X purpose'
<ddevault>
if you do that you'll have a good start for when you're ready to add a second arch
<geist>
i like to see how mature the abstractions are
<kaichiuchi>
I just know that people starting out directly writing to B800 or whatever the hell the VGA device port is
<kaichiuchi>
which I guess is fine
<netbsduser>
that's a rather retro way to go
<geist>
right. and that's of course fine, but it generally shows a level of knowledge/experience of the author
<ddevault>
the mark of a promising project is how quickly they stop writing directly to b8000
<kaichiuchi>
but I don’t know how I can print through a video driver
<geist>
since even simple abstractions are worth a lot, and not really expensive at all
<ddevault>
you can't write a video driver at this point
<geist>
yah
<ddevault>
I've been working on my kernel for 9 months and I don't have a video driver
<ddevault>
all serial all the time baby
<kaichiuchi>
fair
<ddevault>
you have many yaks to shave
<kof123>
bad gog! abstract your concretions, concrete your abstractions...three times in the air. then you will surely have the magical osdev stone
<geist>
yah which is also why an abstraction of at least 'put the characters through this interface' helps, because serial and a screen look the same to the caller
<Bitweasil>
Banging on 3F8... or the MMIO version (probably a PL011?)...
<netbsduser>
i target the limine bootloader for amd64, which helpfully provides you with a terminal so you can get to work immediately on the actually interesting things
<geist>
as a side note this is also why vt100 still lives, since you can now abstract something that is a screen into something that looks like a serial port
<Bitweasil>
I'd rather have serial than video for bringup anyway. You can log serial to a file and sort through verbose amounts of crap without your remote end having to buffer it.
<ddevault>
I actually do still have a console writing to 0xb8000 so I can test on PCs without a serial port (i.e. almost all of them)
<ddevault>
but yeah I usually test with qemu via 0x3F8 on x86_64
<geist>
yah and PCs in particular have a bad tendency to triple fault and clear the screen
<geist>
so you can't see the last thing written
<Bitweasil>
Yup.
<Bitweasil>
Been there, "Wait, you... *ugh.*"
<Bitweasil>
"triple fault" vs "infinite interrupt loop" - I get why they did it, but it's kind of a nasty behavior.
<ddevault>
manually patching your fault handler to write less stuff so that you can see what your kernel did on real hardware is a mood
<Bitweasil>
Zero warning reboot with no way to evaluate state.
<Bitweasil>
And no way I'm aware of to really trap it (on hardware).
<geist>
kaichiuchi: anyway, you're asking the right questions, it's good stuff
* Bitweasil
curls up with the ARMv7 PMSA section.
* gog
pets Bitweasil
<Bitweasil>
Man, these inversions are rough on the solar production. :/
<Bitweasil>
And my backup battery bank inverter idles at 100W.
dude12312414 has joined #osdev
GeDaMo has quit [Quit: That's it, you people have stood in my way long enough! I'm going to clown college!]
joe9 has joined #osdev
fedorafansuper has joined #osdev
fedorafa_ has joined #osdev
gog has quit [Quit: byee]
fedorafan has quit [Ping timeout: 260 seconds]
fedorafansuper has quit [Ping timeout: 246 seconds]
invalidopcode has quit [Remote host closed the connection]
invalidopcode has joined #osdev
danilogondolfo has quit [Read error: Connection reset by peer]
gog has joined #osdev
dude12312414 has quit [Quit: THE RAM IS TOO DAMN HIGH]
<heat>
oh wow I sleep for a bit after almost 24h of no sleep and suddenly everyone decides to do osdev and talk about everything interesting
<heat>
smh
<zid>
Well it stands to reason
<zid>
there was space left in the channel because nobody was talking about how awesome rust is
<heat>
ruuuuus
<gog>
rust rust rust
<heat>
kaichiuchi, have you started
* sortie
merges DNS support to his libc.
<gog>
i don't know how to program
<sortie>
Date: Tue Jul 26 22:47:51 2016 +0200
<sortie>
This took a lil while
<gog>
aaaay
<zid>
I was just thinking of improving my network stack
<zid>
then I remembered I forgot how to set up my bridge.. again
<zid>
bridges are lame
<sortie>
macvtap is the cool thing I use to put one of my VMs directly on the internet
<zid>
Yea I'd use my bridge, if I could remember the command
<zid>
I always forget which things need which addresses
<zid>
whether I run dhcpcd on br0 or the devices, blah blah
<zid>
I should have written it down
<kaichiuchi>
heat: eventually
<heat>
kaichiuchi, start now
<heat>
just do it like the checkmark company
<heat>
or the screaming actor
<heat>
zid, you ran it the other day when you were trying to impress me
<heat>
i told you you should get a script
<kaichiuchi>
right now, I'm trying to figure out how I'm going to have someone home to pick up my packages tomorrow
<kaichiuchi>
because from _two separate carriers_, both packages were delayed
<zid>
heat: yes I couldn't remember then either
<zid>
Also my testing is fucked if I actually enable VBE cus it just hogs all the output and can't be updated easily :P
<zid>
I 'fixed' that by making it dump to serial port instead but a) I don't remember if it's in this tree and b) I don't remember how to log serial
<heat>
qemu -serial stdio > log
<heat>
there should be a "output-to-file" option but who tf cares
<zid>
noo I need to inotify | watch | tee | cat | vim it
terminalpusher has quit [Remote host closed the connection]
craigo has quit [Ping timeout: 246 seconds]
fedorafansuper has joined #osdev
fedorafa_ has quit [Ping timeout: 246 seconds]
fedorafa_ has joined #osdev
fedorafansuper has quit [Ping timeout: 252 seconds]
fedorafansuper has joined #osdev
fedoraf__ has joined #osdev
fedora___ has joined #osdev
fedorafa_ has quit [Ping timeout: 246 seconds]
fedorafansuper has quit [Ping timeout: 246 seconds]
fedoraf__ has quit [Ping timeout: 252 seconds]
fedorafansuper has joined #osdev
* sortie
merges ifconfig(8).
fedora___ has quit [Ping timeout: 252 seconds]
<mjg>
so are you faster than onyx?
<zid>
my code is way faster than onyx
<zid>
it's smaller so it downloads faster, and it's C so it compiles faster, and does nothing so the main loop is faster
<gog>
sophia is faster than boros
<gog>
it does even less
<zid>
is that because the entire codebase is _start: jmp -2; hlt
<gog>
excuse me it's more than that
<gog>
i also can print hello world
<zid>
nice
<gog>
i wrote a very big printf
<gog>
and only use a little of it
<moon-child>
I can print hello world in multiple windows
<zid>
I need to write proper dmesg code and have the screen automatically update and stuff but.. meh