gabi-250 has quit [Remote host closed the connection]
gabi-250 has joined #osdev
wootehfoot has joined #osdev
ss4 has joined #osdev
ss4 has quit [Quit: Leaving]
orccoin has joined #osdev
wootehfoot has quit [Quit: Leaving]
GeDaMo has joined #osdev
stolen has quit [Quit: Connection closed for inactivity]
orccoin has quit [Ping timeout: 256 seconds]
Hammdist has joined #osdev
Hammdist has quit [Client Quit]
valshaped7424 has quit [Quit: Gone]
valshaped74248 has joined #osdev
orccoin has joined #osdev
orccoin has quit [Ping timeout: 250 seconds]
Hammdist has joined #osdev
Hammdist has quit [Client Quit]
orccoin has joined #osdev
Hammdist has joined #osdev
nyah has joined #osdev
<sham1>
hi
<zid>
again?
benlyn has joined #osdev
<sham1>
yes
<zid>
I should book an indian holiday, I've gotta be immune to everything by now
<kazinsal>
I spent a week in vegas so I'm pretty sure I'm unable to contract any disease short of the caner
<zid>
Try india, make sure you book 2 weeks minimum, the first week will be in the bathroom
<kazinsal>
a week of unmasked drunken stumbling through the streets of lost wages rolled me the right feeling for a year
<kazinsal>
managed all of that without becoming a shambling zombie of the meltlung or whatever the psychotics on the internet think you get for exiting your home without a ghostbusters pack attached to a respirator
<zid>
god, is kazinsal antivax too?
<zid>
had to ignore one of those yesterday
<kazinsal>
I get all my shots
<kazinsal>
I just get my shots and not have mental breakdowns when thinking about air that's not filtered through an AC unit
<zid>
I can't tell if you're being a gigantic douche, or realistic
<zid>
like, are you making fun of people who were caused distress by covid?
<kazinsal>
I have seen people so mentally broken by covid that they literally wear a full positive pressure ventilator when leaving the house
<kazinsal>
we're talking backpack and all lookin ghostbusters ass
<kazinsal>
tubes and shit
<kazinsal>
luke I am your father
<zid>
Use it if you got it
<zid>
I would, if half my country was actively trying to murder me, and I had an extant respitotry disease
<zid>
and I'd call people who made fun of me a brainwashed republican shill while I did it
<kazinsal>
I just get the once every eight months shot in my left forearm
<zid>
so you deny that covid is a big deal then?
<kazinsal>
and I have no need to give myself psychosis
<zid>
nobody asked you to
<kazinsal>
we're rolling about 50,000 copies per litre per day in wastewater here
<kazinsal>
for a metro area of four million
<zid>
oh that's fine then, thank god
<kazinsal>
yes that is actually fine
<zid>
take you haven't had family members die of it like I have
<zid>
it*
<kazinsal>
that is hundreds of times below the pre-2020 levels of copies per litre in wastewater of influenza
<zid>
and then watched a giant piece of shit repeatedly tell everybody not to care
<zid>
then have idiots on irc believe him
<zid>
cunt
<kazinsal>
meanwhile an influenza vaccine has an efficacy rate of about 40% against significant disease whereas a modern mRNA COVID vaccine as a 93% efficacy rate against significant disease
<kazinsal>
if your local health authority cannot sort out how to get immunocompromised people additional doses then do the proper american thing and make shit up to get them
<kazinsal>
pretend to have diabetic hypercancer or whatever your local CVS needs to hear in order for you to get eight shots per day
orccoin has quit [Ping timeout: 260 seconds]
<kazinsal>
the rest of us who can breathe on our own and who don't have pre-existing conditions get our biannual shots
<kazinsal>
same as we do with the flu
orccoin has joined #osdev
<kazinsal>
every five years I get a tetanus shot, a diptheria shot
<kazinsal>
you know what? shit's worked for me even dealing in the land of sin and catastrophe
<kazinsal>
but I'm lucky enough to not be immunocompromised or such so I only have to do the standard every six months jam some tasty micro-ribonucleic acids into my shoulder.
<kazinsal>
I'm also lucky enough that most of my elderly relative died right before this all broke out so thankfully they never had to deal with this shit
DrPatater has quit [Quit: Explodes into a thousand pieces]
<geist>
huh just read about libera dropping the matrix portal
<geist>
interesting blog about it at least
<geist>
on liberas page
<geist>
honestly after all these years i dont even know what matrix really is
<nikolar>
I have an interesting algorithmic question
<nikolar>
I'm writing a cow btree, and I need a way to find a diff between two generations of the tree
<nikolar>
Can't really find any information online and what I managed to write is atrociously slow
<geist>
i'm not super clear on it, but it's my understanding that btrfs uses a second data structure there, that tracks per page of the btree (presumably 4K) what generation it is or whatnot
<geist>
and thus can snapshot a tree fairly simple and start peeling off new versions of it by consulting that every time it needs to write a new btree page
<geist>
but presumably you can look through that data structure to see how any two trees differ
<zid>
that was good read
<nikolar>
geist: thanks i'll try to go through btrfs code
<nikolar>
wish they had better documentation lol
<geist>
yeah i only pieced it together by reading lots of various blogs and whatnot
<geist>
but the second tree to track which pages have been COWed is the trick, some variant of it
<nikolar>
this is what i managed to find in case anyone is interested
<nikolar>
i kind of dislike the tower of btrees for the reference counts lol
<nikolar>
would be nice to have something simpler
flom84 has quit [Quit: Leaving]
<gog>
hi
<nikolar>
hello gog
<nikolar>
do you like btrees
<gog>
i do like btrees
<geist>
one of these days i should finish up some btree library i started on years ago
<geist>
was generally looking around and i havne't found a lot of good generic btree codebases
<geist>
and/or the ones that exist have a license i can't use
<nikolar>
i've actually implemented btrees twice now, first in python to see how hard it is and now in c to actually use it in a project
benlyn has quit [Ping timeout: 246 seconds]
<nikolar>
but i need the diff thing to continue and my implementation seems quadratic
<geist>
seems that step one is have the btree never overwrite, always allocates a new block (which rippes up to the root every time) and then the COW part is the fact that you dont reclaim old blocks if it belongs to a snapshot
<zid>
COW
<geist>
so really the auxillary data structure is really just a ref count per block, i guess
<zid>
I think we should name more things like we're shouting them to the driver in a car and pointing
<nikolar>
but that requires an inplace update
<geist>
what requires an inplace?
<nikolar>
to increment the refcoutns
<geist>
oh but the refcount is a *different* data structure
<nikolar>
ah right yeah
<geist>
like a separate btree
<geist>
with maybe different properties, like it does ref count in place
<geist>
er i mean the ref btree maybe does work inplace, or journalled or something
<geist>
naievly i guess a snapshot of the whole main btree just involves incrementing the ref of every block it occupies, dunno if there's an even faster way than doing that
<nikolar>
how does that work with snapshots i wonder
<nikolar>
and clones
<geist>
then you just create a new root pointer to the snapshot. <snapshot X, root of btree -> Y>, etc
<geist>
then as either are modified, they create new roots and the ref counting of their blocks keeps the other part from being reclaimed
<geist>
i think that's basically how btrfs works
<nikolar>
hmm yeah i guess that works
<geist>
and of course the btrfs btree is i'm sure really a b+tree, and has a tremendous fanout, so it probably only goes a few levels deep
<geist>
so more or less the first time you touch either half it probably generates new nodes all the way up to the root, since everything moves position
<nikolar>
yeah, maybe i could make it work with an avl tree for the refcounts or something
flom84 has joined #osdev
<geist>
yeah
<geist>
i think that's basicaly the trick, though most of everything in btrfs (or similar) is in a main big btree, it's not the *only* data structure
<geist>
there are other trees that work alongside it, etc
<nikolar>
yeah i gathered that much
<nikolar>
allocations, etc
<geist>
yah they have the whole translation layer from some sort of logical block to data blocks, where it can distribute stuff across multiple devices, etc
<geist>
and of course ZFS has somehting ike that too, where logical data objects are translated to where they really ive on device(s) via some second layer
<nikolar>
oh yeah raid stuff too
<nikolar>
though in zfs it's more clearly a separate layer as far as i can tell
<geist>
kinda like a MMU for the virtual address space that the main data structures live in
<geist>
yah
<nikolar>
guess i need another btree for refcounts actually
<nikolar>
to avoid obscene number of unnecessary writes
<geist>
yah what i dont grok is when you snapshot how you can efficiently bump the ref count for everything that the main tree occupies
<geist>
dunno if you can do something like store out of the tree some sort of delta offset, such that you dont have to rewrite most of the refcount tree on snapshot
netbsduser has quit [Ping timeout: 246 seconds]
<geist>
but distribute the updates over later rewrites or something
<geist>
but seems like that just delays the need for you to update it
<nikolar>
yeah i need to think more about this kek
<geist>
or maybe each node in the btree has a ref and some sort of delta ref to things downstream fro it? dunno. probabl a lot of optimizations there
<geist>
but the naive thing is i guess to just stop the world and increment all the refs for all the blocks when you snapshot
<geist>
probably the refcount tree is not that large, especially i fyou use relatively small refcounts, like 16 bits, so it may be that it's not really that big of a deal
<nikolar>
yeah maybe, though you can't do it in place
<nikolar>
which means you need another tree to track the changes for it
<geist>
yah though that's probably fine though. a non inplace ref tree would be fine
<nikolar>
and we're back at the btrfs tower of trees
<geist>
well, not really, you can just write an out-place ref tree, you just sort of have to lock it while you snapshot
<geist>
usual rules: bump all the counts, ripples all th way up, set a new root for it, pseudo journalled in that sense
<nikolar>
well i want to the the zfs's double superblock trick for the consitency
<geist>
sure, i think that's kinda implicit
<nikolar>
basically write the first one to begin the operation and then write the second one to commit
<nikolar>
but for this to work, you can't touch the old data
<nikolar>
so no inplace anything
<geist>
the core btrees that the superblock might point to: ref count tree, block allocation tree, subvolume tree, you'd probably double superblock update
<geist>
sure, i think there's probalby no reason to inplace any of these data structures
<geist>
then the main secondary btree is the Big One, but the roots to it are all pointed to by entries from the subvolume tree
<geist>
and eyah, no reason to inplace anything now that i think abot it. no value in that, which is great because you can just reuse your same btree code
flom84 has quit [Quit: Leaving]
<nikolar>
so the tower of trees is kind of unavoidable
<nikolar>
dang it
<geist>
probably some tricky interactions when simultaneously updating multiple of these, especially the block allocation tree
<geist>
well, depends on what you mean by 'tower of trees'
<geist>
and of cours eyou dont have to do any of this, it's just more like 'if you were to design something similar to btrfs, here are the core pieces'
<geist>
you dont *have* to do this
flom84 has joined #osdev
<nikolar>
well if i understood correctly, btrfs has a refcount tree for the main data tree
<nikolar>
and then if it grows too big, it needs another tree for refcounts of the first refcoutn tree
<nikolar>
ad nauseam
<geist>
you could do something more like reiserfs3 where iirc everything is in one bigass btree, but then you have a traditional journal on the siede
<geist>
dont see whty you need refcounts for the main ref tree
<geist>
the refcount tree is only as big as there are blocks allocated to the main data tree
<nikolar>
correct
<geist>
ie, for every block in all of the subvolumes in the main tree, you have a node
<geist>
but the refcountr tree itself doesn't need to be refcounted
<nikolar>
but it's also a versioned btree, which means you need to have it cow, which means you need refcoutns again
<geist>
no, i dont see why you need to version it?
<nikolar>
well what does the old superblock point to in case of a power loss during the transaction
<nikolar>
the refcount could be inconsistant if it's not cow'd
<geist>
the old version of it, but versioning just give 2 versions, the yet to be updated one, and the old one
<geist>
so when you update it you just start writing new blocks out, then once they're flushed you write out the new superblock, thus simultaneously making th enew version active and freeing the old blocks (sine you're also updating the allocation tree)
<geist>
basically requires out of place updates
<geist>
once a block is written it's basically immutable until its reclaimed implicitly by being no longer pointed to in a new version of the tree pointed to by a superblock update
<nikolar>
so i need to find the diff between the old and the new, so i can update the allocation data
<geist>
right, but as you're writing it out you're also updating the allocation tree
<nikolar>
yes, but my initial question is how to efficiently find the diff
<geist>
since there's only two versions, the ref count on the tree is implicitly either in use or freed
<geist>
so as you're updating this tree, every time you need to replace a block, you allocate a new one (and updating the alloc tree/bitmap/etc) and free the old one
netbsduser has joined #osdev
<geist>
but it doens't become live until you write out the superblock
<geist>
(well pseudo free, it's not okay to reuse the block until the superblock is written)
<geist>
so it's not a case f needing to find a diff, it's a case of keeping track of it as you're generating the diffs
<nikolar>
yeah ok, i think that can work
<nikolar>
i just need to reimplement the btree for this specific use case lol
<geist>
which if you're not writing a snapshotting fs, that's basically all you need for the main btree
<geist>
you can just start writing a new btree out, and at particular sync points flip to the new one and free the old blocks implicitly and just continue
<geist>
frankly if i was starting my own btre based fs i'd just do that first, and leave COW out of it
<nikolar>
i want to do snapshotting though
<geist>
sure
<nikolar>
though snapshots will be a part of the main tree so that should be fine
<sham1>
Snapshots are fine since they're explicit
<sham1>
CoW isn't really used enough to justify making it the default policy
GeDaMo has quit [Quit: That's it, you people have stood in my way long enough! I'm going to clown college!]
<nikolar>
geist: thanks btw
<geist>
sham1: thing is with COW snapshots basically 'fall out' of the design
<geist>
so one almost implies the other
<geist>
nikolar: sure, it's good thinking about this stuff again. haven't thought about it in a while
<geist>
had to sort of page it back in, so to speak
<nikolar>
we love os puns
<nikolar>
i have been thinking about it, but you only realise what you need once you start writing it
<sham1>
geist: sure, insofar as every transaction turns into a snapshot effectively
<sham1>
Is it worth it for potential slowdowns and such to have that for every write?
<geist>
oh indeed. i use btrfs almost exclusively here, but i definitely do things like mark files that back VM disks as nocow
flom84 has quit [Quit: Leaving]
<geist>
otherwise the fragmentation is off the charts
<geist>
still kinda is anyway, but much less so otherwise
<geist>
if you dont do it you pretty quickly end up with a 10GB file having like half a million fragments or something
<nikolar>
yeah that makes sense for nested filesystems anyway
<nikolar>
marking nocow
<nikolar>
i like zfs but it's annoying you can't opt into no cow
<sham1>
Yeah, and you'll have to defrag it as if you were suddenly transported back into Windows
<nikolar>
for swap for example
<geist>
thankfully flash has made most of this largely irrelevant
<geist>
yah i think swap on btrfs *requires* nocow
<sham1>
Why are you storing your swap on zfs
<nikolar>
i am not
<sham1>
And/or btrfs
<zid>
the broodwar pro is our savior as ever
<nikolar>
i am using a swapfile on ext4
<nikolar>
just saying you can't use a swapfile on zfs
<geist>
why not? it's not slower necesillary. all the kernel does is basically grab a block list and swap on top of that
<nikolar>
because it might trigger more memory allocations and more swapping
<geist>
i'd think that swap on ext4 vs any other on disk fs is the same speed. the real key is it just needs to tell the fs to not reallocate it and get the underlying block map
<nikolar>
and endless thrashing
<geist>
nope. that's the whole point, it up front 'locks' the file basically
<geist>
the only real overhead is based n how fragmented the file is, so there's an incentive to get the minimum number of frags
<nikolar>
i was talking about swap on zfs, where rewriting a part of the file is not remotely a trivial thing
<geist>
but that's the point, it only really works on nocow files, since it's functionally bypassing the fs
<nikolar>
yeah exactly
<nikolar>
thus, you could have a swapfile on btrfs
<geist>
the swapping ssystem is directly writing to the underlying block device via this translation map that it grabs up front
<geist>
right
<nikolar>
and not on zfs
<geist>
well, okay, maybe not on zfs, i dunno enough about it
<geist>
and probably not on top of stuff that is btrfs raid?
<nikolar>
yeah probably should avoid swap on raid lol
<geist>
anyway once i figured out that it's just bypassing the fs i became not very afraid of it and i generally dont bother with separate swap partitions anymore
<geist>
and i actually tell my linux boxes to use swap, i even set swappiness to 100
<nikolar>
yeah, i never had a swap partition
dude12312414 has joined #osdev
<geist>
with a caveat that that's always on some sort of flash based thing, swapping onto a disk is a bit rough
<nikolar>
i started using a swapfile on this laptop and it did actually save me from many oom situations
<geist>
exactly, and i generally found with swappiness set to 100 over the course of a few days you find a lot of background processes and whatnot will shed some number of MBs into swap and never touch it again
<geist>
which seems like a win for me
<nikolar>
what does setting swappiness to 100 do exactly
<geist>
the default is 60 (/proc/vm/swapiness)
<geist>
you can set it from 0 to 100, and iirc it basically sets a master bias towards when reclaiming pages to handle memory allocations will it tend to reclaim a disk cache page or an inactive private memory page
<geist>
ie, some anonymous memory from some process
<nikolar>
ah makes sense
<geist>
so if you set it to 100 over time it will tned to cause it to write out inactive memory that hasn't been touched in a long time
<nikolar>
yeah good tip
<geist>
but you'll still only generally see it write out to swap as you get below some sort of active memory thresholds
<geist>
or over really. ie, when running a lot of processes that has active memory over say 50 or 75% and it needs to start more aggresively reclaiming
<geist>
or at least i think so
<geist>
and of course once all free memory is basically used with cache
<geist>
what linux doesn't really do is sit around and aggressively pre-swap stuff out to disk before it starts running out, it more than anything else reacts to memory pressure, but doesn't proactively try to deal weith it
<nikolar>
yeah it seems like all my free memory is indeed used by the cache
<nikolar>
so we'll see
<geist>
yah and if you say have 64GB ram but are only using like 10GB of it and 54GB is cache, you probably wont see much swap activity, even in swappiness 100
<geist>
it might write out a few hundred MB over time, since maybe it still decides to go ahead and write out some really really old unused pages
<nikolar>
yeah i have 6gb lol
dude12312414 has quit [Quit: THE RAM IS TOO DAMN HIGH]
<nikolar>
well 8, but 6.6 after igpu takes its cut
<Ermine>
MTP FUSE somewhy reports EIO when there's no space available on device
netbsduser has quit [Ping timeout: 256 seconds]
netbsduser has joined #osdev
bauen1 has joined #osdev
vdamewood has joined #osdev
[itchyjunk] has quit [Ping timeout: 246 seconds]
* Ermine
is migrating stuff to a new library before deploying
<Ermine>
Surely it won't backfire
netbsduser has quit [Remote host closed the connection]
netbsduser has joined #osdev
wblue has joined #osdev
Burgundy has quit [Ping timeout: 248 seconds]
<zid>
make sure not to test it first for more than about 8 seconds
<zid>
otherwise you might find some issues
<nikolar>
Kek
<zid>
make sure significant parts of it are written in interpreted languages too so that unusual code paths can't be checked until used
<zid>
and absolutely *do not* use any sort of type system, this makes your program harder to maintain by enforcing things like numbers not having emoji in them
<mjg>
types are for people who can't keep track of what they are doing
[itchyjunk] has joined #osdev
wblue has quit [Quit: wblue]
gog has quit [Quit: byee]
Turn_Left has quit [Read error: Connection reset by peer]