klange changed the topic of #osdev to: Operating System Development || Don't ask to ask---just ask! || For 3+ LoC, use a pastebin (for example https://gist.github.com/) || Stats + Old logs: http://osdev-logs.qzx.com New Logs: https://libera.irclog.whitequark.org/osdev || Visit https://wiki.osdev.org and https://forum.osdev.org || Books: https://wiki.osdev.org/Books
<qookie> hm after a quick look at musl's source code, it looks like it keeps the handlers around
<qookie> I'll just go with that
<clever> qookie: one tricky problem ive seen, is that threads dont survive the fork, and any mutex held by a thread now wont be released
<qookie> yeah according to the posix docs that was the original intent behind pthread_atfork
<qookie> it was meant to be used to release mutexes etc
<clever> the context i saw that in, was qemu/afl triforce
<qookie> but then all mutex functions apart from semaphores are not async-signal-safe, which means using them in the atfork handlers is UB
<clever> basically, it would temporarily suspend all qemu threads (new code) in a place not holding locks
<clever> then use fork() to clone the entire state
<clever> then re-make the threads in the clone
<qookie> yeah I'm implementing it for qemu lol
<clever> run a short test in the clone, using AFL logic
<clever> then kill it, fork out another clone, and repeat
<clever> youll want to check out triforce then
<qookie> what's that
<clever> afl is a fuzzer, you feed random data into a program, and then check if it goes down a different code path
<bslsk05> ​nccgroup/TriforceAFL - AFL/QEMU fuzzing with full-system emulation. (133 forks/539 stargazers)
<clever> triforce-afl, does the same thing, at the kernel level, using qemu
<zid> it isn't actually random if you let it instrument the code
<zid> it will try to sort of sat-solve
<clever> and its using clone, to rollback the entire VM
<qookie> huh I knew about AFL but not TriforceAFL
<qookie> seems very nice
<qookie> but I'd like to get regular qemu running first ;)
dutch has quit [Ping timeout: 265 seconds]
Izem has quit [Quit: Izem]
dutch has joined #osdev
<qookie> damn yeah this seems really nice for fuzzing the kernel
<qookie> and looks like it's not much work to get running
<MrBonkers> Hmmm yeah that does seem interesting. I didn’t even know that was a thing
<clever> my rough understanding is that you basically just have 3 "hypercalls"?, snapshot, pass, and fail
<clever> so you boot the os, issue a snapshot, and get a payload
<clever> then you do a test, and report pass/fail
<clever> AFL then does its thing, monitors PC, and mutates the payload until you pass
YuutaW has quit [Quit: WeeChat 3.2]
YuutaW has joined #osdev
ElectronApps has joined #osdev
qookie has quit [Ping timeout: 258 seconds]
zoey has quit [Quit: Leaving]
<moon-child> see also syzkaller
<moon-child> fuzzer by google designed specifically for kernels. Uncovered _lots_ of bugs in freebsd and linux (and others afaik?)
eltonpinto has joined #osdev
<geist> yah we've run it against fuchsia too
<geist> though it took a fair amount of massaging to get that to work right since the syscall model doesn't totally line up with what it expects
sts-q has joined #osdev
Izem has joined #osdev
nyah has quit [Ping timeout: 252 seconds]
YuutaW has quit [Quit: WeeChat 3.2]
rorx has quit [Ping timeout: 252 seconds]
ElectronApps has quit [Ping timeout: 265 seconds]
YuutaW has joined #osdev
srjek has quit [Ping timeout: 256 seconds]
immibis_ has quit [Ping timeout: 265 seconds]
Izem has quit [Quit: Izem]
dude12312414 has quit [Quit: THE RAM IS TOO DAMN HIGH]
ElectronApps has joined #osdev
bradd has quit [Remote host closed the connection]
fkrauthan has quit [Quit: ZNC - https://znc.in]
fkrauthan has joined #osdev
fkrauthan has quit [Client Quit]
fkrauthan has joined #osdev
mctpyt has quit [Ping timeout: 258 seconds]
mctpyt has joined #osdev
jeramiah has quit [Ping timeout: 268 seconds]
<klange> gonna work on reviving the ol' toast notification daemon https://klange.dev/s/Screenshot%20from%202021-07-28%2013-36-41.png
vdamewood has joined #osdev
<klange> I have a (rudimentary) marked-up text renderer for formatting simple html-esque tags again (which I've also littered all over the popups for the panel widgets)
<klange> I've got a JSON parser that's not half bad, so I can make the new toast messages just json blobs with that restricted HTML-esque markup
vinleod has joined #osdev
<klange> There was a C toast daemon before the 1.0 release: https://github.com/klange/toaruos/blob/v0.14.0/userspace/gui/core/toastd.c
<bslsk05> ​github.com: toaruos/toastd.c at v0.14.0 · klange/toaruos · GitHub
<klange> It had a struct-based message format and supported a bold subject line and then a few lines of regular text
<klange> The Python version that replaced it added formatting as I had a rich text layout engine in Python that went on to be _almost_ a browser engine.
<klange> The new markup lib isn't as powerful as the one I had in Python, but it'll get there eventually.
vdamewood has quit [Ping timeout: 240 seconds]
<klange> Might even do proper DOM approach instead of just building runs of text.
<klange> I could have sworn I replaced this design with one that was just a fuzzy rounded rectangle and not this one that has crisp edges...
<moon-child> I did a lil responsive gui widget lib thingy a while back, and then decided that widget-based gui was the work of the devil and threw it out
<klange> A widget toolkit has been on my roadmap for so long, and I've built so many prototypes over the years, but I still don't really have one.
<klange> I have these little half-balked widgets that aren't part of a toolkit, like a button, or the menu bars (and general menu system), and I was working on extracting the text input box from my file browser as well (particularly so I could replace an entirely different text input box I have on my login screen)
<klange> On the one hand, I think from a backend perspective widget-based approaches are still what most devs want if they're bothing - and it's definitely what I want for my own apps - but code-driven UIs are terrible and documented-driven ones are better.
<klange> Like... QML, except QML has a bunch of its own issues.
<klange> Or Glade I think is GTK's thing?
<moon-child> I don't mean for building apps, I mean for interacting with them as a user
<moon-child> selecting and clicking buttons with a mouse is slow and unergonomic. Even qutebrowser-style selecting buttons with key sequences is 10 times better (though no one does even that), but
wille has quit [Quit: The Lounge - https://thelounge.chat]
<klange> A lot of research says otherwise, and us old keyboard-focused fogies are demonstrably wrong in thinking our keyboard approaches are faster.
<klange> GUIs are great for discoverability, and most users are terrible at remembering and structuring commands.
<moon-child> I don't mean to imply that command-line interfaces are ideal
wille has joined #osdev
<moon-child> mouse is great. But it's not the right tool for selecting one of a discrete set of options (which is what it's usually used for)
<klange> It depends on the options!
<klange> Mice are great for selecting "one of a discrete set of options" when those options lend themselves to visual scanning and identification.
<moon-child> why?
<klange> I'm not a psychologist / neurologist, so I can't say for sure.
<klange> As a species we're really good at hand-eye coordination and visual pattern detection?
mahmutov has quit [Ping timeout: 240 seconds]
<klange> Notably, my goal these days is to not at all be revolutionary, so I'm somewhat constrained by my objectives to do what everyone else does and build a traditional widget-based UI :)
Izem has joined #osdev
Izem has left #osdev [#osdev]
vinleod is now known as vdamewood
rorx has joined #osdev
GeDaMo has joined #osdev
Vercas has quit [Remote host closed the connection]
Vercas has joined #osdev
tacco has joined #osdev
mhall has joined #osdev
Oshawott has quit [Ping timeout: 258 seconds]
ElectronApps has quit [Ping timeout: 265 seconds]
ElectronApps has joined #osdev
zaquest has quit [Remote host closed the connection]
zaquest has joined #osdev
gog has quit [Remote host closed the connection]
ElectronApps has quit [Remote host closed the connection]
dormito has quit [Ping timeout: 258 seconds]
mctpyt has quit [Ping timeout: 240 seconds]
mctpyt has joined #osdev
sprock has quit [Ping timeout: 268 seconds]
sprock has joined #osdev
flx has quit [Ping timeout: 258 seconds]
archenoth has joined #osdev
sortie has joined #osdev
dormito has joined #osdev
mctpyt has quit [Ping timeout: 245 seconds]
qookie has joined #osdev
iorem has quit [Remote host closed the connection]
iorem has joined #osdev
mctpyt has joined #osdev
isaacwoods has joined #osdev
ahalaney has joined #osdev
nyah has joined #osdev
ElectronApps has joined #osdev
rorx has quit [Ping timeout: 258 seconds]
ckie has quit [Quit: off I go~]
ckie has joined #osdev
gog has joined #osdev
zaquest has quit [Quit: Leaving]
flx has joined #osdev
ElectronApps has quit [Remote host closed the connection]
rorx has joined #osdev
brynet has quit [Quit: leaving]
nismbu has quit [Ping timeout: 240 seconds]
nismbu has joined #osdev
qookie has quit [Quit: leaving]
mahmutov has joined #osdev
iorem has quit [Quit: Connection closed]
pwng has joined #osdev
jeramiah has joined #osdev
mahmutov has quit [Ping timeout: 268 seconds]
brynet has joined #osdev
graphitemaster has quit [Ping timeout: 250 seconds]
LambdaComplex has quit [Ping timeout: 252 seconds]
graphitemaster has joined #osdev
LambdaComplex has joined #osdev
pwng has quit [Ping timeout: 256 seconds]
<NieDzejkob_> clever: "any CoW fs is also GC'd" I don't see why you'd need anything more than refcounting for that.
<NieDzejkob_> (refcounting arguably is a form of GC, but the more exotic designs need a full-blown tracing GC...)
flx has quit [Ping timeout: 258 seconds]
<clever> yeah, i would say that refcounts are also a form of gc
<clever> in the case of zfs, it maintains a free-list, which is a list of blocks that need to be traversed and freed recursively
<clever> so if you delete a large object, it gets tossed onto the free-list (more of a to delete queue)
<clever> and then over time, it will scan its references, and free thngs
flx has joined #osdev
mahmutov has joined #osdev
ZombieChicken has joined #osdev
zgrep has quit [Ping timeout: 240 seconds]
zgrep has joined #osdev
ZombieChicken has quit [Quit: WeeChat 3.2]
drewlander_ has quit [Ping timeout: 250 seconds]
drewlander has joined #osdev
YuutaW has quit [Quit: WeeChat 3.2]
YuutaW has joined #osdev
thaumavorio has quit [Quit: ZNC 1.8.2 - https://znc.in]
thaumavorio has joined #osdev
flx has quit [Read error: Connection reset by peer]
srjek has joined #osdev
GeDaMo has quit [Quit: Leaving.]
gmacd has joined #osdev
<geist> yah agreed. depending on how the internals of the FS works, it'll probably delay the freeing of existing blocks for some time, at least until the new data is written out and appropriate data structures in place
<geist> even non COW fses can have some sort of logic like that: delete a file, mark the blocks it used as freed, but dont yet write that out to the bitmap until the same journal transaction that also updates the dir/inode metadata to mark the file deleted
<geist> since it wouldn't be safe to start reusing those blocks yet for new files
Burgundy has joined #osdev
mahmutov has quit [Ping timeout: 258 seconds]
mahmutov has joined #osdev
Matt|home has quit [Read error: No route to host]
dormito has quit [Ping timeout: 276 seconds]
mahmutov has quit [Ping timeout: 272 seconds]
mahmutov has joined #osdev
gmacd has quit [Remote host closed the connection]
henistein has joined #osdev
dormito has joined #osdev
mahmutov_ has joined #osdev
mahmutov has quit [Ping timeout: 258 seconds]
flx has joined #osdev
Izem has joined #osdev
Burgundy has quit [Ping timeout: 252 seconds]
<NieDzejkob_> wouldn't marking the blocks as freed also be part of the journal transaction?
<clever> NieDzejkob_: the tricky part, is when you have a tree of blocks
<clever> marking them all as free, means reading the intermediate nodes in the tree
<clever> as a shortcut, you can just add the first layer to a list, and then figure that out later
<clever> then as an async task, read a block, clear it from the bitmap, and add every reference to the list
<bslsk05> ​gist.github.com: gist:19248634add01c2f54e2004775b889b1 · GitHub
<clever> NieDzejkob_: in this case, the 2.9gig file has an L3 block at the top, which has a pointer to a pair of L2 blocks
<NieDzejkob_> hmm, so while we're on the topic of allocation, I've read somewhere that Linux uses a clever allocation algorithm that prevents fragmentation, but I've done some searching a few days ago and I couldn't find the details
<clever> that pair of L2 blocks, then contain pointers to 183 L1 blocks
<NieDzejkob_> would there be a problem with the freelist growing too large?
<clever> and those 183 L1 blocks, contain pointers to 23118 L0 blocks
<NieDzejkob_> I suppose a depth-first thing would work best
<clever> zfs has a limit for the number of pointers in a single block
<clever> so the depth is how it references more
<NieDzejkob_> that way the free list is bounded to block_size / ref_size * level_count * top_level_item_count_in_free_list
<clever> if i was to delete this file, it would have to read 1 L3, 2 L2, and 183 L1's, to be able to find all 23118 L0's
Izem has quit [Quit: rcirc on GNU Emacs 27.2]
<clever> NieDzejkob_: i think each filesystem driver in linux has its own algo to avoid fragmentation
<clever> xfs for example, has the inode table spread over the disk disk, and picks an inode slot near the first data block, so you have less seek time between the inode and data
<clever> the algo to convert inode# to block# is a simple formula, and once it reads that inode, it doesnt have far to seek to find the first data block
<clever> but as a side-effect, if the 1st datablock is >2tb into the disk, the inode# is >32bits in size
<clever> xfs also has `xfs_fsr`, which will defrag the filesystem online
<clever> basically, it will just copy a file (using normal api's), then compare the fragmentation (using the filefrag api)
<clever> if the copy is better, it uses a special xfs api to atomicly swap the backing blocks, and only if the original hasnt been modified
henistein has quit [Quit: Client closed]
sortie has quit [Quit: Leaving]
Belxjander has joined #osdev
Belxjander has quit [Ping timeout: 276 seconds]
ahalaney has quit [Quit: Leaving]