klange changed the topic of #osdev to: Operating System Development || Don't ask to ask---just ask! || For 3+ LoC, use a pastebin (for example https://gist.github.com/) || Stats + Old logs: http://osdev-logs.qzx.com New Logs: https://libera.irclog.whitequark.org/osdev || Visit https://wiki.osdev.org and https://forum.osdev.org || Books: https://wiki.osdev.org/Books
<geist> ah just a temporary breakage i guess
<heat> not temporary, I dunno what happened
<heat> maybe LTO doesn't work anymore for older clang versions
gog has quit [Ping timeout: 260 seconds]
wgrant has quit [Quit: WeeChat 3.5]
wgrant has joined #osdev
vdamewood has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
nyah has quit [Ping timeout: 252 seconds]
frkzoid has joined #osdev
FreeFull has quit []
tarel2 has joined #osdev
mzxtuelkl has quit [Quit: fBNC - https://bnc4free.com]
dennis95 has quit [Quit: fBNC - https://bnc4free.com]
friedy has quit [Quit: fBNC - https://bnc4free.com]
andreas303 has quit [Remote host closed the connection]
terrorjack has quit [Quit: The Lounge - https://thelounge.chat]
<wxwisiasdf> where can i find a list of functions gcc require
terrorjack has joined #osdev
<zid> ..what?
<zid> To build it?
elastic_dog has quit [Ping timeout: 244 seconds]
<wxwisiasdf> zid: No, to run it
dennis95 has joined #osdev
<zid> you mean syscalls then? depends on OS
<wxwisiasdf> no i meant as in
<wxwisiasdf> what does cc need to do a sucessful C compilation
<wxwisiasdf> cc piped into as
<zid> where do the functions come in?
<moon-child> wxwisiasdf: memcpy, memmove, memset, memcmp, and abort
<wxwisiasdf> sorry my bad, yes to build it
<bslsk05> ​gcc.gnu.org: Standards (Using the GNU Compiler Collection (GCC))
<wxwisiasdf> everything from the ansi c language?
<wxwisiasdf> i guess gnulib would cover the rest no
heat has quit [Ping timeout: 260 seconds]
<zid> I'm still incredibly confused as to what you're asking
<wxwisiasdf> to build gcc
<zid> Okay. "To build gcc, ..." now fill in the ... part
<Griwes> the answer is "it's whatever you get an error about missing when you try to compile it"
<wxwisiasdf> and run it?
<wxwisiasdf> fair
<zid> it doesn't need any 'functions', or it needs '50 million' of them, depending on what you're asking
<wxwisiasdf> i am asking what gnulib won't cover
<wxwisiasdf> and what i would need to implement to successfuly build it
<zid> That question is much different, and is "gcc, coreutils, binutils, make, sh"
<wxwisiasdf> llast time i tried i did implement a good chunk of the c library but gnulib expected some regex shit
<wxwisiasdf> zid: no, just gcc, building gcc with a libc and getting it running on one's os
<wxwisiasdf> well not gcc, more like just cc
elastic_dog has joined #osdev
<wxwisiasdf> i think i will just grab pdpclib and see what happens
andreas303 has joined #osdev
wxwisiasdf has quit [Quit: Lost terminal]
tarel2 has quit [Quit: Client closed]
[itchyjunk] has quit [Remote host closed the connection]
[itchyjunk] has joined #osdev
opal has quit [Ping timeout: 258 seconds]
opal has joined #osdev
<CompanionCube> gnulib isn't even a c library, thaf's glibc
[itchyjunk] has quit [Remote host closed the connection]
<clever> CompanionCube: making some progress on my zfs code after adding lz4 into the mix, i can now parse the root dnode, which i suspect contains an array of more dnode's, for every object in the pool
mavhq has joined #osdev
<CompanionCube> clever: iirc it's not that simple
<clever> so far, the main thing ive figured out, is that the root block pointer, is just pointing to a single raw dnode
<clever> and that dnode, i believe refers to object 0, the mos
<CompanionCube> yes, and the mos refers to other object sets
<clever> is the mos then an array of dnode's?
<CompanionCube> yes but also no
<clever> where the L0 pointers, point to a block full of dnode's?
<bslsk05> ​utcc.utoronto.ca: Chris's Wiki :: blog/solaris/ZFSBroadDiskStructure
<clever> ive mostly been going off of http://www.giis.co.in/Zfs_ondiskformat.pdf, and its not clear on several points
<clever> like the fact that the root ptr is pointing to a dnode
<clever> reading your latest link...
<clever> > and things aren't always immediately clear from it.
<clever> yep
<clever> CompanionCube: and each dnode is 1024 bytes long i believe?
<clever> oh right, i saw a prop for dnode size...
<CompanionCube> yep, there's a prop for it
<clever> yeah, `man zfsprops`
<CompanionCube> part of the large_dnode feature
<clever> yep, the man page says so
<clever> i assume thats so you can have a larger bonus value on the dnode?
<CompanionCube> i think that was the motivation yes, for things like selinux
<clever> but that then also means, you need to know how large a dnode is on this dataset, and its matching object set
<clever> to translate an object-id into an offset within the L0 blocks, and then with the shifts, an index thru the indirection tree
<bslsk05> ​github.com: lk/io.c at master · littlekernel/lk · GitHub
<clever> CompanionCube: LK's ext2 driver has a similar function, that takes a blocknr, and then computes the index for every indirection level
<clever> slightly complicated, by ext2 storing a few L0, a few L1, and a few L2 in the inode
<clever> so reading block0 can skip the indirection tree always
orccoin has joined #osdev
<clever> but then ext4 went extent based, so you cant predict your path thru the tree at all
<CompanionCube> mhm
<clever> ext2 storing the first few L0's in the inode, is one area where i can see it being slightly better then zfs, in terms of reading the start of a file faster
<clever> at the cost of increased complexity
<clever> with zfs, the number of IO ops needed to read the 1st block of a file, depends on how big the file is
<clever> just as a random test, i created a 1PB file (and shoved a new pool into it for dummy data), then ran zdb on that 1PB file, and i can see it has a single L4 pointer in the dnode
<clever> with a pair of L3's inside that
GeDaMo has joined #osdev
<clever> and a large number of L1's only have a single L0, so this sparse file is very overhead heavy
<clever> CompanionCube: ah, i see the critical line in what you linked, so the MOS will refer to another object set, with a blkptr to a single naked dnode, same as the root blkptr did
<clever> as it said, the object set itself, lacks an object#
<clever> so within the MOS, will be some kind of tree, description every dataset in the pool
<clever> and block-ptr's for the object set of each
<clever> and yep, i see that objectid 1, is an object directory, which has a zap on it, and some L0's
<clever> CompanionCube: but then how does zfs know where object 1 in the MOS is located, that root dnode must somehow have its own object set?
<CompanionCube> clever: uberblock points to it iirc@
<bslsk05> ​gist.github.com: gist:8c24fddc92b41c26542ce2521b05cfb3 · GitHub
<clever> the 1st file, is a snip from zdb, where its showing the MOS and object 0
<clever> the 2nd file is my own code, parsing the uberblocks
<clever> you can see the DVA's match up, so either the MOS is object 0, or the MOS is an object set, and object 0 is just the first element within that
<clever> i have loaded the node from 0:1600:200, but i have yet to parse its block ptr's
<CompanionCube> well, it is caloed the meta *object set* :p
<clever> yeah, so i'm thinking that the root dnode, is pointing directly to an object set
<clever> and like the other dataset objset's, it lacks an object#
<clever> and then there are some hard-coded object#'s within that set, that act as your starting point
<CompanionCube> you can zdb the MOS just like a regular dataset
<clever> zdb -ddddddddbbbbbbbb lk > big-old-dump
<clever> i ran this earlier, it recursively walks every single bit of metadata in the pool, and generates a text file of it all
<clever> CompanionCube: but i'm less clear on how to dump just the mos, and not recurse into every other object within it
<clever> oh, and the mos itself, also has its own indirection tree...
<clever> i dont see that within this dump
<clever> Dataset mos [META], ID 0, cr_txg 4, 106K, 58 objects, rootbp DVA[0]=<0:1600:200>
<clever> looking closer, 0:1600:200 is the location of the lz4's dnode on disk, 58 objects is what is contained within the L0's behind that dnode, and those L0's total to 106kb?
<clever> but its not showing that dnode and its own block ptr's
<CompanionCube> clever: i think zdb takes objset ids and the mos is id 0 so..;
<bslsk05> ​gist.github.com: gist:8c24fddc92b41c26542ce2521b05cfb3 · GitHub
<clever> CompanionCube: but this says object 0 is 208kb, and line 1 says the mos is 106kb?
<clever> but i'm fuzzy on how big a dnode is on this dataset, so i cant say how many objects should fit in 106kb and 208kb
<clever> let me parse the root dnode myself, and see what its DVA is....
<CompanionCube> hm, is 106kb logical or physical?
<clever> it doesnt specify
<clever> size=1000L/200P
<clever> oh wait, there
<clever> thats claiming 4kb, which is just leaving to more confusion
<clever> i think that 1000L size, is the root dnode itself
<clever> and the 106kb, is then within that dnode, which points to more blocks
<clever> CompanionCube: ah, i think i mis-understood dn_nblkptr, there only seems to be one block ptr in this dnode, but that has a value of 3
<clever> or perhaps, it can be 3, but 2 of the slots are allowed to be empty?
<bslsk05> ​gist.github.com: gist:8c24fddc92b41c26542ce2521b05cfb3 · GitHub
<clever> yeah, its looking like the root dnode, IS object 0
<clever> the ptr within that root dnode, is the L1 ptr from object 0
<clever> and with a max block# of 12, this dnode only has blocks 0-12, or 13 total, with a data blocksize of 16kb, the total size is 13*16, or 208kb, the lsize
<clever> oh, its right there in zdb, dnsize=512
<clever> so with 208kb worth of data, and 512 byte dnode's, then the object# can range from 0 to 415
<clever> which fits with 385 being the highest # i see
<clever> and if it was 1 dblock shorter, it would only hold 0-383
<clever> so now this all makes sense
<clever> i just dont see how zdb got a dnode size of 512
<CompanionCube> clever: ah, 512 bytes is the 'original' size for dnodes
<clever> the man page wasnt clear on what legacy meant
<clever> org.zfsonlinux:large_dnode = 0 shows up later in the zdb dump
<clever> object 51 is a zap containing that prop, and object1 has a features_for_read = 51
<CompanionCube> yeah the MOS contains its own copy of the pool config
<clever> object 63 is another zap with org.zfsonlinux:large_dnode = 4, and feature_enabled_txg = 63
<clever> yeah, i did also find a complete vdev_tree a few layers past the MOS
<clever> which cleans up some of the mystery and chance, on finding all vdev's
<CompanionCube> this is used when importing, that's what the 'untrusted' vs 'trusted' is in zdb
<clever> the nvlist at the start of a vdev, doesnt fully describe the layout of the pool
<clever> so if i have a mirror(a1,a2)+mirror(b1,b2), the nvlist on a1/a2, only describe mirror(a1,a2), and the 2nd mirror is entirely absent
<CompanionCube> (iirc this works differently than originally, let me find the article...)
<clever> there are only 2 hints that you need to keep searching
<clever> 1: vdev_children tells you how many top-level vdev's to expect
<clever> 2: top_guid is either a missing object (in the mos) that joins all top-level's, or a single top-level
<bslsk05> ​www.delphix.com: Turbocharging ZFS Data Recovery | Delphix
<clever> i also couldnt find a single reference, on how an nvlist is actually encoded on-disk
<CompanionCube> maybe it's somewhere in illumos-gate?
<bslsk05> ​utcc.utoronto.ca: Chris's Wiki :: blog/solaris/ZFSUberblockGUIDSumNotes
<clever> after staring at the hexdump and zdb enough, i did figure out some basics
<clever> and was able to get it to parse
<bslsk05> ​gist.github.com: gist:8c24fddc92b41c26542ce2521b05cfb3 · GitHub
<clever> this is my debug output, from parsing a single vdev pool
<clever> the nvlists, are kinda printed twice, fist as a pointer+length, then again as key/value pairs
<clever> but unsupported value types, are just silently skipped
<CompanionCube> https://github.com/mharsch/libnvpair this turned up while googling
<bslsk05> ​mharsch/libnvpair - portable, userland-only subset of nvpair from illumos (3 forks/5 stargazers)
<clever> CompanionCube: i found several nvpair libraries on github, they often had the wrong magic# in the header
<clever> and one claimed to be a portable version of the freebsd nvpair, but it was just a bloody shell script that calls subversion, hosted in github, lol
<clever> thats not how version control works
<CompanionCube> yeah but that one has the illumos headers in it
<clever> i'll have to check that out next
<bslsk05> ​github.com: libnvpair/nvpair.h at master · mharsch/libnvpair · GitHub
<clever> taking a break for a bit
<clever> then i'll review all of those links
zaquest has quit [Remote host closed the connection]
zaquest has joined #osdev
frkzoid has quit [Ping timeout: 244 seconds]
\Test_User has quit [Quit: .]
bauen1 has quit [Ping timeout: 244 seconds]
CryptoDavid has joined #osdev
dude12312414 has joined #osdev
dude12312414 has quit [Client Quit]
biblio has joined #osdev
FreeFull has joined #osdev
gog has joined #osdev
biblio has quit [Quit: Leaving]
saltd has joined #osdev
CryptoDavid has quit [Quit: Connection closed for inactivity]
MiningMarsh has quit [Read error: Connection reset by peer]
MiningMarsh has joined #osdev
saltd has quit [Remote host closed the connection]
orccoin has quit [Ping timeout: 250 seconds]
saltd has joined #osdev
saltd has quit [Remote host closed the connection]
nyah has joined #osdev
saltd has joined #osdev
<dzwdz> how hard is it to implement ext2 from scratch?
<dzwdz> so far i've been considering porting libext2fs from e2fsprogs and using it to handle the on disk format but it doesn't seem to be very portable
<zid> Not especially I don't believe, depends if you wanna debug a host of corruption or not while you do it :p
<pitust> yeah its not very hard
<kazinsal> not terribly difficult -- you don't need to know a bunch of b-tree stuff or similar
<pitust> esp. for ro
<pitust> and ext2 has no journaling or anything like that
<pitust> ext4 is the real fun
<zid> if ext4 is so good, why is windows up to version 98
<pitust> i think i never did ext4 lol
<dzwdz> pitust: i'm mostly concerned about rw
<dzwdz> writing seems way more complicated than reading
<kazinsal> I however should not be an authoritative source on filesystem implementations as my most recent one is a fourth edition UFS
[itchyjunk] has joined #osdev
orccoin has joined #osdev
vdamewood has joined #osdev
Raito_Bezarius has quit [Ping timeout: 260 seconds]
freakazoid332 has joined #osdev
Raito_Bezarius has joined #osdev
freakazoid332 has quit [Ping timeout: 250 seconds]
freakazoid332 has joined #osdev
netbsduser has joined #osdev
xenos1984 has quit [Ping timeout: 248 seconds]
xenos1984 has joined #osdev
xenos1984 has quit [Ping timeout: 250 seconds]
xenos1984 has joined #osdev
heat has joined #osdev
<heat> dzwdz, getting rw is always way harder
<heat> BUT ext2 isn't particularly hard in that respect
<geist> the hard parts of ext4 are the btree bits they introduced for large directories
<geist> and then second to that is the journal, though i looked at it and it's fairly straightforward
<geist> (for write at least)
<geist> for read you dont really need to understand the btree much more than being able to follow its structure
<heat> I think you can ignore the journal
<heat> freebsd doesn't touch it afaik
<geist> sort of. if you try to mount it and it's partially replayed that's really bad idea to continue with writing. you'll trash the FS for sure
<heat> for read you don't really need to know what the htree really is
<geist> for reading it'd be easy enough to not replay the journal but just build up a list of blocks that haven't been replaced yet
<heat> you can follow the directory entries traditionally and it Just Works
* geist nods
<geist> yah hadn't looked *too* closely at it
<heat> it's backwards compatible
<geist> noice
<heat> you'll just lose performance when opening
<heat> i'm still "struggling" with ext4 extents writing
<heat> but I think I have things mostly figured out
<heat> it's literally a b+ tree of extents
<heat> so the same rules seem to apply
<geist> cool, yeah it should be simpler in the sense that extents are fairly natural way to think about things anyway, from an allocation point of view, but it's more difficult in the sense that it's less prescriptive precisely what to do in a given situation
<geist> especially the whole 'search the extent list for offset' stuff you gotta do all the time
<geist> oh? i thought it was just a linear list of extents?
<heat> no
<geist> oh. hrm. okay that makes things more complicated
<geist> problem with btrees is not that they're complicated to get right (they kinda are) but when you want to work against an existing implementation you have to do precisely what *they* do
<geist> so it's not as easy as busting out a btree implementation and then just working against existing things
<heat> it's a tree, internal nodes have "this covers from logical block N to N + L", you follow down that road, until you get the leaf nodes, which are the extents themselves
freakazoid332 has quit [Ping timeout: 264 seconds]
<geist> or at least, doing that is even harder, because now you have to build some sort of generic thing
<heat> yeah
<geist> ah but is the tree strictly a btree in that it self balances, or is it more of a direct/indirect/double indirect wthing?
<geist> i thought it was the second, as in the number of levels is determined directly by the size
<geist> ie, some sort of modified radix tree
<heat> no, not radix for sure
<heat> that would be for the ext2/3 block map
<geist> most extent based things i've seen are more of that, or like NTFS a linear list of blocks (spillover file records) with a header at the start that says what logical range this block is
<geist> sure, but my point is you can kinda extend the 2/3 block map into an extent tree
<geist> just replace <pointer to block> with <an extent>
<heat> that's not how they did things
<geist> okay
<heat> that whole u32 i_data[15]; field got union'd with an inline run of 5 extent structs
<heat> (I think it's 5? can't remember)
<geist> makes sense
<heat> then those are either leaf extents, or internal nodes and you need to go down the chain
<geist> ah and the extent itself says 'this is a pointer to more extents'?
<geist> does an extent structure in ext4 describe the logical range it covers?
<geist> or is that implied in the container its in (ie, tree block)?
<heat> not the extent itself, but a "header" structure
<heat> you can't have leaf extents in the middle of internal tree nodes
<geist> yah makes sense. means you still have to linearly scan through the block to find the extent you want, but thats not a big deal on modern machines
<heat> actually no
<heat> they're always sorted
<heat> so you use binary search for everything
<geist> sure, but once you get to a block of extents doesn't it say 'these extents cover logical range X through Y'?
<geist> and then say there's 20 extents, within that block
<heat> yes
<geist> you have to walk through 20 of them to find the exact one that covers your logical block?
<heat> sure. but since it's always sorted, you don't need to walk through the 20
<geist> how so?
<heat> binary search
<geist> i dont get the 'always sorted' part
<geist> how would you binary search those?
<geist> since they're variable sized, you can't just jump into the middle and know their offset into the overall block
<geist> you have to count them up from the start of the block
<bslsk05> ​github.com: Onyx/extents.cpp at pfalcato/ext4-fs · heatd/Onyx · GitHub
<geist> the Nth entry may be 10000 or 15 logical blocks in, based on what 0...N-1 sizes add up
<geist> that code helps me even less
<heat> fuck
<geist> anyway, whatever alas i gotta go
<geist> need to finish up this FAT driver and then maybe i'll tackle this for reals too
<geist> FS work is fun
jimbzy has quit [Quit: ZNC 1.7.5+deb4 - https://znc.in]
<heat> you basically do a "best-attempt" binary search, then at the end of the search you check if you found the correct thing (because this logic may end up giving you an extent that doesnt map the logical block you want)
<geist> ah. sounds easier to just linearly walk until you stop. cpus are even really good at that
<geist> anyway, will talk to ya later, really gotta go now
<heat> for a whole page?
<heat> aight
frkzoid has joined #osdev
jimbzy has joined #osdev
\Test_User has joined #osdev
\Test_User has quit [Quit: .]
\Test_User has joined #osdev
FatAlbert has joined #osdev
<FatAlbert> it's always a pleasure to rub shoulders with the pro's
<FatAlbert> im NOT into computers anymore ... i did some thinking ... and it's NOT me
<FatAlbert> i was lying to myself
<FatAlbert> now i slowly back to what i always liked to do
<FatAlbert> i feel good
<FatAlbert> i even ordered a new Thursday's and a new weightlifting shoes
<heat> why are you in #osdev then
<heat> sounds pretty computer to me
<Ermine> maybe they develop some musle which name can be abbreviated to OS
<Ermine> s/musle/muscle/
frkzoid has quit [Ping timeout: 268 seconds]
<FatAlbert> heat: even though i'm not fooling myself as a "computer guy" anymore
<FatAlbert> i still MIGHT be doing some computer stuff after all
<FatAlbert> plus, there are some interesting conversations going on here from time to time
<FatAlbert> and i learned alot of concepts, and stuff from it
<zid> known troll fyi
<FatAlbert> zid: how you doing ... still have hate in your heart i see
<FatAlbert> zid: im not always on-topic yeah ... but why you always de-legitimize me ?
<FatAlbert> nvm ..
frkzoid has joined #osdev
frkzoid has quit [Ping timeout: 250 seconds]
GeDaMo has quit [Quit: Physics -> Chemistry -> Biology -> Intelligence -> ???]
c2a1 has joined #osdev
<gog> hi
<c2a1> In a process's memory are libraries confined to a single area of physical memory and linked to multiple areas of virtual memory?
<gog> yeah basically
<gog> but it's more like they occuply a fixed number of physical pages and can be mapped into an arbitrary number of address spaces
<gog> the physical pages don't need to be contiguous really
<c2a1> What do you mean a fixed number of physical pages
<gog> size_of_library / page_size = number_of_pages
<c2a1> So the .code segment is located in a single physical address
<gog> roughly
<clever> lets say the .text for libc.so is 5 pages long, by random chance, those pages land into physical pages 6, 2, 10, 7, and 9
basil has quit [Quit: ZNC 1.8.2+deb2+b1 - https://znc.in]
<c2a1> But the data and bss sections have their own different physical memory locations
<gog> they don't know about physical memory, really
<clever> each time something needs to map libc's .text, it then punches 6,2,10,7,9 into the page tables for that process
<clever> and boom, its now showing up as 5 contiguous pages in the virtual space
basil has joined #osdev
<clever> and in general, every process can share those 5 pages
<clever> as long as they are mapped read-only
<gog> yes
<clever> but when low on ram, you may throw physical page 9 out, and blank that entry in every paging table
<clever> since you can always fetch it from disk again
<c2a1> Hmm ok
<c2a1> What process or program usually manages the different memory segments of a given program
<c2a1> The linker?
<clever> co-operation between the runtime dynamic linker and the kernel
<c2a1> Of course. The kernel being mmap and other syscalls presumably?
<clever> the runtime linker map just call mmap(), and ask for libc's .text to be mapped anywhere
<clever> the kernel then picks a free virtual addr, and fills in the paging tables
<clever> the runtime linker then has to setup all of the relocation patching (sometimes via a .plt)
<c2a1> What is the difference between mmap and malloc
<clever> malloc just picks a free area from the heap, which is already part of your mapped virtual memory
<clever> mmap maps a new file into the virtual space
<c2a1> Hmm.
<FatAlbert> :)
<c2a1> I need to read more about this. Thank you all for your help.
<FatAlbert> heat: this is why im here
<FatAlbert> clever why do processe's needs to "share" the pages and they won't punch from phyical memory like the first process
<FatAlbert> ?
<clever> FatAlbert: the sharing is just done to reduce the overall memory usage
<\Test_User> they don't need to, but if they're all the same and read-only there's no need to make a copy and use the extra ram
<clever> yep
<heat> c2a1, so typical VM systems have a concept called copy-on-write (you could've read this in your local mmap(2) manpage as MAP_PRIVATE)
<heat> your programs, shared libraries are all mapped copy-on-write to a file
<heat> you have a master copy, and then if you write to them, it copies it to a new page and maps it (and now it's writable)
<heat> this means your .text, .data all share physical pages until you don't
<heat> this is optimized so you share the most possible and do the least amount of work until you need to actually do it
<clever> ive also noticed that mmap kinda behaves like mini swap files, where the kernel can push data into the mapped file and release ram at any time
<heat> yes, just like the page cache
<heat> for MAP_SHARED that is
<heat> in no way shape or form can that happen with MAP_PRIVATE
<clever> what if its MAP_PRIVATE and you havent modified the page, can it just discard and re-read later?
<heat> MAP_SHARED mappings just map the underlying file's page cache, so they share pages
<heat> clever, yup
<clever> so private acts like shared, until you write
<clever> and then cow's
<heat> well, I don't know if that's the case for linux, but it probably is
<heat> yes, shared = private until you write
<heat> the elf loader and ld.so just map everything MAP_PRIVATE
<heat> it entirely relies on MAP_PRIVATE using COW
<clever> mmap(0x7fc99edab000, 16384, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7fc99edab000
<heat> in theory you can implement it by just allocating anonymous memory and read()ing, like Sortix does
<clever> ah yep, this is a random part of libc being mapped, when i start ls up
<heat> but it's bad and slow
<sortie> you're bad and slow
<sortie> :(
<clever> it also doesnt entirely match what i said earlier, ld.so used MAP_FIXED, so the runtime linker picked where it lands
<heat> :(
<heat> clever, oh yeah that's a fun problem
<sortie> Haha but yeah seriously I really did have a faster PROT_READ on my mind eaarlier so I can mmap binaries
<clever> heat: many years ago, i was digging into why wine was using a lot of ram for WoW, and then i noticed, it wasnt mmap'ing the main WoW.exe binary
<clever> heat: as best as i can tell, the pe32 headers caused the executable pages, to not be page-aligned within the file
<c2a1> What manages the heap, the C library?
<heat> you have to calculate the full size of the elf object, mmap(NULL, full_size), so you reserve a contiguous $full_size of space, then you MAP_FIXED chunks into it
<clever> so the page-cache then isnt aligned right for direct mapping
<c2a1> How is the heap different from a mmap'd file?
<sortie> c2a1, yeah libc
<clever> heat: wine then gives up, and just read()'s it into anon memory
<heat> c2a1, it's anonymous memory
<gog> the heap might be a mmap'd anonymous
<c2a1> Ok. I remembered seeing the heap in some kernel tutorial(probably JamesM) so I had to ask.
<gog> older systems might use sbrk()
<c2a1> Thanks.
<heat> in unix tradition sbrk()
<sortie> c2a1: libc's malloc uses mmap to allocate (big) memory region and then inside those it's able to make arbitrary smaller allocations
<c2a1> Hmm.
<clever> last i looked, linux uses a mix of sbrk and mmap
<sortie> c2a1: You could use mmap for small allocations too but it's super inefficient
<clever> sortie: yep, that
<heat> clever, depends on the malloc
<clever> yeah
<c2a1> Is the heap slower than mmaping your own memory or usually faster due to optimizations?
<heat> some really like sbrk, others can't use it (like scudo IIRC)
<heat> c2a1, faster
<clever> c2a1: using the heap is faster, because you can often use it without any syscalls
<sortie> c2a1, it really varies on your workflow
<c2a1> Or is this a stupid question
<sortie> *workload
<c2a1> Ahh ok.
<c2a1> Thanks.
<clever> c2a1: but the heap can only shrink from one end, and never punch holes, so fragmentation can lead to extra memory usage
<heat> free() doesn't return memory back to the OS for instance
<gog> some applications might reimplement malloc or use their own allocation scheme
<sortie> c2a1, the heap can avoid system calls to allocating more memory and can recycle previously freed memory
<heat> that is super slow
<gog> but generally malloc() is fine for most purposes
<sortie> c2a1, but if you have a really big allocation, it might be faster to mmap it yourself rather than malloc, but you should usually always use malloc unless there's any reason to use mmap in particular
<heat> gog, depends on the malloc()
<gog> yes
<gog> that's why i said generally
<gog> :)
<heat> musl's malloc is fine for most allocations except when threads are involved
<heat> glibc's malloc is fine except when heavily multithreaded applications are involved
<FatAlbert> it is the job of the dynamic linker to keep tabs of what shared libraries are already assgined to process ( in order to prevent redundant instances ) ?
<clever> ive also seen LD_PRELOAD based malloc's for replacing your active malloc
<clever> FatAlbert: yeah, you generally only want one instance of a given library within a given process
<heat> I've seen sizeable performance differences (+multiple-tens of percentage) by replacing glibc malloc with a tcmalloc or so
<clever> any more could lead ot nasty problems
<heat> also tcmalloc for some reason uses fucking spinlocks
<heat> which are like spinlocks but not really
<heat> but they still pessimize the shit out of workloads
<heat> (sometimes)
orccoin has quit [Ping timeout: 250 seconds]
<raggi> greedy optimizations always bench well in microbenchmarks :(
<c2a1> Could one theoretically make a multi threaded application without using any assembly at all
biblio has joined #osdev
<gog> on an existing system or ?
<c2a1> Existing system
<zid> Depends what you count as 'using any assembly'
<gog> yeah multithreaded applications are made all the time without writing assembly code
<c2a1> Without libraries
<gog> ehhh
<c2a1> Or any header files for example.
<zid> In C you need to rely on the compiler having builtin atomics and barriers etc
<zid> but for a brand new platform you will need to write that assembly *for* gcc to use
<gog> yeah multithreading depends a lot on library support
<zid> but if someone else already wrote it, you don't ever need to
<c2a1> I wonder what system calls you need to implement threading itself
<gog> clone()
<gog> for linux
<c2a1> Hmm I guess everything is in sched.h
<FatAlbert> gog: i thought you just #osdev cat :)
nisa has joined #osdev
stux has joined #osdev
<FatAlbert> you know shit ..
<FatAlbert> good for you
<FatAlbert> i remember the days when i actually slept at night ..
<FatAlbert> i miss these days
<c2a1> You aren't drinking enough caffeine
<FatAlbert> i frankenstein deep inside
<FatAlbert> it's probably what i've been through in life .. is coming back at me at the age of 30
<FatAlbert> im not far from the actor from "the mehanic" (no, not with jason statham)
<FatAlbert> i have more muscle definition than him, and i sleep at least once in three days
<FatAlbert> but other than that .. yeah .. it sucks
<FatAlbert> but you know what ?
<FatAlbert> I know i'll survive what is to come
<FatAlbert> i have the best surviving tool that i got from my ancestors
<FatAlbert> im sorry not "The Mechanic"
<FatAlbert> it's called "The Machinist" or something like that
<FatAlbert> c2a1: do you know how i know im going to survive WW3 ?
<FatAlbert> even though it will happen in my country ?
zid has quit [Read error: Connection reset by peer]
zid has joined #osdev
bauen1 has joined #osdev
bauen1 has quit [Ping timeout: 265 seconds]
<heat> raggi, have you tried out scudo in big, server-level super multithreaded environments?
<heat> i'm curious about its performance
<zid> heat can you nmap me
<heat> its clearly suited for desktop level tasks (at least Google trusts it is), but no idea about server stuff
<heat> zid, idk why
<zid> I wanna see what my filtered ports are
<heat> I guess
<heat> is it the ip on your irc whois?
<zid> yus
<heat> seems to be blocking everything?
<zid> yea filters should be rare
<zid> maybe only netbios
<raggi> heat: I have not, although the way you're comparing server and desktop there is not as common now they everyone containerizes everything
<mrvn> There is filtered and just not responding to it because the ISP blocks the control messages
vdamewood has quit [Quit: Life beckons]
<bslsk05> ​www.prnewswire.com: OpenText to Acquire Micro Focus International plc
Andrew has joined #osdev
<FatAlbert> im going to sleep !
<FatAlbert> im going to try
FatAlbert has quit [Quit: if you want peace prepare for war]
bauen1 has joined #osdev
<gog> bye
<heat> raggi, maybe. although I'm talking about big services which do lots of allocations and are /heavily/ multithreaded
<heat> I certainly saw a lot of services at cloudflare with hundreds if not thousands of threads at the same time, same process
<raggi> Yes, I understand, it's just more rare, and often people care more about performance than additional defense in depth there
<raggi> For example the thing probably most common for people in the container universe to look that way is storage like MySQL
<raggi> I've not seen anyone using scudo in those environments
<heat> yeah
<heat> but scudo is allegedly pretty damn fast
<heat> at least with crc32c instructions
<heat> i'm curious as to how fast it is
<heat> no one really seems to publish benchmarks on it
selene_ has joined #osdev
<bslsk05> ​rebeccaweekly.com: Best Hardware Engineering Papers – Rebecca Weekly
selene_ has quit [Client Quit]
netbsduser has quit [Remote host closed the connection]
linkdd has joined #osdev
biblio has quit [Quit: Leaving]
carbonfiber has joined #osdev