Vercas has quit [Write error: Connection reset by peer]
wand has quit [Write error: Connection reset by peer]
gildasio has quit [Write error: Connection reset by peer]
gxt_ has quit [Write error: Connection reset by peer]
Vercas has joined #osdev
gildasio has joined #osdev
masoudd_ has joined #osdev
gxt_ has joined #osdev
wand has joined #osdev
masoudd has quit [Ping timeout: 272 seconds]
archpc2 has joined #osdev
archpc has quit [Ping timeout: 256 seconds]
archpc2 is now known as archpc
masoudd_ has quit [Ping timeout: 250 seconds]
<kingoffrance>
"de-x86 my life" "that's what i mean, cloud providers are doing it, but its generally in house" "not enough to drive the price down." meet the new non-x86, same as the old non-x86 :)
X-Scale` has joined #osdev
X-Scale has quit [Ping timeout: 240 seconds]
X-Scale` is now known as X-Scale
<kingoffrance>
the secret is you must wait for a bubble to burst, and then hit ebay/surplus/etc. https://www.computerworld.com/article/2594832/compaq-boosts-alpha-servers.html A DS20E server based on the new processor, with 9GB of disk space and 256MB of memory, will start at $12,800 <wait x years> $40 ebay/surplus. its all about timing!
* kingoffrance
waves hands wildly " It's just the memory representation of that tree is an associative array." https://cr.yp.to/critbit.html A crit-bit tree can be used as a high-speed associative array and ordered operations such as finding the minimum. , and exact searches, and general suffix searches
<bslsk05>
cr.yp.to: Crit-bit trees
<mrvn>
kingoffrance: not in O(1)
<kingoffrance>
im biased, because i want to find a use for allowing "mixed" format "C strings" containing multiple "character sets" (i.e. no embedded \0). problem in search of a solution :)
<mrvn>
crit-bit sounds like a radix tree except it stores all strings verbatim.
<kingoffrance>
"a prefix-free set of bit strings" the key phrase is prefix-free, e.g. \0 only at end, adam langley's version IIRC it is "as if" each datum has infinite trailing \0
troseman has joined #osdev
<kingoffrance>
so i believe it works fine for "C strings" not necessarily other things
<mrvn>
prefix-free means no string is a prefix of another.
<mrvn>
"{a, b} is prefix-free, but {a, b, ac} is not."
<kingoffrance>
yes, so if \0 is only at end it holds
<mrvn>
Or in other words. Anything with variable length must be length prefixed or have a special end token.
<mrvn>
so basically a NOP since anything else would be unusable.
<mrvn>
(or fixed length, that works too)
<kingoffrance>
*solution in search of a problem
<kingoffrance>
i didnt say anything, but a few days ago the person talking about UUIDs and integers..."integers" can have embedded 0x00
<mrvn>
So if I understand this right you have a struct { Node external; Node internal; char [] data; } layout in memory. And then you build a tree over it.
<kingoffrance>
i need to code and read more, just was saying might be useful. dont ask me, im not there yet...
<kingoffrance>
adam langley's version wants a malloc and a spare bit (e.g. malloc always 2-byte aligned) ...
<kingoffrance>
*spare bit in pointers
<kingoffrance>
that is not needed, might just be an optimization to save a byte
<mrvn>
And the node encodes which bit to look at to decide if you need to go left or right.
<mrvn>
You need to know if the left/right pointer point to nodes or strings. That's 2 bits. And a length. Might be easier to put the 2 bits in the length.
<mrvn>
One thing about this that could be nice is that find will return a pointer to the string in the tress.
<mrvn>
tree
<mrvn>
Other prefix trees will encode the strings as the path and you have to clobber it back together.
<mrvn>
"The simplest general approach is to incorporate these two internal/external selector bits into the length, to have the left string stretching out to the left of the length in memory, and to have the right string stretching out to the right of the length in memory;" this is odd.
<mrvn>
So if I have a leaf/leaf node then it's "hello, world\0<length>hello, world!\0" in memory?
<mrvn>
How does that work? How do you get the left string?
<kingoffrance>
there's code on github, i cant imagine that is the case though
<mrvn>
"Of course, pointing to this node means pointing to the length field in the middle." sure sounds like it
<bslsk05>
agl/critbit - Critbit trees in C (32 forks/297 stargazers)
<bslsk05>
github.com: git/cbtree.c at master · git/git · GitHub
<kingoffrance>
or how it differs from above, etc.
<kingoffrance>
adam has a fractal avatar lol
<mrvn>
The git implementation stores the string as node itself. No left/right, just one string.
<mrvn>
And it uses the lowest bit of the node pointer to say it's a string. So not the odd approach from the paper.
<mrvn>
Hmm, maybe you can just go to the left until you hit the \0 byte for the right string of the next node.
<mrvn>
kingoffrance: Any idea what the external nodes are for and what they contain?
X-Scale` has joined #osdev
X-Scale has quit [Ping timeout: 272 seconds]
X-Scale` is now known as X-Scale
<mrvn>
kingoffrance: Can't find any crit-bit tree implementation that does that strange "stretching out to the left of the length in memory" bit. Or even encoding the 2 extra bits in the length.
<kingoffrance>
what are you reading?
<kingoffrance>
i only glanced adam's pdf
gog has quit [Ping timeout: 272 seconds]
<kingoffrance>
oh i see that is on the djb page. adam's pdf touches on that
<kingoffrance>
the code is tiny, you can compile in 2s and see what it actually does
<kingoffrance>
the "git" version maintains the 2-byte alignment "extra bit in pointer" but "no internal allocations" (does not use malloc )
<kingoffrance>
i dont see a good full-size picture though, the pdf examples are trivial arbitrary for example purposes, and not necessarily actual either, just to demonstrate
<geist>
hmm, will need to grok this thing. whats the intention of storing strings in a tree like way like this?
<kingoffrance>
you guys take me way too seriously lol.
<kingoffrance>
i dont know, just been itching to use it. dont wait up, busy with other things
<geist>
well, it's at least interesting if nothing else
<kingoffrance>
it reminds me of x resources maybe... a.b.c.d
<kingoffrance>
i might be misusing terminology, for OO stuff seems a good fit, hierarchies
<geist>
the length and left walk is kinda interesting
<kingoffrance>
i mean, it would seem to lend it well to that type of hierarchal "lookup"
<kingoffrance>
*lend itself
<kingoffrance>
i dont know, see why does git use it?
<zid>
oh a crit bit tree is just a radix tree
<zid>
I had heard of crit bit tree but never looked it up
<kingoffrance>
it took me a minute to register github git git ...oh thats actually git code itself lol
<kingoffrance>
if we are talking ideas, then a.b.c.d.<hash><size> maybe ... then i believe you would have ability to look up such a datum named with hierarchies....while hash size let you retrieve actual data from somewhere else? so effectively makes it all just pointers add in <version> in there somewhere. again, for c strings, \0 only allowed at end. but fact it can retrieve anything "after" is why i say it seems good for such hierarchal things
nyah has quit [Ping timeout: 250 seconds]
[_] has joined #osdev
[itchyjunk] has quit [Ping timeout: 256 seconds]
smeso has quit [Quit: smeso]
[_] is now known as [itchyjunk]
smeso has joined #osdev
dormito has quit [Ping timeout: 260 seconds]
dormito has joined #osdev
srjek has quit [Ping timeout: 240 seconds]
sonny has joined #osdev
<geist>
haha someone on the local craigslist is looking for 'old vintage computers' and 'anything after 2005'
<klange>
what's the benchmark for vintage, what's their gap of "too new to be vintage, too old to care"?
<geist>
"anything after 2005"
<geist>
that's what i mean, it goes on to say it'll take vintage computers, but then specifies it more concretely as 2005+
<bslsk05>
seattle.craigslist.org: Looking for old vintage computers - computers - by owner -...
<klange>
That is 17 years ago. That's darn-near-adult age.
<geist>
i mean that's fine, was just chuckling that 2005+ is considered vintage
<klange>
For desktops and even laptops, things have been feeling pretty long-tail for a while, so it is interesting think - like I've got this 12-year-old ThinkPad and it's just fine, actually?
<klange>
But when I got it, a then-12-years-old laptop would have been a dinosaur.
<geist>
yah as i think i've mentioned before, i generally consider all x86 PCs to be basically the same in the last 15 years or so
<geist>
and definitely so in 2010+
<geist>
and really to a large extent most consumer electronics. i haven't seen anything really *new* in the last 10-12 years or so
<geist>
just improvements on the same thing
<geist>
no real paradigm changers
<geist>
so maybe apple M1 is at least a major change somewhere, but it's still just a laptop
<geist>
felt like 2000 - 2010 was where a lot of stuff really moved forward. invention of smartphones, finally got smart watches, real actual VR, etc
<geist>
rise and fall of portable MP3 players
<klange>
2000s were also when the x86-64 switchover was happening
<geist>
exactly. since about 2005 or so PCs are basically the same thing. PCIe, 64bit, multicore, etc
<klange>
logorithmic grown :)
<geist>
i guess the rise of SSDs to replace spinny stuff was an early 2010 era thing
<klange>
or logarithmic, i'm too tired to spell right, been up since far too early today
<geist>
though it was already starting to exist in the late 2000s i think. i believe i got my first SATA SSD in about 2009?
<klange>
Swapping those 2010-era laptops over to SSDs is the rejuvination they need to still be solid workhorses.
<geist>
yah amazing how much of a difference it makes
<geist>
it was a huge leap forward at the time. my first SSD (still have it here somewhere) was expensve and totally worth it
<CompanionCube>
uefi is also 2010s no?
<geist>
was starting to catch on in 2009 or so, but yeah
<klange>
yeah, said thinkpad is the last of the BIOS line
<geist>
so even if you consider 2010 to be the line in the sand, there's really nothing new since
<klange>
though lenovo delayed UEFI for the chunky thinkpads
<geist>
the M1 laptops are the only real fundamental change in PC/laptop space
<sonny>
enclaves?
<sonny>
intel sgx
<klange>
I'm also just not really all that convinced UEFI was such a gamechanger in the end.
<geist>
nah. it's just a detail honestly. only real game changer is if you consider secure boot to be a thing
<geist>
and i'm sure some folks do think that's the bees knees
<CompanionCube>
isn't sgx pretty busted?
<Mutabah>
It did allow nicer firmware interfaces
<geist>
yah SGX is as far as i know DOA
<Mutabah>
(As in - UI)
<sonny>
no idea, sounds like a nice idea
<sonny>
oh rip
<geist>
i guess for cpu design the chiplet stuff that AMD is doing (and now apple) is kinda a game changer
<CompanionCube>
can't be vector stuff, especially given amd64 required sse2...
<moon-child>
intel is doing it too iirc :P
<moon-child>
(I think minor differences between all three ... IANAHG)
<geist>
yeah i mean it's not necessarily that AMD invented it. i'm sure it was a known thing and going to happen one way or another
<geist>
someone always gets to it first, but since development goes way back i'm sure everyone was already workign on it in parallel
<CompanionCube>
is there something to point at in gpu terms:
* geist
points at it
troseman has quit [Read error: Connection reset by peer]
xenos1984 has quit [Remote host closed the connection]
xenos1984 has joined #osdev
xenos1984 has quit [Remote host closed the connection]
xenos1984 has joined #osdev
xenos1984 has quit [Remote host closed the connection]
xenos1984 has joined #osdev
the_lanetly_052_ has joined #osdev
the_lanetly_052_ has quit [Max SendQ exceeded]
the_lanetly_052_ has joined #osdev
the_lanetly_052_ has quit [Max SendQ exceeded]
the_lanetly_052 has joined #osdev
hbag has quit [Remote host closed the connection]
the_lanetly_052 has quit [Max SendQ exceeded]
the_lanetly_052 has joined #osdev
the_lanetly_052 has quit [Max SendQ exceeded]
chordtoll has joined #osdev
xenos1984 has quit [Remote host closed the connection]
xenos1984 has joined #osdev
Payam98 has joined #osdev
[itchyjunk] has quit [Read error: Connection reset by peer]
masoudd_ has joined #osdev
sonny has quit [Remote host closed the connection]
sonny has joined #osdev
Electron has joined #osdev
sonny has quit [Remote host closed the connection]
ElectronApps has quit [Ping timeout: 256 seconds]
masoudd_ has quit [Quit: Leaving]
ElectronApps has joined #osdev
Electron has quit [Ping timeout: 260 seconds]
MrBonkers has quit [Remote host closed the connection]
GeDaMo has joined #osdev
bauen1 has quit [Ping timeout: 245 seconds]
JanC has quit [Remote host closed the connection]
JanC has joined #osdev
masoudd has joined #osdev
Matt|home has quit [Quit: Leaving]
Matt|home has joined #osdev
vdamewood has quit [Read error: Connection reset by peer]
vdamewood has joined #osdev
dormito has quit [Quit: WeeChat 3.3]
bauen1 has joined #osdev
gog` has joined #osdev
<zid>
15 years is perfect because that's 2007, when I got my core2
<zid>
and absolutely nothing has changed since except how pci-e is wired
<zid>
The P35 chipset provides updated support for the new Core 2 Duo E6550, E6750, E6800, and E6850. Processors with a number ending in "50" have a 1333 MT/s FSB. Support for all NetBurst based processors is dropped with this chipset.
nanovad has quit [Ping timeout: 240 seconds]
nanovad has joined #osdev
<Griwes>
yesssss finally got blocking mailbox reads working, and my userspace early init process can now correctly print text to the kernel boot log
<Griwes>
I now have a funny bit of the kernel going through the userspace syscall interface to get the blocking effect for a thread that runs in kernel space, but that's fine
dormito has joined #osdev
<Griwes>
...hmmmm, that's all the "client" code written so far working. now I need to figure out what the next piece of "client" code to implement support for is...
the_lanetly_052 has joined #osdev
zaquest has quit [Remote host closed the connection]
Burgundy has joined #osdev
zaquest has joined #osdev
Jari-- has quit [Remote host closed the connection]
dennis95 has joined #osdev
vin has quit [Remote host closed the connection]
masoudd has quit [Remote host closed the connection]
masoudd has joined #osdev
mjg has quit [Remote host closed the connection]
mjg has joined #osdev
FatAlbert has joined #osdev
<mrvn>
I think the last decade the biggest advancement was in so called "mikrocontrollers" like the RPi becoming desktop capable while remaining cheap.
<mrvn>
basically the rise of ARM
<mrvn>
And multicore is now the standard with average core count >4
<mrvn>
Just had a call from someone speaking broken german telling me my windows PC had alerted them and was being hacked.
<mrvn>
*looks left at the RPi4* *looks right at the RPi3* *looks under the desk at the Linux server* *looks straight at the ARM laptop*. Nope.
<GeDaMo>
I didn't know Bill Gates spoke German! :P
<mrvn>
Slowly I'm getting enough ARMs for a centipede.
<GeDaMo>
Shouldn't those be legs? :|
gog` is now known as gog
<FatAlbert>
what is simpler x86_64 or ARM ?
FatAlbert is now known as attaboy
<attaboy>
i just know that x86_64 is more common
<GeDaMo>
That might be debatable, there's a lot of phones about
<attaboy>
but in laptops/pcs it's intel right ?
<GeDaMo>
Mostly, yes
<mrvn>
All PCs are x86 by definition
<j`ey>
chromebooks are often arm, apple is putting arm in most of their computers now
<attaboy>
mrvn: right i forgot that amd is also x86_64
<mrvn>
PC compatible could be amd too
<zid>
bit of a useless question
<zid>
"what's simpler, a ford or a vw?"
<mrvn>
zid: automatic transmission
<attaboy>
zid: you comment is not any more productive though
<attaboy>
arguably even less
<mrvn>
attaboy: x86 is horribly complex to boot up
<zid>
ah just a troll, easily fixed
<mrvn>
attaboy: and getting debug output from a PC is horrible too. Do yours have a serial port so you can capture debug output?
<attaboy>
mrvn: you got me on deep water :o
<attaboy>
but i hope it does
<attaboy>
lol
<mrvn>
then I guess you aren't ready for OS dev.
<attaboy>
you might be right, but ..
<attaboy>
i have one thing, most of the newbies doesn't
<j`ey>
attaboy: well I assume you'll do osdev on an emulator anyway
<bslsk05>
'Is a leap of faith (Spider man - Into the spider verse)' by tota clips (00:04:00)
<attaboy>
but i learned two things today (1 serial port is usefull for debugging purposes (2 x86_64 is harder to boot (3 all PC's are x86 by definition
<mrvn>
I'm pretty sure you don't have a PC.
<ThinkT510>
you don't seem to understand counting either
<mrvn>
kingoffrance: I searched crit-bit a bit more and I think I understand it now. First the "has an external node for each string in S" that then is never mentioned again is just the string. So put your strings somewhere in memory, that is the external node.
<mrvn>
Internal nodes then have: Left: A pointer to (the left child node, if that node is internal; otherwise the string at the left child node.) aka, the external node.
<mrvn>
That crazy "have the left string stretching out to the left of the length in memory" seems to just mean left is a pointer to the external node.
<mrvn>
But then the "Of course, pointing to this node means pointing to the length field in the middle." makes no sense. None of the implementations do it and with external nodes being the strings it makes no sense.
[itchyjunk] has joined #osdev
ravan has quit [Ping timeout: 256 seconds]
gog has quit [Read error: Connection reset by peer]
gog has joined #osdev
ElectronApps has quit [Remote host closed the connection]
<attaboy>
gog: can't hear ya
heat has joined #osdev
attaboy has quit [Quit: WeeChat 3.4]
attaboy has joined #osdev
<graphitemaster>
on x86_64, all jump instructions are coded relative right
<mrvn>
no
<graphitemaster>
when they have an immediate I mean
<graphitemaster>
like jne, je, jb, jae, etc
<mrvn>
I think you can jump absolute too but it's rarely needed/used by the compiler.
<mrvn>
maybe just large memory model
<GeDaMo>
absolute jumps are usually through registers or memory
<mrvn>
GeDaMo: does it have to be memory is the question
<heat>
I'm 99% sure all jumps and calls are relative except those
<graphitemaster>
I have some stand-alone function I compiled position independent, it should not require any fixups or relocations from what I can tell (it's a pure function, only has inputs), when I objcopy just the .text section, and try to call it, it ... runs / works fine, but later on my application state is hosed, like it broke a register or something
<mrvn>
far jumps are absolute.
<heat>
graphitemaster, disassembly?
<graphitemaster>
when I step through it in gdb it is the same disassembly as objdump -d's output of the .o
<graphitemaster>
I mean it's compiled from regular C
<graphitemaster>
that repmov is generated with __asm__ __volatile__("rep movsb\n" : "+D"(d), "+S"(s), "+c"(n) : : "memory");
<mrvn>
FPU enabled?
<graphitemaster>
yes
<heat>
I would step through it but it looks fine
<heat>
maybe the memcpy itself is broken
<mrvn>
Nothing fishy in there that would break the PICness.
<graphitemaster>
The same C code when built as .o and linked in works fine
<graphitemaster>
I'm just trying to monkey patch a binary blob of it
<graphitemaster>
That's when it breaks
<graphitemaster>
Is there some alignment rules about the code itself
<mrvn>
so what is the diff between the two compiles?
<graphitemaster>
wdym
<mrvn>
Is any other code jumping into the middle of the original function?
<graphitemaster>
Nope, nothing jumps into the middle of it, it's called like a regular function
<mrvn>
"compiled position independent" vs. "built as .o and linked in"
<heat>
graphitemaster, depends, you should look at the section headers to find that out
<GeDaMo>
I think there are suggestions about jump targets being 4 byte aligned but that's just for performance, it shouldn't affect correctness
<mrvn>
You might not jump into the middle of it but the compiler might.
<mrvn>
Some use of memcpy() might deduce the first few branches and jump right into 7c.
<graphitemaster>
It's called as ((void (*)(void*, void*, size_t))&data[0])(dst, src, len) and data is a byte array put into the .text section
<graphitemaster>
I cannot see the compiler optimizing across that
<graphitemaster>
The linker maybe
<mrvn>
With LTO it might
<graphitemaster>
Anyways stepping through it in gdb, it's called correctly
<mrvn>
Then call the stand along code with the same arguments
<mrvn>
aeh, your data is in the .text segment? That's read-only.
<graphitemaster>
the .text segment is executable
<mrvn>
How are you binary patching?
<graphitemaster>
I'm not writing into the .text segment
<graphitemaster>
I'm placing this binary data in the .text segment of the binary during compilation
<mrvn>
"I'm just trying to monkey patch a binary blob of it" When are you doing that? At runtime?
<mrvn>
ok, never mind then,
<mrvn>
well, only idea left is to single step it and see where it breaks.
<heat>
does it have any relocations?
<mrvn>
heat: I don't see any in the objdump. Only relative jumps.
<graphitemaster>
Okay something else is broken
Burgundy has left #osdev [#osdev]
bauen1 has quit [Ping timeout: 256 seconds]
Bonstra has quit [Quit: Pouf c'est tout !]
<graphitemaster>
The stack is corrupted
Payam98 has quit [Quit: Client closed]
mahmutov has joined #osdev
<mrvn>
huh? I see no SP in the objdump
<mrvn>
is dst/src on the stack and you buffer overflow?
<graphitemaster>
The calling code is compiled with stack protector and it expected that function to have something
<graphitemaster>
This looks like a gcc bug
<mrvn>
can't link in objects with incompatible compiler flags. *shrug*
<heat>
graphitemaster, hm?
<heat>
what do you mean with "have something"?
toulene has joined #osdev
cyberbanjo has joined #osdev
cyberbanjo has quit [Remote host closed the connection]
cyberbanjo has joined #osdev
Bonstra has joined #osdev
k8yun has joined #osdev
sonny has joined #osdev
sonny has quit [Ping timeout: 256 seconds]
* geist
yawns
gildasio has quit [Ping timeout: 240 seconds]
sonny has joined #osdev
gildasio has joined #osdev
gareppa has joined #osdev
gareppa has quit [Remote host closed the connection]
* heat
yawns back
* mjg
burps
* mrvn
etas chineese food
<mrvn>
Who here maps their ring buffers twice so objects at the end of the buffer continue at the front?
<geist>
not I but it's not a bad idea
<geist>
but i usually dont do it because it relies on an mmu existing etc, and lots of times I have ring buffer libraries that dont make that assumptions (for embedded, etc)
<geist>
question is whether or not the splitting of that last memcpy when wrapping is expensive enough to bother
<mrvn>
If you memcpy both read and write. If you want to emplace objects it becomes hard.
<mrvn>
In the past I had the ring buffer skip ahead to the warp around leaving a hole.
<geist>
yah thats a good point (emplacing objects)
<geist>
usually if it's a fixed sized object i just arrange for the ring to wrap without it
<mrvn>
if (object_size > size - pos) pos = 0; kind of thing
<geist>
er i mean arrange for the ring to be the right size
<geist>
right, or what you said
<mrvn>
for fixed sized objects making the size a multiple of the obejct is a must I think.
<geist>
but that is a good point, by doing the double map page you can get away with having the ring say still be a power of 2 (nice for math purposes) even if the fixed sized objects are not an even divisor
<geist>
well no if you have the wrapping thing you can just sort of have it work out however it wants
<mrvn>
yep. Or if they differ in size. Or you don't even know the size yet, just a maximum. The object might fit or not.
<geist>
maybe there's some spare bytes in there, but as long as you check that the remaining bytes >= sizeof(object) it'll work
<geist>
(and of course dealing with alignment)
<geist>
you're selling me more on this idea the more i think about it
<mrvn>
Don't have a use case for it yet in my kernel but it is an interesting idea for sure.
<geist>
yah
<mrvn>
the code to check front&back might get a bit more interesting.
<geist>
so for zircon, for example, we have a debuglog which is a series of records: timestamp, thread, <variable sized string>
<geist>
we currenlty use a bytewise circular buffer for it with all of the usual wrapping logic
<mrvn>
Ahh, perfect. yeah, that is something I want and can use this.
<geist>
we could easily 'unroll' that to make the wrap logic simpler
<mrvn>
and debug output needs to be fast or it becomes a bother.
<geist>
yah plus it's written to from deep contexts
<geist>
ie, inside interrupt handlers, etc. it's basically where printfs go
<mrvn>
That's where I need it. Can't wait for the UART in the interrupt handler.
<heat>
kinda related, do you output your tracing stuff directly in chrome-compatible json?
<geist>
for zircon or just in general?
<heat>
or is it a binary format that gets translated to json in the end
<heat>
well, zircon
<geist>
in zircon it's the latter. binary format (kernel aint got time for json)
<geist>
and there is a servie in user space that can merge kernel records with various records in user space
<geist>
since they're all time stamped it can reassemble it
<geist>
there's a fairly bespoke mechanism for user space to request kernel records, but the gist is it ends up being a large memory mapped circular buffer that user space can directly tail
<heat>
hmm good point, it might be too slow to dump information in json directly in the tracing points
<mrvn>
geist: You could map the buffer till you hit 64k and then use an atomic<short> as index into the buffer. (for 2^n sized objects)
<geist>
right, also kernel trace records are synthesized in any context in the kernel, inside the scheduler, interrupt handlers, inside spinlocks, etc
<geist>
so it has to be *fast*
<heat>
yup
<geist>
mrvn: possibly though the general optimization guide on arm i think tells you to usually stay away from < 32bit stuff
<heat>
do you do any sort of ftrace/mcount?
<geist>
i dunno, what is ftrace/mcount?
<j`ey>
ftrace allows you to trace arbitrary functions inside the kernel, dynamically
<heat>
compiler inserts calls to fentry (not ftrace, my bad)
<heat>
or mcount
<geist>
ah no.
<mrvn>
might be better than a lock-free read, modulo-add, try-write loop.
<geist>
we have the ktrace stuff for that, but we do it manually
<geist>
ie, ktrace(....)
<heat>
yup
<geist>
inserts a record, and there's a mechanism to quickly mask/unmask particular types of records
<geist>
but at the end of the day it's still an if (event & mask) ...
<mrvn>
For traces I still wish one could tell gcc to instrument all function to call something on entry and exit.
<geist>
yah usde to be a thing way back in the day but i think it was removed in gcc 3.3 or so
<heat>
there is
<geist>
i remember we used it a bit at Danger
<mrvn>
Idealy for c++ a template function.
<heat>
i feel like you could do something akin to jump tables for your if (event & mask) solution
<geist>
heat: probably but then that'd be a spectre issue
<heat>
s/table/labels/
<geist>
and would have to retpoline it
<heat>
i'm mixing my words today
<heat>
like, do a __asm__ __volatile__("jmp target") and nop it out when you don't need it
masoudd has quit [Quit: Leaving]
sonny has quit [Ping timeout: 256 seconds]
* geist
nods
<geist>
in general runtime patching the kernel is at best a huge PITA
<geist>
because we have the kernel mapped XN and whatnot, but of course can always flip the bit and whatnot
<geist>
hoenstly i dunno how linux does it (if they do) with a live SMP system
<mrvn>
can one teach gcc about that part with the arch specific functions?
<heat>
i think they use int3 and fun interrupt handler magic
<geist>
perhaps they reason that if you're fiddling with instruction patching as long as you do it exactly one instruction then it's 'safe' for other cpus to be running at the same time
<geist>
and maybe they do or dont see the XN bit change, etc
<mrvn>
doesn't linux does that SMP/noSMP patching at boot?
<geist>
patching at boot is a known issue, if you patch befoe you boot other cpus that's no big deal. we do some of that in zircon too
<heat>
linux does a lot of patching
<geist>
but runtime patching once everyting is up and going seems a bit more hairy
<heat>
all its ftrace is based around patching
* geist
nods
<heat>
like, you can literally ask linux at any time to trace almost any function, and it does that, even fully booted
xenos1984 has quit [Read error: Connection reset by peer]
<geist>
i guess it's jut a matter of going through a series of thresholds where it's in a weird state and then it's not
<mrvn>
maybe the patch is only doiung atomic changes and shotdowns.
<geist>
yah
<mrvn>
e.g. replace the retpoline.
<geist>
but, for example, if you had a ftrace entry and exit patch in one function
<geist>
and cpu A is patching them and cpu B is potentially running the same code
<geist>
unless you can lock out B, or you go through a eries of patch and flushes
<mrvn>
then you might get the entry but not the exit or vice versa. One can deal with that.
<geist>
then you could maybe have cpu B see the latter change but not the former
<geist>
or the former without the latter
<geist>
yah, exactly
<geist>
not insurmountable, but probably some potentially silly details to make sure it's safe
<mrvn>
maybe they have a flag that the patching is underway and the entry/exit code checks that and does nothing.
<heat>
i honestly can't remember
<heat>
i did look into it for x86
<geist>
also you'd need to fiddle with the page tables, or access the same pages via a different mechanism
<geist>
but that's also doable. maybe arch specific rules you have to follow but there ya go
<heat>
just that they used int3 as a stopper while they patched the rest of the call/nop instruction
<geist>
makes sense
<heat>
for x86 you can just disable cr0.WP
<mrvn>
geist: that's another thing you could play with. Make all the changes in different pages and at the end swap the page tables.
<geist>
indeed
<heat>
that bit also ran with irqs off
<geist>
though with SMP the other cores could still see the pages out of order
<geist>
before you've pushed a TLB flush through
<mrvn>
have to reload page tables on all cores
<geist>
well that would involve building a whole new structure
<geist>
anyway, details
<mrvn>
But it's a "patch everything" "IPI the changed tables"
<geist>
alas i gotta go to work.going into the orifice today!
<geist>
for like, literally the 3rd time in 24 months
<mrvn>
stay safe
<heat>
oh no, not the orifice D:
<geist>
yes the orifice calls
<geist>
it beckons. demands a sacrifice
<geist>
also nice to see cow-orkers i haven't seen in years
<geist>
and meet new ones
<mrvn>
you need a telepresence roboter.
<geist>
hah we actuall yused to, but it fell into disrepair since no one was in the office
<geist>
eventually stopped working and no one was there to fix it
<GeDaMo>
You need two robots so they can fix each other :|
<geist>
i have literally driven up to people hanging out in the break room or having a little mini celebration on my laptop
<geist>
it's actually a pretty trippy experience
<mrvn>
So it didn't lock itself in the closted had not hands to open the door?
<mrvn>
closet even
MrBonkers has joined #osdev
<heat>
fix it and make it run fuchsia
xenos1984 has joined #osdev
sonny has joined #osdev
dennis95 has quit [Quit: Leaving]
sonny has quit [Ping timeout: 256 seconds]
sonny has joined #osdev
MinecraftGamer12 has joined #osdev
<MinecraftGamer12>
hi
<MinecraftGamer12>
;D =*
<MinecraftGamer12>
how do I setup memory allocator??
<bslsk05>
en.wikipedia.org: Virtual memory - Wikipedia
<MinecraftGamer12>
it's Tanenbaum
<MinecraftGamer12>
Tannebaum is something completely different
<j`ey>
MinecraftGamer12: sorry I meant I didnt like the osdev discord
<MinecraftGamer12>
yeah me neither
<MinecraftGamer12>
*Tannenbaum
<MinecraftGamer12>
first time using irc
<MinecraftGamer12>
is this even e2e
<CompanionCube>
discord isn
<MinecraftGamer12>
I know
<GeDaMo>
No encryption here :P
masoudd has joined #osdev
<mrvn>
MinecraftGamer12: as I mentioned there is physical memory, virtual memory, std::allocator and malloc/realloc. They all solve different problems and have different solutions.
<CompanionCube>
discord isn't e2e, this is older
<MinecraftGamer12>
but discord is full with cringelords
<MinecraftGamer12>
I meant physical memory
<mrvn>
I would expect Tanenbaum to be better for the first 2.
<MinecraftGamer12>
Okay
<MinecraftGamer12>
I will read those pages
<CompanionCube>
i tnink most people in the channel use TLS, but perhaps not everyone
<bslsk05>
github.com: s6soc/ksetup.c at master · ZipCPU/s6soc · GitHub
<MinecraftGamer12>
Thank you
<ZipCPU>
It supports alloc, just not free. :D
<MinecraftGamer12>
DAMN
<MinecraftGamer12>
this is really short
<MrBonkers>
MinecraftGamer12 don’t lie you got told off for that link you posted. Here you are bitching about that you got banned for it. It’s you
<MinecraftGamer12>
still have no idea what this code does
<MinecraftGamer12>
MrBonkers stop harrassing me, I dont know who you are
<MinecraftGamer12>
is this IRC always like this?
<CoffeeMuffin>
I read that normally malloc is implemented using a free list
<ZipCPU>
The "heap" pointer is provided initially by the linker. It points to the end of program memory.
<MinecraftGamer12>
full of crazy delusional people?
<MrBonkers>
I’ve done my say, carry on
<ZipCPU>
There's nothing more in memory until the stack.
<CompanionCube>
no not normally
<CompanionCube>
this drama is between you and them, not anyone else in the channel
<MinecraftGamer12>
will I get banned here for posting stupid question
<geist>
Yes lets please not bring any drama here if unnecessary
<geist>
No you will not get banned
<MinecraftGamer12>
ok
<MinecraftGamer12>
good to know
<MinecraftGamer12>
anyways
<MinecraftGamer12>
my questions are clear, Imma head out
<geist>
Clean slate, tabula rasa. We’re here to help but accept help as just that
MinecraftGamer12 has quit [Quit: Client closed]
<geist>
Well okay then
<MrBonkers>
Geist I must say that is a good way to live by, I’ll remember that
<MrBonkers>
and sorry for bringing the drama in here, it’s just that he said to us that he was going to troll here, and I don’t think anyone here wants that
<geist>
Yah he was already starting to push it almost immediately by pushing back with mrvn when they asked questions
<MrBonkers>
so it was just a heads up
<geist>
Good to know
<ZipCPU>
.. and here I thought we'd have some fun discussion memory allocation. :|
<MrBonkers>
Unfortunately not, all he wants is trolling. Always sad to see
<geist>
Funny they were going on about cringlords and then acting like one, but so it goes. Seems like a maturity thing if nothing else. And sometimes immature folks grow up in real time here
<mrvn>
Looks like he really wanted to troll but being caught out he stoped pretending to have questions.
<MrBonkers>
Yep
<geist>
Yah possibly.,
<MrBonkers>
Must be annoying to notice that the moderator that banned you actually was waiting on IRC to warn the rest
<mrvn>
Unless our answers are so magically good he understood the whole problem range.
<geist>
Yah. I’d have noticed it but i left the osdev discord (if it’s the same time) after just a little while
<geist>
Let’s say, there was an impedance mismatch
<MrBonkers>
So I noticed indeed, a shame really. Say what you want about IRC but it does have a higher bar for entry, usually accompanied with a more mature audience too
<geist>
Yah
<geist>
It doesn’t have to, but i think we’ve managed to curate a reasonable set of folks here
<heat>
ZipCPU, what discussion do you want?
<CoffeeMuffin>
People here are definitely a lot friendlier, than in the osdev discord.
<MrBonkers>
I agree. From the lurking I do there is (almost) no drama here
<ZipCPU>
Thank you, heat. I was looking forward to an opportunity to share what I thought was the simplest memory allocator out there. I thought it might be fun to share with someone wanting to learn about the topic.
<ZipCPU>
I posted the link earlier.
<ZipCPU>
Once you start getting into allocation *and* deallocation, things get more complex. That's when I need to start looking up the textbook.
<MrBonkers>
CoffeeMuffin sorry to hear that, we do our best to moderate it over on discord but we’re not perfect and don’t see everything
<geist>
Hehe the ol ‘bump the pointer forward’ allocator
<geist>
Works like a champ when it’s sufficient
<ZipCPU>
geist: So it's not new? And here I thought I'd come up with something novel. :D
<geist>
Yah my main problem with the discord is the moderators, or at least one of them
<geist>
Which I think didn’t like me trying to help and step on their toes. Felt a bit like a turf war which I have no interest in participating in
<MrBonkers>
I’m aware, the discussion about using -kernel in qemu is well known
<geist>
Ah well at least it was noticed
<geist>
ZipCPU: hah sorry. It’s a nice solution for bootloader or firmware that’s only going to run once
<geist>
Or if frees are unnecessary
<heat>
the thing about bump allocators is that they don't work as is for weirder memory maps
<ZipCPU>
Incidentally, for those who don't know me, I've been lurking here under the hope of writing my own OS, and hoping that listening will teach me something.
<MrBonkers>
but, I can say that we welcome people trying to help (it’s just that the moderator in question has actively tried to not promote -kernel in qemu for the reasons that were given, I think that tripped her up)
<CoffeeMuffin>
It's okay MrBonkers, I just accepted that I will most likely be insulted somehow if I write in #lounge
<ZipCPU>
I built my own CPU and ... have visions of getting some kind of OS running on it.
* geist
nods
<CompanionCube>
ZipCPU: #osdev is a 10/10 lurking channel, would recommend.
<MrBonkers>
#lounge is a semi moderated mess yes
<GeDaMo>
ZipCPU: breadboard? FPGA?
<MrBonkers>
on topic channels are a lot better
<ZipCPU>
FPGA
<CoffeeMuffin>
The coding/osdev channels are fine
<ZipCPU>
It doesn't (currently) have an MMU. That's sort of discouraged me. Not sure if I need to build one, or follow in the footsteps of those who've built an OS without one.
<geist>
At the end of the day I think i decided I only have so much time in the day to devote to it and there was too low a signal/noise for my tastes
<MrBonkers>
Understandable
<heat>
ZipCPU, it's super easy to write some sort of embedded OS thing without an MMU
<CoffeeMuffin>
and don't even think about going to #memes
<CompanionCube>
ZipCPU: well, there are even versions of Linux that don't need an MMU :)
<heat>
you don't actually need an MMU to do things
<MrBonkers>
Avoid that mess indeed
<ZipCPU>
CompanionCube: Yes, I've heard that, just ... haven't chased them down yet.
<GeDaMo>
There's a #cpudev channel but only 6 nicks
<geist>
Indeed. For the most part most embedded cpus, even modern ones, don’t do mmu
* MrBonkers
goes back into lurking here
<geist>
And lots of real time systems
* geist
waves at MrBonkers
<ZipCPU>
geist: Thank you. That's encouraging.
dennis95 has joined #osdev
<heat>
of course the things you require from the CPU kind of depend on what kind of OS you want
<ZipCPU>
At least, my first OS build experience had no MMU. ... if you can even call it an "OS". But, task switching was a fun exercise that I'd neve3r done before.
<geist>
Honestly wish arm or riscv would at least add some sort of translating segmentation style system. Even ifit’s a fixed number, like 4 or8
<geist>
That’d make it far easier to at least have a foreground/background process or whatnot
<geist>
even if it’s not fully paging
<ZipCPU>
One problem I had with no MMU was that I had no stack protection. If a processor overran its stack ... it overran my kernel's processor table.
<GeDaMo>
Do you have a C compiler which targets your CPU?
<heat>
like, MMUs are strongly encouraged for userspace and compatibility with modern OSes(mmap and whatnot)
<geist>
Both have a permission based segmentation thing, but doesn’t translate, only lets you set ranges of physical space with particular permissions
<ZipCPU>
GeDaMo: Yes. I wrote a back end for GCC that targets the ZipCPU.
<GeDaMo>
Neat :)
<ZipCPU>
In general, my "market" or niche is small and embedded.
<ZipCPU>
I'd like to be able to support a file system and tasks, but ... I'm not really there yet.
<geist>
Yah you can totally write a kernel or multitasking system on that. Actually fun to do something on cpus where it’s kinda inappropriate
<ZipCPU>
So far, I've only worked with tasks having a fixed and pre-determined memory size. It's not very portable.
<heat>
something cool I found out about: the exynos SoC has physical memory write protection built in that you can ask for using a secure monitor call
<geist>
Like a 6502 is difficult to run a multitasking os on, but it’s fun todo it anyway
<heat>
so linux exynos code write protects the whole kernel as a security measure
<ZipCPU>
I was hoping to be a bit better than a 6502 ...
<geist>
Sure
<j`ey>
heat: not upstream I uess?
<GeDaMo>
Timer interrupt, task swap, what more do you need? :P
<ZipCPU>
heat: That'd cost a clock cycle.
<geist>
Ship it!
<ZipCPU>
GeDaMo: The ability to do it more than once.
<ZipCPU>
I mean ... the CPU supports it well enough (minus the MMU), so I have built multi-tasking stuffs ... it's just that I seem to need to rebuild the infrastructure every time I do it.
<ZipCPU>
Hence my reason for lurking here.
<heat>
j`ey: probably not, there's a whole lot of drivers/soc/samsung which isn't upstream but made available anyway
<geist>
Well it might be hard to build a portable system for it,yes
<geist>
It gets harder the more specialized or esoteric the hardware is
<geist>
But that’s also a fun challenge
<ZipCPU>
Yep!
<geist>
But a lot of low level embedded systems, like freertos, simply leave it up to the implementer to fill in major parts of the kernel
<geist>
Ie, each ‘port’ gets a header file that you fill in a bunch of functions
<geist>
Like task switch, interrupt, etc. and the kernel is mostly the generic parts
<GeDaMo>
I wonder if there could be a modern version of CP/M, something very simple where you filled in the basic I/O routines
<geist>
Not fundamentally different from an arch/ layer in a lot of bigger systems, but they don’t really attempt to even standardize that. It’s simply ‘insert arch bits here’
<geist>
At a fairly high level
<mrvn>
geist: it's even more fun writing the multitasking OS in c++, compile it to x86 and transmute that to 6502 asm.
<ZipCPU>
I need to run for lunch, but I'll be back to lurk again soon enough. geist: Thank you so much for your encouragement. With encouragements like that, I'll enjoy coming back later.
<bslsk05>
blog.sigfpe.com: A Neighborhood of Infinity: The Three Projections of Doctor Futamura
<mrvn>
hlt and catch fire.
<moon-child>
lp0 on fire
<GeDaMo>
Warning: Cthulhu has awoken
<geist>
heat: that’s true.it’s the old question of whether or not tools feel better being maximally used or idle
<gog>
my take is purely materialist, if something is not fulfilling some sort of use-value then producing it was a waste
<gog>
i won't get into whether those use-values serve a socially necessary purpose or not :p
<mrvn>
GeDaMo: In my compiler design class we went the other way. We implemented a tokenizer, scanner, grammar, etc. like you expect. Then an interpreter to evaluate an AST. And last we modified the interpreter to output the code it would execute instead of actually executing it. Voila => compiler.
Teukka has quit [Read error: Connection reset by peer]
<CoffeeMuffin>
That does not make sense for me
<mrvn>
CoffeeMuffin: did you read the three projections of Dr. futamura?
<CoffeeMuffin>
yes
eddof13 has joined #osdev
<mrvn>
So instead of making compilers from interpreters we made the interpreter generate/be the compiler
<mrvn>
Well, maybe it's the same if you look at it sideways. You can say it's a partial evaluation of the interpreter. Ok, forget I said anything.
<mrvn>
So to correct myself: Hey, we did that in out compiler design class. :)
Teukka has joined #osdev
<CoffeeMuffin>
If we are talking about interpreters
<GeDaMo>
I sort of understand the concept but the paper's a bit over my head :/
<gog>
make an interpreter from a compiler
<gog>
or a compiler that generates an interpreter that generates the compiler that generated it
<CoffeeMuffin>
You write an interpreter in a language that gets interpreted
<gog>
oh like pypy
<CoffeeMuffin>
and then that one does the same, but you can go backwards and the interpreted interpreter can modify the interpreter interpreting it
<CoffeeMuffin>
and so on n times
<gog>
yes
<gog>
then you made lisp
<CoffeeMuffin>
exactly
<mrvn>
CoffeeMuffin: An interpreter becomes a compiler if you partially evaluate it and output the code it would execute instead of actually running the code. And if you design the interpreters composable then you can compose that too to collaps the tower.
<gog>
i should learn how to do functional programming
<mrvn>
Gog: well worth it.
<CoffeeMuffin>
well said mrvn
<mrvn>
How to write a CPU emulator in a JavaScript VM in a composable way is another issue.
<mrvn>
This is serious meta meta programming
<gog>
also, unrelated, but i was helping somebody debug some code yesterday and learned a lesson: scanf()ing into buffers on the stack can easily corrupt the stack
<mrvn>
s/into buffers on the stack// s/the stack//
<gog>
fair
<CoffeeMuffin>
I thought scanf only works with files
<mrvn>
In some ways corrupting memory is worse.
<mrvn>
CoffeeMuffin: scanf is file -> memory
<gog>
it was an sscanf reading a 14 byte value from text into a 14-byte buffer on the stack
<CoffeeMuffin>
so the stack is a file?
<gog>
above it was a pointer
<mrvn>
CoffeeMuffin: char c; scanf("%s", &c);
<gog>
no, it aws reading from a file into an array on the stack
<CoffeeMuffin>
oh
<gog>
so it was scaning 14 bytes but writing 16 into the stack becuse of implicit conversion
<gog>
and ruining the pointer above
<gog>
took me a while to find it
<moon-child>
asan!
<gog>
also the code was broken
<gog>
in severarl other ways
<mrvn>
gog: you should have length specified the %
<gog>
it was specified
<gog>
well, rather, the type length wasn't specified
<moon-child>
(also -Wformat I think also checks scanf formats)
<bslsk05>
github.com: vmcam/main.c at master · macm00v/vmcam · GitHub
<gog>
this is not the way i'd have done this also
<mrvn>
Is that the wrong one or the fixed one?
<gog>
that's the wrong one
<gog>
the way i "fixed" it was just made the array 32 bytes so it wouldn't wipe out pass
<gog>
and it worked so i washed my hands of it XD
<mrvn>
isn't that off by 3?
<gog>
uh, actually yeah
<mrvn>
it reads ints into a char array. bad boy.
<gog>
yeeep
<mrvn>
-Wformat should have cought that.
attaboy has quit [Quit: WeeChat 3.4]
<gog>
nope
<gog>
oh wait no it ignored the CFLAGS from env
<gog>
lemme reconfig
<CoffeeMuffin>
on linux you can use write and read syscalls that read/write fixed lengths
<mrvn>
"Wformat Check calls to printf and scanf, etc., to make sure that the arguments supplied have types appropriate to the format string specified
<gog>
aha ok it did
<mrvn>
" if it fails at that simple format string write a bug report.
<gog>
yep
<gog>
whoever originally wrote this code was ignoring all warnings
<mrvn>
hehe.
<gog>
i had to fix a half dozen other things just to get it to build
<mrvn>
gog: If someone comes at me with code like that asking me what's wrong I would just send them back with the compiler output after turning on warnings.
<gog>
the guy i was helping didn't write it either
<gog>
he was trying to get it to work
<gog>
and didn't speak C so well
<mrvn>
poor guy
<gog>
yeh
<gog>
he hasn't come back to me again tho so i think it's sorted for now
<mrvn>
Fixing other peoples code is the worst.
<gog>
the fact that i managed to fix it gave me a bit of a confidence boost though :p
<gog>
i'm half-tempted to fix it for real and submit a pull request but this repo's been inactive since 6 years ago
<mrvn>
gog: at least the pull request would be visible to other users, right?
<gog>
maybe
<mrvn>
There aren’t any open pull requests. go make one
<mrvn>
gog: For code like that don't you wish github had a Black Hole button to un-star something?
<gog>
lol
<gog>
supernova
<gog>
also this is very naughty, but the autogen'd files are included in the repos
sonny has quit [Ping timeout: 256 seconds]
<gog>
shame shame
<mrvn>
Sometimes that is a good thing. If you want to revive the code years later you might not be able to generate it again.
<gog>
hm true
srjek has quit [Ping timeout: 260 seconds]
<moon-child>
why do section names conventionally start with .?
mctpyt has joined #osdev
dude12312414 has joined #osdev
Scripted has joined #osdev
<Scripted>
Hello, how do I implement mmap to my kernel?
<bslsk05>
'Dynamic Memory - 1 (malloc) | Make a 64 bit OS From Scratch!! | Part 20' by Poncho (00:09:33)
<mrvn>
malloc() has been around a lot longer than there are MMUs.
<MrBonkers>
if that is season 1, then please, use season 2 (or, try to avoid youtube tutorials at all, most of the time they aren't that good, Poncho himself said that his tutorials aren't that great)
<mrvn>
urgs, that video is horrible. The top lines of his editor are cropped off.
<mrvn>
plus C from the last millenium.
<mrvn>
uint_64 -- fun
<Scripted>
So... should I watch that cppcon video instead?
<MrBonkers>
and he knows it (both poncho as this guy here). He's been told this video isn't that great before
<mrvn>
you can watch more than one video
<MrBonkers>
cppcon are usually really good
<Scripted>
I hope I will understand it, my C++ needs some refreshing
<MrBonkers>
I haven't watched this one specifically but cppcon is a good resource (not a tutorial and aimed at C++, fair warning)
<mrvn>
Scripted: cppcon videos are great quality but they are more about c++ features than kernel directly.
<Scripted>
yeah well, my kernel is coded in C and x86 ASM
<Scripted>
I honestly have no idea what to implement if all I want is tmpfs
<mrvn>
I would compose a SLAB and malloc like allocator.
<bslsk05>
en.wikipedia.org: Slab allocation - Wikipedia
<Scripted>
Sorry, should've googled it
sonny has joined #osdev
sham1 has quit [Ping timeout: 240 seconds]
<mrvn>
Scripted: the simplest way would be to just allocate 1 page for everything even if that wastes a lot. Memory is cheap, you have a lot. You can trim it later.
<Scripted>
you mean for the heap allocator?
<mrvn>
for tmpfs
<Scripted>
oh yeah, it will take ages till I get there
<bslsk05>
wiki.osdev.org: Going Further on x86 - OSDev Wiki
<Scripted>
So according to the wiki, I need a memory map, a virtual memory allocator and a heap allocator for memory managment
<mrvn>
A file is a control block with name, size and an array of pointer to data blocks. When you need files larger than 8MB you can refine it.
<Scripted>
I doubt I will need anything over 8MB
<mrvn>
see. All that code you write now will get replaced over time. So make that code now as simple as possible.
<Scripted>
got it, so all I need is a heap allocator. dlmalloc and others are too advanced I assume
<Scripted>
So I have to write my own
<mrvn>
My point is: YOu don't need it to be able to e.g. execute a hello-world program.
<Scripted>
That's my goal for the end of osdev
<Scripted>
well, my kernel
immibis has quit [Remote host closed the connection]
<Scripted>
This is off-topic, but is it normal to see the IPs of IRC users?
<mrvn>
totaly.
<gog>
yeah the server tries to get their host with rdns but shows IP if it can't
<gog>
libera doesn't have masks by default
<gog>
you need to register to get a cloak if you want one
<clever>
on some networks, +x will mask it, even if you lack a cloak
<clever>
weird, i cant +x on libera
<Scripted>
ok, I didn't know that
sham1 has joined #osdev
heat_ has joined #osdev
heat has quit [Ping timeout: 260 seconds]
<mrvn>
What are those types like std::vector<T>::reference called again?
<clever>
c++ templates?
<mrvn>
no, the type at the end: reference
gildasio has quit [Quit: WeeChat 3.0.1]
<clever>
thats just a class defined within the template
<clever>
the std::vector<T> template/class, also acts like a namespace
gildasio has joined #osdev
<clever>
cant remember if it had any special name
<mrvn>
Those type aliases inside std classes have a special name.
<mrvn>
or the pattern of doing it has.
dennis95 has quit [Quit: Leaving]
masoudd has quit [Quit: Leaving]
dormito has quit [Quit: WeeChat 3.3]
<gog>
mrvn: proxy
<gog>
?
<heat_>
mrvn, those are typenames
heat_ is now known as heat
<heat>
Scripted: tip: you can totally port a malloc implementation
<Scripted>
heat which one would you suggest?
k8yun has quit [Quit: Leaving]
<heat>
I ported musl libc's old malloc, because it's relatively simple and I was already using musl for my userspace
<heat>
I've also used dlmalloc before
<Scripted>
yeah but dlmalloc requires mmap
<heat>
no
<Scripted>
wdym
<heat>
dlmalloc is ridiculously configurable as you can see if you look at the beginning of the file
<heat>
fairly sure you can run it on brk all the way
<Scripted>
yeah but I dont have brk
<heat>
you need a brk-like thing or an mmap like thing (ideally, both)
dude12312414 has quit [Quit: THE RAM IS TOO DAMN HIGH]
<heat>
brk and mmap will layer on top of a virtual memory allocator, which will layer on top of physical memory allocator
<heat>
note: when I say brk and mmap, I mean kbrk and kmmap, not the system calls themselves (as this is for the kernel itself)
<Scripted>
I will take a look tomorrow on what really to do since I have no idea on how to implement something like mmap or brk
<Scripted>
good to know
<heat>
brk is dead simple
<heat>
it's essentially a pointer that keeps getting incremented, when you increment onto the next page, it maps it
<heat>
when you decrement it onto the previous page, it unmaps the current page
<Scripted>
The thing is, I still lack basic vocabulary like I don't know what the heck a page is
<heat>
ah ok
<heat>
forget malloc then
<heat>
do you have a page allocator?
<Scripted>
No
<Scripted>
heat this is why I'm trying to read endless pages about memory to get a hang out of it
<heat>
ok so essentially your memory is usually divided into pages (each page is usually 4096 bytes long)
<Scripted>
Okay
<heat>
you have a page allocator that deals with allocating those pages (note that pages are always in page-aligned addresses, you don't have page starting at 0x0f0 and ending at 0x10f0)
<Scripted>
I see
<heat>
an example of a page allocator is linux's buddy allocator, although it might be a bit more complex than you want it to be
<heat>
note that this is physical memory
<Scripted>
I found the article for the buddy allocator
<heat>
then, you have virtual memory, which is layered on top of physical memory and on top of the CPU's MMU
<Scripted>
it's better than nothing
<Scripted>
what's the difference between physical and virtual memory?
<heat>
the MMU (memory management unit) is the part of the CPU and essentially lets you do things like separate address spaces for different processes
srjek has joined #osdev
<Scripted>
for multi tasking?
<heat>
yes
<Scripted>
I see
<heat>
for instance, pid 1 and pid 2 might both have a pointer to 0x20000, and because of the MMU, the two processes will read different physical memory pages in memory
<heat>
you manipulate it using page tables (layout depends on the architecture) which are just a weird tree-like structure
<Scripted>
so they don't override each other?
<heat>
no
<Scripted>
why then?
<heat>
when I say "map" I mean "allocate a page and put it in the page tables so that your desired virtual address has a corresponding physical page"
<bslsk05>
libera.irclog.whitequark.org: #osdev on 2022-03-10 — irc logs at whitequark.org
<Scripted>
Now I can (hopefully) understand the osdev articles about memory.
<Scripted>
heat gotcha
<heat>
Scripted, no problem :)
<heat>
note: a lot of osdev articles about this aren't good
<heat>
unless they got exponentially better
<Scripted>
yeah, I noticed at the very beginning of osdev
<heat>
it's important to not overly rely on the wiki
<klange>
heat: be the change you want to see on the wiki :)
<heat>
especially since the quality gets gradually worse as you advance
<Scripted>
telling osdev noobs to read the osdev wiki is like giving elementary school students a wikipedia article about a maths concept and tell them to try to understand it that way they can implement it themselves
<heat>
klange, easier said than done :P
<heat>
Scripted, well, no, there's a lot of stuff that was written by people that really didn't understand it fully
<heat>
that's the issue
<Scripted>
heat yes, but there is a lot of vocabulary that needs to be learned too
<heat>
there's also a lot more complexity to talk about
<Scripted>
that's true
<heat>
the way I explained the vm subsystem is like, the simplest way I can put this
<heat>
if you look into current operating systems, they don't work like this because it's suboptimal
<heat>
note that you can totally design an OS based simply on what I've said
<Scripted>
it's still the same as trying to teach students a new topic via a wikipedia article that uses technical terms without them students knowing anything about it
<heat>
I didn't lie to you, I'm just omitting details
<Scripted>
that's at least how I felt starting osdev
<heat>
yes, starting osdev is rough, I feel you
<zid>
bohr model of the atom :P
<heat>
it all depends on how much prior knowledge you have
<Scripted>
heat there a lot of ways to design a os, which may be a problem for starters
<Scripted>
for example, I didn't even know that there was an entire unix philosophy before starting osdev
<heat>
Scripted, yeah but most of those details don't matter when you're starting out
<klange>
there are a lot of ways to design an os
<klange>
and virtually no one doing osdev is
<sonny>
is there a lot of ways?
<Scripted>
heck yeah
nanovad has quit [Quit: ZNC 1.7.5+deb4 - https://znc.in]
<sonny>
don't worry I won't go on a long discourse anymore
nanovad has joined #osdev
<heat>
actually, physics is a great way to explain this: you learn classical physics before quantum mechanics, because the small details don't matter and so you don't get overwhelmed with information
<Scripted>
you can do it ms dos like, unix like, linux like, and so on
<Scripted>
for instance, linux with systemd tends to be not so unix-like
<sonny>
OS is a class of program that has some very defining characteristics , the areas where you can be creative are smaller than you think
<klange>
Don't confuse user-facing interfaces for the design of the OS as a whole.
<klange>
Linux, and even the BSDs, look nothing like classic Unix on the inside.
<sonny>
(at least per my thought experiments lol)
<Scripted>
I really gotta try the original unix OS once
<heat>
Scripted, that's not true
<Scripted>
What is not true?
<heat>
linux with systemd is still totally unix-like
<klange>
systemd has absolutely nothing to do with how Unix-like Linux is
<sonny>
there is some website that emulates a pdp11 or something with Unix on it
<Scripted>
well yes, linux with systemd has nothing to do how unix-like the kernel is
<Scripted>
but more like the distro you build
<heat>
operating system design has evolved
<heat>
most of the programs you use aren't just small things all piped to each other
<sonny>
heat: are you talking about service management or in general?
<heat>
in general
<sonny>
ok
<sonny>
Scripted: you can even see how stuff like service management is the next step ...
<heat>
going with crappy smaller alternatives just because "muh unix" is dumb because unix was designed in 1970 on a 16-bit processor with no MMU
<Scripted>
sonny no idea what that is
<Scripted>
so much to learn but yet so less time
<heat>
it's what systemd does
<heat>
systemd solves a problem and it solves that problem damn well
<klange>
People just don't like change.
<gog>
i like systemd
<heat>
it started as an init system but evolved into a lot more because the whole "yeah linux is great you grab whatever you want, you can mix and match everything" doesn't work
<Scripted>
I use systemd too
<gog>
i like message-passing
<gog>
i like managed device filesystems
<heat>
turns out fragmentation is not good for progress in general
<gog>
yes
<gog>
exactly
<j`ey>
managed device fs?
<gog>
j`ey: udev
<j`ey>
oh
<gog>
or rather, the part of udev that systemd supplanted
<heat>
gog: you're probably running the devtmpfs mount provided by the kernel
<heat>
most systemds default to it
<gog>
oh so i am
<sonny>
Scripted: systemd
<gog>
anyhow, doesn't matter, systemd good
<gog>
or at least, better than what there was before
<Scripted>
kinda hard to read and play chess
<sonny>
connect computers on a network, you will need someway to administer services
<sonny>
if it's just your own thing then w/e
<Scripted>
sonny: what about systemd
<sonny>
nothing, I replied to you too late
<Scripted>
oh all right
<Scripted>
How come so many people are inside here but only like 10-15 people are talking?
<heat>
also categorizing oses into "msdos like", "unix like" is totally wrong
<heat>
windows is very different from msdos
<klange>
NT is its own category ;)
<Scripted>
MSDOS and NT is different right
<klange>
It's wrong to think of _OS design_ in terms of what interface or API you're providing.
<sonny>
"the only OSes that matter are NT and UNIX" -- my OS slides
<Scripted>
klange: on what should I think of then?
<heat>
sonny, honestly, not that wrong of an assertion :)
X-Scale` has joined #osdev
<sonny>
klange: yeah ... what else then?
<heat>
there's a lot to designing a system
<heat>
it's not just the interface
<sonny>
that's a big part of what an OS provides
<klange>
But it has nothing to do with the design of the OS.
X-Scale has quit [Ping timeout: 240 seconds]
X-Scale` is now known as X-Scale
<sonny>
hmm
<heat>
for example, google's fuchsia provides a posix layer but it's definitely not unix
<sonny>
true
<sonny>
but what about the filesystem?
<heat>
what about it?
<sonny>
that certainly affects how linux and NT are designed differently
<heat>
slashes vs backslashes isn't really a question of os design :P
<klange>
How filesystem access works may _influence_ the design, but they are not the design.
<sonny>
e.g. unix has that virtual filesystem thing
<heat>
windows does too
<klange>
Even DOS _kind of_ did.
<klange>
And the way it did continues to screw Windows to this day!
<sonny>
well, isn't NT all about the object manager?
<heat>
if the object manager and its objects work similarly to a filesystem, what's the difference?
<sonny>
hey, that's a trick!
<mrvn>
NT is posix certified (for some version of NT). Linux, BSD, ... all are not.
<Scripted>
god damn what is happening here
diamondbond has joined #osdev
<sonny>
NT got posix? til
<heat>
macOS is posix too
<Scripted>
this irc client makes everything unreadable
<mrvn>
sonny: They provided all the required bits and basically non of the optional stuff so it was basically useless. But certified.
<sonny>
windows docs say the posix subsystem got deprecated for wsl
<klange>
macOS's certification remains current. The NT thing that was certified a long time ago was a one-off.
<mrvn>
sonny: And yeah, now they just have a Linux layer that actually is usefull.
<sonny>
it's just a vm now iirc
<heat>
yeah
<mrvn>
is it a VM? I thought it was a syscall layer.
<sonny>
drat, I had created two definitions for an OS but I forgot my second one
<heat>
wsl1 is a system call layer, wsl2 is a hyperv vm of linux
<klange>
Didn't Huawei get a Linux distribution certified?
<sonny>
yeah, I remember reading about that
<heat>
probably
<mrvn>
Like BSD has bsd and linux syscalls. Linux on alpha has linux and dec syscalls. Amd64 has x86 and x86_64 syscalls.
<sonny>
HP still has a posix unix too
<heat>
it's not like Linux can't be POSIX, it's just that it costs money and isn't useful
mahmutov has quit [Ping timeout: 240 seconds]
<sonny>
also GNU
<sonny>
that'd be so much work
<heat>
the dreaded GNU HURD
<sonny>
lol
<mrvn>
All the posix cerifications are a one-off. Every time you change one line of code you have to pay again. So totaly useless for non-profit sources.
* kingoffrance
awakens from slumber fee fi fo fum someone highlighted me
<sonny>
I meant making GNU stuff posix
<heat>
GNU diverges from posix a lot
<mrvn>
verry little and with reasons where it does I would say.
<sonny>
is HURD complicated or no interest?
<kingoffrance>
mrvn makes sense re: crit-bit. yes, im not sure you even need "external nodes" if the "strings" are all you care about
<heat>
hurd is simply crap
<sonny>
oh
<klange>
hurd is a joke
<ZipCPU>
heat, klange: Why is that? Curious.
<sonny>
I see
<mrvn>
kingoffrance: it's just a type alias.
<Griwes>
the hurd faq still says they see no reason to ever allow for a 64 bit userspace lol
<sonny>
lmao
<heat>
hurd is a bad operating system that hasn't progressed much
<kazinsal>
hurd's target segment is "people who wanted hurd in 1985"
<sonny>
give me 64 bits please
<sonny>
a full 64
<mrvn>
kingoffrance: That description of a crit-bit tree really makes no sense with the extrernal nodes and then the putting strings in the internal nodes stuff. That contradicts each other.
<kingoffrance>
mrvn: i was trying to say yesterday, if you have "a.b.c=1" "a.b.d=2" "a.b.e=3" then those can be seen as key/value pairs. maybe with external nodes they "point" to somewhere as well, maybe not. and finally, if you can retrieve a.b.* then that can function as a "key" that returns multiple "values" in a sense, is how i see it
V has quit [Ping timeout: 240 seconds]
<kingoffrance>
the ability to retrieve a.b.* is of more interest to me
<klange>
never ported to anything outside of 32-bit x86, remains buggy garbage, offers roughly the same level of features as several hobby OS developed _here_ despite being around for as long as I've been alive and with hundreds of contributors
<mrvn>
You can implement a tree iterator in lexicographical order. That's a plus of the tree. But it's O(length of string) for every it++
<j`ey>
Im assuming its not a steady hundreds of contributors :P
<sonny>
any idea how hurd boots?
<klange>
the only thing hurd has going for it is that it was built to run gnu stuff, so despite being horribly incomplete the bits that are there are enough to run stuff with glibc, so it can technically manage to get a good chunk of Debian userspace
<mrvn>
I really don't see how the crit-bit tree lends itself much to looking up a.b.* actually. The root node might say the kyes differ in bit 17. How do you tell they begin with "a" at all?
<heat>
sonny, normally
<sonny>
so bios and probably via grub
diamondbond has quit [Quit: Leaving]
<kingoffrance>
mrvn: i believe adam's pdf says every crit bit tree has a "unique ordering" so that doesnt matter to me (reproducability) but i believe that is "why" that is possible. sorry i dont know any more details, i am only speculating
<mrvn>
kingoffrance: if you want to find "a.b.*" you have to lookup "a.b.". If you find a node where length > 32 you have to blindly pick a path down to a leaf and then check if the string you found started with "a.b." or not.
<kingoffrance>
s/unique ordering/ there is a single representation used for a tree, that would make that happen/
<mrvn>
If it does then everything below the node starts with "a.b." because the tree is sorted.
<kingoffrance>
^ yeah i think that is what is going on, pardon my vague wording
<mrvn>
But that's something any sorted tree will do.
<kingoffrance>
sure
<mrvn>
The memory requirement given in the paper is odd too. There is one internal node per string with 2 pointers and 1 int. But at the end he claims the tree only needs 1 pointer and 1 int per string.
<mrvn>
And that's basically the only real feature for the tree.
<mrvn>
The memory requirements given only work with that odd <data><length><data> scheme that can't work with strings.
<sonny>
imagine an OS where you memory access is scheduled ... programers would be mad haha
pretty_dumm_guy has joined #osdev
<mrvn>
what do you think 16 cores on the same memory bus do if not schedule?
<sonny>
not in that sense, I control when you do allocation+deallocation
<mrvn>
sonny: every time you call sys_alloc() I will schedule you.
<sonny>
lmao
<sonny>
it would be better overall, but only if you control the entire system
<mrvn>
or more practically on sys_free(). Unmapping a page means invalidating it. If there are more than a handfull then reloading the page table is faster. And if I do that anway I might as well switch processes.
<sonny>
yeah
* mrvn
still chuckles at std::midpoint. They proposed that because it is easy to get wrong. Profed themself to be right by messing it up the first 4 (at least) revisions.
mctpyt has quit [Ping timeout: 252 seconds]
<heat>
oh yeah midpoint is way harder than it seems
<bslsk05>
devblogs.microsoft.com: On finding the average of two unsigned integers without overflow - The Old New Thing
V has joined #osdev
mctpyt has joined #osdev
<kingoffrance>
eh,, i havent look at it forever, but my take on hurd was it seems they kept trying to base things on some upstream...and then upstream would die...and then theyd have to decide to maintain their own "fork" etc. it seems to me lack of NIH is the culprit :) (tongue in cheek...somewhat...)
<kingoffrance>
*looked
<mrvn>
heat: I still think they should have a std::next_larger_integer<T>, check if that exists and use that.
<mrvn>
heat: and it still goes wrong with pointers by the way. On x86 it's only defined for arrays < 2GB.
<mrvn>
(which next_larger would fix)
<mrvn>
I didn't know this midpoint: (a & b) + (a ^ b) / 2;
<mrvn>
Now that C++ only supports 2's complement wouldn't that be the best midpoint code?
sonny has quit [Remote host closed the connection]