<epony>
because.. retirement ages are pushed up towards what second world countries have, and the pay is not updates comparable to what political class get paid while jacking up inflation and spending of public resources on personal estates
<gog>
luv me strikes, luv me direct acksion
<gog>
'ate austerity
<zid>
'ate scabs
<zid>
simple as
<gog>
simple as
<Ermine>
we need more strikes
<zid>
everybody needs more strikes
<heat>
except baseball players
<heat>
hehehehe
<Ermine>
btw did UK govt react on recent strikes?
<epony>
except unlucky strike smokers
<epony>
who get fired from the job and get cancer at the same time
<zid>
they reacted by going "lalalalalaa" I believe
<heat>
uk government is now a react channel
<zid>
"We'll give you 1% next year though? WHY ARE YOU UPSET?"
<epony>
it's a servant to the monarchy
<heat>
oi dis poor blokes are really shtupid innit
<zid>
(inflation 8% and they haven't had a raise in 10)
<epony>
feeding one family very well
<epony>
well, who the fork wanted brexit..
<mrvn>
But good that they quit the EU so the workers can't go and work somewhere with decent wages.
<geist>
lets keep it on topic please
[itchyjunk] has quit [Ping timeout: 260 seconds]
<Ermine>
sorry!
<gog>
heh
<gog>
yeah sorry
<zid>
it's trains that are on topic here right?
<zid>
but no strikes
<mrvn>
is any chip factory on strike?
<immibis_>
full-time OS developers on strike, all 12 of them
<epony>
operating system development as a means to education and raising the skill in the developers learning about the system they are supposed to be programming
<Ermine>
trains have operating systems
<immibis_>
trains are operated with systems
<heat>
them systems really do be operating
<epony>
you can see railroad charts when you find out what EBNF is (and if you read a book about Pascal)
<heat>
geist, did you see the 2GB jump thing?
<heat>
pretty hilariously funny
<geist>
hmm, no. was it a while back here? there's a bunch of nonsense noise
<heat>
posted it to fuchsia #zircon too
<mrvn>
Lastlog:
<mrvn>
End of Lastlog
<mrvn>
01:12 < heat> geist, did you see the 2GB jump thing?
<geist>
on discord?
<heat>
yeah
<mrvn>
can you post it again?
<epony>
did you know that you can overcome the "process limit" barriers with overlay files and other sliding window techniques for working with larger files, so there is really no escuse for what Windows was causing to people for 10-20 years
<geist>
oh heh yeah
[itchyjunk] has joined #osdev
<geist>
i thought you could only relative jump 2GB anyway? or is it 4GB?
<zid>
It makes sense that you probably shouldn't do that
<epony>
"wingoes excuse"
<heat>
geist, it's an indirect jump
<geist>
oh i see
<mrvn>
geist: I thought there is both
<heat>
so jmp *(%rax) or something
<zid>
we have tags and stuff for caching right
<zid>
and they have so many way associativity and crap
<zid>
makes sense that 'lots of other things' are limited by 2GB etc
<geist>
yah i thought it was for relative jumps at first, which ithink you can only 2GB or so
<geist>
but yeah for any sort of absolute or indirect. ouch.
<heat>
probably heavily affects the zircon shotgun mappings right?
<epony>
2GB is the 32bit process size limit, 4GB is the (V)FAT(32) file size limit
<geist>
yah but really if you look at how linux lays out a process, there's going to definitely be a jump between the binary and any lib
<heat>
plus shared libraries on every system
fedorafan has joined #osdev
<heat>
yep
<geist>
yah in that case a shotgun layout isn't going to really be any different, unless some OS clusters everything together
<geist>
and AFAIK they usually dont nowadays
<heat>
it is tho?
<mrvn>
geist: I wouldn't mind a memory model where .text of everything gets mapped into a 2GB block.
<heat>
if you start jumping around libs and they're all > 2GB apart
<epony>
there is one other factor, that is "machine generational" (rough estimate) of the memory that a machine can effectively process for 1-2 seconds as processor throughput and memory access/transfer rates
<geist>
yah that's what i mean. i think most modern systems at least put libs probably >2GB apart
<heat>
not individually, no
<geist>
thoiugh i dunno maybe not
<epony>
that's about those limits for the 32bit machines up to 2000-2004
<mrvn>
geist: mmap usually takes from the end of memory but address space randomization changes that a bit
<heat>
linux does: *put ET_EXEC elf at linked address, put ET_DYN PIE elf at ~0x555555555555*
<geist>
yah i guess you're right. at least on this random linux x86-64 box, the binary is somewhere random, and then the 2 or 3 libs seem to be clustered somwhere else random
<heat>
yeah then all libs are just straight up mmap'd, no magic
<heat>
I was discussing on #musl about a patch to remove the 0x555555555555 thing, and hence remove the separation between the interpreter and the libs
gog has quit [Ping timeout: 252 seconds]
<heat>
erm, sorry. remove the separation between the interpreter/libs and the PIE executable
<mrvn>
heat: do you want to cluster them into a 2GB chunk or make all segments totaly random?
<heat>
as things stand, PIE gets loaded to 0x555555...., interp gets loaded to 0x7fffffff...., and then mmap'd libs are placed under the interp
<heat>
mrvn, wdym make segments random?
<mrvn>
heat: roll a dice and put the interp there, roll again for .text, again for .data, again for each lib
<heat>
you can't do that, shared objects have defined offsets between segments
foudfou_ has joined #osdev
foudfou has quit [Remote host closed the connection]
<heat>
if .text 0, .data 0x200000, I addressof(.data) must be addressof(.text) + 0x200000
<mrvn>
heat: not with PIE/PIC
epony has quit [Ping timeout: 268 seconds]
<heat>
yes with PIE/PIC
<mrvn>
then you didn't include enough relocation information in the elf.
<heat>
the only thing that can get randomized is the load bias/load address
<heat>
bud, there's no relocation information that makes this possible
<heat>
you would never rip-relative address something
<heat>
unless you start relocation patching .text. 1) that segfaults in current ld.so's 2) you broke the point of shared objects
<mrvn>
heat: you patch the got
<heat>
trivia: how do you access the got
<mrvn>
heat: through the global register pointing to the got
<heat>
must be a new register
<mrvn>
got access is different on different archs
<mrvn>
and flags
<heat>
generally, you access the got through pc-relative addressing
<heat>
geist, btw this whole thing sucks even harder because all calls into a shared lib go through the .got, so indirect call :V
<mrvn>
heat: anyway, what is your goal?
<heat>
idk, happiness, build a family, world peace
<mrvn>
if you force libs to be mapped within 2GB of the PIE binary then you could relocate the .text and skip the .got
immibis_ has quit [Quit: HexChat]
<heat>
you cannot because you're also writing on RO segments and killing COW
<mrvn>
You do that at load time. And what COW? it's .text, it doesn't change post load.
<heat>
load time?
<mrvn>
in the dynamic linker
bauen1 has joined #osdev
<heat>
what COW?
<heat>
erm, 100 /bin/cat's share their whole text and rodata
<heat>
because they are mapped COW
slidercrank has quit [Ping timeout: 252 seconds]
<mrvn>
heat: if you don't want a trampoline then you have to sacrifice something.
<heat>
but I do want it
<mrvn>
You could have kernel support so all the 100 cat's use the same mapping. Go straight to the relocated .text on the next exec()
<mrvn>
Or you could keep the trampoline but make it relative jumps instead of indirect
<heat>
did you hear me say I don't want it? I want to exclusively pack shared libraries and the executable near each other as to make .got and this intel cpu fuckery work well
<heat>
really, it's Intel that dropped the ball here
<mrvn>
it sounded like you didn't want to do the indirect jump through the .got
<heat>
I do want it, it's how shared objects work
Burgundy has quit [Ping timeout: 260 seconds]
<mrvn>
if you have the indirect jump then why do you care if the binary and libs are close together? What am I missing?
<heat>
the Intel CPU problem i posted where indirect jumps wider than 2GB mispredict a lot
<geist>
well you have to more or less indirect it through got anyway because it could be far away
<geist>
but if you did pack them tight you could avoid the got if you're okay patching the .text
<heat>
yep, although the got thing really is to just save on dirty pages
<geist>
yes and far away targets (on 64bit)
<mrvn>
heat: so you DO want to cluster everything together so you can have relative jumps (32-bit signed displacement)
<heat>
no
<heat>
well, yes
<heat>
I want jumps and calls to $random-64-bit-addr to be at most 2GB away so performance doesn't go down the drain
<mrvn>
The wording on that page is pretty bad. It makes it sound like a "jmp *($rax)" will be better if the distance is small. But I think what they describe just depends on the opcode, not the actual distance.
<heat>
and this is a much worse problem on non-PIE executables where you'll need to force shared libs down to < 4GB :(
<heat>
mrvn, it will
<heat>
that is the exact problem
<heat>
there's no displacement in indirect jumps, it's all absolute
<mrvn>
heat: You can read the text such that "JMP/CALL imm32" is more efficient.
<heat>
well, no shit. those are not indirect
<geist>
also by def they're within 2GB
<mrvn>
or you can read it that any jump is more efficient when it could possibly be also written as "JMP/CALL imm32" instead of what it actually is.
<heat>
no, you're reading it in random ways
<heat>
it gives you an explicit example
<heat>
wildly different misprediction ratios based on the distance to the PC
<mrvn>
heat: nothing explicit in the example. Where is the opcode that they profile?
<mrvn>
heat: ant those branches are what? "jmp *($rax)"?
<heat>
yes...
epony has joined #osdev
<mrvn>
heat: The text doesn't convince me at all. For one thing the header talks about 21% misprediction but the tools output has no 21% in it at all. Secondly where is the same test with the library mapped close to show the predictor work better?
<geist>
why dont you bring this up with intel and not heat, mrvn?
<geist>
like, heat is just pointing out a thing, he didn't write it
<mrvn>
don't know anyone at intel
<heat>
i'm fairly sure that intel know their processor a lot better than you, such that this is not a random whitepaper nor errata but an actual section in the new optimization manual
foudfou_ has quit [Remote host closed the connection]
foudfou has joined #osdev
<mrvn>
heat: they might very well be and probably are. They can be right without having shown proof.
<mrvn>
yep, that looks like a test of the same code with >2G jumps and <2G jumps.
<geist>
it's an optimization manual, they dont have to *prove* it
<geist>
they just say 'because of this thing you should avoid that'
<mrvn>
They don't have to proof it. But they do give an "How to Detect it" that is inconsistent and doesn't show anything because it's only half the story.
<mrvn>
or maybe there is another tool output on the next page, can't aay.
<mrvn>
say
<epony>
so.. how does the 2GB limit persist on 64bit systems, do you know?
<mjg>
i'm pretty sure the test was only done on 64 bit
<mrvn>
mjg: see the second url heat posted. You can just map things closer without having to go to a 32bit cpu.
<heat>
it's interesting if this will spark any changes into operating system memory layouts
<heat>
who knows if this will be present in future microarchitectures
<mrvn>
How common is the "Golden Cove CPU"?
<heat>
it's alderlake (desktop + laptop) and sapphire rapids (server, just launched a few months/weeks ago)
nyah has quit [Quit: leaving]
<epony>
better use model numbers instead of code names
<epony>
or CPU series / gen / year
<mrvn>
I would really like to see the profile run they did redone with the 2G fix. How much of those 21% aka 22.30% misprediction at 0x555555603664 goes away when the distance gets smaller?
<epony>
ah, the branch predictor artefacts ;-)
<epony>
testing testing..
<epony>
that's why game development for performance targets specific CPU models with particular chipset and periphery, also known as "consoles" which are just PCs with fixed specs and slightly different HW for "captive" reasons
<mrvn>
Another thought: When the library function gets looked up for the got/plt the first time does it generate an "JMP/CALL imm32" when the offset is small enough?
<heat>
no
<mrvn>
that could be another optimization in the dynamic linker then.
<epony>
imagine what happens when people reflash the firmware of their game console with a third party BIOS replacement with regards to that machine specific operation tuning
<epony>
then figure out why are people doing that to themselves with alternate replacement for their system basic programming and expect some "dramatic" gains or achievements
dutch has quit [Quit: WeeChat 3.8]
<heat>
geist, btw what does "wider machine" mean?
<mrvn>
more phys bits? 5 level page table?
<geist>
hmm, in what context?
<heat>
the 1st screenshot
<geist>
*probably* means it has more back end units
<geist>
and/or decodes more instructions per cycle
<heat>
"The Golden Cove CPU is a wider machine and might exhibit a higher Top-down Microarchitecture Analy-
<heat>
sis (TMA) Bad Speculation percentage"
<geist>
ah yeah, it's the 'lots of execution units, runs lots of instructions per cycle'
dutch has joined #osdev
<heat>
ah and because of that it speculates further, and may mispredict more easily?
<Mutabah>
I don't think its quite that bad
<Mutabah>
Well.. it might mispredict more (as there's more in flight)
<geist>
might be the mispredict penalty is higher
<Mutabah>
but the pipeline depth is likely the same, so the penalty stays the same?
<Mutabah>
(in absolute terms)
kanzure_ is now known as kanzure
<epony>
typically width is used for the word, instruction, operand, data types etc and registers, address mode and data buses, so about the "bit width" of various parts of the computation and data transit
elderK has joined #osdev
heat has quit [Ping timeout: 246 seconds]
<zid>
hmm my green has decided today it shall be stripey
<zid>
too cold inside
<zid>
one day the green will totally die and I'll have to try desperately to desolder some other device for matching caps to try :D
<mrvn>
what's a green?
<geist>
a lawn i guess? a golf thing too
<mrvn>
indor? Maybe some virtual golf thingy that needs caps for the electronics.
<ornx>
which EFI memory type is the good one? is it EFI_MEMORY_NV or EFI_MEMORY_RUNTIME
<Mutabah>
Define "good"?
<Mutabah>
Both of those are memory you shouldn't touch afaik
<ornx>
conventional physically-backed memory
<ornx>
wait those are attributes not types... EfiConventionalMemory is the right one i think
<Mutabah>
at a guess from the names, the first is non-voltaile (flash or battery-backed RAM), the second is a scratch space used by EFI runtime servces
gxt has quit [Remote host closed the connection]
gxt has joined #osdev
<epony>
sounds like a VGA cabling problem on an aging PSU in a monitor
<epony>
and of course, cold starts make it happen faster, but summer heat is even more destructive so you experience that at winter (after drying up a bit more)
<epony>
you have approx 1 month
<epony>
to find a spare / replacement solution (or less)
<epony>
peekaboo! one more reason to dislike the winter cold
<epony>
you get the electronics failures just when you can't fix things
<epony>
and that's nothing to do with "murphy's aphorism pseudo-laws"
gxt has quit [Ping timeout: 255 seconds]
gxt has joined #osdev
[itchyjunk] has quit [Remote host closed the connection]
dude12312414 has quit [Quit: THE RAM IS TOO DAMN HIGH]
dutch has quit [Ping timeout: 248 seconds]
dutch has joined #osdev
aws has joined #osdev
les has quit [Quit: Adios]
les has joined #osdev
aws has quit [Quit: leaving]
aws has joined #osdev
bgs has joined #osdev
dutch has quit [Ping timeout: 248 seconds]
dutch has joined #osdev
ryzerth has joined #osdev
ryzerth has quit [Quit: Leaving]
ryzerth has joined #osdev
Burgundy has joined #osdev
slidercrank has joined #osdev
danilogondolfo has joined #osdev
gorgonical has quit [Remote host closed the connection]
gog has joined #osdev
fedorafan has joined #osdev
ryzerth has quit [Ping timeout: 248 seconds]
foudfou has quit [Remote host closed the connection]
foudfou has joined #osdev
nyah has joined #osdev
gmodena has joined #osdev
sinvet has quit [Ping timeout: 260 seconds]
x8dcc has joined #osdev
GeDaMo has joined #osdev
craigo has quit [Ping timeout: 246 seconds]
<x8dcc>
hello, I added paging to my kernel as heat suggested but I am having some problems now. I filled the first entry of the table directory following the paging guide (https://wiki.osdev.org/Setting_Up_Paging)
<bslsk05>
wiki.osdev.org: Setting Up Paging - OSDev Wiki
<x8dcc>
before adding this, I was already using a simple heap allocation system directly on physical memory, and I think that's what broke after adding paging
<x8dcc>
the heap had a fixed start address and size, but now I am not sure how to modify it with paging enabled :(
<Mutabah>
Well, with paging on you now can lay out our memory however you like
<Mutabah>
e.g. Put the kernel image at -2GB (assuming 64-bit) and the kernel heap at something nice and round, e.g. starting right at 0xffff8000_00000000
<sham1>
I think the problem here is that the paging is being set up from inside the kernel, which is delicate to say the least
<sham1>
It's easier to just set up the paging before going to the kernel proper and then just have your heap of pages
<x8dcc>
my heap of pages? I don't understand that
<Mutabah>
I think sham1 means "have your heap be built atop paging"
<x8dcc>
yes, thats what I wanted to do
<x8dcc>
thing is that although the heap is being initialized and all that *after* paging, its still not right
<x8dcc>
pretty sure its because of the location of the heap
<x8dcc>
I can send the code btw, but I'm pretty sure it's just that I don't get how paging really works :/
<sham1>
Paging is weird to wrap your head around at firsr
<x8dcc>
I think I understand the overall concept, but it's hard for me to figure out how to work with virtual memory and what not
foudfou has quit [Ping timeout: 255 seconds]
foudfou has joined #osdev
<x8dcc>
for example, with what I have now (empty page dir except first table filled), could I initialize the heap in any address like I was doing before with physical memory?
<sham1>
Yes, although then you also have to allocate a page for your heap
<x8dcc>
well for example is hard for me to know when I have to allocate a new page
SGautam has joined #osdev
<immibis>
when you want to use memory in a page you didn't already allocate
<x8dcc>
immibis: what do you mean?
<Mutabah>
Virtual memory is backed by physical memory
<Mutabah>
A virtual page needs a physical _frame_ to be allocated and mapped to it to be used
<x8dcc>
I see, thank you
<immibis>
when you want to use address 0x12345678000 you have to map it to a physical page
<immibis>
if you want to also use 0x12345678500 that's the same page so you don't have to change anything yet
<immibis>
if you want to use 0x12345679000 you have to map it because it's a different page
<x8dcc>
I think I understand. Sorry but this is new to me
<immibis>
or you just guess a number of pages and map that many and when they're all used up you say the heap is full
wand_ has joined #osdev
Burgundy has left #osdev [#osdev]
wand has quit [Remote host closed the connection]
x8dcc has quit [Quit: leaving]
bradd has quit [Ping timeout: 248 seconds]
x8dcc has joined #osdev
weinholt has joined #osdev
randm has joined #osdev
SGautam has quit [Quit: Connection closed for inactivity]
craigo has joined #osdev
craigo has quit [Quit: Leaving]
bauen1 has quit [Ping timeout: 248 seconds]
Patater has quit [Quit: Explodes into a thousand pieces]
Patater has joined #osdev
heat has joined #osdev
<heat>
ornx, btw re: efi memory types. you can use a bunch of them. conventional memory is always unused, LoaderCode and LoaderData are your EFI application's code and data (plus by default your pool allocations' data)
<heat>
BootServicesCode and BootServicesData for boot services drivers and implementation, etc
xenos1984 has quit [Read error: Connection reset by peer]
<heat>
the only thing that matters here is whether you're at a stage where you can use these. If you're out of boot services, you can use BootServices* entries too (apart from FW bugs). If you're e.g in your kernel and not your bootloader, you can (probably) use LoaderCode and LoaderData
<heat>
then you also have ACPI reclaim which you may want to reclaim and use up after you got all the ACPI info you want from there
<heat>
I think linux defers usage of BootServices* until after runtime services are initialized as to avoid some buggy firmware out there
<heat>
oh yeah, that too: I assume it's possible to use RuntimeServices{Data, Code} if you're not using runtime services, but you're probably on your own in that regard
<heat>
ok no: "In all cases memory used by the runtime services must be reserved and not used by the OS. runtime services memory is always available to an UEFI function and will never be directly manipulated by the OS or its components. UEFI is responsible for defining the hardware resources used by runtime services, so the OS can synchronize with those resources when runtime service calls are made, or guarantee that the OS never uses those resources"
<heat>
Also typo!
<heat>
actually, 2 typos
Burgundy has joined #osdev
<\Test_User>
> must be reserved and not used by the OS
<\Test_User>
as in you can get more memory booting it off bios and overwriting bios data once you're done with it?
<\Test_User>
(since there's no uefi reserved memory there)
<heat>
\Test_User, sure there is
fedorafan has quit [Ping timeout: 248 seconds]
<\Test_User>
interesting...
<heat>
I think CSM still loads all the runtime drivers
<\Test_User>
or you mean as in still uefi-reserved - but then how would it be told to the OS, just a "don't use this ram path? or?
<\Test_User>
*patch
<heat>
yeah
<heat>
I mean, everything that needs to be reserved here could just be done as E820_RESERVED
<\Test_User>
fun, uefi stealing your memory even if you're not using its features...
<\Test_User>
though ig not a problem if it's just rom
<\Test_User>
or the likes
<heat>
no, it's ram
<\Test_User>
(neglidgible either way, so not a *problem*, but still I don't like unused stuff actively using anything)
<\Test_User>
hmm
fedorafan has joined #osdev
<heat>
in all honesty, you shouldn't be running without EFI runtime services anyway
<heat>
despite Open and NetBSD being so undermanned that EFI runtime services support is super recent
<\Test_User>
and why's that, if you've got drivers/whatnot to do stuff directly yourself?
xenos1984 has joined #osdev
<\Test_User>
anything special they offer?
<heat>
access to UEFI variables and flash
<\Test_User>
ah
<heat>
modern platforms usually lock flash behind SMM so the only way to get there is to handshake with the SMM flash driver (which the runtime services do)
<heat>
also updating your firmware thru UpdateCapsule, and some other helpers
<heat>
ResetSystem is nice (but redundant given ACPI)
<\Test_User>
well, uefi variables doesn't really need to be runtime, you could just copy them over at startup? (unless there's some that actually chnge at runtime or smth weird like that) though sure on flash/etc, would be nice to have a "reload from flash or whatever" to reinitialize it and start runetime services as needed; just onto the "uefi could have been written nicer" list that one goes
<\Test_User>
*runtime
<heat>
at startup = when?
<\Test_User>
bootloader or whatnot, before your OS is running
<heat>
example: you're setting your BootOrder or your RandomSeed variable on shutdown.
<heat>
none of this is actually doable in your bootloader
<\Test_User>
ah setting
<\Test_User>
hmm...
<\Test_User>
true
<\Test_User>
still my latter of reloading available would be nice, but otherwise yeah
danilogondolfo has quit [Remote host closed the connection]
danilogondolfo has joined #osdev
sinvet has joined #osdev
bauen1 has joined #osdev
srjek has joined #osdev
bauen1 has quit [Ping timeout: 248 seconds]
bauen1 has joined #osdev
<epony>
Arm was tier 2 on the overstaffed FreeBSD too, and Linux got SMP unlocked (incomplete yet) and disk journalling at the same time (actually later) than FreeBSD.. and the containers was delayed by a whole 5+ years after the FreeBSD OS level virtualisation
<epony>
so "staffing" is only part of the problem, the other problem is bad loans from Linux and bad vendor preferences for Linux and FreeBSD because of featuritis and "enterprise" marketing at the cost of security failures
<epony>
and.. new shtuff being slow to materialise on the "second hand" consumer computers
<epony>
that said, OpenBSD has its own ACPI and DT support, and is the one that works on the RPi4 in contrast to Linux, NetBSD, FreeBSD and Raspbian/DebPi whatever Windows clone they are doing
<epony>
so.. find out more about OpenBSD's ACPI and DT support for Arm*
<nikolar>
openbsd never disappoints i guess
<epony>
well it has weak points too
<nikolar>
everything does
<epony>
but these are from the LinuxDRM side of imports
<epony>
and the lack of driver updates / porting capacity
<epony>
and changes breaking previous drivers remaining unnoticed because fewer people are testing on the hardware that they used to, mainly this comves from new and gamer developers that are fluff etc
<epony>
and applications targetting mostly Linux and moreso Windows, like Farefux and Chlamidium
<epony>
s/ Farefu/ Fearfu/
<epony>
and the "performance" being the most griping concern
<epony>
and the mitigations not being followed up fast enough by upstream applications, so "not attentive" and "proactive" external software
<epony>
but at least it has decent and modern crypto and export SSH and other important "infrastrcture" project releases
<epony>
can't rely on the commercial wannabe's for reliability and correctness (at all)
<epony>
nothing is perfect, but the goals and objectives are well defined and being tracked
<heat>
openbsd exclusively disappoints
<zid>
That's the same as never disappointing your disappointment
<epony>
so.. knock on wood lock fingers it keeps going, otherwise WSL might end up with SSH fuckups from Debuntu 0-entropy secret recipe of world domination
<epony>
teaching Windows to do W^X before Linux and FreeBSD pick it up for "evaluation"
<epony>
of course, it's not SMP unlocked and has no journalling file system.. yet
<epony>
but is work in progress (slooooow)
k0valski18891 has quit [Ping timeout: 246 seconds]
<epony>
due to insuffiecient skilled deveopers and thoughtful development practice from neophites
<epony>
someday in the distant future, when compilers become more meaningful (and less full of consumer features but more solid in implementation), validation and verification..
<Ermine>
heat: why?
<epony>
the C++ rewrite is not there yet either ;-)
<heat>
it's the erectile disfunction of operating systems
<Ermine>
harsh
<heat>
this is just a joke
<heat>
but I truly see no value in OpenBSD
<heat>
whereas the others seem to have at least *some* value
<epony>
nothing wrong with operating systems though, it's the people and their "specifications missing" development style of "adhoc" work
<heat>
hey, at least OpenSSH is good?
<epony>
there is value
<epony>
but you can't see it because you're unable to take part in that and you do not have these goals and objectives for your own projects
<epony>
for example these are not unique, but are part of the system, integrated, non-optional and they work, unlike "hardon-ing patchkits which are a lie"
<sham1>
OpenBSD has value. Mainly that it is a place to see how much mitigation you can get from a monolithic UNIX whilst being security conscious. Also being a research OS is valuable on its own right
<Ermine>
This is probably subjective whether one or another system has value, but I think I've got your point
<epony>
when you make Gentoo's hardon patchkit and FreeBSD haltening patch kits work.. make a note which applications run on those systems without breaking completely itself, the system and the compilers
<heat>
sham1, but in practice OpenBSD is just a bad operating system with poor security practices, dubious mitigations and a horrible, obscure code review process
<epony>
everything is bad, for your high standards
<epony>
of no standards
<sham1>
Meh, it's a bootloader for Emacs, like the rest of 'em
<epony>
actually, Emacs is usable, and it has "doas" support in some of its modes too
<epony>
might be an indication people are paying attention to it
<epony>
you know doas comes from OpenBSD and sudo lives in OpenBSD as long term maintenance too
<bslsk05>
isopenbsdsecu.re: Quotes and testimonies | Is OpenBSD secure?
<epony>
the applications are not fixing themselves on their own
<epony>
yeah, that site is full of the Sprundel old meta-data from meta-sploit
<heat>
"
<heat>
OpenBSD’s record of mitigations post-drunk-person-in-tent telling them about PaX is nothing to be jealous of. Find any security expert who writes exploits who thinks the silly syscall location “mitigation” makes any difference. Has Theo ever even written an exploit?"
<epony>
whose presentation does not work at the time of its presentation since he did not follow up on this findings collected on mailing lists of hardening kiddies
<epony>
well, the suggestion is: you write one and try it
<epony>
then report findings and see if you get any laughs or serious looks
<epony>
cause, the exploits are not coming all that much from the "grsexperts"
<epony>
tent-or-hike, the point is.. it has working improvements compared to the baseline, which Gentoo and FreeBSD do not have, and Linux does not have, and Windows is adding these too
<epony>
so.. Linux being so darn fast, can spare some percent for "optional security" being designed and developed in kernel too, not from outsiders who can't for the living soul of them make it work without breaking everything
<epony>
Linux has all the funding, all the developers, all the drivers, and all the security experts, why can it not have baseline "toggle on" security?
<epony>
but relies on external patch-kits which are shoddy at best (abandoned even by enthusiasts)
<epony>
it must be some "work to do" right, which others are doing.. so not everything is "words" and "empty value"
<epony>
work that Gentoo and FreeBSD and Linux in general do outside of the system, with super-breakage and inability to apply and use in real everyday use, and as a whole is not there, even though it's claiming the best in class security patching.. of a kernel
<epony>
missing the compilers, the applications, the system add-ons and context and locality aware layers
<epony>
what good is a super patch-kit that does not work and applications run insecure and break / don't run..
<epony>
what modernity is an operating system boasting 5-10 schedulers for the processes, io and networking, none of which can protect you from getting stuffed from remote abuse ;-)
<epony>
it's a funny world, when you think about "net effects" and "end results" isn't it
<epony>
must be "reserved territory" for enterprise vendors who want to be the only source of "quality and security" for commercial overages that are more expensive than anyone can afford individually and in small and medium groups of thousands of developers
<epony>
so.. who's interests are to call it "useless", yours or the big contractor resellers ;-)
<epony>
we need more "applications" that work reliably, and better patchkits that integrate cleanly instead of counter-defamation
<epony>
there is no fame in failure
<epony>
we need more sites and presentations that expose the weaknesses and insecurities and vulnerabilities of OpenBSD, so they are fixed once and for all, showing how it's done
<epony>
so, try more ;-)
<epony>
but it has defects and deficits too, obviously
dude12312414 has joined #osdev
<epony>
just pin them harder so everyone understands how weak and bad it is
bauen1 has quit [Ping timeout: 248 seconds]
<x8dcc>
heat: did you see the messages I sent earlier?
<epony>
so.. axe that "stein" to repeat / revalidate his publications and keep it up to date, it's probably very important work, that someone needs to take under serious consideration
<heat>
x8dcc, no
<x8dcc>
well I kinda added paging but now I can't use my heap allocator and I don't know where to map what lol
<epony>
ground breaking work on virtual memory ;-) nice
<bslsk05>
libera.irclog.whitequark.org: #osdev on 2023-02-07 — irc logs at libera.irclog.whitequark.org
<heat>
x8dcc, ok so it depends on what you want to do. a simple approach is to keep a large 1:1 mapping of physical memory and do your memory allocation directly on top of that
<heat>
page_alloc gives you some pages, you use those in your heap on top of the large direct map
<heat>
Ermine, np :)
<x8dcc>
I think I understand more or less
<zid>
It makes virt to phys and back just an add or subtract, too
<zid>
which is nce
<x8dcc>
and if in the future another process wants to use paging, I could still use it normally having a 1:1 map?
<heat>
you also have other ideas: i.e map things dynamically and use a temporary mapping or the 1:1 mapping to map things
bauen1 has joined #osdev
<x8dcc>
well I don't know what to use, I have been searching a lot but I can't find much
<heat>
x8dcc, yes, the idea is to keep your 1:1 map of $LARGE_AMOUNT_OF_SPACE out of the way of userspace, etc
<epony>
to conclude on the previous "tangent", this need a much stronger voice and a real debate between developers, not strawmen, like "Linux is obsolete" --Tanenbaum.. a scientific analysis and "valid clash of principles"
<epony>
the voice of stein is very faint
<x8dcc>
by the way, I forgot to mention but the kernel is 32 bit
<zid>
oh give up then
<zid>
memory mangement is impossible in 32bit
<heat>
ok, it's much harder but doable
<x8dcc>
well thats nice
<x8dcc>
I feel very motivated now :(
<zid>
just go 64bit
<heat>
so linux 32-bit just maps 800MiB directly (and does its memory management on that) or so and keeps the rest for temporary mappings for higher mem/MMIO
<heat>
it's up to you if you want to deal with this
<heat>
it's certainly an interesting problem
<zid>
Or yea, just support 512MB max memory for a while
<zid>
until you learn more, get ideas etc, and rewrite it to support more
<x8dcc>
why 512mb max?
<zid>
you can map it all
<zid>
without having to think about sliding windows, temporary mappings, blah blah
<x8dcc>
and why is it 512mb?
<zid>
because you have presumably a 3:1 GB user/kernel split, and the kernel needs to map stuff etc
<zid>
and 512 is a power of two
<zid>
so 512 is a nice minimum maximum
<heat>
yeah. so the linux solution is to divide memory into low mem (lower ~800MB) and high mem(the rest). low mem is directly usable by the kernel for slab, etc and is always mapped. high mem needs to be explicitly kmap()'d to be temporarily accessed
<zid>
768MB if you're feeling fancy
<x8dcc>
heat: I see, so that's very hard? I would really like to support 32bit :(
<zid>
windows 98 supports 512MB I think
<epony>
768
<x8dcc>
I don't mind 512 or 800, I just want to understand what I am doing lol
<x8dcc>
most information I find about paging is how it works _overall_
<heat>
x8dcc, it's not very hard
<heat>
it's doable, just do it
<zid>
that's why we're picking 512
<zid>
it's the simplest useful amount that you can just.. go write
<zid>
then you will know what was hard, what was easy, which bits can change, which bits are fixed, etc
<heat>
it's a bit of a pain in the ass, but that's osdev eh
<zid>
the first 3 times you write a thing are prototypes
<x8dcc>
all I implemented is what I sent earlier, whats on the "setting up paging" wiki page
<heat>
ok, so you followed the tutorial?
<x8dcc>
yeah I have that done
<heat>
don't.
<x8dcc>
oh
<heat>
the sooner you stop following tutorials, the better
<heat>
because they almost universally suck and now I don't know if you know paging
<x8dcc>
well it's the only example I could find
<zid>
Do you understand *why* you want paging?
<zid>
or do you just assume it's the next step
<x8dcc>
well 50/50 both, heat told me I should do it, and I also think it will be useful once I add more processes or whatever
<x8dcc>
not blaming heat btw
<zid>
I mean, do you understand what problem it solves
<x8dcc>
well a couple but for example fragmentation
<heat>
I did go through several uses of the MMU
<x8dcc>
yeah
<zid>
So what problem are you going to use it to solve in your OS? Would you have any way to test that you enabled paging correctly etc?
<zid>
or are you just following the steps hoping it all works out magically
<zid>
That's a big problem osdev has in general tbh
<x8dcc>
I don't have any specific problem I need to solve, as I said I just thought it would be useful in the future
<zid>
that's probably why you're finding it hard, tbh
<x8dcc>
and I tried debugging with gdb earlier but gdb is not working correctly for some reason
<zid>
Like, it's much easier to make something if I give you the problem and the tools, than it is if I only give you either the tools or the problem :P
<zid>
do you have processes at all yet or is that planned for later?
<x8dcc>
not yet
<zid>
Hmm
<zid>
I wonder if making some fake processes would be a thing you could do
<x8dcc>
right now my kernel does a bunch of stuff like setting up interrupts, framebuffer stuff, heap, etc. and then at the end call the shell I made (just a normal function call)
<zid>
so like, end up with 4 sets of page tables 'filled out' for some definition of filled out, and then swap to each and print some stuff inside each set
<zid>
then exit :p
<x8dcc>
uhm... right
<mrvn>
if you are only going to have a heap and not allocate memory to processes then just run on physical memory or map all memory at the start.
<zid>
the point is to give you a 'goal' and something to test and get working
<zid>
even if it ultimately isn't useful, the code you write to implement it will be what you want to learn how to write, and will probably be directly reusable
<x8dcc>
zid: I also think that's a good idea, I agree with you that it's harder to understand if you don't have a problem to solve
<epony>
more productive people provide not only a pointer to the problem, but a proof-of-concept of the failure, and a suggestion how to fix it, so it can be immediately resolved
<zid>
so my idea for a first step would just be to.. write some code that can map the kernel into memory somewhere then switch those page tables on, don't worry about being able to allocate memory off it or whatever afterwards
<heat>
x8dcc, do you want a specific problem to solve with paging?
<zid>
dying after you verify that you can just run code with the mmu on is fine
<x8dcc>
heat: well I am not sure haha, but okay
<x8dcc>
zid: I guess I can try that, but I have no idea where to search information about it. Thats probably my main problem, I am not even sure what I am looking for :/
<mrvn>
Tip for bootstraping: Add a 64k block to your .bss segment and make that your early free memory. Then you can parse the memory map and allocate the management structures to add the remaining free memory without running into a chicken & egg problem.
<zid>
x8dcc: well.. do you understand that page tables implement an integer to integer mapping?
<heat>
make it crash on replace_str("Hello World!\n", 'l', 'L');
<heat>
hopefully you understand that's ill formed
<heat>
x8dcc, please do say if there's anything you don't understand
<heat>
this stuff is hard to grasp
<x8dcc>
yeah it kinda is, thank you all for your patience
<chibill>
Question is there a guide to getting from boot all the way to say the 64-bit modes? Its a bit confusing looking around the wiki. (Since things change depending on if your in 32-bit or 64-bit mode)
<immibis>
x8dcc: every time the CPU wants to access a memory address it looks up the address in the page tables. Then it uses the address in the page table instead. Only the top 20 bits are looked up and changed. The bottom 12 stay the same.
<heat>
chibill, I don't think so. although there are some guides in getting into long mode
<zid>
chibill: Main methods are to either efi or grub boot a 32bit bootloader that sets up 64bit then jumps to kmain
<x8dcc>
immibis: yeah, thats what I understood, but it was hard for me to understand how to use that in my case. thank you
<heat>
no. main methods are either efi or doing it yourself. adding a rando stage is not standard
<heat>
x8dcc, so are you doing what I said or nah?
<zid>
there's nothing universal that boots a long mode elf, being the point
<zid>
you have to come up with something that works for your situation
<heat>
oh yeah, sur
<heat>
e
<x8dcc>
heat: well I am not sure what you want me to do, sorry. make sure the ptr is accessible?
<chibill>
Ah, so still have to do a lot of 32-bit stuff (aka set up everything enough to be able to switch into long mode)
<immibis>
x8dcc: if you don't have any particular use for paging, you can just pick a block of addresses, and tell the CPU not to change them (by telling it to change them to the same bits they already are) and then nothing changes within that address block
<zid>
64bit has the issue that paging has to be enabled, so it's easy to accidentally boot something that's trapped in a paging prison with no 'out' :P
<x8dcc>
immibis: I see, thats what I was doing before with my heap allocator
<heat>
x8dcc, make sure that crashes
<immibis>
that's not terribly interesting though. May as well leave paging turned off if that's all you're doing
<heat>
can everyone shut the fuck up
<x8dcc>
lol
<zid>
immibis: we're working him up because he's a little overwhelmed
jjuran has quit [Quit: Killing Colloquy first, before it kills me…]
<zid>
It's intentionally boring
<heat>
x8dcc, you do realize you can't write to string literals right?
jjuran has joined #osdev
<x8dcc>
yeah I do
<x8dcc>
I didn't notice thats what you were asking
<immibis>
zid: yes and I want to make sure x8dcc knows that it's boring and that it isn't the point?
<heat>
ok, so given the facilities the CPU gives you (hint hint *MMU*), make sure that doesn't silently work
<zid>
we covered that
<x8dcc>
heat: I understand what you mean now... I will try
<heat>
(this is a very real problem that the MMU solves. just putting it right in front of you in an obvious manner)
<heat>
x8dcc, good. what are you planning on doing?
<x8dcc>
I was going to search information :(
<heat>
that is a very vague description but go ahead
<bslsk05>
gcc.gnu.org: 59672 – Add -m16 support for x86
<zid>
peter anvin going off at the gcc devs because they can't always make it go first in a file like they need, and gcc people going "well it's a hack and we don't care"
<heat>
sortie, the POSIX committee cannot be trusted with making up new interfaces
xenos1984 has quit [Ping timeout: 248 seconds]
<epony>
clang is so accelerated only conservative operating systems use it
<epony>
as a thin frontend
<epony>
remember when compilers were not python web browsers?
<epony>
let's make the distinction between Clang and LLVM, after all MLIR is on its way to hit the fan
<kof123>
eh, not sure i see a posix problem. why are people writing such code if error returns are all positive? sounds like people going outside of posix
<epony>
GNU is weird
<kof123>
that is not to say either way on whether posix is good or bad or can be trusted, but i dont see an issue there
<heat>
this is 100% a posix problem.
<heat>
1) every function has returned -1 for errors since EVER
<heat>
2) now they make up new functions, but they switch up the return value style
<heat>
3) confusion
<kof123>
ok, that is different then, thanks for background
<epony>
-129
<mjg>
-ELUL
<sham1>
Sadly POSIX is very hard to avoid in osdev, at least if you want to be self-hosting, since you need to somehow port a compiler
<epony>
"applications"
<kof123>
POLA violated
<mjg>
port a real compiler, like msvc
<sham1>
Sure, let me break into Redmond first
<kof123>
beware the corpses of people who died holding ACPI spec or whatever
<epony>
it's called Fedhat
<sham1>
I'll steal an ACPI spec as well, why not
<kof123>
<pitfall IIRC animation, stick figure swinging across rope>
* mjg
burps
<sham1>
Maybe a PCI-e would also be useful to grab…
<sham1>
But seriously, unless one wants TCC, you need something that is Giant, Nasty and Unavoidable
<sham1>
Which in turn most likely requires POSIX
<gog>
do what haiku did and implement POSIX as a second-class subsystem
<gog>
but well-enough to self-host
<heat>
yeah or windows
<mjg>
or linux
<heat>
roasted
<gog>
every sane operating system
<sham1>
Doesn't really suit a μkernel, but that is indeed one possibility
<sham1>
Can't really take advantage of the async
<x8dcc>
heat: I am back. I read a lot of information about the mmu but I could not find anything
dude12312414 has quit [Quit: THE RAM IS TOO DAMN HIGH]
<heat>
x8dcc, did you see that bit that says W or writable?
<x8dcc>
yes
<heat>
what do you want to do, again?
<x8dcc>
make it crash, so in this case read only
<heat>
yes
<heat>
so?
bauen1 has quit [Ping timeout: 252 seconds]
xenos1984 has joined #osdev
<x8dcc>
not sure what you want me to say, set that bit to 0 or...?
<heat>
yes!
<x8dcc>
I am not sure how the string literal is related to paging, sorry
<heat>
lets imagine your kernel is [0x100000, 0x101000, R-X (.text)] [0x101000, 0x102000, R-- (.rodata, string literals)] [0x103000, 0x104000, RW- (.data)]
<sham1>
The string literal is stored in memory. You map the memory with paging
<heat>
are you following me?
<sham1>
Does that help?
<heat>
hello?
<x8dcc>
heat: yes
<heat>
ok. so given that paging gives us the tools to protect memory, what do you think can be done with paging here?
<heat>
in 32-bit x86 all memory is implicitly executable btw, so don't think about the 'X'
<sham1>
*
<x8dcc>
well map those addresses somewhere and decide if we want to enable the write bit in each section
<heat>
yes
<heat>
so what's your question?
<x8dcc>
wdym
<heat>
you said you don't see how paging is related to string literals
<zid>
"I don't get it" *stuff*" "Oh I see" "Do you get it now?" "No"
<zid>
:D
<gog>
just fight with the concept for awhile
<heat>
knock the mofo out
<gog>
it took me a long time to grok paging
<zid>
I haven't even gotten change to link my cool webpaget yet
<heat>
jab jab dodge jab and then make the mofo KO
<zid>
heat do you want to make me a 32bit version
<sham1>
It's called paging because it takes notebooks upon notebooks of diagrams and drawings to understand :p
<bslsk05>
github.com: Onyx/linker.ld at master · heatd/Onyx · GitHub
<x8dcc>
never used readelf
<heat>
x8dcc, do you see the _data_start = .; _data_end = .; ?
Left_Turn has joined #osdev
<zid>
that's readelf -l --wide kernel.bin on my end
<heat>
they place symbols at that location in the linker script
<x8dcc>
heat: yes. I had a look at your os when you sent the gist earlier, btw (not the linker script)
<zid>
It loads two groups of sections into memory, group 00 has .test and .rodata. Section 0 is at 0x1000 int he file, gets loaded to 0xffffffff800000000 and is read+exec and 0x30c3a long
<heat>
so it will be _data_start = .; [contents of .percpu, .data, .bss] _data_end = .;
<zid>
group 01 is at 0x32000 in the file, has .data and .bss, gets loaded to 0xffffffff80031000 has flags read + write, and is 0x300001 long
<x8dcc>
zid: I see, so all that is because of the linker script as well?
<heat>
you can then access these from C. like "extern unsigned char _data_start; // address (&) of _data_start will be the start of .data"
<zid>
yea it's a lot tidier than the default script my linux desktop would hav used
<bslsk05>
github.com: boros/linker.ld at master · zid/boros · GitHub
Turn_Left has quit [Ping timeout: 248 seconds]
<x8dcc>
heat: so you can read that from c... I am going to mess a bit with that, see if I can finally do what you told me
<zid>
matches the readelf output, obviously
<heat>
x8dcc, note that it's not "extern unsigned long _data_start; // use the value of _data_start" since the symbol's value is its address
<zid>
yea extern symbols fromt he linker are odd
<heat>
if you actually do _data_start = 0xDEADBEEF; you're just corrupting whatever is at .data's start
<x8dcc>
and why is it a char?
<sham1>
I do find it annoying that the linker feels the need to merge .text and .rodata for some reason
<gog>
i have a relocatable kernel
<heat>
no particular reason, it's just that it's more correct
<gog>
i'm fancy
<zid>
gog: Fancy pantsy
<zid>
You should relocate my kernel, but you'll have to implement rand() first
<heat>
x8dcc, one of the important bits to note here is that given a linker script like this, if you do . = 0x100000; you're setting the physical base of the image, then every section you place in the SECTIONS is more or less in-order and almost contiguous
<gog>
zid: dang
<gog>
ASLR?
<x8dcc>
heat: so when you declare the extern unsigned char, that will hold the address of _data_start?
<zid>
well why else would yu wanna relocate it
<mrvn>
heat: I prefer making the linker script symbols arrays so they decay into pointers
<heat>
x8dcc, no, &_data_start
<gog>
i mean yeah that's in the plan
<heat>
mrvn, yeah that also works
<zid>
so y ou need to implement rand
<gog>
eventually
<gog>
when i care about that kind of thing
<zid>
let's implement a qemu rand driver
<gog>
i'm still a bit rudimentary
<zid>
in dl, ax or whatever
<mrvn>
it's too bad you can't have a void _data_start[]
<zid>
and just read 32 bits from /dev/random
<heat>
zid, rdrand moment
<heat>
or virtio-rng
<zid>
oh someone already made it
<gog>
i was going to look at how haveged works
<zid>
have ged?
<x8dcc>
heat: oooh, I think I understand, so it's not a pointer to the symbol, but the char would be placed there?
<gog>
i do actually
<sham1>
gog: wait, so you're doing PIC? AMD64?
<gog>
sham1: yeh
<gog>
it actually works somehow idek
<mrvn>
gog: it just dumps some pseudo random bits into the entropy pool
<gog>
mrvn: ah i see
<heat>
x8dcc, yes, the "char" is placed right at the beginning of .data. so &_data_start gets the beginning of .data
<sham1>
mcmodel=large?
<x8dcc>
heat: I see, I see. Very useful
<zid>
gog: did you have any issues with needing to relocate stuff?
<zid>
or did you manage to get it flat with no got
<heat>
x8dcc, "placed". because the char doesn't actually exist nor does it get memory allocated. as I said, _data_start = 0xdead; will corrupt some other bit of data
<x8dcc>
yeah, I see
<x8dcc>
thank you
<zid>
I always fuck up my linker etc invocations when I work with pie and end up needing a GOT even if I don't want one and have to fiddle around with double linking and stuff
<heat>
np
<gog>
it works without a got
<heat>
happy to help
<zid>
is that pie vs pic idk
<gog>
i have totally forgotten how though i don't fk with it
<sham1>
Does a statically linked PIC executable even need a GOT on AMD64?
<sham1>
Because I'd imagine that it wouldn't
<zid>
I think that's the trick sham, fPIC and static
<gog>
it doesn't
<zid>
not fPIE
<gog>
yes
<mrvn>
zid: no, it's different flags and archs behaving different and GOT only getting eliminated sometimes
<heat>
isn't it -static-pie?
<x8dcc>
anyway, I will probably try all this tomorrow since it has been a really long day. Thank you all for your help and patience <3
<zid>
maybe heat!
<zid>
like I said, I always fuck this up
<sham1>
I would like to have some pie right about now
<bslsk05>
github.com: sophia/kc at main · adachristine/sophia · GitHub
<gog>
defaults.mk and all that code has reloc stuff
<gog>
and kc.ld
<zid>
-pic
<zid>
-shared
<zid>
-pie
<zid>
WHAT
* gog
shrug
<zid>
Now I am less certain than when I started
<heat>
x8dcc, <3
<gog>
i just kept doing things until it worked lmao
<sham1>
Story of my life
<zid>
gog: That's my usual plan
<zid>
double linking seems easier
<sham1>
The more I learn and talk here about this stuff, the more I learn that I know nothing. I suppose I should travel back in time and take Socrates' place as the wisest of all the Greeks
<gog>
i'm a really bad programmer
<zid>
binutils is fucked up
<mrvn>
zid: what do you mean by double linking?
<zid>
is the main problem
<gog>
and i only get things right through a stochastic process
<zid>
and ELF reuses fields in weird ways to achieve things like aslr/pie/shared/etc
<mjg>
heat: how about we tank sortix while pretending otherwise
<mjg>
i'm gonna propose a committe-based mandatory code review before any change goes in
<heat>
oooooooooooooooh
<heat>
or private code reviews
<heat>
ok sortie@
epony has quit [Remote host closed the connection]
<zid>
I'll join your startup as a code reviewer
<heat>
commit messages larger than 20 characters are banned and get auto rejected
<sham1>
zid: that video was amazing
<zid>
I'll break out my "This is shit, stars go after the space not before, rejected" stamp
<heat>
... by someone's obscure perl script
epony has joined #osdev
<mjg>
what
<zid>
sham1: he needs to up his glowplug voltage in his bios imo
<mjg>
it is commit messages shorter than 80 lines which get rejected
<mjg>
you describe your work motherfucker
<heat>
you know, I do like "ansi" as the whole commit message
<heat>
so msg length < 8 || msg length > 8000
<mjg>
"ansi" for a commit message which k&r-ifies a func
<mjg>
anyhow review meetings are a bi-weekly afair
<zid>
futa weekly
<heat>
review meetings in microsoft teams biweekly together with the largest computer manufacturer consortium, the UEFI forum
<epony>
you'll make netBDS[m] usable by fixing the cross compilation targets to machines that exist, and find out that's been non-native compilation for 25 years, and the world will choose to use something else for it's world heritage ports tree of unportable packages like "migrantutu unportage the deported have departed"
<epony>
FreeBSD mostly works except when using ZeroFileSeastem
<epony>
"fail"
<epony>
C-stern
<epony>
OpenBSD does not work, as you rightfully adjusted the historic record today, you work around it to make it "cross compile on Arm which has issues with the modern compilers and the older compilers too"
<heat>
mjg, omg openbsd is still on binutils 2.17
<zid>
pfft, me and heat are on 2.40
<mjg>
are they using it?
<mjg>
i do suspect they keep it in the repo
<mjg>
but the toolchain is probably all llvm now
<heat>
yeah, last commit 3 days ago
<mjg>
that does not mean they use it
<heat>
Make ld.bfd default to --execute-only on amd64.
<heat>
ok deraadt@
<epony>
so, for the sake of "world peace" just use Plan6 which is the system that 6oogle uses internally for its superior not-work fail-safety protocol 6pee
<mjg>
lol
<mjg>
ok, that is kind of telling
<zid>
that seems fun
<zid>
bundle binutils with your project, and just default it to your project's flags
<zid>
so that you can do gcc kernel.c -o kernel
<Ermine>
Woah. Even Minix is on 2.23
<gog>
omg
<epony>
because GNU soup contains brainworms
<gog>
i know why bug
<zid>
nice boog
<gog>
T_T
<zid>
I figured boog is better than gug
<gog>
i'm using the wrong params
<Ermine>
bugie wugie
<gog>
i forgot how my own alloc function works
<zid>
that'd do it
<epony>
"dud"?
<zid>
can you make a patch that catches future instances of that
<heat>
mjg, I think --execute-only is actually ok. AFAIK it just makes .text X only, instead of R-X
<epony>
name that acatchy client and start no less than 22 foundations about it
<mjg>
i have no opinon on the commit
<heat>
also seems like they do indeed default to llvm nowadays
<epony>
btw, did you know that you can register a foundation for your new commit tag?
<bslsk05>
marc.info: 'Re: [PATCH 1/2] proc: fix to check name length in proc_lookup_de()' - MARC
<mjg>
look at that fucking routine, holy shit
<mjg>
fortunately they unfucked it at some point
<epony>
in FreeBSD changelogs: sponsored by "fApple", controlled by "coIntel" and.. approved by "Scroogle" (procfs drop when?)
<sortie>
mjg, heat, literally my $DAYJOB is to strictly enforce supply chain security including mandatory code review that not even admins can bypass without approval
<sortie>
I laugh at OpenBSD's code review practices
<sortie>
Oh yes hey let's just have -CURRENT builds that contain whatever sources we're testing today on top of the latest official code
<sortie>
Oh hey yes if someone supposedly said ok on a mailing list you can commit whatever hunk you got
<epony>
that's called snapshots, smartypants
<epony>
try to say that OK without being a developer in that project
<sham1>
Theo will materialise through your monitor to slap you?
<epony>
or better, just commit your proposal by dropping a nameless patch on the misc@ mailing list
<sortie>
My point is that the code review practices we use for work are now much much stronger :)
<mjg>
i don'tk now if your comliant is valid
<epony>
maybe your work produces stronger security than OpenBSD too, but let's see your SSH port
<mjg>
you don't know how much effort went into an "ok" on the list@!
<sham1>
Google SSH. Discontinued after couple of years and doesn't even work right
<epony>
butt hay, Google did BurningSSL
<mjg>
is that areal thing?
<zid>
google layoffs is their current beta
* mjg
never heard of it
<zid>
it'll be discontinued soon though
<mjg>
your ass is discountinued!
<mjg>
ok mjg@
* zid
sits on his pelvis instead then
<sham1>
Well I certainly hope that Google's own SSH project isn't a thing
<epony>
Jello 3 little commits
<epony>
little diff little diff let me in
<epony>
burnt by the halo of a backed out commit
<sham1>
Jello commits
<epony>
yo asswipe your commit bit is mine
<sortie>
epony: https://gitlab.com/sortix/sortix/-/blob/staging/ports/ssh/ssh.patch ← Sortix doesn't have the sandboxing support just yet but here's my port. Although note that obviously I don't hold my hobby work to the same security standards as my dayjob (well actually I do but it's just not there yet)
<sortie>
mjg, the problem is that the ok@is not formally enforced, so one can commit something else than was actually reviewed
<sortie>
And that might be missed
<sortie>
Or even if they get discovered, they might manage to damage or infect CI immediately
<mjg>
:)
<epony>
yes that happens when only one krw@ is working on some fdisk
* sham1
commits straight into main branch instead of feature branch
<sham1>
Oop
<sham1>
oops
<mjg>
sortie: but what if the ok was given at a much higher standard than what you are doing
<mjg>
sortie: then, short of malice, you are worse off!
<epony>
but they are still reviewed
<sortie>
Yeah my point is that code reviews should be done in a proper tool enforcing mandatory code review
<epony>
maybe
<epony>
with drops of blood for validation
<mjg>
sortie: ok mjg@
<sortie>
Right now the supply chain security of the Dart programming language I work on at my job is really really good
<sortie>
The source of truth is not at github but rather googlesource.com which allows us to write exact ACLs that are strictly enforced
<epony>
I think for OpenBSD it's in a couple of CVS servers
<epony>
so GH practically is just a "repeater"
<sortie>
That includes the mandatory code review rules which are set up to enforce mandatory review. It's even configured to not allow admins to bypass it. Because we're not admins.
<sortie>
To perform any sort of admin work, we have to get the admin action reviewed as well
<sham1>
That's my personal biggest problem with OpenBSD's process. Of course, git has its own issues especially for a project which wants to avoid GPL like the plague, but they could probably go with something like Mercurial if they really wanted to
<sham1>
Oh wait, that's GPL too. Damn it
<sortie>
That means every commit has been reviewed by at least two employees (or for an community project, two trusted people)
<sortie>
sham1, they actually hacked up their own git got
<sham1>
Yeah, and it's gone about nowhere from the looks of it
<epony>
most probably such proposals would be met with heartily greets and cheers on misc@ use suJBeck't: Fixing AnonCVS to AnonGIT repository conversion
<sortie>
The really cool security work I'm doing is making really strong guarantees that the final released binaries match the official reviewed source code and there's no way any insider acting alone can tamper with them
<sham1>
But this is you trusting trust
<epony>
you're doing it on Github or or own repositories?
<sortie>
epony, yeah we're using our own company git servers which are much more secure than github
<epony>
fine on that point then
<epony>
but why git?
<sortie>
sham1, very few people have root in these systems, and ideally any admin actions must also be reviewed
<epony>
like are you a distributed team..
<sortie>
Yes and we're an open source project too
<sortie>
git is the thing
<epony>
the sortix I keep hearing good thngs all the time?
<sortie>
Ah I'm not at all talking about Sortix here
<sham1>
Sortix in dart when
<sortie>
sham1, to separate church and state, someone else should do that
<epony>
CVS was "GNU" plague.. so
<sortie>
Although BSD Dart is complicated by our ugh shitty approach to portability
<sortie>
And also that Dart requires Dart to bootstrap
<epony>
it's more about "the textual repository files" and "central repo" than.. must be CVS precisely
<epony>
but then think about it, it's a small team
<sortie>
Sortix currently uses github because it bought the small git host I used many years ago
<epony>
they can't recreate feature complete longopt doofus software and fix all of its complexity
<sortie>
*GITLAB
gxt has quit [Remote host closed the connection]
<sortie>
(Sortix does _not_ use github)
<epony>
so in 1995.. CVS was still a thing
<epony>
GIT did not exist back then
bauen1 has joined #osdev
<sortie>
Sortix technically can be committed to without me, just two trusted people need to review it, and we got three including me
gxt has joined #osdev
<mjg>
sortie: why not a commitee
<epony>
so.. what's the advantage of gitting it fecked up like FreeBSD got screwed themselves with Perforce the enterpise failure and SVN the mega migration to GIT 5 times for a lucky charm
<sortie>
With Sortix you get a promise that either I reviewed (or wrote) it or that two people reviewed it
<sortie>
mjg, honestly we're just 1-3 developers depending on the day
<epony>
or Lindows with their Bitkeeper fantastic choice of marketing debacle needed to spawn Git into existence, like can't happen without Darma
<sham1>
SCCS
<Ermine>
RCS
<sham1>
perforce
<epony>
so OpenBSD has these half-baked even OpenRCS and OpenCVS which are even more limited and incomplete, and also OpenRSYNC too
<sortie>
lemme tell you about piper, <classified information>
<zid>
SVN: CVS done right"
<zid>
Best tagline ever
<epony>
which come from some bsd.lv dev kristaps dzonsons
<zid>
Being disemboweled, the RIGHT way!
<sham1>
For all its flaws…
<sortie>
I absolutely need to set up git.sortix.org hosted on Sortix
<Ermine>
still being hated
<epony>
who also wrote manding doc a.k.a mdoc the man-dog
<Ermine>
sortie: cgit?
<epony>
and some other shtuff
<sham1>
I'm personally partial towards gitea
<Ermine>
one can try sourcehut also
<sortie>
Ermine, yeah someone like that
<epony>
I can only recommend you use what you know and can handle well that suits your project
<sham1>
Oh yeah, I do have a sourcehut thing. Should probably use it for more than just my dots. Might actually be worth the 100 USD
<epony>
so no "this better than that" debacles.. just what gets the job done predictably well
<Ermine>
sham1: i mean self-hosting
<kof123>
"Sortix in dart " which one of these is the church and which is the state?
<Ermine>
Sortix in darth
<sham1>
darth sortix
<epony>
eventually, you end up "working" on the tool you're using too
<sortie>
kof123, I never got around to set up a wacky Sortix religion
<sortie>
All the best hobby operating systems got one
<epony>
so picking a tool that you would be able to work on, is a priority to not overlook
* sortie
idly ponders if the big G is the one true religion and whether he is a non-believer
<epony>
preemptively re-implementing it as a minimal proof-of-concept that works ;-)
<sham1>
sortie: the holy alphabet
<epony>
then.. when you port a couple of "tools" we'll see which tools you pick first from the toolbox next time..
<aws>
An OpenBSD developer is developping got (game of trees), i wonder if they'll end up using that
<bslsk05>
gameoftrees.org: Game of Trees (Got): Goals
<mjg>
aws btw i tried it out recently
<mjg>
in unix fashion their 'got log' does not spawn a pager
<mjg>
but more importantly it failed on my test repo
<sham1>
TBF, I'm not sure whether pager'd output would be the best for a git-style log system anyhow
<sham1>
Too many branches and all thatr
<epony>
if you ask my personal opinion, I think the git is not an option.. but an OpenGITD or AnonGIT or something like that.. which is too much work
<epony>
for something that is already available since 1995-1996 and kind of works
<epony>
especially with the brought in OpenSSH and such
<sham1>
tmux
<epony>
is nice ;-)
<sham1>
I use kitty. It does splits for me and IMO in a nicer manner than tmux
<epony>
I have my points towards it, but.. won't entertain everyone outside "string safety"
<epony>
and "other multiplexors are needed too" (for audio, video, sockets, etc..)
<sham1>
Then again, in the list of eccentricities is that I might be one of the only people left using tcsh
<epony>
"focking X" and what not
<epony>
you're from the FreeBSD land and you've become accustomed to it
<epony>
on OpenBSD it's outside "scrutiny" so you're on your "own"
<epony>
can be understood and judgement avoided etc.. to each their own shell
<sham1>
I do also maintain OpenBSD's ksh port on Fedora. I should probably bump it one of these days
<epony>
oh cool
<epony>
"don't listen to fake talks" like "this bad etc" tools must be present between systems and that's a benefit to everyone if they are being worked on / ported etc
<aws>
mjg: oh ok; i went to the developer talk at fosdem so it caught my attention but i haven't tried it yet
<sham1>
Of course, tcsh in particular can be annoying because, being a csh, it doesn't know how to do things like just redirect stderr. One can deal with it, but it's certainly annoying
<mjg>
lol @ trcsh
<mjg>
are you an old unix luser
<epony>
csh improved with tcsh ;-)
<sham1>
Not even that old, although I certainly have old tastes
<epony>
but.. it's a long story which everyone has read / knows
<epony>
about the shell needing to be different than the system programming language so it can do shell things instead etc
<epony>
that's the whole point, to have a "more scripting" ready system, but then there is also the "convention" camp separation
<epony>
between system programming like (shell for programmers) and system programming unlike (shell for users)
gildasio1 has quit [Ping timeout: 255 seconds]
<epony>
and then it's talked into "but faults, incompletions, defects, nonsense, lies etc my dog ran away because of your shell fuckup bill"
<epony>
the reality is, make both conventions side by side equally good, let users pick, see what advances forward (keep the other around) etc
* geist
yawns
gildasio1 has joined #osdev
<chibill>
I feel like I am going to be pulling my hair out at some point on this journey to start writing my own OS. Although I wonder if I should work on my own OS for https://wiki.pine64.org/wiki/Ox64 or on an ESP32 instead of writing one for x86-64
<bslsk05>
wiki.pine64.org: Ox64 - PINE64
<sham1>
It's like, it's not like I couldn't get all my shell needs done with something like zsh or bash, but I dunno, I like the difference
<epony>
3 plats 1 write
<epony>
sham1, keep at it if you like it, nobody to know better than you
<sham1>
Mmm
<epony>
I've used it too ;-)
Terlisimo has quit [Quit: Connection reset by beer]
<epony>
the benefit of supporting more than one platform is that you learn where the machine dependent and machine indepent separation-handling goes
<epony>
and find your endianness and signedness and word width bugs
<epony>
you can be smart about it and do your "OS shim" on top of another OS
<epony>
or.. directly port out / change that other system you like / disapprove
<geist>
chibill: well, the riscv esp32s would be a lot easier to work with than the pre-riscv ones
<geist>
since the previous ones are a semi-proprietary architecture
<geist>
i'm not sure the esp32-c3s are that well documented
<geist>
but one thing you'll find is it's almost certainly a 32bi only, no mmu (ie, machine mode + user mode only) riscv
<epony>
wouldn't it be nice if the compilers came with standards compliant reference implementation to base your work on ;-)
<geist>
so depending on if that's what you want
<chibill>
Yep, can even write in rust if i wanted xD
<geist>
but back to the original statement, why do you thin you'd pull your hair out?
<geist>
frankly x86-64 is such an extremely well trodden path, there are a bazillion piles of resources for it online, tutorials, examples, docs, etc
<geist>
it's pretty straightforward all told
<epony>
you know compiler writers will one day write their compiler and.. then have to have a thoroughly demanding use case to validate their work (self-host it)
<chibill>
Oh. at least on x86 theres all sorts of stuff that have mixed documentation that is unclear. (Even on the osdev wiki)
<geist>
well we're also here to help if you need help
<sham1>
:)
<geist>
problem with the esp32 is there's a bunch of not documented too, i think
<geist>
i have a few of them and poked at it a bit and was dissapoint
<chibill>
Other problem is I have no idea where to start, I know last time I tried someone recommended me to ignore the 32-bit modes and go straight to long mode.
<geist>
the esp32 programming strategy is to use their libraries
<geist>
well, first establish what you want to do. what's the goal?
gog has quit [Quit: byee]
<chibill>
I use esp32's alot. had a M17 Reflector running on it. End goal, mostly just have something to play around with, long term I am going to need to write a kernel for my homebrew CPU / System.
gog has joined #osdev
<geist>
yah but as i was saying esp32s are unforunately not that friendly for low level hackery
<geist>
i'd consider them a tier 2 or 3 embedded target
<geist>
vs the usual say STM32 or whatnot where the documentation is clear how to run bare metal on it
<geist>
i'm sure you can run bare metal on esp32, but you have to undo their layers of libraries that abstract the hardware
<geist>
and then it wont have paging and whatnot, but also comes back around to what do you want to do with your kernel
<geist>
ie, are you interested in a modern paging system? preemption? SMP? eventually?
gorgonical has joined #osdev
<chibill>
No idea. Thats the thing, this is just one of those, this would be cool to work on type of things. (Thats what most of my projects are.) I do plan on experimenting with paging and multi-processing (as in task switching) since those will be possible on my homebrew system. (Actually based my paging system on x86's a bit)
<chibill>
Alot of the programming I do, I do just to have the experience of doing it.
<geist>
on your homebrew you mean you're going to write an emulator?
<geist>
or do yo umean homebrew kernel?
<geist>
btw i'm not being picky here, just trying to probe the solution space you might be interested in so i can possibly give you some pretty concrete recommendations
GeDaMo has quit [Quit: That's it, you people have stood in my way long enough! I'm going to clown college!]
Terlisimo has joined #osdev
<chibill>
Well, I am writing an emulator now but the plan is to built the CPU in actual hardware ,FPGA or maybe just emulate it on an Arduino with enough pins for the buses.
elderK has quit [Quit: Connection closed for inactivity]
<chibill>
I do have a tiny bit of experience since a few years ago for a college class I had to do work inside xv6 (adding new syscalls and also stuff with the filesystem. Also had to detail the path the CPU took when a specific syscall was done with a specific parameter. )
<geist>
ah. well in that case yeah start with the emulator ad then build your OS on top of that
<geist>
in which case the physical hardware you run on isn't that important
<geist>
or at least the low level details dont matter much
<gorgonical>
I am doing things with screen I shouldn't be doing
<gorgonical>
Somewhere, some unix greybeard screams in pain
<kof123>
does this mean instead of tux/etc. plushes you need a little....greybeard and a set of pins
<bslsk05>
github.com: lk/virtio-gpu.c at master · littlekernel/lk · GitHub
<gog>
nope, it's mad slow
<zid>
neat
<zid>
you can double buffer and hw scroll o the bochs vbe device easily at least
<clever>
gog: so rather then the framebuffer acting as mmio and exiting out of the emulator on every pixel, you can draw it all while staying in emulation (better cache hitting all around), then burst the frame to the host using a host memcpy
<zid>
you get to set the scanout position and oversize the buffer, all you need
<Ermine>
what is hw scrolling!
<Ermine>
s/!/?/
<zid>
Ermine: Think of it like a window looking out onto a gigantic painting
<zid>
you can move the window around to see different parts of the painting
<clever>
Ermine: you can configure the framebuffer to be say 200x100, but then tell the gpu to display a 100x100 slice, starting 50 pixels down from the top
<zid>
(rather than having to redraw the painting that can be seen behind the window)
<Ermine>
so hw can do this for me?
<gog>
yeh the video hardware just has a viewport the size of the display
<Ermine>
okay
<gog>
and you can anchor it to any point on a canvas bitmap
<zid>
yea, so all you're doing is writing a new top left corner to a register somewhere
<clever>
but when you hit the bottom of that virtual buffer, and you need to wrap back to the top, how does that work?
<gog>
mmap
<clever>
can the viewport wrap back to the top, or do you need to memcpy then?
<gog>
i guess it depends on the hardware
<clever>
yeah
<mrvn>
clever: you write everything to y and (y + 100) % 200 so when you scrolled down to 100 you just jump back to 0.
<netbsduser`>
finally got around to something i should've done a while ago
<netbsduser`>
swapping
<clever>
mrvn: yeah drawing twice is one option, so when you snap back to the top, its already drawn
<clever>
in the rpi hardware, you can basically just have 2 viewports, so thats how i solved things
<gog>
but yeah one of the main reasons it's slow is that i'm not double-buffered
<zid>
mario for nes is one giant horizontal slice of level for example, and it uses it in a funny way to let you walk on the title screen
<bslsk05>
'Chaos, 13 sprites randomly bouncing around' by michael bishop (00:00:12)
<clever>
this demo was running on the arm side, but controlling the HVS (composition hw) directly
<clever>
along with the stock firmware, which did hdmi init for me
<mrvn>
clever: guess I have to do that at some point then. Drawing everything twice is kind of wastefull
<clever>
the only problem, is that i dont know how to tell the firmware "no touchy", so if a low-voltage alarm occurs, the firmware restores the rainbow framebuffer
<clever>
until the arm kernel redraws again
<zid>
I need a new usb micro-b cable
<zid>
why the fuck does amazon want £9 for it
<mrvn>
zid: because you accidentally went to amazon.uk
<gog>
why does kms only have 32 sprites?
<mrvn>
gog: because hardware is limited
<gog>
oh
<zid>
£8! but it's 4.5M long lol
<gog>
no i mean like if the hardware could do more than 32 sprites
<bslsk05>
blog.benjdoherty.com: Exploring Hardware Compositing With the Raspberry Pi
<mrvn>
gog: The chip has to check 32 sprites and composite them for every pixel. If you actually have that many sprites at the same position the chip can't even keep up and you get output errors.
<bslsk05>
'ntsc dance v2, interlacing fixed' by michael bishop (00:00:22)
<clever>
in this case, i underclocked it severely, so i could measure where the limit is
<clever>
for something like the (s)nes, its all running in lock-step on some global clock, and it has to fetch sprite image data and composite things on the fly, staying in perfect sync with the electron beam
<gog>
ok, i was confused, i thought you were referring to KMS having 32 sprites , not that chip being limited to 32
<clever>
and because every action can be measured in exact clock cycles, they know where the limit is, and just dont draw any sprite over the limit, for a given scanline
<clever>
kms is limited to 32, because most hw has lower limits, and linux wants to use a bitmask to manage things
<clever>
the rpi is a weird case, where it can handle ~290 sprites, if you follow a series of complex rules
<clever>
linux just gives up, and sets the limit to 32, so the userland doesnt have to be aware of what makes the current platform special
<mrvn>
clever: just don't make them big and overlapping
<clever>
mrvn: yep, overlapping is the major cost
<clever>
if you cut the sprites up, and avoid overlap, you basically cant hit the hw limits
<clever>
but then you cant get alpha
dude12312414 has quit [Quit: THE RAM IS TOO DAMN HIGH]
<mrvn>
clever: well, at 16k your display list runs out
<clever>
290 sprites for the rpi, if you want to double-buffer the display list
<clever>
the display list is just a `uint32_t dlist[4096]`, and an unscaled sprite occupies 7 slots, plus 1 slot to terminate a list
<gog>
ok, NOW i understand XD
<clever>
for the rpi hardware, there is a fifo that holds ~3-4 scanlines of image data
<clever>
the pixel valve will throttle pixels leaving that, to match the pixel clock, and turn flow on/off during blanking periods, and generate syncs
<zid>
pfft, gameboy's fifo is only 8 pixels
<zid>
better latency
<clever>
any time the fifo has room for 1 full scanline, the HVS will parse the display list, and draw each image directly into the fifo
<clever>
4 pixels/clock for unscaled, 2 pixels/clock for scaled, and 1 pixel/clock for palette formats
<clever>
if 2 sprites cover a pixel, you waste clock cycles drawing over it twice
<clever>
if you cant keep up with the pixel clock, for ~3-4 scanlines in a row, the FIFO runs dry, and you get the glitching from the example i linked
<clever>
plus dram latencies, where it has to go out and fetch image data, and share ram with other bus masters
<clever>
its was more bursty then 8bit systems where the gpu is entirely deterministic
<clever>
but part of that, is because of the flexibility, it can drive up to 3 monitors, of basically any resolution/fps
<clever>
so they couldnt design it around certain specs
<nikolar>
that's quite impressive actually
<clever>
when driving multiple displays, each has its own fifo, and the HVS is just time-shared over them
<clever>
drawing 1 scanline at a time, on whatever fifo has room
<clever>
i have accidentally done 10,000 fps before, because it was only 100x100
<clever>
and i have accidentally done an interlaced setup, where one field was 1/60th of a second, and the other field was 2 seconds
<clever>
because i didnt initialize the 2nd field params, and the interlaced bool was undefined (randomly true)
<clever>
assuming unscaled sprites, 4 pixels/clock, and 500mhz, that means it can produce a total of 2 billion pixels/second, now just count how many pixels are in your output resolution, times the fps, add in each display, is that within limits?
<clever>
1280x1024@60 for example, 78 million pixels/second, 1/25th of the limit
<clever>
so at rated speed, it can handle about 25 sprites covering the entire screen, on 1 display
<nikolar>
1080p, 1/16th of the limit
<clever>
as-in, 25 sprites, each 1280x1024, all with per-pixel alpha blending
<clever>
the smaller your sprites, the more you can have
<clever>
this math is ignoring dram latency, i dont know how much impact that has
<clever>
it also doesnt take into account dram bandwidth, and whatever your stealing as you try to build the next frame, if your editing the sprite images
<clever>
most of my demos cheat a bit, and use a static sprite image
<mrvn>
clever: you should use Bad Apple
<clever>
for video playback, i think the problem would be more in the decompression stage
<clever>
but the rp2040 can do it at 40mhz? i think it was, so it cant be that hard
<nikolar>
maybe we should come up with a qoi of vodeo decoding
<clever>
also, i think the rpi supports 1bit palette formats
unimplemented has joined #osdev
<mrvn>
clever: only way to do black/green or black/bernstein
<clever>
oh, and there is something i can do, for bad apple specifically, dont use full frames
<clever>
if the image is 90% white, and there is just a small black object
<clever>
make 1 sprite for that black object, set the bg fill color to white
<bslsk05>
'2d and 3d demo' by michael bishop (00:00:21)
<clever>
if you look very closely, you can see some tearing at the top of the frame, when the triangle tip passes by
<clever>
i render to A, issue a command to swap to A, then render to B almost immediately, while B is still in use
<clever>
flawed double-buffering logic
<clever>
mrvn: but now that i mention the 3d core, that could bad apple even harder, just convert every frame into vector data, slap some polygons on it, and let the 3d core rasterize it
<clever>
the 3d core also lacks all of the timing/limits of the 2d core, it wont glitch out when too slow, it just takes longer to fire the "im done" irq
<clever>
and then its up to the software to decide what to do, if you dont finish a frame in time for the next pageflip
<mrvn>
clever: what I want to to is send that "I'm done" irq to the monitor.
<mrvn>
A fixed frame rate is pretty stupid on a TFT.
unimplemented has quit [Ping timeout: 248 seconds]
<clever>
the rpi does support that as well, but its not documented
<mrvn>
and patented, 2 ways actually
<clever>
it has a non-contiuous mode, where it just scans out one frame, and then stops
<clever>
and there is a dedicated frame-done irq for that
<mrvn>
So first you have to figure out which method the monitor supports and then pay for the patent to use that method.
<clever>
a lot of displays also lack a framebuffer
<clever>
so they simply cant hold an image
<clever>
i think displayport is the start of supporting this at the protocol level
<clever>
where its no longer a stream of pixels with video timings, but just packets of image data
<mrvn>
the TFT cells hold pixels a long time
<clever>
this also reminds me, how does TFT handle shades? is it just turning the crystal half on, and then leaving it floating?
<clever>
or is the TFT (the transistor itself) actually acting like a tiny analog sram cell, and trying to maintain the charge?
<mrvn>
the later I believe.
<clever>
i'm also reminded of an older lcd panel, pre-tft i believe, where you had major streams coming out of any high-contrast pixels
<clever>
i think thats because of a lack of transistors at the pixel, so the row and column lines, impact every pixel they pass thru?
<clever>
the rpi has several video output ports, dpi, dsi, smi, vec, hdmi
<clever>
dpi is just 24 parallel bits of color, hsync, vsync, pixel clock, and data-valid (blanking vs nonblanking)
<sham1>
Damn it, I keep on constantly typing about replicants when talking about replicas in data. I should really watch Blade Runner and/or read the original novel
<clever>
i have accidentally set dpi to have 1/60th of a second for one field, and 2 seconds for another field, and i suspect it could be set to non-contiuous mode, so you just need a display that is willing to accept 1 frame and unstable vsync rates
<clever>
smi isnt documented well, but it involves placing data on up to 18 pins, and pulsing a strobe line, with defined amounts of time around it, it definitely feels like a non-contiuous mode, where your writing to an external framebuffer
<clever>
and SMI can also just access external memory systems directly, sram, hd44 lcd drivers, nand
<clever>
dsi already has a form of non-contiuous mode, where you burst an entire scanline at max speed, then drop to low-speed mode to improve power savings
<clever>
and may have variable fps as well
<clever>
vec does ntsc/pal/secam, and doesnt give you control over the timing parameters, i suspect it wouldnt easily do non-contiuous mode
<clever>
hdmi, i'm not sure...
<clever>
mrvn: so the question then, is what does your display of choice support, from that array, and how does its variable refresh rate work, and how much do you have to pay for answers
<mrvn>
clever: the last part is the biggest problem
<clever>
in the mis-configured interlacing case, i was driving into a crt monitor
<clever>
when it got 2 vynscs 1/60th of a second apart, it woke up
<clever>
but after ~1 second without a vsync, it went back into sleep mode
<clever>
repeat!
<clever>
and given the nature of a crt, it needs to lock a PLL onto the vsync, and turn it into a sawtooth
<clever>
mine was also smart enough to know when things are out of spec, and turn itself off, rather then fry itself
<mrvn>
they stopped frying themself when they invented multisync
<clever>
yep, mine even has multisync printed on the front frame
<bslsk05>
www.adafruit.com: Pimoroni HyperPixel - 4.0 Hi-Res Display for Raspberry Pi : ID 3578 : $64.95 : Adafruit Industries, Unique & fun DIY electronics and kits
<clever>
a DPI display i have, i could experiment and see what happens when i violate the defined timings
<clever>
i think it was based on the ILI9806E driver chip?
<clever>
the display is specced for 800x480, RGB666
<clever>
the driver chip is specced for 480(RGB)x864, "without internal gram", 24bit color
<mrvn>
aren't you pushing pixels to it and toggle the clock after each?
<clever>
that is one mode it supports
<clever>
but i'm not sure this board supports that mode
<mrvn>
would be cool to push pixels at variable speed depending on how many sprites are at each.
<clever>
id have to double-check, but i think the chip is configured to guess the pixelclock, from the hsync clock
<clever>
the dpi hardware isnt able to vary the speed that much, so you would lose all hw accel if you tried it
<clever>
one issue ive seen called "dot crawl" i think, is when you use fractional division
<clever>
basically, on average, the pixel clock is say 2.5mhz, but its doing that by randomly jumping between 2mhz and 3mhz
<mrvn>
clever: think about RLE encoded Bad Apple
<clever>
the issue, is that if the display is not sampling on the rising edge of the clock, but assuming a pure 2.5mhz clock, it will randomly sample the wrong pixel
<clever>
so all of your pixels jiggle left/right a bit
heat has quit [Remote host closed the connection]
heat has joined #osdev
<clever>
i think RLE would be relatively simple to do in something like a pico or fpga
<clever>
for bad apple specifically, just invert the output, load a delay, wait, loop
<clever>
only for short delays, would it become an issue
<clever>
but the rpi isnt good at deterministic timing, so you want to instead just decode into a framebuffer, and let the 2d core scanout for you
<mrvn>
clever: your idea of using 3D + vector graphics sounds way cooler
<clever>
yeah
<clever>
the 3d core also has a triangle fan mode
<clever>
assuming no extreme concave sections, you can just give it a center-point, and an array of edge points
<mrvn>
now where do I get vector data for Bad Apple from?
<clever>
and boom, almost any shape, assuming you can slice it up like a pizza, with flat edges
<clever>
ive seen people do bad apple with a laser before, i assume you just scan the original video off YT
<clever>
or rip it from somethat that already did that
xenos1984 has quit [Read error: Connection reset by peer]
<bslsk05>
github.com: lk-overlay/v3d.c at master · librerpi/lk-overlay · GitHub
<clever>
mrvn: this chunk of code generates a single vertex, in a format the rpi hw can handle, and in this case supplyings 3 varyings, the r/g/b of that one vertex
<heat>
sortie, what were the most fun UNIX utilities to work on?
archpc has joined #osdev
<clever>
to render bad apple, you would just change out the xy and rgb here, and repeat that block, for every point on the vector polygon
<bslsk05>
github.com: lk-overlay/v3d.c at master · librerpi/lk-overlay · GitHub
<clever>
mrvn: then to make actual triangles, you give the index for 3 vertecies from before, and repeat for each triangle (or use a different mode, which ive not tested)
<clever>
adjust the size parameters for both lists, post to the gpu, repeat with new data on each frame, done
<sortie>
heat, mostly I consider them more of a necessity than truly fun programming :)
<heat>
aw :/
<heat>
i've been yearning for some good ol unix fun with C and standard utilities
<heat>
but I already stole most of mine from toybox
<clever>
mrvn: if you want faster rendering, a far simpler fragment shader could be used, such as one that only does grey-scale images, and lacks alpha blending, then you can drop the vertex down to 1 varying
<mrvn>
well, Bad Apple is B/w.
<clever>
or you could even drop the varying entirely
<sortie>
heat, but there's a few categories that are nice 1) Programs with a nice self-consistent data model and algorithm (e.g. find(1), expr(1), nl(1), sort(1)) 2) programs that allow you to do lots of fun design work (e.g. init(8), ifconfig(8), etc.) because you are not quite bound to a particular interface
<clever>
make a shader that is just always black, and use white as a bg fill
<clever>
mrvn: the varyings are 32bit floats, so you cant reduce it down to 1bit
danilogondolfo has quit [Remote host closed the connection]
<netbsduser`>
sortie: service management is my passion
<heat>
i feel sorry for you
<heat>
:P
<netbsduser`>
i have some fun ideas on kernel support for it in my OS (the state of affairs on linux and bsd both is inadequate, in older unix even worse)
<heat>
sortie, problem with the larger programs is that it's a PITA to work on since you have much less helpers in C
<mrvn>
sortie: grep
<sortie>
Grep yeah too
<mrvn>
xargs is fun too
<netbsduser`>
linux's cgroups go some way towards something desperately needed (a way to contain and act atomically on a group of processes which constitute a single service) but better can be done
<clever>
netbsduser`: ive used the cgroup api, and have found that enforcing memory limits to be very clunky, and you need to recursively enable it up the entire tree
<heat>
sortie, your synopsis looks very oopsie?
<heat>
something something semantic man pages :P
<sortie>
heat, just a quirk in this older mdocml port I'm using (need to update it) combined with some bad css
<sortie>
(Or rather css that doesn't try to fix it)
unimplemented has joined #osdev
<netbsduser`>
sortie: i like it, this looks like a good improvement on the init(8) in old unixes
<clever>
mrvn: i had also discovered something useful in the 3d docs a few months ago, to render a scene, you give it a list of commands (a custom bytecode), and a start+end addr, and poke a reset register
<clever>
mrvn: but, if you change the end addr, it will seamlessly resume execution, so you can keep extending the command list without having to worry about any race conditions
<mrvn>
So you can start rednering on the first triangle and then simply keep filling the buffer?
<clever>
yep
<clever>
just feed it vertex data as fast as you can, one whole frame at a time, until the buffer fills up
<heat>
good idea, mdocml is also OK openbsd software
<heat>
we're onto 2 here
<clever>
and wait for frame-done irq's to come back
<heat>
can we get a third?
<sortie>
netbsduser`, yeah def, the intention is to have a basic classic init but with the technology improvements of the modern era (like proper process tracking) done well
<clever>
then track where it likely is (based on frame-done irq), and cicle back around and keep filling the command ring
<sortie>
And dependency tracking with true radiness
<sortie>
readiness
<clever>
mrvn: that way, the 3d rendering is fully pipelined, rather then grinding to a halt between every frame
<clever>
it kinda only works for situations where you dont need low-latency, a game would be unplayable if the output was 10 frames behind the input
<netbsduser`>
sortie: i always thought it a great shame that, although classic init could do proper process supervision, it couldn't do so according to dependency ordering, so we ended up with the hacky approach of pidfiles instead
<sortie>
Yeah I ain't got no pidfiles
<mrvn>
netbsduser`: pidfiles don't do any ordering. That's done by sorting the scripts alphanumerically
<netbsduser`>
mrvn: yes, when it was moved into scripts they didn't use the direct supervision facilities of init anymore
<mrvn>
netbsduser`: pidfiles are used for the stop action because you don't have a handle for the process to kill
<netbsduser`>
that's exactly the problem
<mrvn>
and also to signal readyness of a daemon
<netbsduser`>
it's a poor handle that might continue to exist when the process to which it once referred is not only gone, but has had its PID reused for some other unrelated process
<netbsduser`>
they are named and hierarchically organised, which is enough if the service manager (e.g. systemd) is in control
<clever>
mrvn: i think pid files, are to deal with the case of a complex startup process, or a self-forking service, where the service isnt a direct child of systemd(or whatever you have)
<clever>
and if you can just tell a service to not fork, and use pre-start scripts, the service will be a direct child of the service manager
<mrvn>
clever: all daemons self forked pre systemd
<clever>
yeah
<clever>
because the old init.d system was bash based, and you basically had to
<mrvn>
clever: with systemd the pidfile signals the daemon is ready
srjek has joined #osdev
<netbsduser`>
freebsd has also developed an innovation called procdescs, these are file descriptors created by a new system call pdfork() which refer to the pdfork()'d new process and avoid the problems of pid reuse
<clever>
systemd also has the sd_notify system for that readiness signal
<clever>
where you send a READY=1 over a unix socket when ready
<heat>
netbsduser`, linux also has pidfd
<netbsduser`>
mrvn: not all, d.j. bernstein was crusading for years to discourage the practice because it interfered with his daemontools
<mrvn>
clever: yeah, but then the daemon needs to support systemd directly
<clever>
yeah
<clever>
ive also used that for other status flags
<netbsduser`>
aix also expected daemons to not fork for the sake of its src service manager, and solaris likewise preferred them not to for SMF
<clever>
you can append a custom msg to `systemctl status foo.service`, and change it at any time
<clever>
which is a nice replacement for the ugly old mess with argv
<netbsduser`>
systemd can also deal with a forking daemon without a pidfile, if it forks just once
<clever>
netbsduser`: and cgroups let it track any child of the service, and murderize the whole family at once
<mrvn>
netbsduser`: that's broken though for a daemon. You have to double fork to daemonize.
<netbsduser`>
it can make a best-effort guess in the case of double forks
<netbsduser`>
actually i think it has to guess even in the case of single-fork, in the case that it can't check the cgroup prior to the daemon forking off worker processes or whatever
<mrvn>
heat: with init starting the daemons in cgroups the double fork is indeed a bit pointless
<netbsduser`>
yeah, it's a violence against any sane service manager
<clever>
under normal conditions, i would say that you cant waitpid() for a service, once it double-forks, because your no longer the parent
<clever>
except, pid 1 adopts all orphans, and systemd is pid 1!
<netbsduser`>
clever: systemd had linux add a new 'subreaper' functionality for some other process to become the delegated adopter of all orphans ultimately descended from it
<clever>
ah, hadnt heard of that one
<netbsduser`>
free and dragonfly bsd have the same, which i use for my systemd port to those (but i don't need to; i use kevent with EVFILT_PROC anyway to track forks)
<mrvn>
netbsduser`: wouldn't it be enough to create a thread for that?
<netbsduser`>
mrvn: for systemd the case they needed it for was user service managers
<heat>
netbsduser`, I just want you to know you're the person I've seen that has the most enthusiasm for service management :P
<netbsduser`>
systemd spawns an instance of itself for every user the fisrt time they log in, and that manages services and is controllable by the user, so they wanted to get subreaper as cgroupsv1 had an unreliable exit notification mechanism
<clever>
netbsduser`: ahhh right, that explains why
<netbsduser`>
heat: we all need something stupid to be interested in for some unfathomable reason
xenos1984 has quit [Quit: Leaving.]
x8dcc has quit [Quit: leaving]
<mrvn>
netbsduser`: I was wondering if there was a waitpid() equivalent for a cgroup
<epony>
heat, yes, you can get a third nice OpenBSD tooling, it's called rc.d(8) and rc.subr(8) and avoids the failures of SysV and SysD init scripts
unimplemented has quit [Read error: Connection reset by peer]
<epony>
but it's three part and wanted to link each for quick view
<epony>
these are effectively what RedHat's scripts should have been
<epony>
instead of what they became as systemd units (bummer)
<epony>
the benefits are "shell script" control and handling of the system.. with viewer / modification exposed to a skilled operator
<epony>
nothing wrong with binary tools for speed up and everything useful as solidified implementation, but the loss of front end scripting and system operator control is noticeable and at a serious overhead.. while BSDs keep it reliable and "working"
<clever>
epony: but you can always grab the source for systemd
<epony>
these differ between FreeBSD, OpenBSD and NetBSD.. but is an example that avoids serious peeves with Linux
<netbsduser`>
mrvn: only a "cgroup is now empty" notification
<clever>
just has a higher skill barrier to entry
<epony>
maybe, but it's moving all the time
<mrvn>
netbsduser`: the subreaper seems to fullfill that role
<epony>
so grab it once, keep grabbing it forever ;-)
<clever>
epony: with the nix package manager, its possible to get the exact source that the currently running systemd is using
<epony>
it's overwhelming to individuals and "got to do things" system operators
<netbsduser`>
mrvn: yeah, it obviates the problem, especially since the main PID of a service is usually determinable (either it's not a forker, or it writes a pid file)
<epony>
once the carriage is turned in the river, there are a lot of ways to glue back the pottery
knusbaum has joined #osdev
<netbsduser`>
epony: it is superior to the sysv rc scripts in that they have explicit ordering
<epony>
the rc scripts have ordering too
<netbsduser`>
but they retain fatal weaknesses like not supervising the processes, and their reliance on pid files
<epony>
supervisor setup is not part of the standard tooling (for now)
<epony>
guess you need to make your supervisor an rc script service ;-)
<epony>
it's not a weakness
<epony>
a weakness would be to have a failure in a service and keep respawning it for continuous hammering of the failure (fail open / resume failure)
<netbsduser`>
most supervisors can avoid that
<netbsduser`>
they can use simple heuristics like "if it fails 3 times in 10s, wait a while before trying again" or it can indeed be configurable
<epony>
so it's a different policy of control, or just :justification: for you need to do it to get it that way as "always on"
<epony>
there is also on OpenBSD that "things are not all enabled like an Xmas tree" (hack me I'm lit)
<epony>
that's good overall, I know about these supervisor tools
<clever>
systemd has a retry limit by default, but can be configured to retry forever as well
<netbsduser`>
i also have a certain respect for what apple do with launchd, which is possible mainly because it's their platform and they are in control of what daemons do
<epony>
from the daemontools early implementation back in the high 90ies
<epony>
so.. SysD is kind of late to that party..
<epony>
I have 0 respect for Apple system quirks
<netbsduser`>
they don't do ordering at all, they pre-open all the mach ports and sockets associated with a daemon, they pass those on to the daemon, and the system proceeds in an orderly fashion
eck has quit [Quit: PIRCH98:WIN 95/98/WIN NT:1.0 (build 1.0.1.1190)]
<epony>
but you see both SysD and other such trials are.. Apple featuritis envy
<mrvn>
netbsduser`: systemd can do that too, even for mounting FS and nobody uses it much
<netbsduser`>
since each of the relevant means by which the daemon communicates with the world is already exposed, attempts to communicate just block until the daemon is up
<mrvn>
The big flaw with thise method is that connections time out after a while
eck has joined #osdev
<netbsduser`>
mrvn: yes, i didn't bother with that part when i ported systemd to the bsds
<epony>
SysD is like SysV, ambitious and locks up ;_)
<netbsduser`>
automount units struck me as slightly odd
<epony>
you have not ported it to OpenBSD though..
<epony>
must be some FreeBSD shim
<netbsduser`>
i did
<epony>
how so? OpenBSD does not have a lot of Linuxisms
<netbsduser`>
because systemd (the service manager) is not as linux as people think
<epony>
you know OpenBSD for example does not have PAM
<netbsduser`>
of course it means that all the linux-specific unit options are disabled