<moon-child>
my understanding is one of the problems was i/o and process-making are slow on windows, so linux applications expecting them to be fast were disappointed
<moon-child>
which naturally raises the question: would it be possible to make those things fast in windows generally, if you can manage to wade through enough legacy code?
thegrinchmann has joined #osdev
thegrinchmann has left #osdev [ERC (IRC client for Emacs 27.1)]
<heat>
maybe?
<heat>
welcome to zombocom, you can do everything at zombocom
CaCode has quit [Ping timeout: 246 seconds]
<zid>
yea createprocess is slooow
<zid>
i/o was so slow on windows that I used to play WoW on linux
<zid>
because it turned 30 second zone transitions into 7 second ones
<heat>
IO isn't necessarily slow, it's just that people don't use it properly
<heat>
i.e CloseHandle on a file flushes the page cache
<bslsk05>
'"NTFS really isn't that bad" - Robert Collins (LCA 2020)' by linux.conf.au (00:48:04)
<zid>
so what does that mean in practice, never close files?
<heat>
defer file closing to other threads so you don't serialize on file closing
<heat>
oh yeah also i forgot: virus fucking scanners
<zid>
never close files, got it
<heat>
including the ones that are almost impossible to completely shutdown like windows defender
* zid
has never had a virus scanner besides windows defender, which is set to do nothing unless asked
<heat>
don't worry, it's doing things
<zid>
probably
<heat>
even if you set it all to off, it's still doing things
<heat>
like reading the whole file all at once
<heat>
when you open it
<zid>
I don't think it is
<zid>
if you have it set to not
<zid>
It does turn itself back on constantly though
levitating has quit [Ping timeout: 240 seconds]
<zid>
wow that is the most unique pronunciation of cached i've ever heard
<zid>
good job beardman
levitating has joined #osdev
<zid>
so far this is 'rust is bad' rather than ntfs is gooder than you thought :P
<zid>
tiny i/o buffers on the stack in the stl, people doing redundant stat calls, etc
valshaped7424884 has joined #osdev
valshaped742488 has quit [Ping timeout: 244 seconds]
valshaped7424884 is now known as valshaped742488
Hammdist has joined #osdev
<zid>
I changed the policy, I'll see if that makes it stickier
netbsduser` has quit [Ping timeout: 240 seconds]
terminalpusher has joined #osdev
terminalpusher has quit [Remote host closed the connection]
<heat>
do you think that's not relevant to any other project?
<heat>
stat calls are like abused everywhere
<heat>
even gcc abuses them
<heat>
as soon as you start cutting out the stupid shit you get much better results
<heat>
but then again the anti-fucking-viruses
<zid>
That's precisely the opposite of what I said
<zid>
I said it's a *systemic* problem, unrelated to ntfs
Hammdist has quit [Quit: Client closed]
<heat>
ok well I think this talk helps prove that windows IO isn't doomed in shit
<heat>
just for the sheer fact that you can match linux in tar extraction for many files
<heat>
BUT some parts clearly still suck ass
<zid>
"turns out if you write your own janky installer, rather than using the janky msi installer that also has all these issues and takes 40 years to do anything, it's slow"
<heat>
path walking seems slow, so does *everything that has ever involved AV scanning*
<zid>
Turns out writing things is an skill and you can't do it blindly with no knowledge of how it will perform, and have it peform, same as anything else
<heat>
yes
Hammdist has joined #osdev
heat has quit [Ping timeout: 245 seconds]
Left_Turn has quit [Read error: Connection reset by peer]
[itchyjunk] has quit [Remote host closed the connection]
<geist>
i wonder if in general doing path operations relative to some other path, vs using the entire path has a substantial speedup there
<geist>
should technically matter in linux too, but it has of course mega path traversal
gildasio has quit [Remote host closed the connection]
Vercas has quit [Remote host closed the connection]
blop_ has quit [Remote host closed the connection]
foudfou has quit [Remote host closed the connection]
gxt has quit [Remote host closed the connection]
<zid>
That's why we should move /usr/local/bin to /lbin clearly
foudfou has joined #osdev
Vercas has joined #osdev
gxt has joined #osdev
gildasio has joined #osdev
blop_ has joined #osdev
<zid>
PERF
<geist>
kinda funny all our build servers at work have one letter build paths, i think kinda specifically because of this
<geist>
like /x/f/q/1/... sort of stuff
<geist>
a) it's faster and b) when the paths get baked into binaries as they tend to do in debug stuff they're small and dont leak anything
<geist>
(probably mostly b)
<zid>
yea I have lots of software that will pop an error and ask me for C:\users\takashi\dev\game.pdb
<zid>
which is kinda funny
<geist>
yah at $gameco when i worked there we had a requirement to mount the root of the game at z:
<geist>
so everything was relative to that. wasn't really for a good reason except the build system had tons of hard coded paths in it, i think
<geist>
using `subst`
<zid>
The typex I did a lot of RE work on a game for has an interesting setp
<zid>
it mounts the windows install in a special mode where writes don't fail but they're lost at reboot, and the game itself is mounted as D: read only after decryption and the writeable partition is E: for save data
<zid>
(and it does funny things like hashing the boot sector of the drive in the bios)
friedy has joined #osdev
<geist>
klange: how do i say 'travelling rabbit' in japanese?
<geist>
google translates to `Ryokō-chū no usagi` though elsewhere that seems to be something like 'during'
<geist>
the ryoku-chu part
friedy has quit [Ping timeout: 245 seconds]
gildasio has quit [*.net *.split]
gxt has quit [*.net *.split]
foudfou has quit [*.net *.split]
Vercas has quit [*.net *.split]
blop_ has quit [*.net *.split]
gabi-250 has quit [*.net *.split]
valshaped7424880 has joined #osdev
valshaped742488 has quit [Ping timeout: 245 seconds]
valshaped7424880 is now known as valshaped742488
foudfou has joined #osdev
Matt|home has joined #osdev
Vercas has joined #osdev
netbsduser` has joined #osdev
gabi-250 has joined #osdev
gxt has joined #osdev
gildasio has joined #osdev
blop_ has joined #osdev
bradd has joined #osdev
gildasio has quit [Remote host closed the connection]
gildasio has joined #osdev
netbsduser` has quit [Ping timeout: 240 seconds]
netbsduser` has joined #osdev
linear_cannon has quit [Ping timeout: 240 seconds]
CaCode has joined #osdev
linear_cannon has joined #osdev
CaCode has quit [Ping timeout: 258 seconds]
gabi-250 has quit [Remote host closed the connection]
gabi-250 has joined #osdev
netbsduser` has quit [Ping timeout: 258 seconds]
<Ermine>
I feel that google is still bad at Japanese
<zid>
well, yea, it's AI powered
<Ermine>
it translates 'yokai' as 'monster'
<zid>
so of course it's bad
<zid>
the natural translation of yokai is honestly yokai
<zid>
but you wouldn't write that if you were targetting certain audiences, you'd write monster
<zid>
It's a cultural artefact of japan, it doesn't have a *direct* translation to english, monster is as close as you're going to get
<moon-child>
'which audiences?' and therein lies the problem
<zid>
yea, it's one of the major problems you're *always* going to have with machine translation
<zid>
They don't let you give contextual metadata
<zid>
like, I'd be very happy to get better translations if I could tag shit, a la "I like dates [the fruit]"
gildasio has quit [Remote host closed the connection]
blop_ has quit [Remote host closed the connection]
blop_ has joined #osdev
gildasio has joined #osdev
valshaped742488 has quit [Ping timeout: 240 seconds]
netbsduser` has quit [Ping timeout: 258 seconds]
rustyy has quit [Ping timeout: 260 seconds]
valshaped742488 has joined #osdev
rustyy has joined #osdev
netbsduser` has joined #osdev
Arthuria has quit [Remote host closed the connection]
[itchyjunk] has joined #osdev
goliath has joined #osdev
netbsduser` has quit [Ping timeout: 264 seconds]
grumbler has joined #osdev
cloudowind has joined #osdev
GeDaMo has joined #osdev
Burgundy has joined #osdev
grumbler has quit [Quit: It's time]
netbsduser` has joined #osdev
[itchyjunk] has quit [Read error: Connection reset by peer]
foudfou_ has joined #osdev
foudfou has quit [Ping timeout: 252 seconds]
nyah has joined #osdev
valshaped742488 has quit [Ping timeout: 244 seconds]
valshaped742488 has joined #osdev
grumbler has joined #osdev
grumbler has quit [Remote host closed the connection]
Left_Turn has joined #osdev
q3lont has joined #osdev
bauen1 has quit [Ping timeout: 240 seconds]
gog has joined #osdev
q3lont has quit [Quit: Leaving]
foudfou_ has quit [Remote host closed the connection]
foudfou has joined #osdev
<immibis>
zid: if you know something about the target language, try "i like raisins and dates" then remove the translation of"raisins and"
<zid>
immibis: yea, I know how to game it
<zid>
I'm just saying i'd rather be able to *tell* it
<sham1>
The embeddings are such that it will indeed choose the most common interpretation given the context. And well, dates as in romance are more common than dates as in the edible things
<zid>
I sometimes do it from a language that doesn't have that homograph
leon has quit [Ping timeout: 244 seconds]
tixlegeek has joined #osdev
leon has joined #osdev
<mcrod>
hi
<gog>
hi mcrod
<gog>
may i hug you
leon has quit [Ping timeout: 258 seconds]
grumbler has joined #osdev
leon has joined #osdev
<mcrod>
yes
immibis has quit [Quit: No Ping reply in 180 seconds.]
immibis has joined #osdev
* gog
hug mcrod
Osmten has joined #osdev
kof123 has joined #osdev
bauen1 has joined #osdev
<Ermine>
gog: may I pet you
<klange>
geist: that works; ryokō (旅行) means travel, chū (中) means 'in the process of', no (の) is like a possessive or backwards 'of', usagi (うさぎ) is the rabbit part, 'rabbit of the currently traveling' for an obtuse translation
<gog>
Ermine: yes
* Ermine
pets gog
* gog
prr
cloudowind has quit [Quit: timfe after]
* mcrod
hug gog
<mcrod>
i want fooooood
* gog
hand mcrod a bagel
heat has joined #osdev
netbsduser` has quit [Ping timeout: 244 seconds]
* mcrod
chomp
joe9 has joined #osdev
friedy8 has joined #osdev
joe9 has quit [Quit: leaving]
friedy has joined #osdev
netbsduser` has joined #osdev
orccoin has joined #osdev
friedy8 has quit [Quit: Client closed]
pg12 has quit [Ping timeout: 246 seconds]
pg12 has joined #osdev
[itchyjunk] has joined #osdev
orccoin has quit [Ping timeout: 240 seconds]
Burgundy has quit [Ping timeout: 260 seconds]
tixlegeek has quit [Quit: tixlegeek]
Osmten has quit [Ping timeout: 245 seconds]
bauen1 has quit [Ping timeout: 240 seconds]
gabi-250 has quit [Ping timeout: 252 seconds]
Burgundy has joined #osdev
gabi-250 has joined #osdev
xenos1984 has quit [Ping timeout: 260 seconds]
xenos1984 has joined #osdev
stolen has joined #osdev
rwb has joined #osdev
rb has quit [Ping timeout: 245 seconds]
tixlegeek has joined #osdev
heat_ has joined #osdev
heat has quit [Read error: Connection reset by peer]
Hammdist has quit [Quit: Client closed]
flom84 has joined #osdev
gog has quit [Quit: Konversation terminated!]
heat_ has quit [Remote host closed the connection]
heat__ has joined #osdev
xenos1984 has quit [Ping timeout: 260 seconds]
Hammdist has joined #osdev
netbsduser` has quit [Ping timeout: 240 seconds]
heat__ has quit [Remote host closed the connection]
heat__ has joined #osdev
gog has joined #osdev
netbsduser` has joined #osdev
xenos1984 has joined #osdev
flom__84__ has joined #osdev
flom84 has quit [Ping timeout: 260 seconds]
heat__ is now known as heat
flom__84__ has quit [Quit: Leaving]
gabi-250 has quit [Remote host closed the connection]
<gorgonical>
I'm at an impasse about how to do security for things like the clock and power controllers
<gorgonical>
And I'm considering instead working on the networking sockets delegation thing I have to do
<gog>
hi
<ZipCPU>
You are much farther along than I am ;)
<gorgonical>
like i want to have a secure peripheral over say i2c. but if I want it to be available then how do you make sure linux doesn't go reconfiguring the clocks to starve the peripheral?
<gorgonical>
preventing data access is real easy, but availability for these things is always so tricky
<ZipCPU>
And here I'm back at the point of trying to design my own system calls for a new operating system.
<ZipCPU>
I'm trying to make it so that a task can be suspended to wait for an EVENT (whatever that might be), but that it can also create a timeout as well. Hence, if the EVENT doesn't happen prior to the timeout, the task returns to the READY state anyway.
<ZipCPU>
I have a really cool peripheral designed to make that happen, but ... that requires keeping a sorted list of tasks. You'd think that'd be easy--just sort the list, however I'm trying to think of some "nice" algorithmic way to do it.
<heat>
amap_add | add an anon to an amap
<heat>
amap_unadd | remove an anon from an amap
<heat>
un-fucking-add
<heat>
cranor couldn't think of a synonym for unadd
<pitust>
is there an operation that is the opposite of addition?
<pitust>
would be cool
<pitust>
but i don't think one exists
<pitust>
"remove"? "delete"? "subtract"? what are those fake words!!!
stolen has quit [Quit: Connection closed for inactivity]
Turn_Left has joined #osdev
Left_Turn has quit [Ping timeout: 264 seconds]
<immibis>
gorgonical: i think the A64 chip I looked at had permission bits for the clock and power bits
<immibis>
ZipCPU: isn't a timeout just another kind of event?
<ZipCPU>
immibis: Yes
<ZipCPU>
Exactly
<immibis>
the data structure you're looking for is called a timer queue. There are at least several ways to make one, but the simplest is a heap
<immibis>
well actually the simplest is a linked list and you check every item to find the soonest wakeup, but that's terrible
<immibis>
the simplest sensible one is a heap
<ZipCPU>
But ... I need to keep the list sorted
<ZipCPU>
If I use a linked list, then insertion costs (on average) O(N), since you need to walk through the list to find the right place to insert
* ZipCPU
pulls out the algorithm book to look up how heaps are implemented
<ZipCPU>
Man, is this book ever dusty ...
<immibis>
you may find the name "heap" is used interchangeably with "priority queue"
<ZipCPU>
No, I hadn't noticed that before ... but that's exactly what I need to maintain
<ZipCPU>
immibis: Yes, this looks exactly like what I'm looking for. Thank you.
<geist>
a sorted list works too, but either way the point is you're multiplexing a single hardware timer to multiple virtual ones
<ZipCPU>
Pretty much
<ZipCPU>
So ... I just read the chapter on heaps. Quite fascinating. It's like a partial sorter, only guaranteeing that the end (max/min) is correct.
<ZipCPU>
It'll work quite nicely for me.
<heat>
you probably want to avoid most of that for the kernel
<heat>
fancy data structures often suck for various reasons
<heat>
including memory allocation
<heat>
most of linux is built on doubly linked lists
<geist>
rght, also for the timer queue, N is probably pretty short
<geist>
so not worth spending extra time on it right now
zxrom has quit [Ping timeout: 264 seconds]
<heat>
rb trees are ok here too, if you find yourself having a huge N
GeDaMo has quit [Quit: That's it, you people have stood in my way long enough! I'm going to clown college!]
zxrom has joined #osdev
<ZipCPU>
heat: Seriously? 'Cause ... this would go directly into the scheduler's logic. I figure it's kind of important to get that right.
<geist>
the data structure tends to only matter once N gets large
<geist>
worry about that once you have a lot of N
<heat>
timers should have little to do with the scheduler
<heat>
but yes, what geist said
<geist>
exactly. i think you may be conflating the two
<geist>
in general the strategy is to have a timer queue where you can register a callback at some point in the future
<heat>
also worth noting that usually data structures that scale really well end up being worse in simple situations
<geist>
and that is independent of the data structure your task is waiting on, probably in FIFO order
<geist>
the timer fires and removes your task from the queue
<heat>
for N = 10 a binary tree is probably worse than a linked list
<geist>
right, and for things like mutexes or whatnot, f you have 10 things waiting on a lock you have serious lock contention
<geist>
and that's really what you should be worried about
* ZipCPU
offers a confused look
<heat>
what's wrong?
<ZipCPU>
"in general the strategy is to have a timer queue" ... got it
<ZipCPU>
heat: "data structures that scale really well end up being worse in simple situations" ... can you explain this one to me?
<ZipCPU>
Are you suggesting that walking a linked list will be faster in general?
<heat>
big O says how well the data structure scales in terms of magnitude, but it doesn't say how fast it is
<ZipCPU>
Go on, I'm listening
<heat>
like an O(log n) binary tree walk is slower than an O(n) std::vector (or C realloc'd array or whatever) iteration
<heat>
for N = 10
<heat>
even though it's log n
<zid>
you have to multiply by th cost, in real life
<zid>
how it scales != how it performs
<zid>
do people actually make that mistake irl?
<heat>
if you keep increasing N you'll find that eventually the binary tree will surpass std::vector due to its O(N) properties
<heat>
yes, ofc
<heat>
see: 90% of all usages of a red black tree
<ZipCPU>
So ... I used a straight array the last time I built this. I worried that it wouldn't scale well in general, so I was looking to build it "better" this time. And ... you're saying the straight array walk might be simpler and faster?
dude12312414 has joined #osdev
<zid>
I don't think people use RB trees for direct right now perf, they use it because it won't ever *suck* and cause their program to run super slow, even if they vastly underestimated how many users some code will get
<heat>
sure, you should measure first
<ZipCPU>
and ... if I'm never using more than 5-10 tasks, it probably won't make a difference?
<zid>
straight array only costs money if you constantly realloc it
<heat>
linux uses a red-black tree for this, but linux is also measurably more complex than your OS or my OS
<zid>
are you planning to constantly realloc it?
<heat>
probably not
<ZipCPU>
I'm just trying to build a simple OS that I can use in a small embedded system, to capture SONAR data in real time, store it to "disk" (SSD), and forward it back over a network
<geist>
also once you have it fully implemented/tested/optimzed, there may be an advantage to using something like a RB even if it's overkill, because you have it already
<zid>
sounds like it will run precisely 1 program ever
<ZipCPU>
I'm also a hardware developer first, only reluctant OS developer next ...
<geist>
but without it, a linked list is pretty darn good
<zid>
I'd run it [32]; :P
<heat>
like, if I start adding all these timeouts (which I do have) to my TCP stack, etc then the rb tree will prove itself really good in a loaded scenario
<geist>
yah, linked list
<ZipCPU>
Hmm ... not the advice I was expecting. Reasonable, but not what I was expecting.
<geist>
also it's its a small embedded system you're writing this code for, you need to consider the overhead of the code and data on the relatively small memory space
<heat>
ZipCPU, generally google interview questions are the farthest thing away from kernel hacking
<ZipCPU>
I have an abundance of memory. (2GB+)
<heat>
like, "implement a heap" is something you won't ever do, ever
<geist>
heh 'small' embedded system indeed
<zid>
[64] then
<heat>
*unless* it's a malloc heap lol
<ZipCPU>
heat: It's been decades since I've had a job interview. I'm not expecting any more any time soon ... ;)
<geist>
but all of this you can delay until later if you properly abstract the data structures so you can get it working, test, and then optimize later
<zid>
okay FINE, geez, [128]
<ZipCPU>
geist: Working on that, and getting stuck at this one ...
<geist>
not saying you shouldn't *think* about what to optimize now, but stuff like this probably wont ever matter
<heat>
👏flame👏graphs👏
<zid>
That is my absolute final offer
<geist>
whoa big spender zid
<Ermine>
> unadd
<Ermine>
addn't
<heat>
how about NTIMRS in sys/param.h?
<zid>
creat
<ZipCPU>
One issue I have is that the last time I built my ZipOS, it could only run on one system. I'd like to build something a bit more portable across hardware instantiations
<heat>
anon_addnt is horrendously malicious and i hate you for that suggestion Ermine
<heat>
amap_add and amap_addnt
<zid>
I would say addnt is do not add, rather than subtract, sorry
<Ermine>
(super cursed suggestion don't use it)
<geist>
ZipCPU: in what way was it limited?
<ZipCPU>
It could only run a single fixed set of tasks. The interrupt handling was hardwired, and could only handle the interrupts present on that hardware.
<zid>
sounds perfect
<heat>
sounds like linux 0.1
<zid>
next platform gets a new file in arch/ and its own set of hardwired interrupts
<ZipCPU>
I didn't have proper interrupt controllers, but rather a big interrupt loop tailored for my hardware.
<geist>
i see, yeah you need to start abstracting those into less fixed things
<ZipCPU>
Exactly!
<geist>
like an api to install/remove interrupts. an api to create/destroy tasks, add it to the run queue, etc
<ZipCPU>
Yes!
<geist>
yup
<ZipCPU>
(Sorry, though ... I can't handle dynamic task creation/destruction yet. Tasks will be created initially and ... never again.)
<geist>
for what you're talking about there you need nothing more than arrays and linked lists. dont get too fixated on newer/fancy data structures yet
<geist>
that'll just be a distraction
<geist>
focus on compartmentalization and abstraction, building apis to hide implementation details
<ZipCPU>
What brought about today's discussion is the idea of a system call WAIT_UNTIL(event, timeout) which I'd like to implement
<geist>
sure
<geist>
i have exactly the same thing in my design too
<geist>
event_wait(event, timeout)
<geist>
with a special value of INFINITE_TIMEOUT if you dont want it
<ZipCPU>
Yeah
<ZipCPU>
Or a timeout of zero, in case you just want to check if the event has already tripped
<geist>
right
<geist>
very common pattern, works very well in my experience
<ZipCPU>
So ... I'm walking down a well beaten path. That could be encouraging. ;)
<heat>
see pselect and ppoll and select and poll
vdamewood has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
<ZipCPU>
Yeah ... that's a capability I'd like to build
levitating has quit [Ping timeout: 260 seconds]
levitating has joined #osdev
grumbler has quit [Quit: It's time]
<gog>
hi
<sham1>
hi
heat has quit [Ping timeout: 260 seconds]
* kof123
scans backscroll...noone even mentioned "least addimal"
heat has joined #osdev
terminalpusher has joined #osdev
Burgundy has quit [Ping timeout: 260 seconds]
<heat>
amap_unadd!!!!!!!!!!!!!
<zid>
_dad, short for de-add
<gog>
uncreate
<zid>
reverse_increment
<heat>
unincrement
<zid>
and if anybody asks why it isn't called decrement, insist that your program deals heavily in french wine, and you don't want it confused with remove_crémant
heat has quit [Remote host closed the connection]
Turn_Left has quit [Read error: Connection reset by peer]
<gog>
cwasson()
<gog>
descwasson()
dude12312414 has quit [Quit: THE RAM IS TOO DAMN HIGH]
terminalpusher has quit [Ping timeout: 245 seconds]