<sikkiladho>
on armv8, where it L1 cache is divided into icache and dcache and there's l1 and l2 cache. probably like this: https://i.imgur.com/r5QtAxd.png
<sikkiladho>
there are sctrl.{i,c,m} bits to cotnrol instruction cacheablity, data cacheblity and mmu on/of respectively.
<sikkiladho>
What is we turn off mmu and data cache off, would the unified caches like l2 and l3 will work or will they be off too?
<sikkiladho>
*sctrl.d for data cacheablity control
gog` has joined #osdev
Ellenor is now known as Reinhilde
vdamewood has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
wootehfoot has joined #osdev
sonny has joined #osdev
gildasio has quit [Ping timeout: 268 seconds]
gildasio has joined #osdev
gareppa has joined #osdev
gareppa has quit [Remote host closed the connection]
chartreuse has joined #osdev
gmacd_ has joined #osdev
heat_ has joined #osdev
sonny has quit [Quit: Client closed]
gareppa has joined #osdev
srjek has joined #osdev
gmacd_ has quit [Ping timeout: 255 seconds]
heat_ is now known as heat
gmacd_ has joined #osdev
gareppa has quit [Quit: Leaving]
frkazoid333 has quit [Ping timeout: 244 seconds]
FreeFull has joined #osdev
wootehfoot has quit [Ping timeout: 244 seconds]
gog has quit [Quit: byee]
gog` is now known as gog
sikkiladho has quit [Quit: Connection closed for inactivity]
<dzwdz>
does anyone here use plan9 mk for building their kernel?
<dzwdz>
it seems like a pretty nice alternative to standard make
gmacd_ has quit [Ping timeout: 264 seconds]
gmacd_ has joined #osdev
sikkiladho has joined #osdev
dude12312414 has joined #osdev
gmacd_ has quit [Ping timeout: 268 seconds]
foudfou has quit [Remote host closed the connection]
foudfou has joined #osdev
gmacd_ has joined #osdev
gmacd_ has quit [Ping timeout: 252 seconds]
gmacd_ has joined #osdev
[itchyjunk] has joined #osdev
dude12312414 has quit [Remote host closed the connection]
dude12312414 has joined #osdev
frkzoid has joined #osdev
Matt|home has quit [Ping timeout: 244 seconds]
gmacd_ has quit [Ping timeout: 252 seconds]
<geist>
sikkiladho: in general i think it's stated int he ARM ARM that only the icaches work with the mmu off
<geist>
it's because otherwise the cpu doesn't know what memory regions to cache or not for data accesses, but it's assumed that wherever the icache is fetching is valid memory
gmacd_ has joined #osdev
gmacd_ has quit [Ping timeout: 268 seconds]
CellBE_boy has joined #osdev
frkzoid has quit [Ping timeout: 244 seconds]
gmacd_ has joined #osdev
wootehfoot has joined #osdev
<geist>
i think it's kinda a misdesign of the ARM arch but they chose to use paging as the only way to determine cachability of any region of physical memory, for better or worse
<geist>
it's powerful, but also annoying, requires mmu to get any dcache at all, etc
<geist>
x86 and even riscv have the notion that the cpu already knows what physical regions are cacheable or not (or have their own out of mmu mechanism like MTRR)
<geist>
though you can of course override some of that with PAT and whatnot (involving paging)
<mrvn>
but do you really want to run without MMU?
<geist>
indeed., in the grand scheme of things it's not a big deal, since you generally do want to enable it, but it does mean there's more nonsense to deal with
<geist>
especially at the bootstrap stage
<geist>
where there are problems down the line is you have to be very very careful about how you map things with and weithout cache, since the arch doesn't fix your mistakes
<geist>
ie, if you map a page with caching in one spot and without somewhere else, that's at best UNDEFINED.
<mrvn>
well, any software controlled caching has that problem
<geist>
whereas if the cachability parameters was just implicit per physical address the page tables dont matter, etc
<heat>
it's entirely defined in intel cpus iirc
<geist>
right. but that's my point, this is a case where arm gives you the keys at the virtual point, not the physical level, so you have to be very careful
<geist>
btw did we ever figure out what the issue is with the channel logger?
gmacd_ has quit [Ping timeout: 268 seconds]
<geist>
somsone else asked me about it yesterday since they follow the channel via the logger
<mrvn>
yeah. But at least they didn't steal 2 bits from the physical address as cache attributes limiting everything to 1GB.
<heat>
geist, the bot wasn't in the channel
<heat>
but it magically rejoined
<geist>
ah it's back. yay
<heat>
maybe it was my email, idk
matt__ has joined #osdev
<geist>
yay perfect
matt__ is now known as freakazoid333
gmacd_ has joined #osdev
<sikkiladho>
giest: I was confused by some article where it said that if we turn of dcache and keep icache on(both are like attached to each cpu or called l1 cache), it would also disable unified per-cluster l2 cache but then how does icache then connectes to l3 cache and then memory? Looking at this picture might clear my question https://i.imgur.com/r5QtAxd.png. I think article might be wrong here. If we keep icache=on, dcache=off, l2 and
<sikkiladho>
l3 cache must also be able to work so icache can be connected memory.
gmacd_ has quit [Ping timeout: 252 seconds]
<geist>
most likely the L1 icache in this case just skips the unified caches
<geist>
loads directly
<geist>
note that in arm when you disable either i or d cache it doesn't necessarily 'turn off' the caches, it just causes the cpu to bypass them when fetching
<geist>
so it stands to reason that the L1 icache (per cpu) just simply issues a direct read across the bus and doesn't consult the unified L2 or L3
<geist>
there are various other places for optimization purposes where levels of caches can be skipped as well anyway, the PLD instryction for example has hints to tell the cpu to bypass levels of caches, etc
<geist>
the issue i think is that picture implies that all caches fetch through the lower level, this isn't always true. it may be for what is called an 'inclusive' cache vs an exclusive one, but really ARM doesn't define how that works as part of the arch spec. it allows implementations to be free on how some of that is implemented
<geist>
but really the icache being inclusive or not is independent of wht happens when the dcache is enabled, etc
<geist>
also keep in mind on ARM the L1 icache and L1 dcache are *not* synchronized like they are on x86. if the icache fetches something that the dcache also has and you modify it afterwards in the dcache (or memory) the cpu does not automatically fix the icache
<geist>
so really think of the L1 icache as just doing what it needs to do to keep the cpu fed, and it somewhat independently fetches from wherever it can get data
<sikkiladho>
yes, it makes sense now
<geist>
cool!
<sikkiladho>
and yeah the pic gave me the impression that cache can't be bypassed
<geist>
yeah cache heirarchy and whatnot is a really complicated topic, but i have found it to be very rewarding to have a good understanding of the general shapes of how modern caches work
<geist>
since a lot of problems (and solutions) in modern low level cpu stuff comes from this area and having a good understanding lets you derive solutions from first principles
<sikkiladho>
yes, it's a lot more liberating to actually understanding things from ground up/first principles.
<sikkiladho>
I have been learning about kernel and arm64/armv8 by translating some really good Chinese article and they make a lot more sense that official arm docs :P
<sikkiladho>
*than
gmacd_ has joined #osdev
GeDaMo has quit [Quit: A program is just a bunch of functions in a trenchcoat.]
<mrvn>
keeping cache synchronized is also horribly complicated and doesn't scale all that well. Not having it makes the CPU a lot simpler on the outside.
<heat>
thats horrific sikkiladho
<heat>
D:
<heat>
I know the ARM ARM docs are tough to read, I've been struggling with that as well
<heat>
but you get there bit by bit
<heat>
you also don't need to understand everything
<heat>
(it's also a notoriously poor way to learn - would you explain C++ to a beginner by going through each and every line of Hello World and explain it bit by bit?)
<mrvn>
and yet that's how they do it
<heat>
is it?
<heat>
I've never seen a hello world tutorial explaining operator overloading
<mrvn>
well, they start with "you have to include bits.h" just do that.
<heat>
or how #include works
<mrvn>
c++ doesn't have bits.h but who cares
<geist>
yah but usually they just say 'you gotta add this' for the #include stuff
<heat>
most tutorials I've seen just say "this is main, its your program's entry point, this cout thing is how you print stuff, return 0 returns 0"
<geist>
and then get down to business
<geist>
i think the general idea is you learn hello world not as a way to learn the language but a way to learn the tool
<mrvn>
what return 0? stop teaching C
<geist>
editor, compiling, running a program, etc
<heat>
mrvn, ?
<geist>
i think mrvn is in a contrarian mood right now
<mrvn>
the return in main() is optional in c++
wootehfoot has quit [Ping timeout: 252 seconds]
<heat>
its also optional in c
<j`ey>
i
<mrvn>
heat: no
<heat>
yes
<geist>
ESTOTERIC LANGUAGE FIGHT: GO!
gmacd_ has quit [Ping timeout: 268 seconds]
* geist
tosses a flashbang in the room and runs away
<mrvn>
<source>:1:13: warning: control reaches end of non-void function [-Wreturn-type]
<mrvn>
1 | int main() {}
<mrvn>
| ^
* geist
runs away before being pulled into this
<geist>
later! gotta go take some old flourescent tubes to the recycling place. been putting this off for years
<mrvn>
heat: it compiles but the result is UB.
<geist>
yay getting rid of flourescent lighting
<heat>
:D
<mrvn>
in c++ it's well defined to return 0
<sikkiladho>
heat: yeah I think I need to learn ignoring if I don't understand something. xD but the chinese articles I was talking about are really though :P
<heat>
mrvn, I can't reproduce your warning
<sikkiladho>
Take a loook at this:https://www-wowotech-net.translate.goog/memory_management/458.html?_x_tr_sl=auto&_x_tr_tl=en&_x_tr_hl=en-US&_x_tr_pto=wapp&_x_tr_sch=http
<sikkiladho>
just the diagrams and translation works really well
<mrvn>
heat: -std=c89?
<heat>
i'm not passing anything
<heat>
just using the default
<mrvn>
or -ansi -pedantic
<heat>
mrvn, ok, that behavior is C89 only
<heat>
"main(){}" is a perfectly valid C99 program
<geist>
(fairly certain the real issue is whether or not C or C++ lets you define main as returning int or void and similiarly with the args)
<\Test_User>
gcc also fails to complain about `char* main(char argv, int** argc) {return "Hello, world!";}` when you don't pass any args :P
<geist>
(you still have to return based on how the function is declared, however C lets you implicitly declare the return type)
<mrvn>
heat: true. The code gen is different too: C89: "ret", C99: "xor %eax, %eax; ret"
<heat>
because for C89 it's UB and for C99 it's defined
<geist>
(also IIRC C++ has a hack where whatever function you declare called 'main' is always implicitly extern "C" so it'll work in existing runtimes)
<heat>
yes
<mrvn>
geist: <source>:1:1: error: '::main' must return 'int', c++ is stricter
* geist
nods
<geist>
anyway really gone now.
<mrvn>
<source>:2:5: warning: 'int main(int)' takes only zero or two arguments [-Wmain] 2 | int main(int argc) { }
<mrvn>
<source>:2:5: error: conflicting declaration of C function 'int main(int)'
<mjg>
you land in a routine which walks a linked list of all known eventhandler types, doign strcmp to find dev_clone
<mjg>
and the locking idiocy starts
<mjg>
then*
<mjg>
there is also a documented feature where a handler can deregister *itself*
<mjg>
while being called from the above
<heat>
what is this?
<mjg>
freebsd bullshit from 90s
<heat>
the best kind of bsd
<mjg>
probably least screwed in terms of internals tbh
gildasio has joined #osdev
<mjg>
people like to think the 4.2bsd codebase was great
<mjg>
or the other old unix stuff
<mjg>
but reality is these people made really weird decisions, some of which perhaps were justified at the time
<heat>
the 4.2bsd was great
<mjg>
i'm confident some of the lollery is just them not knowing any better though
<heat>
code is unreadable, but that's the bsd we all know and love
<heat>
proper bread() shit
<mjg>
what is probably justified is "type stable" objects, i.e. stuff which never gets really freed
<mjg>
at best it gets reused for the same type of object
<mjg>
in particular vnodes were implemented like this. so if you rack up, say, 1mln vnodes, you NEVER recover that memory for smething else
<mjg>
there is a lot of technical debt of the sort in there and freebsd removed more of it than any other bsd afaics
<mjg>
openbsd is dead last with still not being able to free vnodes
<mjg>
pretty funny
<heat>
lol
<mjg>
no joke
<heat>
openbsd is dead last in everything
<heat>
it's the openbsd way
<mjg>
to their credit, it's not everything
<heat>
Security through lack of features(tm)
<mjg>
one positive thing i can say about it is that their devs use it on their laptops
<mjg>
while it's not great of course, it kind of works, better than the rest of the "family"
<heat>
100% more dev usage than freebsd's devs
<heat>
:V
<mrvn>
what do freebsd devs use?
<heat>
solaris obviously
<mjg>
depends. some freebsd, others os x, others linux
<mjg>
i don't know ofa nyone with windows, but there are probably some
<heat>
cursed?
<mjg>
personally i'm on a linux laptop
<mjg>
logging in to freebsd stuff to do actual work
<mjg>
if i did not have to fuck around with suspend/resume and wifi, i would use it
sonny5 has joined #osdev
<mjg>
linux is not that great tbh :P
<mjg>
for example it keeps going crazy with wifi from time to time, just dropping the connection every few seconds
sonny has quit [Killed (NickServ (GHOST command used by sonny5))]
sonny5 is now known as sonny
<mjg>
this lasts for fwe minutes and then it works again
<heat>
its the linux desktop experience
<moon-child>
I have freebsd on my old laptop
sonny has left #osdev [#osdev]
<mjg>
perhaps you are too young to know the experience of configuring xfree86
<mjg>
that shit was the epitome of "why"
<mjg>
you install fucking windows 3 and you have a mouse
<mjg>
even in the installer
<mjg>
it seems like such a no-brainer
<mjg>
but you can't just have a mouse in xfree86
<heat>
the oldest I've installed was an old mandriva or so
<heat>
late 90s
<heat>
also UnixWare but that was super smooth
<heat>
Motif worked
<mjg>
look, if you did not keep fucking with x11 config, you don't know what i'm talking about here
<mjg>
also you could not just have a monitor either
<mjg>
you had to configure a combo of graphics card + monitor + keyboard + mouse afair
<moon-child>
yuppp definitely before my time
<mjg>
also with virtual resoluition
<mjg>
absolutely bizzare
<mjg>
note it was not an option for interested, you had to fuck with it to achieve what everyone else got automatically
<mjg>
re mouse vs xfree86, you had to pick the protocol used (apart from the port)
<netbsduser>
then of course there was the art of writing a modeline that didn't make your CRT explode
<mrvn>
mjg: and hsync and vsync and blank and overscan values
<mjg>
right
<heat>
god bless kernel modesetting
<heat>
and DRI/DRM
<mjg>
say what you want about windows 9x et al, it did not ask you crazy questions to get a basic setup
<heat>
windows systems have always Just Worked(tm)
<heat>
even if they had some technically dubious decisions
CellBE_boy has quit [Quit: Leaving]
<mjg>
what
<mjg>
was you first windows xp by any chance
<heat>
yes
<heat>
well
<heat>
no
<heat>
I had w98 I think
<heat>
or 95?
<heat>
something like that
<mjg>
windows still based on dos was utterly bad
<heat>
but most of my early experiences were after that, with xp
<mjg>
xp was the first windows for home users based on the nt kernel
<mjg>
that worked mostly fine
<mjg>
9x would fuck itself up without you even doing anything
<raggi>
mjg> look, if you did not keep fucking with x11 config, you don't know what i'm talking about here --- yessss so much this
justache has joined #osdev
<mjg>
osdev xfree86 support group?
<raggi>
It always blows my mind in the Wayland arguments once someone tries to espouse some kind of x11 config perfection
<raggi>
About 8 years ago I told myself I was never again going to run a personally configured Linux desktop unless I can completely avoid reconfiguring any of that crud, so far nothing has bust through that
<raggi>
Gotta put a max cap on time lost to fixing config files
<heat>
fixing xorg config files to get nvidia to work is peak linux desktop
<heat>
thankfully it has worked pretty nicely for some years
<raggi>
Honestly Nvidia stuff people complain about is less bad than it was in the past
<heat>
yeah
<heat>
totally
<heat>
nvidia even supports optimus now, for X11
<heat>
I dont think they have it for wayland, it's the last thing blocking me to switch
<mjg>
nvidia drivers always worked for me with no hassle fwiw
<raggi>
Pre-edid configuration for high res/high frequency :'(
<mjg>
now ALSA is a different story
<mjg>
fwiw ubuntu is almost usable out of the box for me, i only add some packages and change the wm to i3
wootehfoot has quit [Read error: Connection reset by peer]
<raggi>
What was the root cause of 9x being often unable to configure irqs correctly?
heat has quit [Remote host closed the connection]
<mjg>
are you going to quote raymond chen? :)
<mjg>
afair he described some of the hackery needed to support dos drivers under windows
<raggi>
Haha, no I never really got deep enough at the time, I used 2k at home since the betas
<mjg>
well then check out the blog -- oldnewthing
<mjg>
absolute madness what these peolpe had to make work
<mjg>
basically maintain binary compat with DOS, which provides no real abstraction for squat
<mjg>
*small* tidbit is that they had their own file handle counter, which they had to copy over to a location where it would normally be in DOS
<raggi>
"the hell devnode" hah
heat has joined #osdev
heat has quit [Remote host closed the connection]
heat has joined #osdev
<heat>
ok prime offloading works on wayland
<heat>
im now a wayland user
<heat>
future is now
<mjg>
i3 does not work on wayland a the supposed replacement is total crap
<mjg>
and*
<heat>
whats the replacement?
<heat>
sway?
<mjg>
yea
<psykose>
it's the exact same thing
<mjg>
no, for one it does not show titles on my windows
<mjg>
i print some stuff which identifies the terminal, i get a bunch of them and i stack them
<psykose>
windows when floating or in what position
<bslsk05>
linux-haters-redux.blogspot.com: Linux Hater's Redux
<heat>
"Microsoft Windows Services for Unix"
* heat
closes the tab
<psykose>
was this written by a highschoole
<psykose>
r
<psykose>
>LAWL!!
<heat>
freetard!!!!!!
<psykose>
Oh snap!
<ptrc>
clearly a canonical employee
<psykose>
no u
<heat>
shut up alpine inc employee
<psykose>
mjg: more seriously i have no idea why they would be missing but it's not part of the configuration but rather something else being completely broken
<heat>
why dont you go live in the alps
<psykose>
i thought employees got paid
<heat>
you do
<heat>
from the musl foundation
<psykose>
i get paid $0
<heat>
plus free lambos straight from rich felker
<heat>
you're not fooling anyone psykose
<psykose>
just because he's named rich does not mean he gives away lambos!!!
<mjg>
my actual name is dick dollars
<mjg>
i may be a dick, but i don't have dollars
<heat>
your real name is matthew garrett
<geist>
I look away for an hour and what has everything devolved to!
* geist
adults you kids back to responsibility
<mjg>
this is a windows channel now
<heat>
oh yes how about that BeOS virtual memory system
<geist>
Yay beos virtual memory system
<psykose>
virtual memory is for losers i prefer rawdogging the memory
<geist>
Actually I was just thinking of digging out the old beboxen and making sure they work still
<mjg>
[serious question: is there someone here familiar with nt kernel internals and willing to rant?]
dude12312414 has quit [Remote host closed the connection]
<geist>
A bit, but I’m not really interested in ranting
<mjg>
:)
<geist>
And please whoever may participate do *not* link to any illegal source drops or whatnot
<geist>
It comes up every few years
<mjg>
heh
dude12312414 has joined #osdev
<mjg>
welp, some time ago i ran into claims that git on windows is very slow
<geist>
Someone will be like ‘I’m being helpful! Here’s link to …’
<mjg>
"justified" with file access on windows being slow compared to linux
<mjg>
curious what's actually going on there
<heat>
windows is better with different access patterns than the usual linux ones
<geist>
I generally think of it the other way, lots of posix systems are very very optimized for file system stuff
<heat>
i.e it flushes file caches when you close the file
gildasio has quit [Ping timeout: 268 seconds]
<geist>
Yah, dir caching, statting tons of files, etc are posixy stuff’s forte
<heat>
there was an interesting post about this on rustup
<mrvn>
heat: why would it do that? that's not in the specs.
<mjg>
it's not hard to open a file fast, provided it was already open previously
<geist>
It’s all shades of ‘fast’
<mjg>
i mean as long as you don't have to do actual i/o
<geist>
There’s fast and then there’s optimize the whole thing for doing it really really fast
<geist>
And when you do a million stats in a row, the difference starts to add up
<mjg>
but is there something stemming from the architecture which slows it down?
<mrvn>
geist: if you do a million stats then bloddy well use the syscall for stating a range of files in a dir
<geist>
Something that takes 10usec here and 100usec there is a 100x difference when you sequentially do it in a row a million times (like git and other SCMs tend to do)
<mjg>
not just code not being optmized here and there
<bslsk05>
'"NTFS really isn't that bad" - Robert Collins (LCA 2020)' by linux.conf.au (00:48:04)
<heat>
there it is
Vercas65 has joined #osdev
<mrvn>
A lot of that stuff is O(N^2) without optimizing
<geist>
Yes. A lot of the way posixy vfses work is highly tuned for taking the work out of the fs implementation
<mrvn>
1 million squared is kind of a lot
<heat>
a lot of optimizations took rustup from 3m30s to install down to 14s
<geist>
Ie, the walks are done in the vfs, now the vfs itself can cache the shit out of the dir walks, for example
gildasio has joined #osdev
<geist>
Whereas NT’s layer for example tends to push the path parsing into the fs itself
<geist>
It’s a up front design decision that trades flexibility here for speed there, etc
<mjg>
you could argue path parsing in the fs allows to make it faster
<geist>
Flexibility is nice in some cases: walking across network shares is somewhat more flexible since you can basically pass the path to the server and let it figure it out
<mjg>
notably you don't need to do an indirect func call for permission checks
<mrvn>
but then you have to reimplement it N times, once per FS and half of them will get it wrong
<geist>
Exactly, but then NT doesn’t really support that many fses. So it sort of works out
<mrvn>
And they have to consider chroots and filesystem namespaces
<mjg>
you can export all of it as utility funcs
<geist>
It’s clearly something that it’s a MSFT first thing
<heat>
mjg, you don't need to do an indirect call for permission checks in posix designs either
Vercas6 has quit [Ping timeout: 268 seconds]
Vercas65 is now known as Vercas6
<mrvn>
mjg: but then you just call the FS to have it call you back. 2 pointless function calls
<mjg>
heat: you don't *need* one, but it happens in practice
<geist>
But in summary I think they just haven’t optimized the VFS/FS layer for the same set of things
<heat>
mjg, why?
<mjg>
the fs at hand might or might have funzies like ACLs
<mjg>
or might not*
<geist>
So lots of stat heavy, tons of files, deep directory heirarchy thing aren’t really a ‘first party’ thing that windows and NT worried about as much
<geist>
And that usually doens’t matter until you get a work load that it’s just not optimized for
<mjg>
is there publicly available profiling data from windows?
<geist>
Yah the ACL stuff in NTFS may hurt too, perhaps it’s hard to optimize those walks
<heat>
geist, the slowness is not NTFS's fault
<geist>
Okay. I’m not really saying it is
Matt|home has joined #osdev
<heat>
you should all see the talk I linked, or at least go through the slides
<geist>
It’s more like the base assumptions about how FS stuff works is not tuned for lots of little operations
<geist>
Since that’s not something that win32 leans on that much
<heat>
it's pretty obvious that a lot of patterns that unixy things use don't go hand in hand with windows's fs
<geist>
Yah
<heat>
like stats for instance
<mrvn>
and then you have things like NFS where you have a file handle to open files directly without path walking.
<geist>
But even posixy systems dont necessarily pull out all the stops to make stuff blindingly fast. A good example is macosx and it’s fs layer
<geist>
Derived from BSD, but it makes some explicit decisions that are more about data safety than blinding speed
<heat>
yay data safety
<geist>
Since it’s a non-goal to try to build a Linux competitor in server space
gxt has quit [Ping timeout: 268 seconds]
<mjg>
bsd has very slow vfs layer to begin with
<geist>
And perhaps that too
<mjg>
no really, it's incredibly baroque
gxt has joined #osdev
<heat>
bread() moment
<mjg>
and multicore awareness was bolted onto it
<mjg>
[i did largely fix it in freebsd fwiw]
<geist>
Even back in the earlier days of Linux (later 90s, early 2000s) when it started really competing, a lot of Linux design decisions in the fs area was about being fast as hell, data safety be damned (ie, ext2)
<mjg>
heat: check out lockmgr
<mjg>
geist: haha, ext2
<mrvn>
it's too bad they didn't make it async from the start
<mjg>
i got a distro, forgot the name, which came with async by deafult
<geist>
At the time it was en vogue to diss on Linux as being a toy you used if you didn’t care about keeping your data
<geist>
Since a power pull was pretty disaster out
<mjg>
as a high schooler i did not understand why the system keeps fucking itself up
<heat>
ext2 isn't bad :(
<geist>
Indeed, even work glinux was for a while explicitly mounting ext4 in not ordered data mode
<heat>
it's just super simple
<heat>
like ufs
<geist>
And i saw at least one person’s git repo get totally corrupt with a power pull, even on ext4
<mrvn>
geist: It's not like any other system had a better FS.
<geist>
since without data ordered it made no guarantees that file data was in sync
<geist>
mrvn: at the time absolutely
<heat>
is glinux google's internal linux distro?
<geist>
But ‘better’ was usually ‘not faster’
<mrvn>
geist: ext2 isn't a linux invention
<geist>
Ext2 was just very very simple. A cut down FFS that had no safety at all
<geist>
Ext2 is absolutely a Linux invention
<geist>
It was the Linux implementation of the classic FFS
<geist>
But in a non compatible way
<geist>
But it’s big thing at the time (and still is) was no journaling, no ordering of data vs metadata, etc
<geist>
It was just pure yolo throw it in the write back cache and hope it works
<mrvn>
on the other hand it's what POSIX says an FS does
<mrvn>
if you want to have the stuff on disks you must sync
<geist>
Thing was even at the time if you had a BSD or Solaris or whatnot system you never ran you FS that way
<geist>
They were typically fairly sync mode. Terribly slow by today’s standards, but its just What You Did if you cared about data safety
<mrvn>
.oO(mount -o sync)
<geist>
But you can build a FFS looking fs implementation around sync being fairly fast if you’re careful. See soft journaling, etc
<geist>
Which is what you had with a BSD at the time, etc
<geist>
Ie, a BSD sync mode was somewhat faster than ext2 in pure sync mode
<mrvn>
when you say FFS I always read "Fast File System" from Amiga OS.
<geist>
Even something like FAT you can build an intermediate mode where you are fairly sync in the metadata department but fairly a async in the data department, but y ou usually have to design the driver around that notion
<geist>
By at least tracking what is going on and inserting barriers in particular spots
gildasio has quit [Ping timeout: 268 seconds]
<heat>
fwiw ext4 is pretty safe these days, ordered at least
<heat>
at least for my purposes
<heat>
never really lost any data
<mrvn>
except the journal is not written to diks in order so you might be broken blocks mid journal.
<mrvn>
So even though you wrote to the file, called fsync(), the FS said it's all written to the journal so fsync() returns when the power fails you might come back and there is a checksum failure in the journal a block before and your data isn't replayed and is lost.
<mrvn>
broken harddisk caching
<geist>
If you have a broken HD caching that’s an issue, but frankly you shoul;dn’t get HDs that have it
<geist>
OTOH modern SSDs, i do wonder. Would love to see if someone has written a test case to determine if modern SSD vendors lie about cache flushes
<mrvn>
geist: almost all rotary disks lied
<geist>
heat: yeah ordered journaling is the way to go. Also i think with modern SSDs a lot of the need to be blindingly fast is consumed somewhat by the SSDs being fast
<mjg>
tangent, but i had seen a paper claiming wear leveling makes things worse
<geist>
mrvn:i 100% disagree.
<mrvn>
Good ones use the rotational energy of the disk to finish writing the last block on power loss
<geist>
I think in general they are very very good, but there were some external usb enclosures and whatnot that lied
<geist>
I remember apple having a whole blacklist for precisely this
<geist>
I think the answer is ‘some lied’ but i think the some is very very low
<geist>
and/or questionable quality stuff
<geist>
But as i said random usb enclosures are a different can of worms. A lot of those did
<mjg>
so what you would normally see in home computers?
<geist>
Since they sit between the OS and the disk
<mrvn>
geist: the some was very high (in desktop disks) till the feature was actually used.
<geist>
Yes but the feature has been used heavily over the last 15-20 years
<geist>
So sure, maybe in the 80s or 90s
<geist>
But anything remotely recently has needed the flush operation to work properly
<mrvn>
well, have you bought a rotary disk in the last 10 years? :)
<bslsk05>
minnie.tuhs.org: [TUHS] The evolution of Unix facilities and architecture
<geist>
Yes
<geist>
Fairly recently in fact
<mrvn>
but yes, 80s, 90s, type disks lied a lot
<mrvn>
or didn't even have the right features at all
<geist>
Indeed, key is by the 2000s journalled fses starting becoming the norm, and a proper journal *relies* on cache flushing and barriers
<geist>
So fairly quickly the bad vendors got named and it got fixed
<mrvn>
that's the period I was refering to
<geist>
Okay, please in the future specify periods. It’s a bit disingenuous to say ‘most disk drives suck’ and then upon pressing say ‘well okay mostly in the 80s and 90s’
<geist>
That’s a big different
<netbsduser>
the 'physical journalling' (where you write 'change block 49345 from value whatever to value whatevernewvalue) is significantly more resilient than logical journalling (set bit whatever of the allocation bitmap ) against some very unpleasant corner cases
gildasio has joined #osdev
<geist>
By the same token, ‘all computers are unreliable’ because the tubes in eniac used to burn out
<mrvn>
I think you still get journal checksum errors today though
<geist>
What i *do* think happened is when SSDs started to become more ubiquitous the same thing happened all over again: some of them lied, they got found out, vendors got shamed, stuff got fixed
<geist>
But indeed, i still dont entirely trust a SSD as much as a spinny disk, which is precisely why i also still buy them
<mrvn>
it's also the problem of what to do when you loose power while writing.
<mrvn>
There is this nice fix to have a little capacitor that you use to finish the write but then you have to pay license fees.
<geist>
Indeed. It quickly became mandatory that spinning disks at least finish he block they were on. It’s an implicit assumption in secondary storage that you at the sector level get either old or new data but nothing in between
<geist>
I think that’s been the norm since at least the 80s
<geist>
But early SSDs, etc etc. or random SD/MMC cards, etc
<mrvn>
geist: i don't think desktop disks (IDE) did that at all. SCSI / server disks more so.
dude12312414 has quit [Ping timeout: 268 seconds]
<geist>
They did. And do
<geist>
It’s independent of the interface
<geist>
Again you wouldn’t be able to build any semi modern fs if the disk corrupts itself that way. NTFS wouldn’t work, etc
<mrvn>
In the 80s you still had different disk hardware and not just different controler boards.
<geist>
Of course. but IDE itself has the notion that the controller is on board, so it can also finish writing the block it was on
<geist>
Something like ESDI would probably have the issue that it’d just get cut off mid stream
<geist>
Or floppy disks, for that matter
<mrvn>
geist: doing that is actually not that easy. You a) need energy, b) the platter slows down, c) using the motor as generator slows it down even more, Yout bit rate on the write head becomes variable.
<geist>
Anyway i only feel really strongly about this because my first job was doing FS stuff, and this sort of question was very very much on the forefront
<geist>
mrvn: hard != impossible
<mrvn>
hard == costly. ==> not for cheap desktop disks :)
<geist>
Fine., anyway tired arguing about this
FreeFull has quit []
<geist>
I’m just trying to keep the amount of FUD down a little bit. Whenever I see stuff like ‘all Xs sucked because Y’ I like to point out that well no there are shades of grey in almost everything
<geist>
And because something in one time period was notoriously bad didn’t mean it got fixed 5 years later and became a non issue
dude12312414 has joined #osdev
<mrvn>
In the 90s / 2000s you had IDE server disks and the same disk hardware with just the controller board swapped for ide/scsi and that stuff got a lot more mixed up
<mrvn>
or desktop quality scsi disks
<mrvn>
and remember ZIP disks?
* geist
senses a bit of latent ‘Amigas had SCSI and therefore …’
<mrvn>
click click click click click
* gog
clicks
<geist>
Keep in mind that everything that wansd’t an Amiga at the time was not intrinsically garbage
<geist>
Despite what you may remember
<mrvn>
By that time I got my first Alpha
<geist>
Side note i wish i hadn’t gotten rid of my scsi Zip disk
<geist>
The one thing i kinda wished i hadn’t tossed a few years ago in a hardware purge
<mrvn>
Still have mine and it wasn't clicking. :)
<geist>
Turns out they’re really really handy for moving data between modern machines at old scsi stuff
<geist>
Same, i tossed it anyway
<gog>
the scsi ones are kinda rare these days
<gog>
:)
<gog>
:(
<geist>
Exactly, now they’re hard to find
<geist>
OTOH i guess i have the scsi2sd board, so it really serves the same purpose
<mrvn>
but now you have scsi2sd
<gog>
i was just gonna say
<geist>
And is even more flexible, so i guess that
<gog>
those are things
<mrvn>
at the amagzingly slow speed of 1MB/s
<geist>
Was probably about the transfer speed of a 100MB Zip drive too
<mrvn>
I know the parallel ZIP was that slow. but I think scsi was faster.
<geist>
Yah at some point though it’d be capped by the transfer speed off the platter
<geist>
And yeah my first Zip disk was a parallel one. Iirc it was only really that useful in dos, where you weren’t really doing anything else anyway
<geist>
Later on you could use it under win95 and Linux and maybe NT, but the PIO overhead of bit banging the parallel port was pretty atrocious to a multitasking OS
<mrvn>
good enough to boot linux on the universities NT system
<jjuran>
Don't try printing a file stored on a parallel Zip disk
<mrvn>
jjuran: on your parallel printer?
<geist>
i also think this and the Laplink file transfer stuff was where things like ECP and whatnot modes came into play on your LPT port
<jjuran>
mrvn: Right.
<jjuran>
At least, it didn't work the one time I saw it tried.
<geist>
I never fully grokked that but there were all these DMA assisted transfer modes that various monos had
<geist>
Mobos
<geist>
EPP and ECP i think
<jjuran>
mofos
<geist>
Stupid autocorrecting client
<geist>
Am on ipad in coffee shop after dropping some stuff off at the recycling center
terminalpusher has quit [Remote host closed the connection]