<esden[m]>
I honestly was not bohtered by it before. (especially considering that I run it in a loop for hardware testing)
<esden[m]>
But I guess I have the bitstreams pre generated... this is usually the biggest slowdown, and I am not sure what latency we are exactly talking about.
<whitequark[m]>
it could probably be more like 300-400 ms
<whitequark[m]>
re: pregenerated bitstreams, we have caching now, so you should not end up hitting the bitstream generation more than once per config. and even if you did, running the same applet in a loop has always been supported
<whitequark[m]>
s/supported/special-cased to be fast/
<attiegrande[m]>
have you investigated where the time is spent?
<attiegrande[m]>
(it's more like 1.2 - 1.5s for me...)
<whitequark[m]>
it's O(n) where n = applet count because it loads the code for every applet
<whitequark[m]>
in case you want to run --help
<attiegrande[m]>
yeah, i suspected
<whitequark[m]>
we have glasgow.support.lazy which can be used to delay that loading
<whitequark[m]>
at fairly little cost
<whitequark[m]>
or, well, may be able to delay it, depending on how argparse is built
<attiegrande[m]>
worth parsing the required info into a cached blob, so we don't need to parse so much every time? i feel a little surprised it's so intensive
<whitequark[m]>
how do you compute the cache key?
<whitequark[m]>
that was my first instinct to build
<attiegrande[m]>
that's indeed the tricky one... for "normal" installs, perhaps it's not required(?? 🙈), for editable installs, perhaps we get the startup penalty of no cache?
<attiegrande[m]>
(is it even possible to differentiate?)
<attiegrande[m]>
*not required -> no cache key required... just build the blob and call it a day
<attiegrande[m]>
or check for version changes only
<whitequark[m]>
what's a normal install?
<whitequark[m]>
our instructions specifically tell everyone to use editable ones
<whitequark[m]>
also, didn't you previously argue pretty hard against potential issues with bitstream caches?
<whitequark[m]>
(I agreed back then which is why the bitstream cache is incredibly robust; I would like to apply the same rigor to help text caching)
<attiegrande[m]>
i did, and i'm very pro robust caches... i was also keen for some thin cache management tool (e.g: just a clean would suffice, but is unnecessary - good job!)
<whitequark[m]>
having the help text be outdated is pretty bad considering that you may end up missing a warning about data loss
<whitequark[m]>
like, it's basically as bad as picking the wrong bitstream from the cache
<attiegrande[m]>
"normal" installs is a tough one - as you say, we promote editable installs, and many / most people will likely want to make changes and thus it's not going to be the bonus for many
<attiegrande[m]>
"normal" to me, is installing a versioned package from pypi or something... you don't expect any of the files to change, and if they do change the version should change too
<whitequark[m]>
i'm also not super interested in building a complex caching layer which would not benefit me
<attiegrande[m]>
... again though, we don't (i don't think!) publish like that
<whitequark[m]>
this isn't even something we ever plan to provide, currently
<attiegrande[m]>
i honestly really like the one file for the whole applet ... how much benefit could we gain from pushing the argparse code to a separate file? (i don't _want_ to do that... just throwing ideas)
<whitequark[m]>
how would that help?
<attiegrande[m]>
(also trying hard to avoid us producing an alternative to argparse, which I know you've mentioned in the past)
<attiegrande[m]>
less python to parse to startup - just the argument stuff, no applet machinery
<whitequark[m]>
oh, i see
<whitequark[m]>
that's still O(n) though the constant is less
<whitequark[m]>
there's a solution
<whitequark[m]>
make bare glasgow let you repeatedly enter commands while frontloading the startup latency
<whitequark[m]>
the main issue with that is that currently our ^C handling is super fragile
<attiegrande[m]>
another-another idea... which i'm not a fan of initially due to the probability of extra-extra complexity...
<attiegrande[m]>
a daemon, which starts (if not already running) and handles providing the argument data only
<whitequark[m]>
jruby calls this "shotgun"
<whitequark[m]>
i've considered it. the problem is that your stdio becomes a nightmare
<whitequark[m]>
like, imagine trying to use I/O redirection
<attiegrande[m]>
oh no
<whitequark[m]>
now imagine you're on windows.
<attiegrande[m]>
my thinking is more basic - on startup produce a daemon which parses / gathers the whole argparse, and sticks around for a while
<attiegrande[m]>
then on subsequent runs, transfer that argparse structure via RPC
<whitequark[m]>
this is just a transient cache
<attiegrande[m]>
yes, but we can watch for changes - quit or reparse
<whitequark[m]>
and has the exact same problems as storing it in a file, just... more complex
<whitequark[m]>
how
<attiegrande[m]>
inotify or (hopefully) a windows alternative
<whitequark[m]>
what will it use on windows exactly
<whitequark[m]>
also, inotify can and will miss events by design
<whitequark[m]>
(which is liable to happen if you git checkout)
<whitequark[m]>
also, you can't actually reload modules into a running python process. like. you sort of can but in practice you cannot
<attiegrande[m]>
hm, okay... is that [missing changes] related to moving / swapping files rather than altering them in place?
<whitequark[m]>
so you end up with at least two levels of process management: daemon itself and the subprocesses it spawns to generate argparse stuff
<whitequark[m]>
there is a fixed size buffer in the kernel
<whitequark[m]>
if it overflows inotify drops changes on the floor
<attiegrande[m]>
yeah, okay - ignore that one then...
<attiegrande[m]>
oh huh, didn't know it was so fragile
<whitequark[m]>
like many linux apis it's kind of garbage
<attiegrande[m]>
🙈
<attiegrande[m]>
is it worth me giving the "argparse into a separate file" approach a try? just roughtly to evaluate the potential speedup?
<attiegrande[m]>
seems like the best option discussed just now, but still "not brilliant"
<attiegrande[m]>
i also wonder how much of the startup time is spent scanning the usb tree... do you have a feel for that? or might be worth me looking into?
dne has quit [Remote host closed the connection]
dne has joined #glasgow
<whitequark[m]>
i feel like it's not a good use of your time
<whitequark[m]>
it's just a lot of effort to do that
<attiegrande[m]>
okay / true
<whitequark[m]>
I'm thinking I could look into whether argparse can be combined with `lazy`
vegard_e[m] has joined #glasgow
<vegard_e[m]>
`ReadDirectoryChangesW` is apparently the windows equivalent to inotify, I recently used this thing for a project: https://github.com/gorakhargosh/watchdog
<whitequark[m]>
thank you, this is useful
duskwuff[m] has joined #glasgow
<duskwuff[m]>
hot reloading Python applications is a fairly common ask in web development, and I suspect some of the libraries used for that could be reused/reworked for use here if you wanted to go down that road
<whitequark[m]>
I've thought about this
<duskwuff[m]>
I imagine some of them have already solved the cross-platform issues, at least
<whitequark[m]>
it's well known to be unreliable in the general case, which is why I'm hesitating
<whitequark[m]>
I know I could get it to mostly work
<whitequark[m]>
"well known" as in "the python docs for `importlib.reload` tell you to not use it"
<duskwuff[m]>
oh, I just mean to the extent of "detect that a file we care about has changed" not the actual process reloading
<whitequark[m]>
oh! yeah, watchdog should solve that though
<whitequark[m]>
you do still need to get a list of files though
<whitequark[m]>
like, nobody stops you from haivng an applet whose help text depends on another applet
<whitequark[m]>
and we have multiple in-tree applets which do this already
<whitequark[m]>
i feel like not loading every applet until and unless --help is requested is the best option here, let me just see if i can implement it