<SamantazFox>
jrei: The goal is to have "workers" to palliate the lack of real multithreading. It's for the invidious project ^^
<FromGitter>
<Blacksmoke16> why not just try it with the MT flag?
<FromGitter>
<Blacksmoke16> that would prob be easier/more robust and help find any bugs
<SamantazFox>
Blacksmoke16: already tried, and of course, it doesn't work x)
<FromGitter>
<Blacksmoke16> what didnt work exactly?
<FromGitter>
<Blacksmoke16> i.e. it's entirely possible invidious just isnt designed to handle true multithreading, or is there an issue with Crystal's implementation itself?
<SamantazFox>
naqvis: MessagePack looks nice! Protobuf is way too complicated, and also too slow (packing and unpacking the protobuf format takes a *TON* of time)
<FromGitter>
<Blacksmoke16> so writing bytes around to share data between processes just feels kinda meh in the long run...
<SamantazFox>
Blacksmoke16: I don't know exactly. My knowledge of the crystal internals is too limited. In the same way that the `--release` flag seems to break the database, I have no idea where it can come from, or where to look at. There are so much interactions all over the place.
<FromGitter>
<Blacksmoke16> i would think figuring that out would be more beneficial :P
<FromGitter>
<Blacksmoke16> tho yea, prob be hard to debug :/
<SamantazFox>
I've been cleaning and refactoring invidious' code over the last year or so, so the general stability has improved, but there is still many things that don't make sense to me, or which are hard to easily reproduce (e.g only happens after running the server for 5h with 200 users connected to it)
<SamantazFox>
hard to reproduce*
<SamantazFox>
My idea is to separate the various components to (try to) isolate the problems.
<FromGitter>
<Blacksmoke16> ahh okay, so like a microservice approach?
<FromGitter>
<Blacksmoke16> that makes more sense compared to what i was thinking you were going to do 😅
<SamantazFox>
sort of, yes
<FromGitter>
<Blacksmoke16> 👍 gotcha
<SamantazFox>
the youtube backend (JSON parsers and scrapers) would be multiple separate module, to which "tasks" could be sent in a round-robin fashion. Image/video proxies could be completely separate modules, as they don't need any interaction with the main code. And the front-end(s) (HTML/JSON) would be more autonomous modules
<FromGitter>
<Blacksmoke16> i never really worked with it, but always looked pretty neat and fairly straightforward
<SamantazFox>
Problem is that it's quite complicated, and I'd prefer to KISS for now.
<FromGitter>
<Blacksmoke16> cant go wrong with JSON then for now
<riza>
mosquito welcomes you ;)
<riza>
the serialization is _basic_ though, so anything that's not easily stringable is going to put you out of the well-worn pathing
<riza>
github.com/mosquito-cr/mosquito
<SamantazFox>
riza: mosquito looks interesting! The redis/job parts could be of a great use already ^^
<riza>
awesome, glad to hear it. let me know if you have any questions
<SamantazFox>
riza: can it run without redis?
<FromGitter>
<Blacksmoke16> no
<FromGitter>
<Blacksmoke16> as far as i know, as thats what it uses for job queueing
<FromGitter>
<Blacksmoke16> if all you want to do is execute some code on a schedule, is always good ol' cron :shrug:
<SamantazFox>
erf :/
<SamantazFox>
Maintainin very large apps is somewhat a complex nightmare, lol
<FromGitter>
<Blacksmoke16> yes, esp with all the tech debt they usually come with :P
<SamantazFox>
yes lol
<SamantazFox>
Especially since I need to take care of the server-side performance, but alos of the user experience
<SamantazFox>
also*
<FromGitter>
<Blacksmoke16> a lot of it comes down to system design versus actually writing code
<SamantazFox>
yup!
<SamantazFox>
Thing is, I don't want to engage in something quite complex, as I know that later on it will change no matter what
<SamantazFox>
Hence why I want a simple way to send/receive serialized data over an unix socket, so I could separate the various parts of an incredibly large program.
<FromGitter>
<Blacksmoke16> JSON + HTTP would prob be the simplest approach then id say
<SamantazFox>
The problem I have is that the stdlib implementation of an HTTP client/server is lacking...
<SamantazFox>
Currently, we have QUIC, (which is terribly outdated), and that is causing all sorts of issues (especially since it requires a modified SSL library)
<FromGitter>
<Blacksmoke16> getting the code less coupled to each other would help a lot. Also ensuring you avoid depending on class vars and such for sharing state as thats not concurrently safe
<SamantazFox>
Yep, that's the goal! Also, I don't think we're using class vars.
<FromGitter>
<Blacksmoke16> what features are you missing from the http server? the client has some known limitations but there are other libs to handle that stuff if you need like redirect/proxy support etc
<FromGitter>
<Blacksmoke16> 👍
<SamantazFox>
HTTP2 and 3 (QUIC) support.
<SamantazFox>
that's the major drawbacks of the HTTP library in crystal.
<SamantazFox>
(We also have our own proxying code, which I haven't had the time to look at)
<FromGitter>
<Blacksmoke16> fair
<SamantazFox>
Some of the problems we have comes from the double BoringSSL / OpenSSL linking.
<SamantazFox>
That's something I'm also hoping that "workers" using unix sockets will solve.
<SamantazFox>
That way, the back-end HTTP logic can't interact badly with the front-end's
<FromGitter>
<Blacksmoke16> sounds like a plan
<FromGitter>
<Blacksmoke16> now have to see how it goes ha
<SamantazFox>
Lol
<SamantazFox>
Problem is I have everything to learn about paralellism, scheduling, inter-process communication and asychronous stuff.
<FromGitter>
<jrei:matrix.org> also multi process may be more efficient if done properly than multi threading, depending of the use case
<FromGitter>
<jrei:matrix.org> SamantazFox: how multi process will be used for invidious?
<SamantazFox>
jrei: in short, I'd see the JSON parsers (YT parsers) to be sub-processes that run independently, but are "tasked" by the main process using unix sockets.
<SamantazFox>
(And the video/image proxies would be entirely separate binaries, but that doesn't matter as they're almost independant)
<FromGitter>
<jrei:matrix.org> I see. A sub process worker
<FromGitter>
<jrei:matrix.org> the goal is to have a faster experience for the user I guess? ⏎ Because now I suppose you have one invidious process running per core on the server to maximize efficiency
<FromGitter>
<jrei:matrix.org> then each user goes randomly to one
ur5us has joined #crystal-lang
<riza>
SamantazFox: there's some work in process to decouuple it from redis. It's an abstract layer which can be swapped out with an implementation of whatever storage backend you'd like, but no other storage backends have been implemented.
<riza>
my personal belief is that redis is better suited to this kind of work than a typical sql database, but I know it's not a terribly popular tool
<SamantazFox>
riza: I agree with you ^^ I'd like to have redis implemented in the future as a proper caching system. Though, for now I have to live with the technical debt, and requiring our user to deploy a mandatory PG DB + redis is a bit too much!