OCamlPro[m] has quit [Quit: You have been kicked for being idle]
bartholin has quit [Ping timeout: 240 seconds]
bartholin has joined #ocaml
mro has quit [Remote host closed the connection]
<d_bot_>
<wokalski> companion_cube: regarding lwt+tracing; what were your thoughts about the integration? We talked about this exact topic a couple of weeks ago with my team and we thought about creating a wrapper (`type 'a t = (ctx * 'a) Lwt.t`) that will carry the context around behind the scenes but maybe there's a smarter way.
gentauro has quit [Read error: Connection reset by peer]
gentauro has joined #ocaml
mro has joined #ocaml
mro has quit [Ping timeout: 252 seconds]
gravicappa has joined #ocaml
bartholin has quit [Ping timeout: 268 seconds]
bartholin has joined #ocaml
waleee has joined #ocaml
kurfen has joined #ocaml
<companion_cube>
@darrendl tough ask :D
<d_bot_>
<darrenldl> (:
<companion_cube>
@wokalski: @antron has some cool tricks in Dream, maybe Lwt.key might be a good fit?
<d_bot_>
<Anurag> @companion_cube Lwt.key is something I looked at for similar needs in the APM library I worked on. But I see it marked as deprecated <https://github.com/ocsigen/lwt/blob/master/src/core/lwt.mli#L1691-L1700> so I didn't push forward with that approach. I instead ended up using `with_` style functions and manually forwarding the trace context everywhere I needed to (including forwarding it to logs for adding the trace context as a
<companion_cube>
yeah I guess
waleee has quit [Quit: WeeChat 3.4]
szkl has quit [Quit: Connection closed for inactivity]
bartholin has quit [Ping timeout: 240 seconds]
Haudegen has joined #ocaml
bartholin has joined #ocaml
<companion_cube>
you need some manual work anyway, if you're forwarding trace_id/span_id to remote hosts
<d_bot_>
<Anurag> That is true. So far my approach has been to create http clients from a given trace context. Then all calls made using that client will automatically add a trace context http header when making outgoing calls.
<companion_cube>
so I'm looking at Logs, and the reporter interface seems very promising
<companion_cube>
one interface can serve both sync and async needs
d_bot_ has quit [Remote host closed the connection]
d_bot has joined #ocaml
olle_ has joined #ocaml
mro has joined #ocaml
mro has quit [Remote host closed the connection]
mro has joined #ocaml
mro has quit [Remote host closed the connection]
dextaa_ has joined #ocaml
<d_bot>
<mbacarella> I kinda wish libraries that provided independent I/O endpoints just implemented a synchronous interface so that you can wrap them in `In_thread.run (fun () -> ...)` if you want an Async version
<d_bot>
<mbacarella> dealing with functorized promise modes in every single library in the world is kind of a drag
<companion_cube>
the Logs thing should work, actually, in this case
<companion_cube>
with a small Lwt wrapper
<companion_cube>
I just need an async reporter now :D
Serpent7776 has quit [Read error: Connection reset by peer]
<companion_cube>
a callback to return a value (here, a promise); and a callback to signal the job is done (here, to wakeup the promise)
<d_bot>
<orbitz> I think that would be a subpar experience. My preference is for people to offer an API that takes bytes and gives back decoded frames, and then you can implement a sync or async API on top of that.
<companion_cube>
depends for what
<companion_cube>
for a protocol, I agree, I guess? not sure it's easier than the functor though
<d_bot>
<orbitz> I haven't seen it done very successful as a functor. Cohttp, for example, is a pain to implement, IMO
Serpent7776 has joined #ocaml
<companion_cube>
well if you want to see an incredibly large scale and robust system (lolol), see: irc-client
<companion_cube>
(warning: a smidge sarcastic here :D)
<d_bot>
<orbitz> It was hard to tell over the lol's 🙂
<companion_cube>
you never know!
<d_bot>
<mbacarella> the reason dealing with functorized promise monads is kind of a drag is because probably the author only implements either lwt or async
<d_bot>
<mbacarella> but also if you're trying to release a library you feel a little guilty because you've implemented only async or lwt bindings
<d_bot>
<mbacarella> that is, there's pressure to do the functorized promise monads. the concept alone creates some angst
<d_bot>
<orbitz> To support companion_cube and not support my claim: dns-client does something not terrible in that its API surface is fairly small so you do implement a functor with monad + their API and then it takes care of calling the right things. I think it works ok there, but it's delicate work and, for example, I had to request (and was thankfully obliged) to change the API in a way that made handling cancellation correctly possible
unyu has quit [Quit: brb]
<companion_cube>
@mbacarella it does mean you can relatively easily add the other one, whereas if you don't functorize (or do the stateless IO-less version) you have to rewrite everything
<d_bot>
<mbacarella> sorry, rephrase?
<companion_cube>
you might functorize and do only lwt
<companion_cube>
but adding async is then easier than if you had no functor to start with
<d_bot>
<mbacarella> what i'm saying is there's no shame in your wrapper being `In_thread.run (fun () -> let h = Sync_io.create () in Sync_io.send_request h request; let reply = Sync_io.read_response h; Sync_io.close h; reply)`
<companion_cube>
for a stub, sure
<companion_cube>
for actual use, meh :D
<d_bot>
<mbacarella> i mean, the functorized monad version will be doing something close to that behind the scenes anyway
<d_bot>
<mbacarella> i will reiterate i meant for independent I/O endpoints. if you have to share handles between many calls yeah that has obvious issues
<companion_cube>
hmm, In_thread means using a thread pool? might actually scale decently, I guess
<d_bot>
<mbacarella> IIRC all I/O in Async is dispatched to a thread-pool
<d_bot>
<orbitz> No
<d_bot>
<orbitz> FIle I/O probably is
<companion_cube>
on linux, probably
olle has quit [Quit: Lost terminal]
olle_ has quit [Remote host closed the connection]
rgrinberg has joined #ocaml
rgrinberg has quit [Client Quit]
<d_bot>
<mbacarella> hmm, indeed, async_unix src/unix_syscalls.ml is heavy on the In_thread. stuff except for socket stuff
mro has joined #ocaml
<d_bot>
<mbacarella> I wonder why the break-out for network stuff. Guessing the payloads are tinier and more latency sensitive.
mro has quit [Remote host closed the connection]
<d_bot>
<orbitz> @pilothole No, it's because most OS's have functioning non-blocking network APIs, but not for file systems
<d_bot>
<mbacarella> ah, that sounds more plausible
<companion_cube>
io_uring is supposed to change this for linux
<companion_cube>
with some exciting new vulnerabilities, too
<d_bot>
<orbitz> It's not progress without a security hole or two
<d_bot>
<mbacarella> you would still probably rather use the threaded interface instead of managing a non-blocking state machine if you could get away with it? I imagine it doesn't perform as awesome
<d_bot>
<orbitz> Threads do not performa wesome
<d_bot>
<mbacarella> actually, why not? you have context switches whether you use threads or non-blocking I/O
<d_bot>
<mbacarella> in fact, i could imagine one argument where it's better to have the kernel operate the state machine for you and let you have concurrency through threading because the kernel knows more about what's going on and is heavily optimized
<d_bot>
<orbitz> @pilothole The information around why is more efficient to use the non-blocking APIs is pretty detailed on the internet. You can start with C10k problem, which is quite old
<companion_cube>
@orbitz depends for what, i guess
<companion_cube>
doesn't scale as well, for sure
<companion_cube>
(although c10k is doable)
<d_bot>
<orbitz> Sure, c10k is not large these days, but the principles still apply
<d_bot>
<orbitz> My point is: this isn't some whacky decision made by Ocaml developers, there is a pretty robust discussion that's gone on over the decades
<d_bot>
<mbacarella> i didn't say it was a wacky decision, just trying to remember from first principles
<d_bot>
<mbacarella> for the sake of argument
<sim642>
Uhoh, this is confusing: Error: This expression should not be a function, the expected type is Ppx_deriving_ord_helper.t -> Ppx_deriving_runtime.int
waleee has joined #ocaml
<d_bot>
<Anurag> @pilothole async uses `In_thread` when performing file io since that doesn't support nonblocking operations. For socket io (If Fd.t supports nonblocking mode) it doesn't go through In_thread (unless you are performing a writev call with a large number of iovecs)
<d_bot>
<Anurag> Ah, nvm, you just said that a little while ago (I should have scrolled further 😄 )
<companion_cube>
sim642: weird indeed
<d_bot>
<Anurag> async (and lwt) does make working with nonblocking IO pretty straightforward so as a user of Lwt_io/Reader/Writer (or any other wrapper over some lwt/async primitives), one doesn't really have to think about managing IO threads, or orchestrating non blocking state machines.
<d_bot>
<mbacarella> anyway, haven't threads traditionally sucked for c10k level concurrency because the OS doesn't make smart decisions about what to schedule next?
<d_bot>
<mbacarella>
<d_bot>
<mbacarella> i recall an old observation that if you had N processes waiting on accept the kernel would wake all of them only to put N-1 of them back to sleep, because only one can get the next client
<d_bot>
<mbacarella> (the kernel here being linux)
<d_bot>
<mbacarella> or, stated another way, you could blow my argument up where when i say "it's better to have the kernel operate the state machine for you and let you have concurrency through threading because the kernel knows more about what's going on and is heavily optimized" i'm being hopelessly naive
<d_bot>
<orbitz> The general issue is threads are heavy and and if you have lots of connections, you use up a lot of RAM + context switches
<d_bot>
<mbacarella> right. but assuming you had a perfectly executed concept, why should 1000 threads waiting on 1000 different descriptors be heavier than one thread waiting to find out about an event in 1 of 1000 descriptors? why does one model require more context switches than the other?
<d_bot>
<mbacarella> (true 1000 different threads means 1000 different stacks)
mro has joined #ocaml
<d_bot>
<orbitz> It depends on the specifics. But let's say you wrap "read" in an In_thread.run, you need to (possibly) spawn a thread, wait for its response, context switch over to it, read (syscall), notify that the result is ready, context switch back, handle the notification is ready, and continue, and maybe kill a thread. The first and last are alleviated by a thread pool to some degree
<d_bot>
<orbitz> if you batch more operations, then context switching becomes less dominating
<d_bot>
<orbitz> But per thread you have 4 context switches in that scenario: into and out of the thread, and the syscall
<d_bot>
<orbitz> at least 4
<companion_cube>
otoh if you read gigantic files in a few threads, I don't think lwt would help
<companion_cube>
performance isn't 1 dimensionnal
<d_bot>
<orbitz> Excuse me, I'm referring specfically to network here. I should have been more explicit
<d_bot>
<mbacarella> ah, fair, in the ocaml model dispatching things to thread pools is 4x as many context switches as needed
<d_bot>
<orbitz> @pilothole what I just described has nothing to do with ocaml
<d_bot>
<mbacarella> sorry, i meant the ocaml models we have been talking about
<d_bot>
<orbitz> What I described is inherent to any unixy thread model
<d_bot>
<orbitz> Go and Erlang handle pushing I/O to an event loop for you for example. High performance networking APIs like Java's Jetty use an even tloop as well
<d_bot>
<orbitz> Netty, maybe I menat
<d_bot>
<rudy> Does anyone have a nice resource on what OCaml is good for? I'm looking for something in-depth, and perhaps with comparisons to other famous languages like Java or C.
<d_bot>
<orbitz> @rudy It's a general purpose programming language, so it's good for all the things!
<Corbin>
rudy: Yes. Additionally, it can.
waleee has quit [Ping timeout: 240 seconds]
<d_bot>
<mbacarella> kind of? the reason for the 4x context switches in this approach is because you're throwing away the possibility of sequencing by dispatching everything to an I/O threadpool
<d_bot>
<orbitz> @pilothole I don't undertand what you mean. The scenario I described was pretty specific.
<d_bot>
<orbitz> As I said, context switches become less dominating the more amount of work you do in the thread, but if you are running a specific syscall in a thread, you have a bunch of context switches
<d_bot>
<Et7f3 (@me on reply)> it is not one vs one but it show arguments that help you choose
<d_bot>
<Et7f3 (@me on reply)> The conclusion I took from this: when a langage is static and include many warnings it become safer by design.
<companion_cube>
@orbitz yeah yeah, for networking it's quite clear
<d_bot>
<mbacarella> what i'm saying is you could probably eliminate a context switch if you don't have to enter and exit a threadpool if you do two system calls in a row if they depend on each other (e.g. a thread dedicated to reading a request and then sending a reply), but i now agree with you/remember threads suck because you wrap each system call context switch in a thread entry/exit context switch
<d_bot>
<orbitz> @pilothole That would be "doing more stuff in the thread" as I mentioned which will reduce your context switches.
<d_bot>
<mbacarella> right
<d_bot>
<orbitz> But even still, depending on your scale, you're using more resources than are necessary
<d_bot>
<Et7f3 (@me on reply)> log4j come from a hyper dynamic language where deserialization can run arbitrary code. It is not the first time https://opensource.googleblog.com/2017/03/operation-rosehub.html @rudy The time you earn by developing quickly is the time you will lose later in worse. JS was coded in 7 day and now many webdev spend time on creating tool to fix it.
<d_bot>
<mbacarella> yes, i agree. i will withdraw the contention. threads sux non-blocking I/O roolz
<d_bot>
<orbitz> @pilothole Consider you have 100 connections and they all have something to read at the same time: you can do 1 kqueue call + 100 reads (202 context switches) or you can do 100 * 4, at least, context switches
<d_bot>
<mbacarella> yes right clearly worse
mro has quit [Remote host closed the connection]
mbuf has quit [Quit: Leaving]
<d_bot>
<Et7f3 (@me on reply)> @Bluddy description of #débutants is not clickable you can write <# 913078886345085008> without espace. And why we have notifications in #rules did you modified something ?
zebrag has joined #ocaml
Haudegen has quit [Quit: Bin weg.]
<d_bot>
<mbacarella> neat article
<d_bot>
<Et7f3 (@me on reply)> reposted in #share
rgrinberg has joined #ocaml
oriba has joined #ocaml
kakadu has quit [Remote host closed the connection]
Techcable has quit [Remote host closed the connection]
Techcable has joined #ocaml
chrisz has quit [Ping timeout: 252 seconds]
chrisz has joined #ocaml
Haudegen has joined #ocaml
dextaa_ has quit [Remote host closed the connection]
bartholin has quit [Quit: Leaving]
<d_bot>
<mbacarella> so you have a `type foo = Foo | Bar | Baz` exposed in a library. you want to get free sexp conversions for it without copy/pasting it into your library. sadly `type foo = Library.foo [@@deriving sexp]` doesn't work
<d_bot>
<EduardoRFS> ppx_import
<d_bot>
<mbacarella> that's awesome
olle has joined #ocaml
mro has joined #ocaml
mro has quit [Ping timeout: 240 seconds]
bobo has joined #ocaml
spip has quit [Ping timeout: 252 seconds]
t-j-r has quit [Quit: quitting]
infinity0 has quit [Remote host closed the connection]
infinity0 has joined #ocaml
mro has joined #ocaml
rgrinberg has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]