Anarchos has quit [Quit: Vision[]: i've been blurred!]
bobo_ has quit [Ping timeout: 252 seconds]
bobo_ has joined #ocaml
Haudegen has quit [Ping timeout: 246 seconds]
spip has joined #ocaml
bobo_ has quit [Ping timeout: 246 seconds]
neiluj has joined #ocaml
<neiluj>
hi! has anyone used pthreads in C stubs called from OCaml?
<neiluj>
it looks like the more threads are spawned, the longer the C stub function takes time to execute
<neiluj>
and looking at the graph of cpu usage there's a spike with all cores used for 2-3 seconds, way less than the actual execution time of the C stub
<neiluj>
so I'm wondering if OCaml garbage collector might interfere a bit or not? I'd say no
ewd has quit [Ping timeout: 252 seconds]
<neiluj>
cannot switch to ocaml5 yet unfortunately
<neiluj>
or could this be the time it takes for the os to load things in cache for the threads?
<neiluj>
os->OS
<pgiarrusso>
neiluj: far too much for "loading caches" — RAM bandwidth is measured in tens of _gigabytes_ per second, and all your caches take megabytes
<pgiarrusso>
OTOH, spawning threads _in the same process_ sounds rather unsafe unless OCaml explicitly supports that. So my first (super-naive) question is "could you use a separate process instead"
<pgiarrusso>
neiluj: also are your threads calling C functions (what I assumed above) or OCaml functions (much much more dangerous)
<neiluj>
thanks! C functions only
<neiluj>
actually the wait then spike is normal, I missed some intermediate computation
<neiluj>
however, the timings say that the function executes in 20secs while it looks more like 3 secs
<neiluj>
from the graph of cpu usage
<neiluj>
moreover the sum all the intermediate execution time is bigger than the total execution time
<neiluj>
so it might be a false alert, although it's a bit strange that the timing is not accurate
<pgiarrusso>
"1 second wall-clock/real time, 10 seconds CPU time" would just mean you're running 10 thread on 10 cores
<neiluj>
oh
<pgiarrusso>
"10 seconds wall-clock time, 1 second CPU time" would mean the thread is not running in the other 9 seconds — either blocked on IO, or descheduled because something else is running
<pgiarrusso>
etc
<neiluj>
that must be it, in C I print measure the time in clock cycles
<neiluj>
and in ocaml with Sys.time () which should do the same then
<pgiarrusso>
> Return the processor time, in seconds, used by the program since the beginning of execution.
<hannes>
memory representations.. that reminds me, is there a least recently used implemenetation where the weight of an element is the size in memory? since from a systems perspective, it makes much more sense to be able to say "this cache maximum 2MB" than "this cache maximum 1250 entries" (esp. when each entry may be of different size)?
<zozozo>
hannes: not that I know, but it would be complex to build such a cache, for a few reasons : 1) determining the size of a value takes atime linear in the size (so not great when the value is big), 2) how would you account for memory shared by mutliple values ?
<Armael>
no idea, but that sounds possibly tricky in presence of sharing?
<Armael>
argh zozozo was faster
<zozozo>
Armael: ^^
<hannes>
ok, thanks. to answer the sharing, this is really easy :)
<hannes>
just account it multiple times.
<hannes>
since what I'd like to express is _this cache takes at most 2MB_
<zozozo>
hannes: well, actually, with the ocaml representation, if the cache is, say basically, an array, then you only have to bound the size of the array, and that's it, the cache/array will take a fixed size
<zozozo>
whether the cache keeps some values alive that take more space is another question
<hannes>
zozozo: I can bound the size of the keys of the arrays, but not the values, no?
<hannes>
so if I have 500 keys, uniform ints, but their values are lists (or more complex structures), I don't know how much memory the array (keys + values) will consume!?
<zozozo>
hannes: same difference, whether the cache keeps some keys alive or not is a question separate from the size of the cache itself (in my opinion)
<Armael>
I wonder how bad of an overapproximation you may get if you ignore sharing + store many elements of Set/Map in the cache that share subparts
<Armael>
maybe it's fine in practice
<hannes>
zozozo: hmm, I don't understand what you mean with "same difference"... the issue I try to solve is to have an upper bound of memory for both keys and values
<hannes>
so maybe my question is instead how I can compute the in-memory-transitive-no-sharing-size of a (non-function) value
<Armael>
Obj.reachable_words?
<zozozo>
hannes: the point I'm making is that: if the key is alive (i.e. also present/referenced from another part of your program), then why would it count toward the size taken by the cache ? since even without the cache, you would need to keep the value since it is alive
<hannes>
zozozo: ah, ok. thanks.
<Armael>
zozozo: even in that case I think it makes sense to count it as part of the cache size
<hannes>
in my specific use case, a DNS resolver, the keys and values are not alive in any part of the program
<zozozo>
hannes: you might be interested in reading the following PR (that talks about computing sizes and accounting for size shared by values) : https://github.com/ocaml/ocaml/pull/10031
<hannes>
zozozo: thank you :)
<zozozo>
Armael: well, naively, I'd say you'd need a cache backed by a weak hashtbl, and that tries and count only dead value as "taking up space", because else you don't really count the space used by the cache itself
olle has joined #ocaml
<zozozo>
hannes: ah, in the case where keys and values are not values references from other part of the program that makes sense
<zozozo>
maybe I'm too used to manipulating ASTs and having caches of terms/expressions, :p
troydm has quit [Ping timeout: 252 seconds]
jlrnick has joined #ocaml
<hannes>
So indeed the Obj.reachable_words looks promising, and I've been here before. The now underlying issue is (still) that we use BigArray.t that aren't accounted for appropriately in Obj.reachable_words.
<hannes>
Thus, yet another reason to remove the deep usage of BigArray.t.
calvnce has quit [Quit: Client closed]
jlrnick has quit [Ping timeout: 252 seconds]
wingsorc__ has quit [Ping timeout: 252 seconds]
xd1le has joined #ocaml
calvnce has joined #ocaml
azimut has quit [Remote host closed the connection]
spip has joined #ocaml
bobo has quit [Ping timeout: 252 seconds]
azimut has joined #ocaml
neiluj has quit [Ping timeout: 252 seconds]
Haudegen has quit [Quit: Bin weg.]
calvnce has quit [Quit: Client closed]
troydm has joined #ocaml
xd1le has quit [Quit: xd1le]
jlrnick has joined #ocaml
jedb has quit [Read error: Connection reset by peer]
neiluj has joined #ocaml
waleee has joined #ocaml
calvnce has joined #ocaml
jlrnick has quit [Ping timeout: 260 seconds]
Haudegen has joined #ocaml
jao has joined #ocaml
rgrinberg has joined #ocaml
random-jellyfish has joined #ocaml
waleee has quit [Ping timeout: 252 seconds]
azimut has quit [Ping timeout: 255 seconds]
alexherbo2 has joined #ocaml
Tuplanolla has joined #ocaml
mro has joined #ocaml
Haudegen has quit [Quit: Bin weg.]
random-jellyfish has quit [Quit: Client closed]
mro has quit [Remote host closed the connection]
mro has joined #ocaml
bobo_ has joined #ocaml
szkl has joined #ocaml
spip has quit [Ping timeout: 260 seconds]
bartholin has joined #ocaml
thizanne` is now known as thizanne
jao has quit [Remote host closed the connection]
kakadu has joined #ocaml
Anarchos has joined #ocaml
olle has quit [Quit: Lost terminal]
jao has joined #ocaml
mro has quit [Remote host closed the connection]
mbuf has quit [Quit: Leaving]
kakadu has quit [Quit: Konversation terminated!]
Anarchos has quit [Quit: Vision[]: i've been blurred!]
Haudegen has joined #ocaml
gwizon has joined #ocaml
rgrinberg has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
alexherbo2 has quit [Remote host closed the connection]
alexherbo2 has joined #ocaml
mro has joined #ocaml
mro has quit [Remote host closed the connection]
alexherbo2 has quit [Remote host closed the connection]
alexherbo2 has joined #ocaml
troydm has quit [Ping timeout: 260 seconds]
olle has joined #ocaml
oakyy has joined #ocaml
oakyy has left #ocaml [#ocaml]
mro has joined #ocaml
mro has quit [Ping timeout: 260 seconds]
ewd has joined #ocaml
mro has joined #ocaml
mro has quit [Remote host closed the connection]
ewd has quit [Ping timeout: 248 seconds]
mro has joined #ocaml
alexherbo2 has quit [Remote host closed the connection]
alexherbo2 has joined #ocaml
calvnce has quit [Quit: Client closed]
ewd has joined #ocaml
wingsorc__ has joined #ocaml
waleee has joined #ocaml
neiluj has quit [Ping timeout: 252 seconds]
bgs has quit [Remote host closed the connection]
cedric has joined #ocaml
troydm has joined #ocaml
xd1le has joined #ocaml
mro has quit [Quit: Leaving...]
rgrinberg has joined #ocaml
bartholin has quit [Quit: Leaving]
azimut has joined #ocaml
cedric has quit [Quit: Konversation terminated!]
azimut has quit [Remote host closed the connection]
azimut has joined #ocaml
azimut has quit [Remote host closed the connection]
azimut has joined #ocaml
alexherbo2 has quit [Ping timeout: 260 seconds]
alexherbo2 has joined #ocaml
alexherbo2 has quit [Remote host closed the connection]