<neiluj>
Problem is, for a file just under 1Go, this explodes memory consumption
<neiluj>
Is there a more efficient way to achieve that? RPCs goes through http
<Armael>
you want to stream the file rather than loading it entirely in memory
<neiluj>
Thanks Armael!
<neiluj>
hmmm, not sure if the RPC library I'm using support streaming
glassofethanol has joined #ocaml
<neiluj>
maybe break the payload into smaller ones
<Armael>
I don't know much about your problem domain, but since http supports streaming, either the RPC library on top to leverages that and has a streaming API that you could use, or it doesn't, and then you have to implement streaming at the "protocol" layer that you're implementing on top of the RPC library
<Armael>
(i.e., as you said, explicitly break your payload into chunks)
gravicappa has quit [Ping timeout: 252 seconds]
<neiluj>
thanks, going to check that :)
<neiluj>
oh and another question related to debugging: I've noticed that the p2p library I'm using can't send payloads of more than 1Go through curl. Are there better ways than pinning deoendencies patched with some Printf to see what's going on?
gravicappa has joined #ocaml
favonia has quit [Ping timeout: 252 seconds]
<neiluj>
huh curl is irrelevant in this case
favonia has joined #ocaml
bartholin has joined #ocaml
favonia has quit [Ping timeout: 255 seconds]
favonia has joined #ocaml
favonia has quit [Ping timeout: 255 seconds]
favonia has joined #ocaml
<d_bot>
<Alistair> What does "non-expansive" mean in the context of `typecore.ml` in the typechecker?
favonia has quit [Ping timeout: 240 seconds]
favonia has joined #ocaml
gravicappa has quit [Ping timeout: 255 seconds]
PinealGlandOptic has quit [Quit: leaving]
favonia has quit [Ping timeout: 272 seconds]
favonia has joined #ocaml
favonia has quit [Ping timeout: 240 seconds]
favonia has joined #ocaml
gravicappa has joined #ocaml
mbuf has quit [Quit: Leaving]
Haudegen has quit [Quit: Bin weg.]
<companion_cube>
wow, what kind of p2p library is that
<companion_cube>
ah, can't
<companion_cube>
like Armael says, streaming
<companion_cube>
or, even better, not http
<neiluj>
In fact, the http RPC module works fine, with curl you can upload stuff to a node without issues. It's the client that needs to read a file and store it in memory to upload it.
<neiluj>
The client basically implements curl
<companion_cube>
you're using a http client library, right?
<neiluj>
right
<companion_cube>
then you just need chunked encoding I think
<companion_cube>
that wouldn't work over http, would it?
<sadiq>
no?
<sadiq>
if you've written the response headers already and have the file size ahead of time, would there be an issue?
<sadiq>
I mean it's only going to work in a blocking setup
<companion_cube>
the issue is on the other side, imho
<companion_cube>
a server won't expect a large payload in one chunk, when chunk-encoding exists precisely for that reason
<sadiq>
that's true.
Leonidas_ is now known as Leonidas
Leonidas is now known as Guest5726
terrorjack has quit [Remote host closed the connection]
Guest5726 has quit [Changing host]
Guest5726 has joined #ocaml
terrorjack has joined #ocaml
Guest5726 is now known as Leonidas
favonia has quit [Ping timeout: 255 seconds]
favonia has joined #ocaml
terrorjack has quit [Remote host closed the connection]
terrorjack has joined #ocaml
unyu has quit [Read error: Connection reset by peer]
terrorjack has quit [Remote host closed the connection]
terrorjack has joined #ocaml
terrorjack has quit [Remote host closed the connection]
terrorjack has joined #ocaml
<neiluj_>
okay, so yeah the chunked transfer encoding must be implemented in this case :/
<neiluj_>
tempted to call curl in the client code :p
unyu has joined #ocaml
<companion_cube>
there's ocurl
<neiluj_>
oh nice, thank you :)
terrorjack has quit [Remote host closed the connection]
<neiluj_>
and you made this library!
terrorjack has joined #ocaml
<companion_cube>
ocurl? no :D
<neiluj_>
my bad, ezcurl
favonia has quit [Ping timeout: 272 seconds]
favonia has joined #ocaml
<companion_cube>
oh I forgot about that :s
<olle>
I have a vague preminition I'll never code in ML again :/
<olle>
premonition*
* olle
:(
<neiluj_>
olle, why?
<olle>
neiluj_: I'm moving up the ladder in PHP-land, managing etc
<olle>
Buuuut
<olle>
My boss recently expressed an interest in React...
<olle>
React --> Reason --> OCaml
<olle>
\o/
<olle>
And now they denied the |> operator in PHP too -.-
<companion_cube>
don't they have `.` already?
terrorjack has quit [Remote host closed the connection]
<olle>
xD
<olle>
not sure if joke, but `.` is string concat
<companion_cube>
ah, -> maybe
terrorjack has joined #ocaml
<companion_cube>
neiluj_: I think ezcurl could be modified to send a stream
<olle>
-> is method/property access
<companion_cube>
yes
<companion_cube>
so you can chain them, no?
<olle>
Sure
<olle>
Like with builder pattern, $query->select()->from()->etc()
<neiluj_>
olle, Reason is nice! hope it'll work out!
mro has joined #ocaml
<olle>
neiluj_: They just changed name, or what happened? It's always something with those guys. ^^
terrorjack has quit [Remote host closed the connection]
<neiluj_>
companion_cube, thanks, will look into it
<neiluj_>
but I'll also see if it's doable to patch the library that's already in use
<neiluj_>
olle, you mean, ReScript? :D
terrorjack has joined #ocaml
<neiluj_>
their transpiler compile to outdated ocaml tho
mro has quit [Remote host closed the connection]
<olle>
hm
<olle>
yes, rescript
<d_bot>
<thangngoc89> Try melange 🙂
<d_bot>
<thangngoc89> I hope 4.12 is new enough for you
<companion_cube>
neiluj_: what do you currently use?
<neiluj_>
thangngoc89: that's cool :)
<neiluj_>
a thin layer on the Resto http/json RPC client/server from the Tezos codebase
<neiluj_>
afraid it'll be too difficult to add this feature
favonia has quit [Ping timeout: 245 seconds]
favonia has joined #ocaml
<companion_cube>
what do _they_ use for http?
<neiluj_>
cohttp-lwt
<d_bot>
<thangngoc89> What should I look into if I want to encode/decode something into bytes? The data should last several years and I don’t want to encode 100GB of data on every compiler upgrades
<companion_cube>
neiluj_: I guess you can ask someone knowledgeable about cohttp if it supports *sending* streams
<neiluj_>
good idea! it seems that the server can produce chunked answers, so that's a starting point to add this to the client
favonia has quit [Ping timeout: 272 seconds]
favonia has joined #ocaml
favonia has quit [Ping timeout: 240 seconds]
favonia has joined #ocaml
terrorjack has quit [Remote host closed the connection]
mro has joined #ocaml
terrorjack has joined #ocaml
glassofethanol has quit [Quit: leaving]
<d_bot>
<darrenldl> @thangngoc89 depends on your objective: i was gonna make something for long term data storage in ocaml, but not fully committed to the idea yet
<companion_cube>
is acid-state from Haskell something that would make sense?
Haudegen has quit [Quit: Bin weg.]
<d_bot>
<darrenldl> ye, really depends on what the something is, and how static it remains
gareppa has joined #ocaml
Corbin has joined #ocaml
mbuf has quit [Quit: Leaving]
Guest1511 has quit [Quit: Konversation terminated!]
spip has joined #ocaml
spip is now known as Guest7130
mro has quit [Quit: Leaving...]
favonia has quit [Ping timeout: 240 seconds]
favonia has joined #ocaml
gareppa has quit [Quit: Leaving]
bartholin has quit [Quit: Leaving]
<d_bot>
<thangngoc89> @darrenldl it's some 100GB key-value data that I am going to store in levelDB.
<d_bot>
<thangngoc89> it's read only data
<d_bot>
<thangngoc89> computing the data from raw data is very time consuming so I'm hoping not to that very often because I do like using latest compiler version
favonia has quit [Ping timeout: 240 seconds]
chrde has joined #ocaml
favonia has joined #ocaml
Haudegen has joined #ocaml
<chrde>
I have these types
<chrde>
type foo =
<chrde>
| Red
<chrde>
| Green
<chrde>
type bar =
<chrde>
| Red
<chrde>
| Blue
<chrde>
I am having troubles trying to disambiguate between both usages of `Red`, specifically in the context of a menhir
<chrde>
rule, e.g :
<chrde>
rule_1:
<chrde>
| HELLO { Red } (*this one should be of type `foo` *)
<chrde>
rule_2:
<chrde>
| BYE { Red } (*this one should be of type `bar` *)
<chrde>
renaming them is always an option - but it is very obvious for the reader which type it is depending on the context
<chrde>
I tried adding these rules, but they dont change anything - they seem to be hints for the error messages
<chrde>
%type <foo> rule_1
<chrde>
%type <bar> rule_2
<companion_cube>
chrde: I'd recomment using different modules
<companion_cube>
so you get Foo.Red and Bar.Red
<companion_cube>
and you can still locally open Foo when it's all you need
<chrde>
companion_cube ok thanks :)
chrde has quit [Quit: Client closed]
favonia has quit [Ping timeout: 255 seconds]
favonia has joined #ocaml
cedric has joined #ocaml
terrorjack has quit [Read error: Connection reset by peer]
terrorjack has joined #ocaml
olle has quit [Ping timeout: 252 seconds]
gravicappa has quit [Ping timeout: 252 seconds]
zebrag has joined #ocaml
adanwan_ has joined #ocaml
adanwan has quit [Ping timeout: 244 seconds]
favonia has quit [Ping timeout: 240 seconds]
favonia has joined #ocaml
favonia has quit [Ping timeout: 252 seconds]
favonia has joined #ocaml
favonia has quit [Ping timeout: 276 seconds]
favonia has joined #ocaml
<d_bot>
<sarna> anyone using [neo]vim for ocaml and has their dotfiles publicly available? I'd be thankful for some resources
<companion_cube>
if you're not afraid of staring into the abyss of a 13y old config file: https://vrac.cedeela.fr/.vimrc
<d_bot>
<sarna> stupid ios won't let me change the extension so I can read it, lol
<d_bot>
<sarna> gonna have a look tomorrow on my pc, thanks
dhil has joined #ocaml
Stumpfenstiel has joined #ocaml
<cemerick>
Is there any way to constrain the scope of an `include` or `let open`?
<cemerick>
(context being, a lot of library modules like to provide operators, and it'd be nice to have control over which takes precedence beyond the ordering of includes or opens)
<d_bot>
<Shon F> Doesn’t `let open ` give very fine grained constrain on scope?
<d_bot>
<Shon F> You can also use the bracket delimited opens: `Mod.( c * b + c )` wherein those operators will be the ones provided by `Mod` if it happens to provide such.
favonia has quit [Ping timeout: 240 seconds]
favonia has joined #ocaml
terrorjack has quit [Read error: Connection reset by peer]
favonia has quit [Ping timeout: 250 seconds]
favonia has joined #ocaml
terrorjack has joined #ocaml
favonia has quit [Ping timeout: 255 seconds]
favonia has joined #ocaml
oriba has joined #ocaml
favonia has quit [Ping timeout: 255 seconds]
favonia has joined #ocaml
Stumpfenstiel has quit [Ping timeout: 252 seconds]
<cemerick>
@leviroth: right, but afaict I'd have to replicate the module sig modulo the vals i don't want...pretty onerous if the module in question has a large surface area
terrorjack has quit [Read error: Connection reset by peer]
<cemerick>
It wants something like `let open (M : module type of M without [foo, bar]) in... `
terrorjack has joined #ocaml
cedric has quit [Quit: Konversation terminated!]
<d_bot>
<Alistair> I'm currently working on a ppx w/ the ability to do something like that