<adam12>
They clobber global namespace on some too, which is unfortunate. I know they plan on them being used for Eventide but it would still be nice to not have them all at the top.
<rapha>
the parallel gem is nice, but DB=Sequel.connect('sqlite://23GB_of_text.sqlite3');something=Parallel.map(DB[:table].all){|rows| do_stuff} will have to read the whole 23GB into memory _first_ before it can begin with do_stuff. Is there a way to have it start doing stuff _while_ it's still reading the data, at least as fast as the data will come in?
sgt_chuckles has joined #ruby
Ziyan has quit [Ping timeout: 256 seconds]
Ziyan has joined #ruby
jpn has quit [Ping timeout: 240 seconds]
cocalero has quit [Quit: Going offline, see ya! (www.adiirc.com)]
hanzo has joined #ruby
dionysus69 has quit [Ping timeout: 250 seconds]
giorgian has quit [Ping timeout: 260 seconds]
Ziyan has quit [Ping timeout: 248 seconds]
<leah2>
use .each?
<leah2>
instead of .all
Ziyan has joined #ruby
<leah2>
or just pass the DB to Parallel.map i guess
giorgian has joined #ruby
<ox1eef>
a lazy Enumerator comes to mind.
Ziyan has quit [Ping timeout: 260 seconds]
<sgt_chuckles>
im trying to scaffold a new package in my rails app with `rails g package package_name` but I get an error saying: "Could not find mysql2-0.5.3 in any of the sources. Run `bundle install` to install missing gems."
Ziyan has joined #ruby
<sgt_chuckles>
When I run bundle install, I get another error saying: "An error occurred while installing mysql2 (0.5.3), and Bundler cannot continue.
<sgt_chuckles>
Make sure that `gem install mysql2 -v '0.5.3'` succeeds before bundling."
<sgt_chuckles>
when i run that, i get another error message -_-
<sgt_chuckles>
lost for ideas here
<ox1eef>
what platform are you on? it sounds like the mysql native extension is probably failing to build.
<sgt_chuckles>
im on macos
<ox1eef>
i dunno about macos
gr33n7007h has quit [Ping timeout: 256 seconds]
giorgian has quit [Ping timeout: 240 seconds]
sgt_chuckles has quit [Quit: Client closed]
sgt_chuckles has joined #ruby
<havenwood>
sgt_chuckles: Have you install mysql?
<havenwood>
Not the gem, the DB?
<sgt_chuckles>
how do i know if i have it?
<havenwood>
sgt_chuckles: brew install mysql
<sgt_chuckles>
thing is, im using a remote db, so i don't think i need?
noname has joined #ruby
<havenwood>
I mean, you can ask brew if it's installed, but betcha it isn't.
<ox1eef>
you need mysql-client at least
<havenwood>
sgt_chuckles: the gem needs the library headers
<sgt_chuckles>
aah, i see
<sgt_chuckles>
okay, im installing now
Ziyan has quit [Ping timeout: 248 seconds]
Sankalp has quit [Ping timeout: 260 seconds]
Ziyan has joined #ruby
<sgt_chuckles>
So when I try running `rails g package echo` again, I get: "Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2) (Mysql2::Error::ConnectionErr"
oxfuxxx has joined #ruby
<havenwood>
sgt_chuckles: You'd have to be running MySQL to connect to it.
<havenwood>
You have to install MySQL to build the gem extension and you have to run MYSQL to connect.
<havenwood>
Make sure you can connect to the DB without Ruby first and the Ruby part should be easy.
<havenwood>
(from this machine)
Sankalp has joined #ruby
giorgian has joined #ruby
jpn has joined #ruby
oxfuxxx has quit [Client Quit]
graywolf has joined #ruby
jpn has quit [Ping timeout: 276 seconds]
giorgian has quit [Ping timeout: 276 seconds]
noname has quit [Quit: Leaving]
<adam12>
I must be out of Rails a while now because I don't recall rail g package..
giorgian has joined #ruby
Ziyan has quit [Ping timeout: 260 seconds]
Ziyan has joined #ruby
protektwar has quit [Ping timeout: 240 seconds]
giorgian has quit [Ping timeout: 248 seconds]
<weaksauce>
i don't think that's standard adam12
<weaksauce>
probably a gem
hololeap_ has quit [Ping timeout: 240 seconds]
hololeap_ has joined #ruby
giorgian has joined #ruby
giorgian has quit [Ping timeout: 260 seconds]
Inoperable has quit [Quit: All your buffer are belong to us!]
Ziyan has quit [Ping timeout: 248 seconds]
Ziyan has joined #ruby
giorgian has joined #ruby
Inoperable has joined #ruby
hololeap_ has quit [Remote host closed the connection]
giorgian has quit [Ping timeout: 246 seconds]
hololeap_ has joined #ruby
dionysus69 has joined #ruby
Ziyan has quit [Ping timeout: 256 seconds]
Ziyan has joined #ruby
giorgian has joined #ruby
jpn has joined #ruby
Buggys has quit [Quit: ZNC 1.6.3+deb1+trusty0 - http://znc.in]
hanzo has quit [Quit: Connection closed for inactivity]
mixfix41 has joined #ruby
sgt_chuckles has quit [Quit: Client closed]
graywolf has quit [Quit: WeeChat 3.5]
<rapha>
leah2: wait, how does that work ... passing the DB to Parallel.map? Using .each was the thing I first thought about, but it doesn't work like that ('LocalJumpError: no block given (yield)'). And there's a bigger problem anyways as the stuff inside of the Parallel.map doesn't fit into memory, apparently not even into 64GB of it.
<rapha>
havenwood: something is already "wrong" with DB[:table].all.paged_each(rows_per_fetch: 2){|row| pp row; exit} ... it takes a looong time for the pp to happen (in fact i'll Ctrl-C it now after waiting 30 minutes)
<rapha>
time sqlite3 file.db 'select * from table limit 2 offset 1000;' comes back after 0.009s
<rapha>
oooh i think it needs to be without #all
<rapha>
okay, needs to be .order().paged_each()
<rapha>
and only works with sqlite3, not with extralite
roadie has joined #ruby
<rapha>
this help nicely with the RAM problem, but it doesn't work with Parallel
<rapha>
+s
roadie has quit [Ping timeout: 248 seconds]
<weaksauce>
doesn't offset get really slow the further in the offset you are?
<rapha>
i don't know weaksauce but that's what Sequel does here under the hood
<weaksauce>
yeah I am pretty sure it has to fetch all the records and then discard the results after counting n offset
<rapha>
fwiw i have this running now with rows_per_fetch:80000 and it just got to half a million records processed and is still going reasonably fast. ofc it'd go faster on 20 CPUs than on just 1 :)
<weaksauce>
has it been slowing down each iteration?
<rapha>
it's fast enough to not be able to really tell (as a progress report, i have the number of iterations showing as print "\\033[K\r#{i}")
<rapha>
hmm it does pause every now and then ... safe bet to say to fetch the next 80000?
<leah2>
isn't to_enum enough?
jpn has quit [Quit: leaving]
<rapha>
leah2: for passing this to Parallel.each you mean?
<leah2>
yes
jpn has joined #ruby
jpn has quit [Client Quit]
<rapha>
seems to just take a giant, long while again before anything starts happening
emcb549 has joined #ruby
RickHull has joined #ruby
emcb54 has quit [Ping timeout: 256 seconds]
emcb549 is now known as emcb54
roadie has joined #ruby
<weaksauce>
rapha what's the structure of your data?
<weaksauce>
if you can use an indexed column it's much faster
<weaksauce>
like instead of select * from foo limit 80000 offset N
emcb540 has joined #ruby
<weaksauce>
you'd so something like lastrecord = row.timestamp; select * from froo limit 80000 where timestamp > timestamp
<weaksauce>
er timestamp > lastrecord
<weaksauce>
and you'd have to order by the time
emcb54 has quit [Ping timeout: 246 seconds]
emcb540 is now known as emcb54
hololeap_ has quit [Ping timeout: 240 seconds]
idiocrash has joined #ruby
hololeap_ has joined #ruby
idiocrash_ has joined #ruby
giorgian has quit [Ping timeout: 248 seconds]
roadie has quit [Ping timeout: 248 seconds]
idiocrash has quit [Ping timeout: 256 seconds]
z4kz has joined #ruby
giorgian has joined #ruby
roadie has joined #ruby
giorgian has quit [Ping timeout: 246 seconds]
roadie has quit [Ping timeout: 248 seconds]
<leah2>
shouldnt a plain select * already stream the result?