00:45 <FromGitter> <wwalker> Well, maybe not. That requires that I have /usr/lib64/crystal on the aarch64 machine. If I had crystal on the aarch64, I wouldn't be cross compiling :-(

00:47 <FromGitter> <Blacksmoke16> you sure? my understanding of cross compiling was its already compiled and it just needs linked on the target system

00:49 <FromGitter> <wwalker> It has me run this: ⏎ ⏎ ```code paste, see link``` [https://gitter.im/crystal-lang/crystal?at=637ebfa3426f9311ffbb86e5]

00:49 <FromGitter> <Blacksmoke16> does it work if you just exclude the `-L`? pretty sure thats not actually the crystal binary, but the source of crystal libs

00:51 <FromGitter> <Blacksmoke16> idk if that's expected or not, feels a bit weird its there...?

00:54 <FromGitter> <wwalker> @Blacksmoke16 it felt weird. And I was getting "not found" for libs that were installed. but now it builds fine. ⏎ ⏎ Thanks! I was about to switch all 20 machines to x86_64 (that is the nice part of AWS EC2... :-)

00:55 <FromGitter> <Blacksmoke16> :shrug: huh

00:55 <FromGitter> <Blacksmoke16> any reason to run on aarch64? amd64 would deff be easier ha

01:12 <FromGitter> <wwalker> the same performance level server for aarch64 costs 80% of what the x86_64 servers cost...

01:12 <FromGitter> <Blacksmoke16> oof

01:12 <FromGitter> <Blacksmoke16> yea thats a fair reason ha

01:24 <FromGitter> <wwalker> yeah, I need to set up 300 machines. so it is the difference between 217,700 USD and 174,200 USD per month. $45,500 / month.

01:33 <FromGitter> <jrei:matrix.org> You could also use qemu, compile statically and put it on the server

01:46 <FromGitter> <stellarpower> Earthly can be useful for this stuff:

01:46 <FromGitter> <stellarpower> https://earthly.dev/

01:47 <FromGitter> <stellarpower> I think it has a way of latching into QEMU to do cross-compilation without as much effort as it often takes

01:47 <FromGitter> <moe:busyloop.net> you need to set up 300 machines with an app called slimer. do you work for the ghostbusters? 🤔

01:47 <FromGitter> <stellarpower> I might be being stupid; is there a nice way to pairwise multiply two arrays without doing a zip then a map?

01:48 <FromGitter> <stellarpower> It's actually called slimmer

01:48 <FromGitter> <stellarpower> It optimises

01:48 <FromGitter> <moe:busyloop.net> ok that sounds less exciting then :P

01:48 <FromGitter> <stellarpower> And the sign of a good optimiser is when it optimises away redundant information in itso wn name

01:48 <FromGitter> <moe:busyloop.net> ha

01:50 <FromGitter> <wwalker> earthly looks interesting!

01:50 <FromGitter> <moe:busyloop.net> if cost is a concern, have you considered hetzner tho? around $60k would buy you the same compute there. probably less.

01:50 <FromGitter> <stellarpower> I've been having a play. IT still suffers form some of the arseyness of dockerfile syntax.

01:51 <FromGitter> <stellarpower> But it's a step in the right direction at lweast

01:51 <FromGitter> <stellarpower> We wanna use it to get reproducible builds, across differnt people's machines, but then the ease of debugging in a local IDE for development work

01:54 <FromGitter> <wwalker> Nope, I am writing a pipeline to ingest about 100 TB a day into an elasticsearch cluster (editing all 100 billion ~1KB json documents) ⏎ The project is named Taz (Tasmanian Devil from Looney Tunes) and the first piece of the pipeline is Slimer because it consumes those 100E9 docs the same way Slimer donsumed hot dogs, just shovels them in and they just fall right through. The analogy was better in my head....

01:55 <FromGitter> <moe:busyloop.net> that's quite some data indeed

01:57 <FromGitter> <wwalker> the data we consume is already in AWS, so the cost to exfiltrate the data would kill any cost savings most likely, and we've already done all our compliance paperword around AWS...

01:59 <FromGitter> <wwalker> However, Hetzner looks nice. I'll keep it in mind for any of my old clients looking for cheap high end cloud servers.

02:01 <FromGitter> <stellarpower> I already stole this idea ;)

02:01 <FromGitter> <stellarpower> I think ours are settled, and struck a deal, but it's interesting as we do need quite some power.

02:01 <FromGitter> <wwalker> Yeah, currently we're doing about 5% of that (5 TB / day, with the ability to ingest about 15 TB/day (in case we get behind because something broke, so we can catch up in a reasonable time), versus 35 TB/day with the ability to ingest 100 TB / day for "catching up")

02:02 <FromGitter> <moe:busyloop.net> hm yea, 100T egress day is hefty

02:03 <FromGitter> <wwalker> well 100TB is our burst, but 30-35 TB / day is still hefty.

02:04 <FromGitter> <moe:busyloop.net> yea, that'd be something like $45k/mo

02:05 <FromGitter> <moe:busyloop.net> could still work out, considering how much you could save on compute

02:05 <FromGitter> <moe:busyloop.net> but if you also have compliance stuff to deal with, and if it's not your own money, may not be worth it ;)

02:06 <FromGitter> <wwalker> Yeah, even talking with other IT folks, when I say things like "I need to move 1.5 Petabytes over the next 5 days" they say things like "you mean terabytes?"

02:08 <FromGitter> <moe:busyloop.net> i would probably say something like "have you tried gzip?" :P

02:12 <FromGitter> <moe:busyloop.net> i'm curious about that elasticsearch cluster tho. do you rebuild it from scratch every day or does it permanently hold an index of that size?

02:13 <FromGitter> <moe:busyloop.net> my experience with ES durability is not great, wonder how many nodes that takes and how you keep them alive lol

02:29 <FromGitter> <wwalker> Currently 120 nodes. 112 with data storage. 48 have 12 TB of hard drives each, 64 have 2 TB of SSDs each. ingest 3 to 5 TB/day, times 2 for storage, as we have 1 replica of each index. each node has 16 cores and 128 GB RAM. Data comes in through the "hot" nodes (SSDs) and sits there for 3 days, then as it exceeds 3 days old, it moves to the "warm" nodes (HDDs) and sits until it is 30 days old and is deleted.

02:30 <FromGitter> <wwalker> I'm expecting roughly 300 nodes to meet our ingest and retention goals.

02:31 <FromGitter> <moe:busyloop.net> ouf.

02:31 <FromGitter> <Blacksmoke16> 💰

02:31 <FromGitter> <moe:busyloop.net> that def sounds like fun :D

02:33 <wwalker> it is fun until you realize that to start up your "test/scaling" env you will be spending $11K a day. And if you forget and leave an "extra hundred or so..." machines running for a week...

02:34 <FromGitter> <moe:busyloop.net> yea. i also don't want to imagine having to reshard or such on that size.

02:38 <FromGitter> <moe:busyloop.net> but still sounds like fun anyway. with some sweaty palms in between. ;)

02:38 <wwalker> yes, our ES is "write once". No updates, no document deletes.

02:38 <wwalker> yeah, this is my sweaty palms week for this year and next year!

02:39 <FromGitter> <moe:busyloop.net> ah good. that def makes life easier. updates are evil.