10:57 <Try2Code> hi! can some enlighten why builders get into 'RETRY' mode? The docu gives a connection loss as the most likely reason. A normal build takes 30-60 mins and 50% of the builds end up in a RETRY (some of them 2 times).

10:58 <Try2Code> I have an older system running (0.8.x) which does not show that behaviour. but 'automatic retry' might not be available in these old versions ...

11:47 <Guest5> Hi guys, just wanted to post one short question: It looks like that sending the logs of a build-process to the master actually needs longer than building my application. I see that, because the buildbot-worker already finished building (theres no load anymore on htop) and the buildbot-master does print logs for another 20 Minutes or so. Is there a

11:47 <Guest5> way to speed that log-send-process up a bit? :)

12:01 <Try2Code> @Guest5 : Do you use "follow": True in the server config?

12:02 <Try2Code> I also noticed some lag, but not too such an amount like you

12:04 <felixfontein> Guest5: how many log lines do you have? and which database are you using?

12:05 <felixfontein> I remember similar troubles when I used sqlite in the beginning; switching to MySQL/MariaDB fixed that

12:18 <Try2Code> @felixfontain: Why did you switch in the first place?

12:20 <Guest5> felixfontein there are about 145,000 lines per build. In addition, the build is highly parallelised. We deployed buildbot with sqlite3.

12:22 <Guest5> Try2Code: I don't know for sure. We pretty much deployed standard configuration and I don't remember that I changed anything named like that. Where can I look up that option?

12:23 <Try2Code> http://docs.buildbot.net/current/manual/configuration/steps/shell_command.html?highlight=shellcommand#using-shellcommands

12:23 <Try2Code> ShellCommand is the basic tool to execute stuff remotely AFAIK - at least thats what I use

12:24 <Try2Code> 'logfiles' is one kwarg of it

12:35 <sknebel> what kind of storage is the DB on?

12:36 <tardyp> sknebel: can be postgres, mysql or sqlite

13:17 <sknebel> was a question to Guest5, had missed that they left

13:18 <sknebel> because DB on a filesystem with crap IOPS can cause such problems too

13:22 <Try2Code> @sknebel: So you say sqlite3 is in general a bad idea for a production system?

13:26 <sknebel> can be a problem with other databases too

14:51 <felixfontein> it's definitely worse with sqlite

15:01 <Try2Code> I currently have a silent crash of my 3.5 server. There is nothing in the log. systemd detects a core-dump and restarts

15:30 <Try2Code> I have no clue if my setup is heavy or not: 1 master, 9 workers (on 5 hosts), 27 builders, the server runs on a VM using sqlite

15:35 <Try2Code> a single complete build was fine, 2 competing ones do not make it though