<RP>
tardyp: I applied "send buildrequest events at once", "precompute getBuilderById in a dictionary" and have a 0.5s asyncSleep in activity loop. Between the buildset being added to the DB and builds starting it takes 17s. It is calling into prioritizeBuilders() many times during that 17s and it does look like it does end up with the whole list before starting builds so is doing the right thing. The 17s seems a little long given other thi
<tardyp>
which will reset for each buildrequest slowly streaming (if they still do)
<RP>
tardyp: right, that one was more invasive and I was curious if the problem still existed without that
<RP>
tardyp: thanks for looking into this btw, it is much appreciated and our builds will be happier :)
<tardyp>
the commit is not that invasive, there is just a small refacto in it which makes it more heavy that it would need
<tardyp>
and also the tests change were quite annoying, as adding a delay requires manual taking care of the time in tests
<tardyp>
re: The 17s seems a little long given other thing.
<tardyp>
the brd does a bunch of db fetch for each loop, so it can take a while indeed. this is a well known performance issue in BB
<RP>
tardyp: it isn't that invasive, that is the wrong word I guess. I just meant it was harder to apply to the system I'm testing it on and I also wanted to see if the other pieces worked ok
<RP>
tardyp: if the 17s is expected then we're good. I only know the well known performance issues bitbake has! :)
<tardyp>
ah!
sphet has joined #buildbot
<sphet>
Hey tardyp are you around??
<tardyp>
hi sphet
<sphet>
Hey I have some questions about branches in the UI and multimaster/worker issues. Can I run them by you quickly?
<sphet>
We have multiple projects in our company, each of which have multiple branches. Historically we have used a master-per-branch and manually moved workers from one master to another. This is cumbersome and we are seeing lots of workers sitting idle in one branch's master while another pool of workers is over taxed.
<sphet>
One idea we had was to put all branches under one master, but then the UI becomes cluttered and confusing - changes for all branches are showing up in the UI and there is no way to filter them. Are we missing something or is that how buildbot UI works?
<sphet>
An alternative approach we are considering is to subclass the Worker, and create our own system of arbitrating the avialability of workers between masters by having multiple worker per physical machine, where each is mapped to a different master. I'm not really keen on writing this solution as it seems like a lot of work to address a UI issue in
<sphet>
Buildbot.
<sphet>
So my real question is - does Buildbot have a way to filter all the UI elements by Branch or Project?
<tardyp>
probably you need a custom dashboard page
<sphet>
tardyp that's exactly what we guessed (although sad :) )
<sphet>
What do you think a realistic maximum number of worker under a single master is these days?
<tardyp>
you can start from console and quite easily add that branch filtering
<tardyp>
it is more imho depending on the amount of log spitted by your build
<tardyp>
multimaster is working quite well and easy to scale
<tardyp>
you just add more masters when things are slowing down
bdbaddog has quit [Quit: Leaving.]
<sphet>
tardyp if we have a custom dashboard parameterized by branch name, and run a unique 'UI' master for each branch, but a central master is used for all the workers and the task scheduling. Does that make sense?
<tardyp>
I guess that would work, but you can as well have 4 dashboards in the same UI master
<sphet>
How can we parameterize the dashboards in the same UI since the www services is a dict by type
<tardyp>
a single plugin can produce several pages
<sphet>
Hmm - are there are documentation or examples? We cloned the console_view but didn't see anything like that.