tardyp changed the topic of #buildbot to: A Software Freedom Conservancy Project | Buildbot-3.2.0 | docs: http://docs.buildbot.net/current/ | tutorial: http://docs.buildbot.net/current/tutorial | irclogs: https://libera.irclog.whitequark.org/buildbot
<verm__> couple of emails may have been sent out i had to kick mailman and fix a space issue.
bdbaddog1 has quit [Quit: Leaving.]
aakashjain has joined #buildbot
bdbaddog has joined #buildbot
aakashjain has quit [Remote host closed the connection]
aakashjain has joined #buildbot
aakashjain has quit [Read error: Connection reset by peer]
aakashjain has joined #buildbot
aakashjain has quit [Ping timeout: 240 seconds]
aakashjain has joined #buildbot
vmeson has quit [Quit: Konversation terminated!]
vmeson has joined #buildbot
<aakashjain> tardyp: p12tic: rjarry: cmouse: skelly: Is there a way to have non-disruptive buildbot restart in some kind of multi-master buildbot configuration?
<aakashjain> We frequently make changes to our buildbot configurations and need to restart buildbot to pick them up. This is disruptive since our busy buildbot instance is always running hundreds of builds (some of which often takes many hours to finish). I am wondering if there is some kind of multi-master configuration in which we can do restart without disruption ongoing builds.
<aakashjain> Maybe while running multiple masters with same configuration (behind a load-balancer), restart one master, meanwhile other master takes over whole traffic, first master finish the restart (with new configuration), and then restart other master in similar manner. I wonder if that can work and ongoing builds will keep running smoothly with old configuration, and new builds with new configuration.
<aakashjain> wondering if someone have solved this problem already
<bdbaddog> buildbot reconfig ?
<aakashjain> reconfig isn't that reliable, only works for very simple changes
<bdbaddog> aakashjain: does work, but you need to take some extra steps in your buildbot config.
<bdbaddog> The issue is that reload doe not refresh any `from x import y,z`
<bdbaddog> I've done this in the past where reconfig was extremely usable with fairly complex builds.
<aakashjain> bdbaddog: interesting. Can you please tell more about what did you find about reconfig, and how to make it work better?
<aakashjain> is using "import x" and then using x.y better than using "from x import y" from reconfig perspective?
<bdbaddog> yes.
<bdbaddog> from x import y is effectively
<bdbaddog> import x
<bdbaddog> y = x.y
<bdbaddog> with the exception that via the mechanism reconfig uses the y=x.y bit isn't rerun
<bdbaddog> so you end up with y pointing to the "old" x's y instead of the one you reloaded.
<bdbaddog> I think additionally I needed to use reload() for all my imported modules in my bulldbot master configs.. https://docs.python.org/3/library/importlib.html?highlight=reload#importlib.reload
<bdbaddog> a little experimenation will help iron out what's not properly being reloaded in your config.
<bdbaddog> of course there are some items where restart was required.. but I'm not remembering right now what those were.
<bdbaddog> Gnerally reconfiguring my builders wasn't not one of those, nor adding/removing workers.
tflink_ has quit [Quit: ZNC 1.8.2 - https://znc.in]
tflink has joined #buildbot
<aakashjain> bdbaddog: Interesting, thanks for the details! I'll try it out when I get a chance.
<bdbaddog> No problemo. I've long meant to write this all up, but alas the chaos monsters rarely provide me the breathing room..
<aakashjain> bdbaddog: Yeah, I think the import part is definitely worth a bug on https://github.com/buildbot/buildbot/issues
<aakashjain> I'll try to file it if I'm able to reproduce that issue
<bdbaddog> I don't think it's anything buildbot can do.. it's python's import mechanisim.
<aakashjain> bdbaddog: oh ok. If you have any link where I can read more about it, that would be great.
<bdbaddog> nope. like I said. Haven't gotten around to writing it up. You can look at the python doc I pasted above..
<aakashjain> bdbaddog: ok, will do. Thanks for the details.