772467 - Figure out (and fix!) stale buildslave connections in AWS

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Chris AtLee [:catlee]]

If you reboot a build slave in AWS without first shutting off buildbot, the master doesn't know that the old instance disconnected, and will prevent the new instance from connecting forever.

Comment 1

[Chris AtLee [:catlee]]

Attachment: give up on old slave connections

I'm not at all sure why the AWS slaves hit this problem more frequently than our other machines, but this patch seems to work around the issue.

Instead of relying on callRemote() to cause the old tcp session to die, we add a timeout (30s here), and if we haven't heard back from the slave before the timeout we disconnect it. This then allows the next slave connection to succeed.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 640910 [details] [diff] [review]
give up on old slave connections

lgtm.  getPeer includes both the remote IP and port, so a collision is unlikely (since slaves don't re-use ports)

Comment 3

[Chris AtLee [:catlee]]

Comment on attachment 640910 [details] [diff] [review]
give up on old slave connections

landed on bm35 only for now

Comment 4

[Chris AtLee [:catlee]]

Not sure if we want or need this on other buildbot masters as well?