Repository navigation
Buildbot web page: connection lost after 1 minute, then display "Connection restored" popup #85867
Description
| BPO | 41701 |
|---|---|
| Nosy | @db3l, @pitrou, @vstinner, @encukou, @zware, @ewdurbin, @pablogsal |
| PRs |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2021-04-21.15:23:50.551>
created_at = <Date 2020-09-03.10:21:19.805>
labels = ['tests', '3.10']
title = 'Buildbot web page: connection lost after 1 minute, then display "Connection restored" popup'
updated_at = <Date 2021-04-21.15:23:50.551>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2021-04-21.15:23:50.551>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-04-21.15:23:50.551>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2020-09-03.10:21:19.805>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 41701
keywords = ['patch']
message_count = 15.0
messages = ['376293', '381966', '381967', '381972', '381973', '385256', '385269', '385271', '385276', '385278', '385279', '391483', '391487', '391524', '391526']
nosy_count = 7.0
nosy_names = ['db3l', 'pitrou', 'vstinner', 'petr.viktorin', 'zach.ware', 'EWDurbin', 'pablogsal']
pr_nums = ['24119']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue41701'
versions = ['Python 3.10']
Activity
vstinner commentedon Sep 3, 2020
Since the buildbot server migrated to a new machine, the web page losts is connection and the whole web page is reloaded every minute. Try for example:
https://buildbot.python.org/all/#/release_status
The new buildbot.python.org machine is now behind a load balancer.
TCP connections closed after 1 minute already affected clients: see bpo-41642. It seems like HTTPS connections (tcp/443) are also affected.
db3l commentedon Nov 27, 2020
I was wondering if there was any update on whether or not this new behavior can be corrected?
I was attempting to review a buildbot failure today and it's actually pretty tough to "race the refresh" when trying to review the build steps and logs.
vstinner commentedon Nov 27, 2020
We need someone of the PSF Infra team to increase the load balancer delay. Or fix the load balancer.
ewdurbin commentedon Nov 28, 2020
I am away from my computer at the moment, but there is a direct access hostname for the buildbot host that was announced to the build server owners. Configuring that bypasses the load balancer.
ewdurbin commentedon Nov 28, 2020
Apologies, wrong issue. I’ll have to take a closer look at this.
encukou commentedon Jan 19, 2021
Is there anything I can help with to move this forward?
Investigating buildbot failures continues to be very annoying.
ewdurbin commentedon Jan 19, 2021
OK, I've confirmed that HAProxy seems to be the issue. WebSockets opened to the nginx proxy on the server or directly to the twisted server successfully remain indefinitely.
If anyone familiar with HAProxy would be interested in helping debug, the current relevant information is:
Here is the current HAProxy configuration: https://gist.github.com/ewdurbin/d8a42c30a04d6cb5763431200acaecde which is generated from this salt state: https://github.com/python/psf-salt/tree/master/salt/haproxy and this pillar data: https://github.com/python/psf-salt/blob/master/pillar/base/haproxy.sls
I can provide any additional information that would be helpful to someone trying to sort out what's going on.
vstinner commentedon Jan 19, 2021
I looked at these files.
HAProxy configuration, "defaults" section contains:
These timeouts are not overriden in "frontend main" nor in "backend buildbot-master" sections.
pillar/base/haproxy.sls seems to try to override these variables:
https://github.com/python/psf-salt/blob/d89e5ef2e86f45c1766c8b93d6e9621b0ab1bb09/pillar/base/haproxy.sls#L7-L11
I don't see these in the rendered HAProxy config, but I see "timeout tunnel 3600s" is rendered as "timeout tunnel 1d" in "backend buildbot-master".
ewdurbin commentedon Jan 19, 2021
Apologies, I gisted a version that was from attempts to debug the timeouts. It's been updated.
vstinner commentedon Jan 19, 2021
Ee:
Aaaah :-D No problem. Can you please try:
in buildbot-master server of the pillar config:
https://github.com/python/psf-salt/blob/d89e5ef2e86f45c1766c8b93d6e9621b0ab1bb09/pillar/base/haproxy.sls#L7-L11
vstinner commentedon Jan 19, 2021
Hum. I'm not sure if "1d" syntax is accepted. Maybe use "3600s".
pablogsal commentedon Apr 20, 2021
Ee, apparently this issue in the buildbot repo may be related: buildbot/buildbot#4078
Could you investigate if we can use this on our PSF server?
pablogsal commentedon Apr 21, 2021
I'm going to try it our on the server but apparently there are some problems: buildbot/buildbot#5991
pablogsal commentedon Apr 21, 2021
Can someone confirm if they still have this problem on the buildbot server?
vstinner commentedon Apr 21, 2021
Oh wow, I don't have the issue anymore! Thank you very much :-) Previously, the website was barely usable :-(