Conversation
…read
The metrics HTTP server runs as a thread in the Gunicorn master, which
forks the workers. Its default WSGIRequestHandler writes an access-log
line to stderr for every scrape. A fork while that thread holds the
stderr buffer lock leaves the new worker with the lock held forever: it
blocks on its first log line ("Booting worker") and never serves.
Use a request handler with a no-op log_message for both the IPv4 and
the dual-stack server built by _make_metrics_httpd, like
prometheus_client's own _SilentHandler.
Fixes feast-dev#6928
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Alexandr Borgatin <a.borgatin@yandex.ru>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
The metrics HTTP server runs as a thread in the Gunicorn master, which forks the workers. Its default
WSGIRequestHandlerwrites an access-log line to stderr for every scrape. A fork while that thread holds the stderr buffer lock leaves the new worker with the lock held forever: the worker blocks on its first log line ("Booting worker") and never serves a request. Details, stacks and a reproduction in #6928 (same class of problem as #6647).This adds
_QuietWSGIRequestHandler(no-oplog_message, like prometheus_client's own_SilentHandler) and uses it for both the IPv4 and the dual-stack server built by_make_metrics_httpd. Scrapes are no longer written to stderr; nothing else changes.Tested on Kubernetes with
--max-requests 20and a scrape storm: 5 hangs in 318 forks before, 0 hangs in 2,000 forks after. Locally (steps in #6928): a hang after 12–29 forks before, none in 336 forks after.Which issue(s) this PR fixes:
Fixes #6928
Checks
pytest sdk/python/tests/unit/test_metrics.py: 117 passed;ruff checkandruff format --checkpass;mypy feastreports no errors inmetrics.py)git commit -s)Testing Strategy
TestMetricsHttpdDoesNotLog: a request to the server built by_make_metrics_httpdwrites nothing to stderr (fails before the change, passes after)🤖 Generated with Claude Code