Skip to content

fix: Do not log /metrics requests from the pre-fork metrics server thread (Gunicorn deadlock) - #6929

Open
aborgatin wants to merge 1 commit into
feast-dev:masterfrom
aborgatin:fix/metrics-server-quiet-handler
Open

aborgatin wants to merge 1 commit into
feast-dev:masterfrom
aborgatin:fix/metrics-server-quiet-handler

Conversation

@aborgatin

Copy link
Copy Markdown

What this PR does / why we need it:

The metrics HTTP server runs as a thread in the Gunicorn master, which forks the workers. Its default WSGIRequestHandler writes an access-log line to stderr for every scrape. A fork while that thread holds the stderr buffer lock leaves the new worker with the lock held forever: the worker blocks on its first log line ("Booting worker") and never serves a request. Details, stacks and a reproduction in #6928 (same class of problem as #6647).

This adds _QuietWSGIRequestHandler (no-op log_message, like prometheus_client's own _SilentHandler) and uses it for both the IPv4 and the dual-stack server built by _make_metrics_httpd. Scrapes are no longer written to stderr; nothing else changes.

Tested on Kubernetes with --max-requests 20 and a scrape storm: 5 hangs in 318 forks before, 0 hangs in 2,000 forks after. Locally (steps in #6928): a hang after 12–29 forks before, none in 336 forks after.

Which issue(s) this PR fixes:

Fixes #6928

Checks

  • I've made sure the tests are passing. (pytest sdk/python/tests/unit/test_metrics.py: 117 passed; ruff check and ruff format --check pass; mypy feast reports no errors in metrics.py)
  • My commits are signed off (git commit -s)
  • My PR title follows conventional commits format

Testing Strategy

  • Unit tests – TestMetricsHttpdDoesNotLog: a request to the server built by _make_metrics_httpd writes nothing to stderr (fails before the change, passes after)
  • Integration tests

🤖 Generated with Claude Code

…read

The metrics HTTP server runs as a thread in the Gunicorn master, which
forks the workers. Its default WSGIRequestHandler writes an access-log
line to stderr for every scrape. A fork while that thread holds the
stderr buffer lock leaves the new worker with the lock held forever: it
blocks on its first log line ("Booting worker") and never serves.

Use a request handler with a no-op log_message for both the IPv4 and
the dual-stack server built by _make_metrics_httpd, like
prometheus_client's own _SilentHandler.

Fixes feast-dev#6928

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Alexandr Borgatin <a.borgatin@yandex.ru>
@aborgatin
aborgatin requested a review from a team as a code owner October 2, 2026 10:09

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feast serve: Gunicorn worker deadlocks on startup when forked during a /metrics request (follow-up to #6647)

1 participant