Skip to content

Add IPv6/dual-stack support: Operator-rendered and metrics server commands both hardcode 0.0.0.0 #6862

Description

@dbbvitor

Is your feature request related to a problem? Please describe.

Yes, in two places, both hardcoding an IPv4-only bind:

  1. start_metrics_server()
    binds via wsgiref.simple_server.make_server("", port, ...), which defaults
    socketserver.TCPServer.address_family to socket.AF_INET. On our IPv6-primary cluster,
    Istio's sidecar scrapes the pod's IPv6 address and gets connection refused — confirmed via
    Envoy access logs (503 delayed_connect_error:_Connection_refused on every scrape), while
    curl localhost:8000/metrics from inside the same container succeeds every time.

  2. The Feast Operator's
    getContainerCommand()
    hardcodes -h 0.0.0.0 in the startup command of every service container it renders (online,
    ui, offline, lineage), with no CRD field to change it. On the same IPv6-only cluster, none
    of these servers bind an address anything can reach.

We work around (2.) today with a Kyverno ClusterPolicy that rewrites each container's command
at admission time — functional, but fragile: it doesn't self-heal already-running Deployments
(skipBackgroundRequests: true), and an earlier version of this exact policy replaced the entire
command with a hardcoded string instead of mapping over the operator's own argv, silently
dropping --workers, --metrics, and other flags for months before we caught it.

Describe the solution you'd like

feast.metrics.start_metrics_server() — bind dual-stack instead of IPv4-only:

import socket
from wsgiref.simple_server import WSGIServer, make_server

class DualStackWSGIServer(WSGIServer):
    address_family = socket.AF_INET6

    def server_bind(self):
        self.socket.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, 0)
        super().server_bind()

httpd = make_server("::", port, app, server_class=DualStackWSGIServer)

Still works on IPv4-only hosts — Linux dual-stack sockets fall back cleanly with
net.ipv6.bindv6only=0, the default.

Feast Operator — add a BindAddress/DualStack field to ServerConfigs (parallel to other
per-service settings) that getContainerCommand() uses instead of the hardcoded 0.0.0.0
literal, accounting for the uvicorn-based ui/lineage servers rejecting a bracketed host (::)
unlike the gunicorn/Flight servers ([::]).

Describe alternatives you've considered

For the Operator half: the Kyverno ClusterPolicy above. It works, but every environment that
needs IPv6 needs Kyverno installed and a policy kept in lockstep with the Operator's own
command-building logic, and a bug in the policy (we've had one) doesn't self-heal already-running
Deployments.

Additional context

Test plan

  • start_metrics_server() binds successfully and serves /metrics over both 127.0.0.1 and
    ::1 on a dual-stack host.
  • Same binary still starts on an IPv4-only host (no IPv6 stack).
  • A FeatureStore CR setting the new bind-address field renders the configured address in
    each affected container's command.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions