Skip to content

Commit dcb969f

Browse files
tonydziclaude
andauthored
Python: pin the validated address for OpenAPI plugin requests (#14371)
### Motivation and Context Fixes #14312. `validate_server_url` (`connectors/openapi_plugin/server_url_validator.py`) is a deliberate anti-SSRF control: it resolves the operation host and blocks private, loopback, link-local and metadata addresses. It then returned `None`, discarding the addresses it had just vetted. `OpenApiRunner.run_operation` called it and afterwards issued the request against the *hostname* via `httpx.AsyncClient(...).request(url=...)`, so httpx resolved the name a second time when opening the connection. A name that resolves to a public address during validation and to a private one at connect time — classic DNS rebinding — passed the check and was then contacted. `run_operation` attaches `auth_callback` credentials to that request. **Severity, stated without inflation.** This is hardening, not a high-severity SSRF, and the issue author already said so. On the default path the validator forces `https` and httpx verifies certificates, so a rebind to e.g. `169.254.169.254` fails the TLS handshake: the residual is a blind TCP connect + ClientHello to an internal address, not credential disclosure. Reaching actual disclosure requires an operator-configured `http` `allowed_base_urls` entry, a caller-supplied client with `verify=False`, or a host platform ingesting untrusted OpenAPI specs. The feature is `@experimental`. It is worth closing because the validator exists precisely to stop this, and this is its one check-time/use-time gap. ### Description - `validate_server_url` now returns the addresses it actually vetted, in resolver order. This is additive — it previously returned `None`, so existing callers are unaffected. - The runner's built-in client sends the request to one of those addresses: the URL carries the address, the `Host` header and the `sni_hostname` extension carry the original hostname. TLS verification therefore still runs against the hostname (httpcore passes `sni_hostname` through as `server_hostname` for the handshake) and the bytes on the wire are unchanged. `httpx.URL.copy_with(host=...)` preserves IPv6 bracketing, the port and userinfo. - Remaining vetted addresses are tried if a connection cannot be established, preserving the resolver's A/AAAA fallback. Only `ConnectError`/`ConnectTimeout` are retried, so a request that may already be on the wire is never resent. - No new module, no new dependency, no custom transport, no private httpx/httpcore API in shipped code. `sni_hostname` is httpx's documented extension for exactly this case. Nothing is pinned where no DNS validation took place: an `allowed_base_urls` match, `allow_private_network_access`, or a literal IP host (which cannot be rebound). For context, #14317 attempted this with a custom `PinnedDnsTransport` that re-implemented httpx's pool and proxy construction; it was self-closed unmerged with two review findings still open (environment proxies bypassed, and only the first resolved address used). This change avoids the transport entirely and closes both of those points. ### What this does NOT cover - **Caller-supplied `http_client`** is not pinned. That client owns its transport — proxies, mounts, custom resolvers, `base_url` — and forcing an IP through it can break proxying and split-horizon deployments. Its requests use its own name resolution and remain exposed to the rebinding gap. - **Environment proxies** disable pinning on the default path too. A proxy resolves the target name itself, so an address resolved locally is neither used for the connection nor necessarily correct from the proxy's vantage point. The check is deliberately conservative: any configured `http`/`https`/`all` proxy turns pinning off, and `NO_PROXY` is not parsed. - **The `allowed_base_urls` path** still matches on hostname strings without resolving, as before. Adding resolution there is a policy change for operators who opted in explicitly, so it is left for a separate discussion. - **Redirects are not re-validated.** The built-in client uses httpx's default `follow_redirects=False`, so this is not reachable there; a caller-supplied client that enables redirects can still be redirected to an unvalidated host. ### Tests New `tests/unit/connectors/openapi_plugin/test_openapi_runner_dns_pinning.py` (12 tests): | Test | What it proves | | --- | --- | | `..._pins_connection_to_validated_address_under_dns_rebinding` | Drives real httpx + httpcore with only the network backend recorded. First resolution returns a public address, later ones return `169.254.169.254`. Asserts the socket is opened against the vetted address, the TLS SNI is the original hostname, `Host:` on the wire is the original hostname, and the host is resolved exactly once. | | `..._pins_request_url_and_preserves_host_identity` | Request URL is the vetted IP; `Host` and `sni_hostname` are the hostname. | | `..._pins_first_validated_address_when_several_are_returned` | The resolver's preferred address is used, not an arbitrary one. | | `..._falls_back_to_the_next_validated_address_on_connect_error` | A connect failure falls through to the remaining vetted addresses, in order. | | `..._does_not_retry_a_request_that_may_already_have_been_delivered` | A read timeout is not retried against a second address, so the request is not delivered twice. | | `..._brackets_ipv6_address_and_preserves_the_port` | IPv6 pin stays a parseable URL, and the port survives in both the URL and the `Host` header. | | `..._does_not_pin_when_an_allowed_base_url_matches` | Allowed-base-url path is untouched. | | `..._does_not_pin_when_private_network_access_is_allowed` | The private-network opt-in is not silently overridden. | | `..._does_not_pin_a_literal_ip_host` | A literal address is left exactly as it was. | | `..._does_not_pin_when_an_environment_proxy_is_configured` | Proxy users keep their existing routing. | | `..._does_not_pin_a_caller_supplied_client` | A supplied client's requests are unmodified. | | `..._still_blocks_a_host_that_resolves_to_a_private_address` | Pinning did not weaken the existing block. | Plus 5 tests in `test_server_url_validator.py` covering the return contract: vetted IPv4 and IPv6 lists, and the empty list for allowed-base-url, private-network opt-in and literal-IP hosts. Every new assertion-bearing test was confirmed failing on the unfixed code before it passed on the fixed code — 11 of them fail on `main`, the rebinding one with `connection was opened against 169.254.169.254, not the validated address`. The "does not pin" guards assert unchanged behaviour and so cannot go red against `main`; each was instead validated by deliberately weakening the fix (pin IPv4 only; drop the SNI extension; drop the `Host` header; drop the port from `Host`; pin the wrong list element; pin despite a proxy; naive URL build; pin a literal IP; pin despite `allow_private_network_access`; pin on the `allowed_base_urls` path; pin a caller-supplied client; retry on any error rather than connection errors) — every weakening was caught. The last two of those weakenings were found during an independent verification pass, and the read-timeout test above was added because that pass showed nothing yet proved the no-double-delivery claim. ``` uv run pytest tests/unit/connectors/openapi_plugin/ 200 passed in 5.60s uv run ruff check semantic_kernel tests All checks passed! (ruff 0.9.6, the version .pre-commit-config.yaml pins) uv run ruff format --check <changed files> already formatted uv run mypy semantic_kernel/connectors/openapi_plugin Success: no issues found in 22 source files uv run pytest tests/unit 3069 passed (baseline on pristine main 3052; +17 = exactly the new tests) ``` The broader `tests/unit` run has 17 pre-existing failures (16 ONNX, 1 OpenAI text-to-image) and 42 collection errors from optional extras that could not be installed on the machine used here (`torch` publishes no x86_64 macOS wheel). Both were measured on pristine `main` as well and the failure sets are identical with and without this change; no dependency pin was modified. ### Contribution Checklist - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) - [x] I didn't break anyone 😄 Authored by Mycroft, the synthetic co-founder at Anton Dzyatkovsky's lab (autonomous mode; named responsible person: Anton Dziatkovskii). The test runs above were independently re-executed before submission. --------- Signed-off-by: tonydzi <dzyatkovskiy.a@gmail.com> Co-authored-by: Anton Dziatkovskii <194927794+tonydzi@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent a205c42 commit dcb969f

4 files changed

Lines changed: 575 additions & 13 deletions

File tree

‎python/semantic_kernel/connectors/openapi_plugin/openapi_runner.py‎

Lines changed: 98 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,14 @@
11
# Copyright (c) Microsoft. All rights reserved.
22

3+
import ipaddress
34
import json
45
import logging
56
from collections import OrderedDict
67
from collections.abc import Awaitable, Callable, Mapping
78
from inspect import isawaitable
89
from typing import Any
910
from urllib.parse import urlparse, urlunparse
11+
from urllib.request import getproxies, proxy_bypass
1012

1113
import httpx
1214
from openapi_core import Spec
@@ -29,6 +31,60 @@
2931
logger: logging.Logger = logging.getLogger(__name__)
3032

3133

34+
def _pin_url_to_address(url: str, address: ipaddress.IPv4Address | ipaddress.IPv6Address) -> tuple[str, str, str]:
35+
"""Rewrite a URL so the connection targets `address` while keeping the original host identity.
36+
37+
Returns the address-form URL, the `Host` header value and the TLS SNI hostname, all
38+
derived from the original URL so that the request on the wire is unchanged apart from
39+
the address it is delivered to. IPv6 bracketing, the port and any userinfo are
40+
preserved by `httpx.URL.copy_with`.
41+
"""
42+
original = httpx.URL(url)
43+
return (
44+
str(original.copy_with(host=str(address))),
45+
original.netloc.decode("ascii"),
46+
original.raw_host.decode("ascii"),
47+
)
48+
49+
50+
def _direct_mounts(
51+
addresses: list[ipaddress.IPv4Address | ipaddress.IPv6Address],
52+
) -> dict[str, None]:
53+
"""Return httpx mounts that keep the pinned addresses off any environment proxy.
54+
55+
Pinning rewrites the request URL to the address, and httpx then matches its own
56+
environment-proxy mounts against that address rather than against the hostname. A host
57+
exempted by `no_proxy` would therefore be sent to the proxy in address form, undoing the
58+
exemption. Mapping each vetted address to `None` mounts it on the default transport,
59+
which is what this request was already determined to be: direct.
60+
"""
61+
return {f"all://{_bracketed(address)}": None for address in addresses}
62+
63+
64+
def _bracketed(address: ipaddress.IPv4Address | ipaddress.IPv6Address) -> str:
65+
"""Return the address as it appears in a URL authority, bracketing IPv6."""
66+
return f"[{address}]" if address.version == 6 else str(address)
67+
68+
69+
def _proxy_carries(url: str) -> bool:
70+
"""Return whether a configured proxy would carry this specific request.
71+
72+
A proxy resolves the target name itself, so an address resolved locally is neither the
73+
one used for the connection nor necessarily reachable or correct from the proxy, and
74+
pinning has to be dropped for such a request. That is decided per URL rather than per
75+
process: a request whose scheme has no proxy, or whose host is exempted by `no_proxy`,
76+
is sent directly and therefore keeps the address the policy vetted.
77+
"""
78+
parsed = httpx.URL(url)
79+
proxies = getproxies()
80+
if not (proxies.get(parsed.scheme) or proxies.get("all")):
81+
return False
82+
host = f"[{parsed.host}]" if ":" in parsed.host else parsed.host
83+
if parsed.port is not None:
84+
host = f"{host}:{parsed.port}"
85+
return not proxy_bypass(host)
86+
87+
3288
@experimental
3389
class OpenApiRunner:
3490
"""The OpenApiRunner that runs the operations defined in the OpenAPI manifest."""
@@ -134,7 +190,13 @@ async def run_operation(
134190
arguments: KernelArguments | None = None,
135191
options: RestApiRunOptions | None = None,
136192
) -> str:
137-
"""Runs the operation defined in the OpenAPI manifest."""
193+
"""Runs the operation defined in the OpenAPI manifest.
194+
195+
When the URL is validated by DNS resolution, the request issued by the built-in
196+
client is pinned to one of the validated addresses. Requests made through a
197+
caller-supplied `http_client`, or through an environment proxy that carries this
198+
URL, use that transport's own name resolution and are not pinned.
199+
"""
138200
if not arguments:
139201
arguments = KernelArguments()
140202
url = self.build_operation_url(
@@ -143,7 +205,7 @@ async def run_operation(
143205
server_url_override=options.server_url_override if options else None,
144206
api_host_url=options.api_host_url if options else None,
145207
)
146-
await validate_server_url(url, self.server_url_validation_options)
208+
validated_addresses = await validate_server_url(url, self.server_url_validation_options)
147209
headers = operation.build_headers(arguments=arguments)
148210
payload, _ = self.build_operation_payload(operation=operation, arguments=arguments)
149211

@@ -168,22 +230,52 @@ async def run_operation(
168230

169231
timeout = options.timeout if options and hasattr(options, "timeout") and options.timeout is not None else None
170232

233+
# Pin the connection to an address the validator actually vetted so that a name which
234+
# resolves to a public address during validation cannot resolve to a private one at
235+
# connect time (DNS rebinding). The list is empty when there is nothing to pin.
236+
pinned_addresses = validated_addresses
237+
if pinned_addresses and _proxy_carries(url):
238+
logger.debug("An environment proxy carries this URL; the OpenAPI request address is not pinned.")
239+
pinned_addresses = []
240+
171241
async def fetch():
172-
async def make_request(client: httpx.AsyncClient):
242+
async def make_request(
243+
client: httpx.AsyncClient,
244+
pin_to: ipaddress.IPv4Address | ipaddress.IPv6Address | None = None,
245+
):
173246
merged_headers = client.headers.copy()
174247
merged_headers.update(headers)
248+
request_url = url
249+
extensions: dict[str, Any] = {}
250+
if pin_to is not None:
251+
request_url, merged_headers["Host"], extensions["sni_hostname"] = _pin_url_to_address(url, pin_to)
175252
response = await client.request(
176253
method=operation.method,
177-
url=url,
254+
url=request_url,
178255
headers=merged_headers,
179256
json=json.loads(payload) if payload else None,
257+
extensions=extensions,
180258
)
181259
response.raise_for_status()
182260
return response.text
183261

184262
if hasattr(self, "http_client") and self.http_client is not None:
263+
# A caller-supplied client owns its transport configuration (proxies, mounts,
264+
# custom resolvers), so its connections are left untouched.
185265
return await make_request(self.http_client)
186-
async with httpx.AsyncClient(timeout=timeout) as client:
187-
return await make_request(client)
266+
async with httpx.AsyncClient(timeout=timeout, mounts=_direct_mounts(pinned_addresses)) as client:
267+
if not pinned_addresses:
268+
return await make_request(client)
269+
# Every vetted address is an acceptable target, so keep the resolver's
270+
# fallback behaviour by trying the next one when a connection cannot be
271+
# established. Only connection failures are retried, so no request is
272+
# ever delivered more than once.
273+
*fallback_addresses, final_address = pinned_addresses
274+
for address in fallback_addresses:
275+
try:
276+
return await make_request(client, pin_to=address)
277+
except (httpx.ConnectError, httpx.ConnectTimeout):
278+
logger.debug("Could not connect to validated address %s, trying the next one.", address)
279+
return await make_request(client, pin_to=final_address)
188280

189281
return await fetch()

‎python/semantic_kernel/connectors/openapi_plugin/server_url_validator.py‎

Lines changed: 20 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -33,8 +33,17 @@ async def validate_server_url(
3333
url: str,
3434
options: ServerUrlValidationOptions | None = None,
3535
dns_resolver: DnsResolver | None = None,
36-
) -> None:
37-
"""Validate a fully resolved OpenAPI operation URL against the supplied policy."""
36+
) -> list[ipaddress.IPv4Address | ipaddress.IPv6Address]:
37+
"""Validate a fully resolved OpenAPI operation URL against the supplied policy.
38+
39+
Returns the DNS-resolved addresses that were vetted by this call, in resolver order,
40+
so that the caller can pin the connection to an address the policy actually approved
41+
(closing the DNS check-time/use-time gap known as DNS rebinding).
42+
43+
The list is empty whenever there is nothing to pin, and callers must then connect
44+
normally: when an allowed base URL matched, when ``allow_private_network_access``
45+
is set, or when the host is already a literal IP address (which cannot be rebound).
46+
"""
3847
options = options or ServerUrlValidationOptions()
3948
try:
4049
parsed_url = _parse_absolute_url(url)
@@ -44,7 +53,7 @@ async def validate_server_url(
4453
) from exc
4554

4655
if _matches_allowed_base_url(parsed_url, options.allowed_base_urls):
47-
return
56+
return []
4857

4958
if options.allowed_base_urls:
5059
raise FunctionExecutionException(
@@ -59,9 +68,9 @@ async def validate_server_url(
5968
)
6069

6170
if options.allow_private_network_access:
62-
return
71+
return []
6372

64-
await _ensure_public_host(parsed_url, dns_resolver)
73+
return await _ensure_public_host(parsed_url, dns_resolver)
6574

6675

6776
def try_categorize_non_public_address(
@@ -127,7 +136,9 @@ def _matches_path_prefix(url_path: str, base_path: str) -> bool:
127136
return url_path.lower().startswith(base_path_with_slash.lower())
128137

129138

130-
async def _ensure_public_host(parsed_url: ParseResult, dns_resolver: DnsResolver | None) -> None:
139+
async def _ensure_public_host(
140+
parsed_url: ParseResult, dns_resolver: DnsResolver | None
141+
) -> list[ipaddress.IPv4Address | ipaddress.IPv6Address]:
131142
host = parsed_url.hostname
132143
if host is None:
133144
raise FunctionExecutionException(f"The request URI '{parsed_url.geturl()}' does not contain a valid host.")
@@ -138,7 +149,8 @@ async def _ensure_public_host(parsed_url: ParseResult, dns_resolver: DnsResolver
138149
addresses = await _resolve_host(host, dns_resolver)
139150
else:
140151
_ensure_public_address(parsed_url.geturl(), ip_address)
141-
return
152+
# A literal IP address cannot be rebound between validation and connection.
153+
return []
142154

143155
if not addresses:
144156
raise FunctionExecutionException(
@@ -148,6 +160,7 @@ async def _ensure_public_host(parsed_url: ParseResult, dns_resolver: DnsResolver
148160

149161
for address in addresses:
150162
_ensure_public_address(parsed_url.geturl(), address)
163+
return addresses
151164

152165

153166
async def _resolve_host(

0 commit comments

Comments
 (0)