fix(docker): report a refused seed as one failed result, not a dead batch - #2299

Fixes #2288

Summary

_normalize_and_validate_seeds validated every seed up front and raised on the
first one the destination check refused, so a single internal address — or a
single hostname that simply doesn't resolve, which is what a dead domain in a
stale sitemap looks like — failed the whole /crawl request with a 400 and zero
results. The caller couldn't tell which seed it was, so its only recovery was to
split the batch and retry.

A refused seed now comes back as one failed result among the batch, the way a
robots.txt refusal already does inside a batch, on both /crawl and
/crawl/stream:

POST /crawl  {"urls": ["https://example.com/", "https://no-such-host-12345.example/"]}

200 {"success": true, "results": [
  {"url": "https://example.com/",                "success": true,  ...},
  {"url": "https://no-such-host-12345.example/", "success": false, "status_code": 403,
   "error_message": "URL blocked (SSRF protection): URL blocked",
   "response_headers": {"X-Egress-Status": "Blocked by egress policy"}}
]}

Four decisions a reviewer would otherwise have to reverse-engineer:

• The message stays opaque, so this isn't a resolution oracle. detail from
  validate_url_destination is carried through verbatim. egress_broker._resolve
  turns socket.gaierror into EgressBlocked, so resolved to an internal
  address and does not resolve both produce the same URL blocked. A caller
  learns which seed was refused — it already knows the hostname it sent — and
  nothing about why. Two tests pin this: one message across all six internal
  targets, and equality between the internal and NXDOMAIN cases.
• A request with nothing crawlable left is still a 400. There's no batch to
  report per-URL failures in, and it keeps a single blocked URL a 400 — the
  contract /md, /llm, /crawl/stream and the existing SSRF tests rely on.
  _require_crawlable_seeds is the only place that decides this.
• Refused results are appended, not spliced into their original index. I wrote
  the index-based version first and it was wrong: MemoryAdaptiveDispatcher.run_urls
  appends in completion order, so results never lined up with the caller's
  urls to begin with. Callers already have to match by result.url — which is
  exactly the field the 400 gave them no way to recover.
• crawler_configs passes through unfiltered. A caller that sent per-URL
  configs still means them, so the branch keys off the original url count. The
  list is deliberately not filtered by surviving seeds:
  BaseDispatcher.select_config pairs via url_matcher, not position, so
  index-filtering would drop the config that actually matches. The test uses real
  url_matcher patterns and asserts the pairing still resolves.

A refused seed never reaches the crawler — urls is rebound before
get_crawler is acquired, asserted on both the batch and streaming paths. There's
also one greppable logger.warning per refused batch, each URL truncated to 200
chars so a blocked-URL scanner can't write megabytes into the log.

List of files changed and why

• deploy/docker/api.py — _normalize_and_validate_seeds partitions instead of
  raising and returns a _SeedBatch. New helpers: _require_crawlable_seeds,
  _refused_seed_result (a real CrawlResult, so the key set matches every other
  entry and survives Crawl4aiDockerClient.crawl's CrawlResult(**r)),
  _append_refused_results, _prepend_refused_results. Both handlers unwrap the
  batch; arun/arun_many selection is reworked around the surviving count.
• deploy/docker/tests/test_security_seed_batch.py (new) — the regression plus
  the properties around it, behavioural over the real ASGI app.
• tests/test_docker_pdf_crawler_pairing.py, tests/test_issue_2127_docker_pdf.py —
  their stubs of _normalize_and_validate_seeds returned a bare list, which the new
  contract replaces. Test-only, no assertions changed.

How Has This Been Tested?

New suite, 32 cases, fully offline — a local resolver fixture models literal IPs,
NXDOMAIN and internal hostnames, and the crawler pool is mocked so no browser
launches:

• the issue's exact input as a client, asserting 200 + one failed result carrying
  the refused URL;
• a refused seed is absent from what the crawler received (batch and stream);
• internal address vs. NXDOMAIN produce a byte-identical detail; all six internal
  targets produce the one opaque string;
• every refusal is collected, not just the first;
• raw: URLs and bare-host normalization unchanged;
• a single refused URL, and an all-refused batch, are still 400 — on both paths;
• per-URL crawler_configs survive a refusal and still pair by url_matcher;
• the SDK's CrawlResult(**r) round-trip;
• an empty urls list still returns the empty success it always did.

That last one is a regression guard on myself. Keying arun/arun_many off
len(urls) > 1 instead of != 1 turned POST /crawl/job {"urls": []} —
reachable, since CrawlJobPayload.urls has no min_length and that route has no
emptiness check — from an empty 200 into an IndexError 500. I confirmed the
test fails at api.py:874 when the != 1 form is reverted.

deploy/docker/tests/           404 passed, 1 xfailed   (was 372 + my 32)
the two touched test files     10 passed
pyflakes                       clean on the new code

Repo-wide tests/ has 21 pre-existing collection errors and test_docker*.py has
7 failures / 8 errors — I diffed both against a stashed baseline and the numbers
are identical with and without this change.

Not run: anything needing a real browser or the Docker image. The behavioural
tests drive the real FastAPI app with the pool mocked, so the seed gate and the
response shape are covered end to end; actual Chromium egress is not.

Breaking change: a partially refused /crawl batch now returns 200 with
success: true and per-URL success: false where it previously returned 400 —
and /crawl/job correspondingly reports status: "completed". A client that
relied on 400 to mean "at least one of my URLs was bad" should check each
result's success. An all-refused request is still a 400, so a single blocked URL
is unaffected.

Security: unchanged. The check is still a gate in front of every fetch, the
message is still opaque, and nothing new about the server's network reaches the
caller beyond which of its own hostnames was refused. Single-URL behaviour is
identical, so the batch path can't be used to probe internal addressing faster
than before — asserted directly. resolve_and_pin still raises for every other
caller.

Not changed on purpose: docker_client.py logs url_status(success=True) for
every yielded stream result including the {"status": "completed"} marker.
That's already wrong today for robots.txt refusals, so it's pre-existing and
client-side; worth its own issue. No CHANGELOG.md entry either — every commit
that has touched it is a release commit.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation — no user-facing docs described the old 400-for-the-whole-batch behaviour; the docstrings, the _SeedBatch contract and the operator log line are updated in place
• I have added/updated unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes