fix(dotnet-sdk): stop pagination on repeated next URLs - #4849

Why

The .NET SDK follows next URLs without remembering which pages it has requested. If an API response repeats a cursor, CrawlAsync, BatchScrapeAsync, and auto-paginated monitor checks request the same page indefinitely, append duplicate results, and never return unless externally cancelled.

Change

Track requested pagination URLs for each operation and throw a FirecrawlException when a URL repeats, before making another HTTP request. This covers the crawl, batch scrape, and monitor-check paths without changing their successful pagination behavior.

Reproduction and tests

A fake HTTP handler returned the same next URL on successive pages. The three new cases failed against main: each operation fetched the page again and hit the handler's guard. After the fix:

• Focused .NET tests: 3 passed, 0 failed.
• Full Firecrawl.Tests project: 70 passed, 0 failed.
• git diff --check passed.

No API or wire format change.

Summary by cubic

Fixes the .NET SDK paginating indefinitely on repeated next URLs, which could loop forever and append duplicate results.

• Tracks visited pagination URLs and throws a FirecrawlException before re-fetching a repeated page.
• Applies to CrawlAsync, BatchScrapeAsync, and monitor checks without changing successful pagination behavior.
• Adds tests covering all three paths, including a fake handler that repeats the same next URL and one that progresses normally.

^{Written for commit 44c9bad. Summary will update on new commits.}