Skip to content

fix(python-sdk): forward robots_user_agent from crawl and start_crawl - #4839

Open
RaphaelFakhri wants to merge 1 commit into
firecrawl:mainfrom
RaphaelFakhri:fix/py-crawl-robots-user-agent
Open

RaphaelFakhri wants to merge 1 commit into
firecrawl:mainfrom
RaphaelFakhri:fix/py-crawl-robots-user-agent

Conversation

@RaphaelFakhri

@RaphaelFakhri RaphaelFakhri commented Sep 29, 2026 •

Copy link
Copy Markdown

Summary

FirecrawlClient.crawl and FirecrawlClient.start_crawl list every crawl option explicitly, but robots_user_agent is missing. CrawlRequest and _prepare_crawl_request already support it (mapped to robotsUserAgent), and the async client forwards it through **kwargs, so only the sync wrappers reject it with a TypeError.

The change adds the keyword-only parameter to both methods, documents it, and passes it into the CrawlRequest. When the argument is unset, it is dropped from the payload as before.

Test plan

  • Added test_crawl_client_options.py with tests that check the request body sent by start_crawl and crawl, and that robotsUserAgent is absent when the argument is unset.
  • Before the change, 2 of the 3 new tests fail with TypeError. After the change all 3 pass, and the SDK unit suite passes (559 passed).

Command: python -m pytest firecrawl/__tests__/unit -q from apps/python-sdk.


Summary by cubic

Fixes FirecrawlClient.crawl and FirecrawlClient.start_crawl rejecting the robots_user_agent option. The parameter is now accepted, documented, and forwarded to the request body, matching the existing async client behavior. Adds unit tests verifying the option is sent and omitted when unset.

Written for commit 0877431. Summary will update on new commits.

Review in cubic

CrawlRequest and the request builder support robots_user_agent, but FirecrawlClient.crawl and FirecrawlClient.start_crawl did not accept it, so the option could not be set from the sync client. Add the parameter and pass it to the request.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 2 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Re-trigger cubic

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant