fix(python-sdk): forward robots_user_agent from crawl and start_crawl - #4839
Open
RaphaelFakhri wants to merge 1 commit into
Open
RaphaelFakhri wants to merge 1 commit into
RaphaelFakhri wants to merge 1 commit into
Conversation
CrawlRequest and the request builder support robots_user_agent, but FirecrawlClient.crawl and FirecrawlClient.start_crawl did not accept it, so the option could not be set from the sync client. Add the parameter and pass it to the request.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
FirecrawlClient.crawlandFirecrawlClient.start_crawllist every crawl option explicitly, butrobots_user_agentis missing.CrawlRequestand_prepare_crawl_requestalready support it (mapped torobotsUserAgent), and the async client forwards it through**kwargs, so only the sync wrappers reject it with aTypeError.The change adds the keyword-only parameter to both methods, documents it, and passes it into the
CrawlRequest. When the argument is unset, it is dropped from the payload as before.Test plan
test_crawl_client_options.pywith tests that check the request body sent bystart_crawlandcrawl, and thatrobotsUserAgentis absent when the argument is unset.TypeError. After the change all 3 pass, and the SDK unit suite passes (559 passed).Command:
python -m pytest firecrawl/__tests__/unit -qfromapps/python-sdk.Summary by cubic
Fixes
FirecrawlClient.crawlandFirecrawlClient.start_crawlrejecting therobots_user_agentoption. The parameter is now accepted, documented, and forwarded to the request body, matching the existing async client behavior. Adds unit tests verifying the option is sent and omitted when unset.Written for commit 0877431. Summary will update on new commits.