Automating Search and Infinite Scroll with Crawlee and Playwright

Part 3: Interacting with a JavaScript-driven marketplace, discovering dynamically loaded products, and extracting structured data with a real browser

In Parts 1 and 2, I used BeautifulSoupCrawler to process pages whose required content was available in the returned HTML.

That approach worked for category discovery, pagination, and product extraction on server-rendered pages. However, some websites depend more heavily on JavaScript and user interaction.

In Part 3, I use Crawlee with Playwright to automate a search, process a dynamically updated results page, trigger additional loading through scrolling, and extract structured product data from the rendered page.

The purpose of this part is to show when browser automation becomes useful and how it can fit into the same Router-based structure used in the previous parts.

What Changes in Part 3

This crawler performs the following sequence:

• It opens the marketplace’s home page in Chromium.
• It locates the website’s search input.
• It enters a predefined search query.
• It submits the search form.
• It adds the resulting URL to the request queue with the CATEGORY label.
• The CATEGORY handler discovers product links.
• It scrolls the page to trigger additional product loading.
• Each product URL is processed by the PRODUCT handler.
• The extracted records are stored in Crawlee’s default Dataset.

Unlike the previous parts, this implementation uses a real browser page rather than parsing only the initial HTML response.

Why Playwright Is Needed

Some websites render important content only after JavaScript executes.

A normal HTTP request may return the basic document structure, while product listings, prices, images, and interactive controls appear later in the browser.

Playwright allows the crawler to interact with the page after it has been rendered. It can locate elements, enter text, press keys, wait for page states, scroll the document, and read data from the updated DOM.

Browser automation is more resource-intensive than static HTML parsing, so it should be used when the target workflow actually requires browser execution or interaction.

Configuring the Playwright Crawler

The crawler is created in main.py:

crawler = PlaywrightCrawler(
    request_handler=router,
    headless=False,
    browser_type="chromium",
    max_requests_per_crawl=1000,
    request_handler_timeout=timedelta(
        seconds=60
    ),
)

The important settings are:

• request_handler connects the crawler to the shared `Router`.
• headless=False keeps the browser visible during execution.
• browser_type=”chromium” selects Chromium.
• max_requests_per_crawl=1000 limits the total number of processed requests.
• request_handler_timeout gives each handler up to 60 seconds to complete.

Keeping the browser visible is useful while developing this example because the search and scrolling behavior can be observed directly.

The crawler starts from:

start_urls = [
    "https://torob.com/",
]

Automating the Search

The default handler is responsible for the initial search interaction.

It first looks for the search input using page.query_selector():

input_element = await page.query_selector(
    "input[name='query']"
)

If the element is found, the crawler fills it with the predefined query and presses Enter:

query = "Iphone 17 Pro Max"

await input_element.fill(query)
await input_element.press("Enter")
await page.wait_for_timeout(5000)

The explicit five-second wait gives the website time to update the interface and navigate to the search results.

This is a simple waiting strategy. In a larger project, it may be better to wait for a specific results selector or URL condition instead of relying only on a fixed delay.

Preventing the Search from Running Repeatedly

The module contains a counter named search_query_counter:

search_query_counter = 0

The default handler returns immediately if the search has already been performed:

if search_query_counter > 0:
    return

After a successful search, search_query_counter is incremented.

This prevents the same browser interaction from being triggered repeatedly during one execution.

The counter is sufficient for this small single-process example, but global state would require more careful handling if several crawler instances or processes were running at the same time.

Adding the Results Page to the Queue

After submitting the search, the crawler takes the browser’s current URL and creates a labeled request:

await context.add_requests([
    Request.from_url(
        url=page.url,
        label="CATEGORY",
        unique_key=page.url,
    )
])

The resulting page is labeled CATEGORY because its responsibility is similar to a category page: it contains multiple product links and may load additional products as the user scrolls.

The Router then sends this request to the CATEGORY handler.

Discovering Product Links

The CATEGORY handler initially looks for links containing /p/:

await context.enqueue_links(
    selector="a[href*='/p/']",
    label="PRODUCT",
    unique=True,
)

Each discovered link receives the PRODUCT label.

The unique=True option helps avoid repeatedly adding the same URL when the page is scanned again after scrolling.

Handling Infinite Scroll

The results page loads additional products when the browser reaches the bottom of the document.

The crawler handles this with a bounded loop.

Before scrolling, it counts the currently visible product links using page.query_selector_all():

current_count = len(
    await page.query_selector_all(
        "a[href*='/p/']"
    )
)

It then waits for network activity to settle, scrolls to the bottom of the document, waits five seconds, and counts the links again:

await page.wait_for_load_state(
    "networkidle"
)

await page.evaluate(
    "window.scrollTo("
    "0, document.body.scrollHeight"
    ");"
)

await page.wait_for_timeout(5000)

new_count = len(
    await page.query_selector_all(
        "a[href*='/p/']"
    )
)

If new_count is greater than current_count, new product links have appeared. The crawler runs context.enqueue_links() again and continues scrolling.

If the count does not increase, the crawler assumes that no additional products were loaded and exits the loop.

Bounding the Scrolling Process

The implementation limits the scrolling process to 10 attempts:

scroll_attempt_cnt = 10

This limit is important because a dynamic page may behave unexpectedly.

Without a bound, a scrolling loop could continue indefinitely if the stopping condition never became true.

The loop may also stop earlier when no additional product links appear.

The value stored in scroll_attempt_cnt is an example configuration rather than a guarantee that every available product will be loaded.

Extracting a Product Identifier

The PRODUCT handler first attempts to extract an identifier from the product URL.

Product URLs use a path containing /p/, so the code applies a regular expression:

sku_match = re.match(
    r".*/p/([^/]+)/?",
    context.request.url,
)

If the pattern matches, the captured path component becomes the SKU-like identifier.

If it does not match, the crawler generates a deterministic fallback using the URL:

url_hash = hashlib.md5(
    context.request.url.encode()
).hexdigest()

sku = url_hash[:10].upper()

The same URL produces the same fallback value across repeated runs, as long as the URL does not change.

MD5 is used here only to produce a compact and repeatable identifier. It is not being used for passwords, encryption, or any security-sensitive purpose.

Extracting the Product Title

The product title is selected from the rendered page using page.query_selector():

title = await page.query_selector(
    "div[class*='Showcase_name'] h1"
)

title_text = (
    await title.text_content()
    if title
    else "No title found"
)

The selector uses a partial class match because the generated class name contains Showcase_name.

If the element is not found, the crawler stores “No title found” instead of immediately stopping the entire handler.

Normalizing Persian Numbers

Prices on the target website may contain Persian digits, separators, spaces, and currency-related text.

The clean_number() helper converts Persian digits to English digits and removes non-numeric characters:

def clean_number(text: str) -> int:
    if not text:
        return 0

    table = str.maketrans(
        "۰۱۲۳۴۵۶۷۸۹",
        "0123456789",
    )

    cleaned = re.sub(
        r"[^\d۰-۹]",
        "",
        text,
    ).translate(table)

    return int(cleaned) if cleaned else 0

For example, a formatted price string can be converted into an integer that is easier to store, compare, and analyze.

The clean_number() helper assumes that all remaining digits together represent the intended numeric value. Currency units and possible decimal values would need additional handling in a more general implementation.

Extracting Price and Seller Information

The crawler selects text elements from the website’s cheapest-seller section:

price_elements = await page.query_selector_all(
    "div[class*='Showcase_cheapest_seller'] "
    "div[class*='Showcase_buy_box_text']"
)

Instead of assuming that specific list positions always represent the seller and price, the crawler first collects the available text values:

price_texts = []
for element in price_elements:
    text = await element.text_content()
    if text:
        cleaned_text = text.strip()
        if cleaned_text:
            price_texts.append(cleaned_text)

This avoids depending on a fixed number or order of matching elements.

Availability is determined by checking the returned texts for the unavailable indicator:

if any(
    "ناموجود" in text
    for text in price_texts
):
    is_available = False

The seller is then identified using text associated with the seller link rather than a fixed list index:

for text in price_texts:
    if (
        "مشاهده در" in text
        or "خرید از" in text
    ):
        cheapest_seller = (
            text
            .replace("مشاهده در", "")
            .replace("خرید از", "")
            .strip()
        )
        break

If the product is available, the crawler searches the remaining text values for a numeric price:

if is_available:
    for text in price_texts:
        if (
            "مشاهده در" in text
            or "خرید از" in text
        ):
            continue
        numeric_value = clean_number(text)
        if numeric_value > 0:
            price = numeric_value
            break

This is more defensive than relying on price_elements[0] and price_elements[1].

If only one matching element is returned, the crawler no longer raises an IndexError.

The resulting record still distinguishes between:

• The seller offering the displayed price
• The normalized price value
• Whether the product is currently available

This logic remains site-specific because it depends on the current structure and text used in the marketplace’s buy box.

Detecting Used Products

The code also checks for an element associated with product stock or status:

is_used = False

used_element = await page.query_selector(
    "div[Showcase_main_info] "
    "div[class*='stock-status']"
)

if used_element:
    is_used = True

The result is stored as a Boolean value in the used field.

Because this logic depends on the target website’s current DOM structure, the selector should be tested again whenever the website changes.

Extracting Images

The PRODUCT handler first searches for image elements inside the product gallery:

image_elements = await page.query_selector_all(
    "img[class*='ImageVideoSliderGallery_media']"
)

The src attribute of each matching image is added to the image_urls list.

If no URLs are found, the crawler uses a fallback selector for <source> elements and reads their srcset attributes.

For a srcset value containing multiple candidates, the implementation selects the final entry and removes the 2x descriptor.

This provides two extraction paths for different gallery structures that may appear on product pages.

Building the Output Record

The final record contains:

• sku
• url
• ts
• title
• is_available
• cheapest_price
• cheapest_seller
• used
• images

The data dictionary has this structure:

data = {
    "sku": sku,
    "url": context.request.url,
    "ts": datetime.utcnow(),
    "title": title_text,
    "is_available": is_available,
    "cheapest_price": price,
    "cheapest_seller": cheapest_seller,
    "used": bool(is_used),
    "images": image_urls,
}

The record is stored using context.push_data():

await context.push_data(data)

The default Dataset therefore receives one structured record for each product request successfully processed by the PRODUCT handler.

Logging the Workflow

Loguru writes crawler events to both:

• logs/crawler.log
• The terminal

The logged events include:

• Starting and completing the crawl
• Processing URLs
• Loading the search results
• Starting and finishing the scrolling process
• Detecting newly loaded products
• Extracting product identifiers and titles
• Errors encountered during link discovery or scrolling

These logs make it easier to understand which stage of the browser-driven workflow is currently running.

Limitations of This Implementation

This project is an educational example and has several limitations:

• The search query is hard-coded.
• The browser runs with headless=False.
• The search-result wait uses a fixed five-second delay.
• The scrolling loop is limited by scroll_attempt_cnt.
• The global search_query_counter is designed for one crawler process.
• The CSS selectors depend on the current target website.
• Price and seller extraction still depends on the current DOM structure and marketplace-specific text such as مشاهده در, خرید از, and ناموجود.
• The crawler does not include automated tests for the selectors.
• Website changes may require updating the interaction and extraction logic.

Documenting these limitations is important because browser automation can fail in ways that static parsing does not, including timing changes, delayed rendering, redesigned interfaces, and altered selectors.

Why Browser Automation Is More Expensive

A PlaywrightCrawler starts and controls a browser, executes JavaScript, renders the page, maintains browser contexts, and waits for dynamic events.

This generally requires more memory and processing time than requesting and parsing static HTML.

For that reason, browser automation should not automatically replace BeautifulSoupCrawler.

A useful approach is to choose the simplest crawler that can reliably access the required data:

• Use static HTML parsing when the response already contains the data.
• Use browser automation when interaction or JavaScript execution is necessary.
• Consider direct API requests when the website exposes structured endpoints suitable for the task.

Project Structure

The Part 3 directory contains:

Part-3-Search-Automation/
├── main.py
├── routes.py
└── README.md

The responsibilities are divided as follows:

• main.py configures logging, creates the PlaywrightCrawler, defines the starting URL, and starts execution.
• routes.py contains the search, scrolling, product-discovery, normalization, and extraction logic.
• README.md documents the example and its main design decisions.

Source Code

The complete code for Part 3 is available here:

https://github.com/Saeiii-d/crawlee-python-masterclass/tree/main/Part-3-Search-Automation

The complete series is available in the main repository:

https://github.com/Saeiii-d/crawlee-python-masterclass

What Comes Next

Part 3 uses browser automation because the workflow requires search interaction and dynamically loaded content.

In Part 4, I use a hybrid approach for a different website. HTML parsing is used for page discovery and basic product information, while direct asynchronous API requests retrieve variant-level prices and stock quantities.

This avoids using browser rendering for data that can be requested directly from structured endpoints.

Connect

You can find my projects and future updates here:

• GitHub
• LinkedIn
• Medium