How Do You Use Scrapy Playwright to Scrape Dynamic Websites?

Quick Answer

Scrapy Playwright lets selected Scrapy requests load in a browser before Scrapy parses the rendered HTML. Install `scrapy-playwright`, register its download handler, and mark browser requests with `meta={"playwright": True}`. Use `PageMethod` for waits or interactions, while leaving static requests on Scrapy's ordinary downloader.

Key Takeaways

• Scrapy Playwright adds browser rendering without replacing Scrapy's scheduler, selectors, pipelines, or exports.
• Only mark requests that need JavaScript, scrolling, clicking, or another browser action.
• Wait for a visible result instead of relying on fixed delays.
• Browser contexts control proxy routes, cookies, storage, and temporary session state.
• Limit concurrent pages, validate required fields, and measure cost per usable record.

When Should You Use Scrapy Playwright?

Scrapy Playwright is useful when Scrapy needs a browser to render JavaScript or complete an interaction before parsing the response. Static pages should normally stay on Scrapy's lighter downloader.

Scrapy Playwright is not a universal upgrade. Browser rendering costs more CPU, memory, bandwidth, and time than a direct request. It also changes the response into the browser's serialized DOM.

The main advantage is selective rendering. A spider can send ordinary requests through Scrapy, then reserve browsers for pages that require them. This keeps web scraping jobs easier to schedule, deduplicate, export, and monitor.

Check the initial response before adding a browser. Data may already exist in HTML, embedded JSON, or a public endpoint. Scrapy's dynamic-content guidance recommends inspecting those sources before accepting browser costs.

Choose direct Playwright when the project is mostly one headless browser workflow. Choose Scrapy Playwright when many URLs need queues, pipelines, exports, and selective rendering. The Python web scraping libraries comparison explains how each layer fits a larger collection system.

What Do You Need Before Installing Scrapy Playwright?

A working Scrapy Playwright project needs compatible Python packages, an installed browser, and a normal Scrapy project. The tested setup uses Python 3.12.14, Scrapy 2.19.0, and scrapy-playwright 0.0.48. Playwright 1.64.1 supplies the browser API.

Confirm each command resolves inside the same virtual environment. `python -m pip --version` should point into `.venv`, not a global installation. This avoids installing Scrapy in one interpreter and running another.

Current scrapy-playwright releases require Python 3.10 or newer. The tutorial uses `async def start()`, which requires Scrapy 2.13 or newer. The official Scrapy Playwright documentation lists the integration's current minimum versions.

Create and activate an isolated environment on macOS or Linux:

Use these commands in Windows PowerShell:

Install the exact versions tested for this walkthrough:

The Python package and browser binary are separate installations. Linux systems may also need supported browser libraries. Run `python -m playwright install --with-deps chromium` when Playwright reports missing system dependencies.

Pinning the tested versions makes the example reproducible. Review release notes before upgrading one package because browser and integration changes can interact. Recheck the installed browser after every Playwright upgrade.

Create the project after the browser installation succeeds:

Keep the project directory writable because screenshots and exports are local files. Run later commands from the directory containing `scrapy.cfg`.

How Do You Configure Scrapy Playwright?

Scrapy Playwright needs its download handler and Scrapy's asyncio reactor before marked requests can open a browser. Requests without the Playwright metadata continue through Scrapy's standard handler.

Add these values to `playwright_scraper/settings.py`. Replace an existing assignment instead of defining the same setting twice.

`DOWNLOAD_HANDLERS` registers the integration for both schemes. Request metadata still decides whether Chromium handles a request. The reactor lets Playwright's asyncio tasks run within Scrapy's event loop.

`PLAYWRIGHT_DEFAULT_NAVIGATION_TIMEOUT` covers page navigation, while individual actions can use separate timeouts. A 15-second selector wait can fail before a 30-second navigation limit.

Setting `USER_AGENT` to `None` lets the browser send its normal user agent. The separate robots setting supplies Scrapy's robots middleware with a nonempty identity. The example also disables automatic retries, making failures visible during development.

These conservative settings protect the practice site and the local machine. Scrapy allows one concurrent top-level request per domain, while each browser context allows two concurrent pages. Browser subrequests bypass Scrapy's scheduler, so review target rules and authorization before increasing concurrency or reducing the delay.

`scrapy-playwright` sends a Scrapy request through Chromium only when its metadata contains `meta={"playwright": True}`. This selective design distinguishes the integration from a crawler that renders every page. The Scrapy web scraping tutorial covers requests that do not need JavaScript.

Keep these settings in version control so reviews catch accidental changes. Store credentials outside the file.

How Do You Build a Scrapy Playwright Spider?

A Scrapy Playwright spider marks selected requests for browser rendering, runs page actions, and parses the rendered response. This example scrolls a public practice page and confirms that another quote batch appeared.

Create `playwright_scraper/spiders/quotes_browser.py`, then add the complete spider below:

`PageMethod` runs each browser action before Scrapy constructs the final response. The custom function returns the first quote count for later validation. Scrapy then parses the browser's serialized DOM with its normal CSS selectors.

The helper waits for the first quote, records its count, scrolls, and waits for a larger count. That final condition proves the action changed the rendered page. A fixed delay could finish before the next batch appears.

The screenshot provides a visual record, but the exported fields remain the acceptance test. The parser rejects any quote missing text or author. It strips surrounding whitespace from those fields before yielding each item.

The response URL records the final page location after redirects. That field helps diagnose unexpected navigation without retaining the live page.

The page methods execute in their listed order. Moving the screenshot earlier would capture the page before scrolling.

The example leaves `playwright_include_page` unset. Scrapy Playwright therefore closes the page after processing, including supported failure paths. Manual page access requires explicit closure on both success and failure.

How Do You Run and Validate a Scrapy Playwright Spider?

A verified Scrapy Playwright run should prove that the browser action changed the page and produced complete records. Process completion and HTTP 200 are insufficient validation signals.

Run the spider from the directory containing `scrapy.cfg`:

The `-O` option replaces the output file. Scrapy's feed exports support JSON, JSON Lines, CSV, XML, and several storage backends. Use an explicit encoding when text may contain non-ASCII characters.

Validate the exported records before accepting the job:

The verified run started with 10 rendered quotes and exported 20 unique records after scrolling. It also created `quotes.png` and closed the Playwright page. The final logs reported no failed browser requests or spider exceptions.

Compare the exported count with Scrapy's `item_scraped_count`. Any mismatch deserves review before downstream storage or analysis. Replace the example's text-and-author key with the dataset's stable business key.

The target controls how many records each scroll loads. Future runs may return another valid count. Treat success as an increased quote count, complete fields, unique records, and a clean crawler log.

For longer collections, define a stop condition before repeated scrolling. Stop after a known page total, an empty response, a repeated cursor, or no new records. The pagination in web scraping guide covers those patterns separately.

How Do You Use a Proxy With Scrapy Playwright?

Scrapy Playwright proxy settings belong at the browser or context level because browser requests ignore Scrapy's `meta["proxy"]` setting. A context-level proxy covers the document and browser-generated assets within that context.

First, complete a direct test without a proxy. Then append this configuration to `settings.py`. Supply the actual endpoint and credentials through environment variables.

Playwright accepts HTTP and SOCKS5 proxy servers through context options. The Playwright network documentation lists the supported fields. Use the scheme, hostname, and port shown by the provider.

The credential example uses HTTP proxies because Playwright documents username and password for HTTP or HTTPS proxy authentication. Use source-IP allowlisting for SOCKS5 when the provider supports that route. Playwright applies the selected proxy across the full browser context.

Create `run_with_proxy.py` beside `scrapy.cfg`. The helper prompts for credentials, passes them through the child environment, and preserves the crawler's exit status.

Run the proxy test from a terminal:

The browser context does not automatically cover ordinary Scrapy downloads. A robots request or unmarked request may still use Scrapy's standard handler. Test both paths when every request must follow a defined route.

Use one context for browser actions that share cookies and temporary storage. Create another named context when an independent task needs a different proxy route. Proxy settings cannot change after Playwright creates the context.

Proxy errors can appear as authentication failures, navigation timeouts, or empty exports. Check the observed route separately, then validate the target content. Never print passwords, commit secrets, or place credentials in a public URL.

How Do You Run Scrapy Playwright at Scale?

Scrapy Playwright scales best when browser rendering stays selective, concurrency remains bounded, and every result receives content validation. More browser pages can increase memory, bandwidth, and target pressure without increasing valid output.

Start by classifying each URL. Send static HTML and predictable endpoints through Scrapy's ordinary downloader. Reserve Playwright for pages requiring JavaScript, browser storage, or a measurable interaction.

Use `CONCURRENT_REQUESTS`, `PLAYWRIGHT_MAX_CONTEXTS`, and `PLAYWRIGHT_MAX_PAGES_PER_CONTEXT` as separate controls. Browser processes consume memory outside Scrapy's usual process measurement. Monitor system memory, browser crashes, navigation time, and open-page counts directly.

Queue browser jobs separately from lighter requests when their resource profiles differ. This prevents slow rendering from occupying every worker slot. Set per-domain limits before raising global concurrency.

Retries also need classification. Retry temporary network failures with capped attempts, backoff, and jitter. Stop retrying when selectors changed, authorization ended, or the target rejects the workflow.

Browser requests may load images, fonts, scripts, and API responses. Blocking unnecessary resources can reduce traffic, but it may also change page behavior. Test the extracted data after every routing or resource-filter change.

Measure successful records rather than completed navigations. Track page validity, duplicate rate, missing fields, retries, browser time, proxy bandwidth, and total cost. The useful metric is cost per accepted record, not requests per second.

How Do You Troubleshoot Scrapy Playwright Errors?

Scrapy Playwright errors become easier to fix when logs separate installation, browser, selector, proxy, and output failures. Change one layer after identifying the failed condition.

Avoid replacing a selector timeout with a long fixed delay. Wait for the required element, count change, URL change, or response. A precise wait fails sooner and explains what did not happen.

Enable debug logging during diagnosis, but keep secrets out of logs. Capture the final URL, relevant status, rendered element count, and Scrapy statistics. Screenshots help when the returned page differs from the expected content.

How Does Proxidize Fit a Scrapy Playwright Workflow?

Proxidize supports Scrapy Playwright when a crawl needs controlled locations, stable sessions, or broader route diversity. The browser still handles rendering, while Scrapy manages crawling and extraction.

The proxy endpoint works without a proprietary scraping SDK. Standard Playwright context options carry the route configuration.

Browser contexts keep cookies and storage together, but they do not guarantee one exit IP. Use a sticky proxy session for dependent scrolling or navigation that needs route continuity. Use another context for independent work that needs another eligible route.

Residential Proxies are the normal starting point for global public-data collection. They support country, city, and ISP targeting, plus rotating and sticky sessions. Their bandwidth model also makes browser asset control important.

Best For: Residential Proxies suit global dynamic-page collection requiring location selection or broad route diversity.

Mobile Proxies suit authorized tests requiring a United States mobile-network route. They support city and mobile-network selection, with rotating or sticky behavior depending on the plan. They should not replace residential routes when another country is required.

Best For: Mobile Proxies suit United States mobile-page testing where the exit network affects the returned experience.

Choose a Proxidize plan after the direct spider passes its content checks. Compare valid records, retries, browser time, and bandwidth across the intended locations. A proxy changes routing, not the target's rules or the spider's correctness.

What Should You Remember About Scrapy Playwright?

A reliable Scrapy Playwright workflow uses browsers selectively, waits for observable changes, and validates every exported record. The integration is most valuable when only part of a crawl requires browser execution.

• Use ordinary Scrapy requests whenever the required data already exists without browser rendering.
• Mark only dynamic requests with `meta={"playwright": True}`.
• Wait for a specific page change instead of sleeping for an arbitrary duration.
• Keep browser-page concurrency below the machine's tested capacity.
• Configure proxies at the browser or context level, then confirm the observed route.
• Measure complete, unique records rather than successful process exits.

Scrapy handles the crawl, Playwright handles browser behavior, and the proxy controls the network route. Keeping those responsibilities separate makes failures easier to diagnose. It also prevents browser costs from spreading across every request.