The Web Scraping Ecosystem for Professional Developers

Overcome any web scraping challenge quickly and efficiently with the Zyte coherent web scraping eco-system.

Step 1

Write your spiders code with a scraping framework

We recommend you start with Scrapy an open-source web scraping framework for Python, created and maintained by Zyte. Check our learn Scrapy tutorials or join the Extract Community on discord to connect with web scraping experts.

1from scrapy import Field, Item, Spider
2
3class MyItem(Item):
4    price = Field()
5
6
7class Scraper(Spider):
8
9   name = "scraper"
10   start_urls = ["https://the-best-store.com/"]
11
12   def parse(self, response, **kwargs):
13       item = MyItem()
14       item["price"] = response.css(".price_color::text").get()
15       return item

Copy

Step 2

Deploy to Scrapy Cloud (optional)

Host, monitor and QA your Scrapy spiders in Scrapy Cloud the perfect solution to scale web scraping projects quickly and reliably with Zero vendor lock-in. Deploy code to Scrapy Cloud via your command line or directly with GitHub or access AI-Powered Smart Spider templates.

Try Scrapy Cloud Free Learn more

Step 3

Handling bans and blocks

Configure Scrapy requests to use Zyte API to automatically extract data from websites of all complexity levels, using only the leanest tech to handle bans. Plus render Javascript, automate browser actions and take screenshots. Available as either a REST API or HTTP API.

Try Zyte API Free Visit Docs

1def start_requests(self):
2        yield scrapy.Request(
3            url="https://quotes.toscrape.com/",
4            meta={
5              "zyte_api": {
6                "geolocation": "CA",
7                "browserHtml": True
8              }
9            }
10        )

Copy

Step 4

Turn web pages into JSON with AI Extraction

Switch on automatic extraction for articles, product pages or job listings and our patented ML will structure it for you. Forget about writing (and fixing) parsing code for your website.

Massively reduce time to build spiders
Minimize maintenance overhead per site
Can be extended and overridden

Try Zyte API: Automatic Extraction Learn more

1from scrapy import Request, Spider
2
3
4class MySpider(Spider):
5    my_spider = "toscrape_com"
6
7    def start_requests(self):
8        yield Request(
9            "https://books.toscrape.com/",
10            meta={
11                "zyte_api_automap": {
12                    "httpResponseBody": True,  
13                    "productList": True,  
14                },
15            },
16        )
17
18    def parse(self, response):
19        http_response_body: bytes = response.body
20        productList = response.raw_api_response["productList"]

Copy

Step 5

Go 'All-in' on AI for product data

Our complete solution to extracting e-commerce product data with AI

Radically scale your web data with our end-to-end solution to web scraping products using AI.

Write e-commerce spiders 3x faster,
Generate 50% less maintenance overhead
Customise AI Extraction with Scrapy

Try Zyte API - AI Scraping Learn more

Step 6

Ready to scale up?

When you want to level up and tackle advanced use cases or scale quickly, technology alone often isn’t enough. With our Enterprise plans, we combine our leading technology with our industry expertise to give you:

Developer-to-developer training and support
Enhanced SLAs
Enterprise pricing and volume discounts

Get a quote Learn more

Step 7

Outsource to Zyte

For organizations that want to outsource some (or all) of their web data collection, Zyte offers web scraping services backed by 13+ years of experience plus the largest team of 100+ web scraping experts. Let our experts understand your needs, and build the best solution to get the data for you.

Zyte Data - Talk to us Learn more