Explore resources by topic or category

One of the things that takes more time when building a spider is reviewing the scraped data and making sure it conforms to the requirements and expectations of your client or team.

We're excited to introduce Dash, a major update to our scraping platform.
This release is the final step in migrating to our new storage back end and contains improvements to almost every part of our infrastructure. In this post I'd like to introduce some of the highlights.

We are proud to introduce Zyte Smart Proxy Manager , a smart web downloader designed specifically for web crawling.

We often have to write spiders that need to login to sites, in order to scrape data from them. Our customers provide us with the site, username and password, and we do the rest.

After a year considering it, we have decided to go ahead and drop support for Python 2.5 in Scrapy. Starting from 0.15, Scrapy will require Python 2.6 or above.