Update: Zyte Smart Proxy Manager has been retired and is no longer available. Zyte API replaces it, and you can use it as a drop-in proxy with proxy mode. This post is kept as originally published.
Introducing Zyte Smart Proxy Manager, a smart page downloader
We are proud to introduce Zyte Smart Proxy Manager , a smart web downloader designed specifically for web crawling.
Smart Proxy Manager routes requests through a large, distributed pool of IPs, throttling access by introducing (carefully crafted) delays and discarding IPs from the pool when they get banned from certain domains, or have other problems. As a scraping user, you no longer have to worry about tinkering with download delays, concurrent requests, user agents, cookies or referrers to avoid getting banned or throttled, you just configure your crawler to download pages through Smart Proxy Manager and relax.
Smart Proxy Manager supports HTTPS and POST requests, and a HTTP Proxy API for seamless configuration. It also supports custom region routing (only US supported for now, more regions to come).
To learn more or sign up (public beta is open!) please visit the Smart Proxy Manager home page.
.png&w=3840&q=75)
