Designing a fair and predictable pricing model for Web Scraping APIs

The architecture of a Web Scraping API pricing

Designing a fair and predictable pricing model for Zyte API.

When it comes to web scraping, uncertainty can drive over-spending. Customers often overspend on infrastructure, unsure of what they truly need. This issue was a key topic in a recent conversation between James Kehoe, Senior Product Manager at Zyte, and Zyte’s CEO, Shane Evans.

That was one of the key insights I got as I listened to James and Shane discuss how they arrived at the pricing model for Zyte API.

I also gained a deeper understanding of the three core principles: predictability, fairness, and transparency, which shaped their approach to designing a customer-centric pricing model.

Below, I’ll outline the key takeaways from their conversation.

Balancing complexity and simplicity

“When people aren’t sure what they need, they tend to oversubscribe. Again and again, we saw customers second-guess their web scraping infrastructure requirements and overspending just to err on the side of caution. We realised there is a major infrastructure efficiency gap that customers don’t know they’re facing, and we can close that gap for them.

From a technical standpoint, we knew how to assemble the pieces, but just as crucial was figuring out how to embed predictability, transparency, and fairness into the Zyte API pricing model.” James reflected.

As a "Swiss army knife" for extracting data from millions of websites, Zyte API caters to a wide range of use cases, each with different needs and expectations. The challenge of designing a pricing model for Zyte API lay in balancing this functional sophistication with a simple, fair user experience. 

It took time, customer feedback, and extensive data analysis to get it right. Here’s how Zyte approached the problem.

Predictability: reducing uncertainty

“Our customers need predictable costs,” James said. “Scraping needs can change overnight—seasonal spikes like Black Friday, technical changes on websites, or new business priorities all play a role. But we made a choice to absorb as much of that variability as possible, rather than passing it directly onto customers.”

For example, even during periods of high demand like Black Friday, Zyte avoids price hikes. “If a website becomes temporarily harder to scrape, we don’t immediately bump its tier. Only if the change is permanent—for instance, when a website introduces significant new restrictions—we do move it up, and even then, we give customers two weeks’ notice,” he added.

Enterprise customers, in particular, value this stability. “They told us, ‘We don’t want pricing surprises disrupting our annual budgets,’ so we locked their pricing in for a year as we launched it. That stability is non-negotiable for them.”

Fairness: charging for what worked

James emphasised that fairness is central to Zyte’s approach. Speaking of Zyte’s five-tier pricing model: “Most websites—about 80%—fall into our simplest tiers. It wouldn’t be right to have those customers subsidising the few highly complex sites.”

“Each tier reflects the complexity of scraping a website,” he continued. “This wasn’t based on guesswork. We ran extensive data science analyses to figure out what truly drives cost and structured the tiers accordingly.”

You can explore Zyte’s 5-tier system in more detail here. In essence, websites with advanced anti-bot measures demand more sophisticated scraping techniques, which in turn raises costs. Zyte’s five tiers range from simple sites that require only data center proxies to more complex ones that need residential proxies, headless browsers, and geo-targeted proxies.

Another way Zyte ensures fairness is by eliminating penalties for overages. “We only charge for successful requests,” James explained. “If traffic exceeds expectations, there are no surprise fees or overage penalties. That’s not just a pricing decision—it’s a way to show we’re on the customer’s side.”

Many customers begin with complex scraping problems. “We’ve heard customers say, ‘We came to Zyte because nothing else worked—Tier 4 and Tier 5 challenges.’ But once they see how well our API handles those, they often delegate simpler sites to us too. That’s how we build long-term partnerships.”

Transparency: simplifying without dumbing down

Web scraping APIs are inherently technical, but Zyte worked hard to make its pricing transparent. “The pricing table lays it all out—no hidden fees, no drilling into fine print. Customers can see the cost factors for every edge case upfront,” James explained.

Still, transparency has its hurdles. “Some customers told us, ‘Why not just sell us “bundles of requests” like before?’ They were used to older models like requests-based or bandwidth-based pricing. But those approaches don’t reflect real costs or outcomes. Our tiered model takes some getting used to, but it’s better in the long run.”

Addressing resistance

The transition to this pricing model wasn’t without challenges. Some customers initially worried about frequent tier updates. “We heard concerns like, ‘Are you going to adjust our tier every month?’” James shared. “So we made stability a priority”.

To address these concerns, Zyte reviews tiers once per quarter, and even then changes are rare. On average, only 0.1% of websites move to a different tier—upwards or downwards. When adjustments are necessary, Zyte ensures customers are informed well in advance to maintain transparency and trust.

Others assumed the new model would be more expensive. “Customers often start with their biggest challenges—high-tier sites—and think that reflects all costs. Over time, they realise how affordable it is for simpler sites too”. That shift in perception has made customers add even more websites to their pipelines, due to the affordable costs of the API.

What’s the future?

The future of pricing in this space is likely to shift further away from technical metrics and units, focusing instead on simplicity and customer value. “I think the industry will move toward tying pricing to outcomes, offering predictability and aligning with how customers measure success,” James explained.

This trend reflects a growing emphasis on paying for value rather than infrastructure complexity. As businesses talk to more customers and understand their needs, the path forward becomes clear: abstract away unnecessary layers, focus on predictability, and ensure pricing models are easy to manage and justify financially. “Ultimately, when you listen to enough customers, the decision is obvious,” James concluded.

Curious how the pricing model works for your needs? Click here to try out the cost estimator on your list of websites and see the pricing model in action.