How Price Comparison Websites Work
In recent years, there’s been an explosion in the number of online price comparison sites and the market shows no signs of slowing.
As bargain-hunting consumers become ever savvier and increasingly cost-conscious, price comparison engines are starting to corner markets in everything from insurance to TV and media providers.
Specialised sites
Over the years, comparison sites have become increasingly specialized to offer targeted price reductions in specific sectors. For example, comparison-providers Utility Bidder have a specialised business water division, offering firms reduced rates on their water supplies. It seems highly likely this trend will continue as providers aim to bolster their branding and become recognised as the go-to expert in specific fields.
Common tools used by comparison engines
As most people realise, price comparison engines work by gathering costs from multiple providers to present the best market rates for products and services – but just how do these sites grab this data? Here are three common tools employed by price comparison providers to scour markets and display deals.
Direct feeds from merchants
The largest comparison services typically take feeds directly from suppliers’ websites to minimise the risks of price errors or omissions. For many internet-based retailers, price comparison engines provide substantial opportunities for new business and increased revenue, so partnerships between providers and the comparison sites are quite common and make good commercial sense.
In the direct feed model, businesses grant comparison providers access to their Application Programming Interface (API) to directly deliver data from the source. However, they will also often charge a premium if the provider’s server calls place too heavy a load on resources, which can create a considerable additional overhead for comparison sites. Also, merchants often provide data in multiple formats, which can cause parsing headaches on the part of the comparison service.
Third-party API data feeds
Another extremely common method for comparison providers to gather up-to-date product and service prices is to rely on feeds from a third-party API service company. Third-party companies constantly scan the most popular retailer websites and in turn grant the comparison provider access to their database of current prices.
Using a third-party provider can result in considerable time and money savings for comparison providers by reducing development and integration costs. However, the third-party company will charge the comparison provider based on the volume of data supplied and the fees can quickly add up if excessive requests are made.
Gathering data via web scraping
If a comparison service is operating on a tight budget, they might rely on web scraping services to literally ‘scrape’ data from merchant sites. The technique of web scraping involves using automated bots to crawl merchant providers’ websites to gather data and send it back to servers for further processing.
Specialist data-as-a-service (DaaS) companies can provide reliable, current pricing data to comparison companies for a small fee. Alternatively, it’s also possible for the comparison provider to build their own web scraping systems, thereby further reducing their overheads.
The major drawback of web scraping is that many merchant sites have started employing anti-crawling techniques on their sites to reduce the additional server-load caused by bots and prevent them from making repeated data requests.
