Choosing your Scraping Type
Price Comparison Pro provides multiple scraping types. Below is a guide explaining when to use each. You’ll find the Scrape Type on your Price Comparison Pro -> Price Comparison Pro settings page:
Once you have websites added, you’ll see the Scrape Type drop down where you can configure your scraping type.
Standard is the default approach to scraping. It fetches the HTML using your server. This should be your default choice as it’s the cheapest, i.e. Free. Only change from this to another approach if it is not working for some reason.
Reasons it may not work include:
- The website you are targeting produces different output per country and does not give you the data you need because of the location of your server – in this case any of the proxy solutions would help
- Your web server has been blocked by the target website. In this case, either the proxy solutions or either of the Scraper API solutions will help
The proxied scrape still uses your server, but loads the data through a proxy. This can be useful to simulate traffic coming from a specific country – e.g. if the website produces country-specific prices.
It can also be useful to avoid bot-blockers. If you scrape 1000 prices per day from your own server, most websites will end up blocking you. If you use 20 proxies, you can reduce this to 50 prices being scraped per proxy IP address.
Phantom JS Proxied
This is the same as Phantom JS except the scrapes will be rotated through the proxies you configure.
This 3rd party company do automatic proxy-rotation and offer very affordable prices. They have a free plan too for 1000 scrapes per month. They will avoid your website scrapes being stopped by bot-blockers.
Scraper API Rendered
This is the Scraper API version of our own Phantom JS service.