How to debug Price Comparison Pro scraper issues from your price comparison table
Directly under the prices in the price comparison tables produced by Price Comparison Pro you’ll see a ‘refresh price’ button. If you click this, it’ll perform a fresh scrape of the source URL and update the prices. This is probably the easiest place to debug any scraping issues.
Once you expand this object, you’ll see a lot more info related to the actual scrape. There are quite a few things that could affect your scrapes, but probably the most common is that you got blocked by that websites bot blocker.
You can see in the example below that amazon.co.uk is returning an HTTP STATUS of 503:
A 503 HTTP STATUS means “Service unavailable”. Often that will mean the server is overloaded, but with Amazon we can be quite sure it’s not and in fact they are returning a 503 status to suspected scrapers.
You can also expand other items inside this debug view.
Here’s how a successful scrape would look – it will return an HTTP STATUS of 200 most likely:
Then if you expand the ‘scrapedpage’ variable, you’ll see some more detail like this: