How to debug Price Comparison Pro scraper issues from your price comparison table

Directly under the prices in the price comparison tables produced by Price Comparison Pro you’ll see a ‘refresh price’ button. If you click this, it’ll perform a fresh scrape of the source URL and update the prices. This is probably the easiest place to debug any scraping issues.

To view the debug, open the JavaScript console (CTRL+SHIFT+I or F12 then click the Console tab) then click the Refresh price button. You’ll see something like this:

Once you expand this object, you’ll see a lot more info related to the actual scrape. There are quite a few things that could affect your scrapes, but probably the most common is that you got blocked by that websites bot blocker.

You can see in the example below that amazon.co.uk is returning an HTTP STATUS of 503:

A 503 HTTP STATUS means “Service unavailable”. Often that will mean the server is overloaded, but with Amazon we can be quite sure it’s not and in fact they are returning a 503 status to suspected scrapers.

You can also expand other items inside this debug view.

blank
Expanding curlinfo shows all the debug from curl.

Here’s how a successful scrape would look – it will return an HTTP STATUS of 200 most likely:

blank

Then if you expand the ‘scrapedpage’ variable, you’ll see some more detail like this:

blank

Did this article answer your question?

Leave a Reply

Your email address will not be published. Required fields are marked *


Visit our Discord server to chat directly to us!

We use cookies

We use cookies for various things on our site, including our on-site chat bubble (if you use it), our comment forms (if you use them) and for session handling (if you log in).

Other than that, we use cookies to identify where traffic came from to help us understand which traffic turns into sales, we use a cookie for Google Analytics traffic analysis and we use a cookie to customise adverts for our own products we think you'll be interested in.

You can read more detail in our privacy policy page. Please click 'Accept' or 'Decline' to continue.