Scraping queue not processed

WP Intense Q & ACategory: Price Comparison ProScraping queue not processed
Lucas Vos asked 2 months ago

Hi,

When trying to setup the first product to scrape prices for, I'm getting stuck because whatever I do, the scraping queue doesn't seem to get processed.

I checked the price comparison settings

  • Removed https etc.
  • Used Selector Gadget for the right CSS selector
  • Used Scraper API to check if it is my server or something

Whatever I do, the queue doesn't seem to get processed, no log is created, no errors appear. The scraping urls remain in queue, and it keeps trying until I close the page.

Can you help me out?

Thanks!

Dave H. Staff replied 2 months ago

Do you have your WP Cron running? If you install the WP Crontrol plugin or something similar you can see your cron jobs in the queue and any error messages from them.

Lucas V. replied 2 months ago

Just installed the WP Crontrol plugin, but I don’t see any error messages from them either. Just nothing seem to happen, although the cron is running every minute, as far as I can see. Pretty clueless to be honest..

Mark M. replied 2 months ago

been having the same issue described here for a while too…

Lucas V. replied 2 months ago

@dave H. Wondering when you get back to us, because the problem is definitely still there, leaving the plugin unusable at the moment.

Lucas V. replied 1 month ago

Very disappointed not to hear from you @Dave H. for almost a month now, leaving the plugin unusable at the moment. I feel a bit scammed, especially given the license fees paid. When could we expect a response to help us solve the problem?

3 Answers
Lucas Vos answered 2 months ago

Just installed the WP Crontrol plugin, but I don’t see any error messages from them either. Just nothing seem to happen, although the cron is running every minute, as far as I can see. Pretty clueless to be honest..

Dave Hilditch Staff answered 1 month ago

For anyone who experiences this issue, please check the following:

  1. Confirm you can scrape pages normally, e.g. on a woocommerce product detail page or a page where you have placed the comparison shortcode
  2. You can view debug from the JavaScript console – there is a lot of extra debug gets printed to the console when you click ‘Check Prices’ manually in the comparison table
  3. If the issue persists, please raise a question and provide wp-admin access so I can view what’s going on.

If you don’t provide wp-admin access, it’s v hard to diagnose issues. You can provide access to a dev or test server, but I’ll most likely need access to debug.

Lucas (OP), if this is still persisting, can you please update your question and provide creds in the hidden area please.

Dave H. Staff replied 1 month ago

For extra debug info, if you enable debug logging then any queue processing or scraper processing errors will get output there: https://www.wpintense.com/2020/09/02/how-to-enable-debug-log-to-collect-wordpress-error-messages/

Dave Hilditch Staff answered 3 weeks ago

It’s not finding the config at all – in the domain, you have included a forward slash / at the end of the domain name.

https://i.imgur.com/7jGSQye.png

Instead of http://www.urbanebikes.nl/ it should be http://www.urbanebikes.nl (for example) or even just urbanebikes.nl

I’ve created a bug for this, I’ll fix it so others don’t fall foul of this.

https://www.wpintense.com/idea/prevent-users-entering-dodgy-domain-data-in-pcpro-domain-area/

Next thing I had to alter – you didn’t have a price for this example product and you didn’t have an external URL set – without those, WooCommerce will not display the add to cart button and you had Price Comparison Pro configured to display before the add to cart button – so nothing was displaying!

Then when I view the page, I see an error of ‘decimal symbol not defined’ – this is related to the config not being picked up because you entered the / on the domain name:

https://i.imgur.com/N4jPmNg.png

I’ve created a bug report for the default config missing the decimal symbol here:

https://www.wpintense.com/idea/fix-default-config-to-include-a-decimal-symbol/

For now, I altered your domain config to remove the / and then refreshed the page and I see a price on one of the products (the other is 404ing to the URL you gave). The price is 4.75 which for a motorbike is incredibly cheap!

https://i.imgur.com/FdJqKSK.png

The reason for this is because in your (now working) config, you have , entered instead of . for the decimal separator. The page you are scraping uses commas for thousands and periods for decimals, so you should enter a period here.

Once I’d changed the config, I see the correctly scraped price appearing:

https://i.imgur.com/FXyGUot.png

Sorry it took me so very long to help you with this. I’ve hired 2 people to help me now.

Lucas V. replied 1 week ago

Thanks Dave for helping me out. Hopefully this helps others as well. With this, I’ve setup other products as well with correct scraping.

Dave H. Staff replied 1 week ago

That’s great news! Thanks for letting me know

We use cookies

We use cookies for various things on our site, including our on-site chat bubble (if you use it), our comment forms (if you use them) and for session handling (if you log in).

Other than that, we use cookies to identify where traffic came from to help us understand which traffic turns into sales, we use a cookie for Google Analytics traffic analysis and we use a cookie to customise adverts for our own products we think you'll be interested in.

You can read more detail in our privacy policy page. Please click 'Accept' or 'Decline' to continue.