A customer has found a site with ‘Im under attack’ mode enabled.

We should still be able to scrape these pages – except when a captcha is requested – although even then, it would be useful to be able to display the captcha to the admin and ask them to enter it to re-enable scraping from their server.

The example site here is:

https://www.size.co.uk/product/white-puma-x-ralph-sampson-low-og/140715/

The library to use to perform the bypass would be this:

https://github.com/Anorov/cloudflare-scrape

This service will have to be enabled through our own Phantom JS scraping service.

An additional check should be added to the start of any scraping session to check for cloudflare protection and then use the library.

The process is pretty much:

1. Check if cloudflare protection exists
2. Use the cloudflare-bypass library
3. Store the cookies for this target site for this source IP
4. When scraping after that, append the cookies to the cURL request (along with any other cookies specified by user in normal way) so that all requests bypass the protection page.
5. If cookies fail, re-run the cloudflare-bypass library to get a fresh set of cookies for this site + server IP

In some cases – digital subscriptions, for example – products have different levels of pricing, but only one source (ie. you can only sign up directly from the vendor of that software, so there’s no price comparison between different stores, but between different tiers). It’d be useful to be able to display these as comparisons – and to be able to pull in other additional data – as well as a single price for a single product.

It would be great if the price comparison pro could send out push notifications. So visitors can follow their selected products and receive a push notification when a product goes on sale or reach a certain price.

I see the foundthru site currently failing to show prices. I believe I’ve seen PC pro uses transients. if transient price data has been purged, and the script is not able to fetch prices, it will fail to show prices.

It would be great if PC pro would just store prices in postmeta. It could render it server side making, also making it easier for search engines to see it (may be beneficial for SEO if combined with rich snippets markup).

Transients could still be used as an enhancement, ie it could still fetch these after page load and update prices if succesful. Pricing already rendering server side could be hidden until the ajax completes or returns an error.

For the people using your plugins in countries that don’t have english as their primary language it would be great if you could include the necessary strings to make it ready for translation.
This could be easily accomplished by wrapping translatable strings in special gettext functions.
For a start it would be enough to make the front-end texts translatable.