WPI Scrapping Service

WP Intense Q & ACategory: Price Comparison ProWPI Scrapping Service
Mark asked 4 years ago

I have purchased this service as recommended by one of your support clerks after explaining what I was looking at achieving. The following website http://www.luluwebstore.com is what I am having issues with when grabbing the pricing from the product page; it is giving me the following currency AED (United Arab Emirates), where I want it to give me the BHD (Bahraini) currency.

Correct me if I am wrong, but I Believe that website relies on the country selection input to be able to display the correct currency and not by accessing the website using a proxy?

Any clarification on how I can proceed with this using the Scrapping Service is appreciated…

Regards,

6 Answers
Vasil Yordanov answered 4 years ago

Hey Marvin. If you select Bahrain as the country and then grab the correct CSS selector from the price you wish to scrape, and use the correct Proxy from the settings page, it should work without a problem. Let me try and walk you through the process:

First of all, please make sure you have the correct CSS selector for the price selected for the particular product you wish to scrape.

Second, go to the Price Comparison Pro settings page in your WP admin area and scroll all the way down until you see "Proxy and Scraper Configuration".

Under this, you will see the following:

"Choose the option you wish to use below. If in doubt, Standard Scraping is what you want. If the scraped page is using Ajax to load prices (you can check by viewing the source of the page and if you can’t find the price, it’s using ajax) then you’ll need to use our Scraping Service. If you wish to simulate scraping from a different country, or you wish to rotate the scraper IP address to avoid detection then you’ll have to use proxies."

The thing which you need in particular is this line:

" If you wish to simulate scraping from a different country, or you wish to rotate the scraper IP address to avoid detection then you’ll have to use proxies."

You will see four options right under this, you can tick each one of them in order to use a different method for scraping. Try the following:

Basic proxied – Uses random proxy from specified proxy list to request page. (You must have at least one specified proxy in your list)

Basic PhantomJS Parsing – Loads data after page has been parsed and javascript executed on Price Comparison Pro’s servers.

Advanced PhantomJS Proxied Pre-parsing – Uses random proxy from specified proxy list to request before loading in PhantomJS as above. (You must have at least one specified proxy in your list)

One of those three settings should work to grab the price in Bahrain currency.

You can test them out one by one and see which one will display the price, but always make sure you grab the CSS selector from the BAHRAIN version of the website.

Vasil Y. replied 4 years ago

P.S – if none of this works, please contact me on my personal email – [email protected] with an admin access information to your website, and I will see what might be wrong with your configuration.

Mark M. replied 4 years ago

Alright, this clears it up a bit, but my dilemma now is that i do not have any proxy sources to use for Bahrain, do you have suggestions for any?

Also is their any place where i can track the requests i have used for the scrapping service purchased?

Vasil Y. replied 4 years ago

The key provided (API key), after you bought the subscription, is all that you need. Please try using “Basic Proxy” or “Basic PhantomJS Parsing” and see if you can fetch the price that you need? Technically what proxy you use shouldn’t matter, as we are using a phantom JS server for the PCP.

Mark M. replied 4 years ago

I am grabbing the CSS Selector for Bahrain, the Basic Proxy Does not yield any results and keeps showing “checking…” only and the Basic PhantomJS Parsing yields the same results as the standard scrapping which is with the AED Currency and not BHD.

Mark M. replied 4 years ago

Reminder!

Mark answered 4 years ago

Reminder….

Vasil Y. replied 4 years ago

Since nothing else seems to be working, I will need admin access to your website in order to determine what’s wrong with your configuration. Please share it on my personal email – [email protected] – and i will have a look.

Mark M. replied 4 years ago

sent

Mark M. replied 4 years ago

reminder

Mark M. replied 4 years ago

Reminder!!!!

Vasil Y. replied 4 years ago

Reminder about what? The issue was resolved with an email to the customer. We are awaiting an answer with the requested information which we have not yet received.

Mark M. replied 4 years ago

i haven’t received any emails!

Vasil Y. replied 4 years ago

I just received your email after you forwarded it again. We are getting to working on the issue asap, expect a solution within a day. Sorry for the wait.

Vasil Yordanov answered 4 years ago

Hey there, here is the direct answer from the dev who had a look at your issue:

" I don’t think there is a direct solution for the lulu webstore site. The problem is that the site is always delivered first with the AED currency set. This is rendered on the lulu web store server. When you select a language, it stores a cookie on your browser and then reloads the site. The site can then get the stored cookie and render the desired currency.

Generally speaking, it is possible to get a url that links to a product with a desired language / currency setting. Unfortunately the luluwebstore doesn’t work like this – the url for AED is the same as BD. As far as I can tell, PCP doesn’t support setting cookies in the request header at the moment.

The only workaround I can suggest just now would be just to fetch the AED price, and inject a script that performs a conversion to BD – this wouldn’t be ideal, as BD prices might not follow an exact or static conversion formula, but it might be close enough. It wouldn’t take me long to write a script to do this and insert it into the customer’s site, if they think this approach would be acceptable."

Please let us know if you would like us to create the script mentioned here, as this would be the only work around at the moment.

Kind regards.

Mark M. replied 4 years ago

The script will not be ideal at all, that particular business has stores in Multiple Countries. The AED currency is meant for products sold in the country of Dubai and the BHD currency is meant for Bahrain which are two different countries / Pricing for any particular product… Please get back to me if there will be any implementation / ETA on being able to have PCP Support cookies as mentioned (if possible at all).

Additionally i’d like to get a refund on the Purchased Scrapping Service. Please get back to me on what i need to do from my end to make it happen.

Vasil Y. replied 4 years ago

Hey there. I will direct your request for the feature to the dev team and see what can be done about it. Meanwhile, I will refund your 3.99 service later today, you don’t need to provide us with any information. 🙂

Mark M. replied 4 years ago

Anything new regarding the PCP implementation?

Anon replied 4 years ago

There was an update to Price Comparison Pro released last week including a number of fixes. There is another update due out end of this week which will add an extra selector to identify SALE prices.

Anon replied 4 years ago

There was an update to Price Comparison Pro released last week including a number of fixes. There is another update due out end of this week which will add an extra selector to identify SALE prices.

Mark M. replied 4 years ago

but still no update on the issue i have encountered?

Anon replied 4 years ago

Hi – my apologies – I had not read this properly. I see the issue you are describing – and the solution, to allow users to pass a cookie along with the scrape. If we modify the code to give you the option to add a cookie when scraping would you be able to identify the cookie name and value needed to enter into the scraper config?

Mark M. replied 4 years ago

Sounds Promising, i believe i should be able to grab the site-config cookie details which includes the location id for that website through the chrome browser, just to check i was able to access the detail = Content: {“LocationId”:”13241″,”DeliverySlot”:null,”DM”:”H”}.

Does this sound right?

Anon replied 4 years ago

Yes that sounds right. I’m bringing Jed here to comment as he’s inside price comparison pro code right now working on an upgrade.

Anon replied 4 years ago

Hi Moka,

I am implementing a new setting into the Price Comparison Pro settings page, to enable the input of a cookies.txt formatted file (see: http://www.cookiecentral.com/faq/#3.5). If you then use any of the scraper PhantomJS based methods while using the plugin it will then automatically load this file for all requests.

We will let you know when the new version is ready for use.

Thanks,
Jed.

Mark M. replied 4 years ago

Great, Any ETA i should be looking at?

Mark M. replied 3 years ago

any news regarding the last comment/implementation?

Anon replied 3 years ago

Yes sorry – it’s implemented – run your updates. Let me know if any issues. There’s an article with all the changes here: https://www.wpintense.com/2017/04/28/major-new-upgrade-price-comparison-pro-released/

Dave Hilditch Staff answered 3 years ago

This has been implemented – an article discussing the changes was released here:

https://www.wpintense.com/2017/04/28/major-new-upgrade-price-comparison-pro-released/

To use cookies, upgrade your Price Comparison Pro and paste the cookies you wish to use into the little cookie text box.

Mark M. replied 3 years ago

that’s great news, is there anywhere i can test it out before repurchasing it again? as i have already received a refund because if this.

Mark M. replied 3 years ago

also reading through the update, i believe i would have to be subscribed to the PhantomJS Scraping Service to be able to use the cookies? can you please clarify on this?

Mark M. replied 3 years ago

and please elaborate on how the cookies.txt file should be setup?

Anon replied 3 years ago

Hi Moka – we’re using the standard cookie file format. The easiest way to generate your cookies.txt file is to visit the page you need to scrape, generate the cookie, then copy it from the browser e.g. using this guide: http://stackoverflow.com/questions/21919156/how-do-i-copy-cookies-from-chrome

Mark M. replied 3 years ago

do i need any additional services to be able to use the cookies option?

Anon replied 3 years ago

Currently, you need to use the scraper service to use the cookies file. Next release I’ll add the cookies to the direct-scrape code. Sorry, I thought it was fully implemented, but just checked and it’s only implemented on scraper service. There’s a free trial of scraper service.

Mark M. replied 3 years ago

alright, i have subscribed for that service, is there a write up on the cookie option, i cannot seem to be able to get it to work. also the update woocommerce price/images option. does woocommerce have to be setup in a specific way or am i misunderstanding this feature?

Anon replied 3 years ago

Hi – for woocommerce price/images option, you need to enter an images selector. Visit your settings page, and choose to save prices and/or images. For the cookie option, paste in your cookies you would like to persist in the standard format as per the link I provided.

If you need further help – email me on [email protected] and provide login details so I can see your configuration and provide a URL where I can see an example of your price comparison in action.

Mark M. replied 3 years ago

noted, mail sent

Mark M. replied 3 years ago

not sure if i am understanding it correct, but the update woocommerce price option should reflect the lowest price from PCP in the woocommerce item and update its price as well?

Anon replied 3 years ago

Yes that’s correct. If you’re having this many issues, there must be something wrong with your config. I’ll be able to look at the email you sent shortly – I’m in the middle of a release just now.

Mark M. replied 3 years ago

thanks

Mark M. replied 3 years ago

Kind reminder…

Mark answered 3 years ago

Kind Reminder

Mark answered 3 years ago

Hello? not sure why i have not been getting any responses about the cookie setup here or through the tickiting system…

Kind reminder….

We use cookies

We use cookies for various things on our site, including our on-site chat bubble (if you use it), our comment forms (if you use them) and for session handling (if you log in).

Other than that, we use cookies to identify where traffic came from to help us understand which traffic turns into sales, we use a cookie for Google Analytics traffic analysis and we use a cookie to customise adverts for our own products we think you'll be interested in.

You can read more detail in our privacy policy page. Please click 'Accept' or 'Decline' to continue.