I need to scrape specific search results from this website: https://forms.justice.govt.nz/search/TT/
However, after searching for the results, the URL does not change.
Is there some sort of hidden URL that I can find and link to?
Welcome
#mixonic posted a great answer to this over here.
What you need to do is automate some of the actions on your page. You can do this by finding their id or classname, setting a value and triggering the click to submit.
After that you can scrape the result as needed.
Related
I'm using HTMLUnit to extract href attributes under anchor class elements from Google Shopping results. To get a list of all the URLs of the shopping results, I use:
List<HtmlElement> urls = page.getByXPath("//a[contains(#class,'Lq5OHe eaGTj translate-content')]");
but some of the links turned out to be wrong, or more accurately, distorted. For example: this is the link I was supposed to retrieve based on what I saw using Inspect
https://www.universitysupplystore.com/shop_product_detail.asp?catalog_group_id=MjE&catalog_group_name=VGlkZSBUZWNo&catalog_id=599&catalog_name=TWFjQm9vaw&pf_id=202680&product_name=MTMtSW5jaCBNYWNib29rIEFpciBBcHBsZSBNMiBDaGlwIFdpdGggOC1Db3JlIENwdSBBbmQgOC1Db3JlIEdwdS84R2IgVW5pZmllZCBNZW1vcnk&type=1&target=shop_product_list.asp
but this is what the scraper got (not working link)
https://www.universitysupplystore.com/shop_product_detail.asp%3Fcatalog_group_id%3DMjE%26catalog_group_name%3DVGlkZSBUZWNo%26catalog_id%3D599%26catalog_name%3DTWFjQm9vaw%26pf_id%3D202680%26product_name%3DMTMtSW5jaCBNYWNib29rIEFpciBBcHBsZSBNMiBDaGlwIFdpdGggOC1Db3JlIENwdSBBbmQgOC1Db3JlIEdwdS84R2IgVW5pZmllZCBNZW1vcnk%26type%3D1%26target%3Dshop_product_list.asp&rct=j&q=&esrc=s&sa=U&ved=0ahUKEwjtzMqfoLD5AhVIBUQIHXbUBmoQ2SkIww4&usg=AOvVaw2Bom-oAHHYL7PLWixwFHes
And this is the href link when I clicked "Edit as HTML" on the Elements tab after Inspect. (not working link)
https://www.universitysupplystore.com/shop_product_detail.asp%3Fcatalog_group_id%3DMjE%26catalog_group_name%3DVGlkZSBUZWNo%26catalog_id%3D599%26catalog_name%3DTWFjQm9vaw%26pf_id%3D202680%26product_name%3DMTMtSW5jaCBNYWNib29rIEFpciBBcHBsZSBNMiBDaGlwIFdpdGggOC1Db3JlIENwdSBBbmQgOC1Db3JlIEdwdS84R2IgVW5pZmllZCBNZW1vcnk%26type%3D1%26target%3Dshop_product_list.asp&rct=j&q=&esrc=s&sa=U&ved=0ahUKEwiDjOz0oLD5AhU2KkQIHdZJDjAQ2SkIjxY&usg=AOvVaw0l1Boo5XTmiOrL3GU8XMuq
My observation is that it added different characters like % to the original link. My goal is to extract this original link. How do I do that?
What you are facing here is more or less URL encoding (https://en.wikipedia.org/wiki/Percent-encoding).
Please clarify what you are mean by 'not working link'. What do you try to do with the URL?
I have a typeform form that has a different score per answer. At the end of the form I want to redirect to a page on my website (on Wordpress). On that page I want to have a section that has the form results (eg "you scored 5 out of 10").
Is there a way to carry these results onto my website?
I've tried researching web hooks, but I am lost.
I think this rather a Typeform question than a pure programmatic question.
On your Typeform you can create as many Thank you screen as you like.
Each thank you screen could have its own text and its own redirect link.
On the redirect link, you can pass a query parameter taking the value of the score calculated on Typeform. Click on the + button to select either a response to a question, a hidden field, or the calculated score.
And you can redirect the user to your Wordpress with a URL like
https://mywebsite.com?score={value from typeform}
If you have more questions you should contact Typeform support directly.
Is it possible to have a search form on every page of my WordPress site, in the header.php for instance, that contains a search form for Google Maps? When someone adds their postcode into the box they get redirected to the search results page which shows their results.
If so, does anyone know how this works or can anyone provide any code?
Many thanks
M
This is quite possible, and is described here in google's API. You can redirect the user using data gathered by your form, or you can actually even embed a map into your own webpage (which seems like a better user experience, and lets the user spend more time on your site).
I'm trying to put the FB comments box on the bottom of each of my products pages on my site. I have managed to do this but when set to a static url it says the same comments on every product. I deleted the url and now it kind of works but says the plugin is in compatabilty mode.
Heres whats in my products template:
My site is trickscooter.co.uk so you can see. The comments box is at the bottom of every product.
I want people to be able to ask questions on the item shown so I can answer them so all can see.
PS - It says this:
Warning: this comments plugin is operating in compatibility mode, but has no posts yet. Consider specifying an explicit 'href' as suggested in the comments plugin documentation to take advantage of all plugin features.
Thanks !
The static URL won't do for the 'href' parameter as that parameter tells Facebook which comments to show. If you always use the same URL, of course you get always the same comments. You should have a unique url per product and use that dynamically.
If you generate the href parameter dynamically, you need to be aware that you will loose comments if your page is accessible under different domains.
The best way is to configure a 'facebook domain' (e.g. 'www.foo.example') and use that to generate your href attribute for Facebook comments.
If I misunderstood you please explain your current embedding code and what you tried. If you like the answer, please mark it as 'solution'.
I've searched high and low for an answer for this problem and need the help of experts.
The website I'm helping to oversee has a registration in a frame on their main website. When a user visits the site, they can choose which course to take and click on a "Register" link to be taken to the externally hosted form.
I was able to get the GA tags changed on the framed pages, so that step is done.
However, I realise that in order to properly cross-domain track with Google Analytics, I need to also add an onclick event to any links that bring users to those pages, such as the example below.
Link
However, the site I'm working with is in Wordpress, and when I view the link in there it looks like this:
[iab_event link="http://www.EXAMPLESITE.com/course/4105/register.asp" text="Register"]
And when you view the page source of the live site this link is translated to this:
Register
I tried adding the second onclick event to the one above, but that didn't work. Is there any way to add the onClick="pageTracker._link('http://www.EXAMPLESITE.com/course/4105/register.asp'); return false;" to this link without breaking it?
Thank you in advance for any advice given!
Try stacking them in the onclick event like this:
Register
It should run all of the commands as long as they're separated by semi-colons.