I have done quite a few instances of scraping some data for projects, I was adding a plugin for WP to add a card to the woocommerce store and wanted to get basic data for a MTG card from TCGplayer.com. Not trying to scrape mass pages just want the basic card data and price info from a url entered. Using curl I do header data back but no content. Looking in chrome I do not see any browser loading activity that curl would not be retrieving.
URL I have tried to gather data
https://www.tcgplayer.com/product/240037/?Language=English
any thoughts are appreciated on this one.
Related
I have a website with Wordpress and I want to add members cards that they can fill their information and can download it from the website, but not sure how to do it I look for a plugin but I did not find.
also, I have 2 data on my website 'Event and Users' could I add them in one excel sheet or no?
I tried to look for a plugin
you need to separate your questions and add more information about your problem.
I'll try to help you:
** members cards **
You can use a form plugin to let the user fill the data (for example: name, surname, birth date, ecc..) that you need to print in the card. I suggest you to use this: https://formidableforms.com
the difficult way: you can intercept the form submission with formidable specific function "https://formidableforms.com/knowledgebase/frm_after_create_entry/" and with the collected data you can create image in the fly with php and imagemagik (or GD or other) https://www.php.net/manual/en/book.imagick.php. This part need that you have a little bit of experience in development. Check next point for a much simpler alternatives
the simple way: With formidable you can create a post (or better custom post) with the collected data without working on code:
https://formidableforms.com/knowledgebase/create-posts/ then you can redirect user to that page.
Invite the user to print that page and setup a specific css for print: https://benfrain.com/create-print-styles-using-css3-media-queries/
I would like to create a webpage that displays the data from another website's RSS feed and prettify the content and maybe remove some entries etc.
Is this something that would be possible with Github pages and jekyl?
I suspect not but I'm a little out of date with the latest web-tech so would appreciate any advice.
(I am unable to do this with javascript/ajax alone because the request is cross-origin and I get "Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at X" errors.)
Thanks
For Example
Let's say i want to create page at https://news.github.io that when requested fetches the data from the BBC's RSS news feed at http://feeds.bbci.co.uk/news/rss.xml?edition=uk and displays the title of the first 10 results in a table with a given font.
I want to scrape this website - https://recorder.co.clark.nv.us/RecorderEcommerce/default.aspx.
I need to simulate clicking the 'Parcel #' link first then entering a value (i.e. 1234) into the Parcel # textbox and clicking search.
I need to scrape the data in the table which is shown at the bottom.
I'd like to write this in ASP.NET so I can push the Parcel # etc parameters through as part of the request. Once I get that request back, I'm confident I can parse it myself, I'm just not sure how I should exactly send the original request as it's not as simple as sending across parameters?
In your question you've specified both Javascript and asp.net so I really have no idea what technologies you're planning on using. I'd recommend HtmlAgility pack. It has a download from url option. It'll help with the parsing too.
HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
doc.Load("https://recorder.co.clark.nv.us/RecorderEcommerce/default.aspx");
I am trying to write a program that extracts shipping container information from a specific site. I've had success with several shipping companies wbsites that use POST methods to submit searches. For these sites I have been using cURL, a PHP libary. However, this one site http://www.cma-cgm.com/eBusiness/Tracking/ has been very difficult to interact with. I have tried using cURL but all I retrieve is the surrounding html without the actual search results.
A sample container I am trying to track is CMAU1173561.
The actual tracking URL seems to be http://www.cma-cgm.com/eBusiness/Tracking/Default.aspx?ContNum=CMAU1173561&T=292012319448 where ContNum is the shipping container and T is a value constructed from current time.
I also noted the .aspx. What is the best approach for retrieving these search results programatically?
I originally asked this question on Super User but was told that it might be better placed here...
I have a running blog and to help me track and write about my runs I've recently bought a Garmin GPS watch. The setup works a treat and I'm able to share links to my runs in my blog such as:
http://connect.garmin.com/activity/23842182
Is there an easy way for me to capture the map itself out of the Garmin Connect site (see the link) and display it in my blog posting? I can take a screenshot but an interactive map would be heaps better. It's obviously a Google Map with the run info overlayed so there must be a way... right?
To created an embedded interactive Google Map to render your run polylines, you will need to extract the data that the Garmin site is using to render the line.
From the Garmin site, there are two Javascript files that do the work:
http://connect.garmin.com/resource/garmin-js-lib/map/MapsUtil.js - Bunch of utility functions for rendering Google maps based on data in the Garmin system
http://connect.garmin.com/api/activity/component/mapLoader.js - Uses Garmin.service.ActivityClient to grab the JSON data describing the polyline. It feeds this data into Garmin.map.MapsUtil.addEncodedPolylineToMap to render the map.
So do do this on your blog, you will need to either request the JSON data from the Garmin site (and trust that the URI format doesn't change) or grab the data and store it on your own site. The URI format is currently:
http://connect.garmin.com/proxy/activity-service-1.0/gpolyline/activity/<activity id>?full=true
Where activity ID is the last number in your original URL. So:
http://connect.garmin.com/activity/23842182
http://connect.garmin.com/proxy/activity-service-1.0/gpolyline/activity/23842182?full=true
This data request will return some JSON that you can then use to render a Google Map.
Once you have decided how you want to store the JSON data, you will need to write some Javascript to request the JSON and, in the callback, feed it into the GPolyline.fromEncoded method. Once you have a GPolyline object (that is populated from the encoded JSON data), you can add it to a Google Maps GMap2 with the addOverlay method.
I realize that this answer is fairly technically involved and might be overwhelming if you haven't played with Google Maps before. If this is the case, I suggest heading over to the Google Maps API intro page for some hints on getting started.
Since this question was first posted, Garmin Connect has since added a quick code snippet to embed in your WordPress site to display your maps and course data. If you're having issues getting the code snippet to stay in the post after saving - check out these instructions for embedding Garmin Connect activities in WordPress.