I want to download CSV file from a webpage, where I have to select the time frame described by the data as well as the columns I want to download.
The page is the following:
https://www.transtats.bts.gov/DL_SelectFields.asp?Table_ID=258&DB_Short_Name=Air%20Carriers
I wanted to ask, how I can achieve downloading this table for years 2015 and 2016, with columns passenger, carrier, origin and dest.
Using the Chrome Developer Tool I found out, that when clicking upon the button "Download", a function "TryDownload()" is being called in the background, which should be callable using a POST request. However, I dont understand, how I can call this function using R as well as changing the default selected columns.
Thank you for your help.
Related
I have a situation where I need to extract tables from 13 different links, which have the same structure, and then append them into only one table with all the data. This way, at first I extracted the links from a home page by copying the link from the respective hyperlink, and then import the data through the Web connector on Power BI. However, 3 months later, I realized that those links changed every quarter but the link from the homepage where they are listed remain the same.
This way, I did some research and I found out this video on YouTube (https://www.youtube.com/watch?v=oxglJL0VWOI), which explained how I can scrape the links from a website, by building a table with the header of the link as a column and the respective link as another column. This way, I can have the links automatically updated, whenever I refresh the data.
The thing is that I am having issues to figure out how can I use this links to extract the data automatically without having to copy them one by one and then import the data using the Power BI Web connector (Web.BrowserContents). Does anyone can give me a hint of how can I implement this?
Thanks in advance!
I'm working on a dashboard in Google Data Studio and my manager would like to have the last edited time of the Data Studio report displayed on a page in the report. I'm using Google Sheets as the data source for all the charts used. (I do not want to display the date when the Sheet was last edited, but the date when the Data Studio Report was last edited.)
I've tried adding a scorecard and messing around with the fields and formulas to display the date, but I'm only able to show the current date and not the date when the report was last edited.
I'm guessing maybe there's a way to connect to Google Analytics to make this possible, I'm not really sure how. Although, if there's a way without using Google Analytics, that would be wonderful.
I'm not a Data Studio expert, so there might be a way.
But from what I gather from https://support.google.com/datastudio/answer/9265174, easy access to version history requires report edit permissions:
Limits of version history
In order to see version history, you must have edit permission to the report or data source .
This question already has answers here:
Scraping data to Google Sheets from a website that uses JavaScript
(2 answers)
Closed last month.
Trying to import weather forecasts for multiple sales markets, but the site I was using blocked Bot Crawl, so my ImportHTML function can't fetch the URL.
I found another site, but the table is formatted in calendar view instead of the list view.
Can I still pull this information into Google Sheets (GS) somehow? I've gotten it to pull information, but it just comes up as [TABLE] in GS.
This is the code I was using to pull changing dates:
=CONCATENATE("https://www.wunderground.com/calendar/us/ca/eureka/KACV/date/",$B$3,"-",$C$3,"?cm_ven=localwx_calendar")
And the code to pull the completed URL's table into GS:
=IMPORTHTML(A2, "Table", 1)
I want the first string of code to pull today's year and month from B3 and C3, and Concatenate, and then the second string of code pulls all that together and then pulls the desired table from the website, but I get a bunch of cells with [TABLE].
There is an API that returns json. Look into documentation to see if there is an endpoint that meets your needs. For example, network tab shows the following for 15 day forecast
https://api.weather.com/v3/wx/forecast/daily/15day?language=en-US&apiKey=6532d6454b8aa370768e63d6ba5a832e&geocode=40.95%2C-124.11&units=e&format=json
You would probably need to write your own script to handle this response though or use a tool like ImportJSON. With a little research it is highly likely you will find something suitable.
Explore 15 day forecast JSON here
I am trying to pull related to trade (imports and exports) from different central banks or statistical offices' websites in RStudio.
This is not a problem when an URL link is associated to a file (.pdf, .csv, .xls, ...). However, I can't find a solution when the user has to manually specify manually the filters (e.g. years, months, sectors,...) and no URL link is associated with the query.
For example, I am trying to load the imports and exports of El Salvador at this url: http://www.bcr.gob.sv/bcrsite/?cdr=38
It appears that the data is not stored in the html code of the web page. I have tried web scraping, but the data cannot be found this way as the user has to first make a query and then click "Export the results".
How I can automatically load these datasets into RStudio?
Looks like you need to use http://www.bcr.gob.sv/bcrsite/downloadsxls.php?exportCDR=1&cdr=38&xls=1 to get the XLS file which you can then parse. Make sure they are ok with this service being used as an API.
First time question here! I'm new to R and am trying to have some fun in it with some NBA data. I want to automatically download shot tracking data and put get it into R so I can produce images, run analysis, etc.
I have been unable to find a website that displays all of the necessary data that I could just grab through web scraping so I turned my attention to finding files that have all of the data.
I found a website that has exactly the data I am looking for, but to download it I have to click a download button. In the upper right corner of the chart at the below link there is a "download csv" link that gives me all of the data related to every shot taken in the NBA. Can anyone please help me to figure out how I can automatically get this with R instead of going and manually downloading it each day?
Below is the link.
http://nbasavant.com/shot_search.php?hfST=&hfQ=&hfSZB=&hfSZA=&hfSZR=&ddlYear=2017&txtGameDateGT=&txtGameDateLT=&ddlGameTimeGT_min=&ddlGameTimeGT_sec=&ddlGameTimeLT_min=&ddlGameTimeLT_sec=&ddlShotDistanceGT=&ddlShotDistanceLT=&ddlTeamShooting=&ddlTeamDefense=&hfPT=&ddlGroupBy=player&ddlOrderBy=shots_made_desc&hfGT=0%7C&ddlShotMade=&ddlMin=0#results