I am a beginner in R.
I am trying to download NetCDF4 files from the NASA subset wizzard (https://disc.gsfc.nasa.gov/SSW/#keywords=TRMM_3B42RT_Daily) - in my case, I am looking for TRMM 3B42 precipitation data for South Africa. I will need to download thousands of datasets and work with them in raster format in R and therefore want to use the URL's provided by the subset wizzard. e.g.:
http://disc2.gesdisc.eosdis.nasa.gov/opendap/TRMM_RT/TRMM_3B42RT_Daily.
7/2016/10/3B42RT_Daily.20161001.7.nc4.nc4?precipitation[777:867][99:173],
precipitation_cnt[777:867][99:173],uncal_precipitation_cnt[777:867][99:173],
lat[99:173],lon[777:867]
I have tried
url1 <- "http://.."
dest <- "C:\\Users\\User\\Documents\\3_UP\\2016_Masters\\
Dissertation\\Satellite Data\\TRMM Precipitation\\TRMM 3B42 Daily RT\\Try.nc4.nc4"
download.file(url=url1, destfile=name, mode="wb")
And here I receive an error message "cannot open URL" and additional Warning Messages ending with "HTTP status was '401 Unauthorized'.
Which led me to suspect that the browser needed some login details. In Chrome- the URL works as is. in Internet Explorer (R's default) I typed in my username and password once and after that - the URL also works as is.
However, no success with R functions. The errors remain. I have tried to use other R packages and functions - however, this is my first time trying something like this and I am not seeing the light yet. e.g. most RCurl functions see illegal characters found in the URL.
Does anyone have more experience in working with these specific files ?
best regards
Marion
Related
I'm trying to download all the zipped csv file from the following pg: http://mis.ercot.com/misapp/GetReports.do?reportTypeId=12301&reportTitle=Settlement%20Point%20Prices%20at%20Resource%20Nodes,%20Hubs%20and%20Load%20Zones&showHTMLView=&mimicKey
I've started by trying to download one file as an example before I move on to downloading multiple. This site contains prices from specific locations in Texas - interesting, given recent power outages due to cold weather in Texas.
url <- "http://mis.ercot.com/misapp/GetReports.do?reportTypeId=12301&reportTitle=Settlement%20Point%20Prices%20at%20Resource%20Nodes,%20Hubs%20and%20Load%20Zones&showHTMLView=&mimicKey/cdr.00012301.0000000000000000.20210220.141704636.SPPHLZNP6905_20210220_1415_csv.zip"
temp <- tempfile()
download.file(url,temp, mode = "wb")
data <- read.csv(unzip(temp, "cdr.00012301.0000000000000000.20210220.141704.SPPHLZNP6905_20210220_1415.csv"))
unlink(temp)
Keep receiving the following error message: "error 1 in extracting from zip file."
I'm relatively new to R, so any advice would be helpful.
Edit: If the link above doesn't work, another way to get to the link is following this: http://www.ercot.com/mktinfo/rtm and going to "Real-Time Price Reports" and selecting the last option "Settlement Point Prices at Resource Nodes, Hubs, and Load Zones." Might look a little overwhelming, but my goal for right now is just to download and open the first zipped csv file on there (and ignore all the other files on there)
Trying to open data from government website (hud.gov) about Rental Assistance Demonstration:
http://radresource.net/pha_data.cfm.
The document appears to be an xls file but opening it in Google Docs failed so I tried R. Both readxl::read_xls and readr::read_tsv (as suggested on GitHub) just gave me this BS (Google Doc file or you can just download from radresource above). Any help would be muy apreciado!
I think there is something wrong with the files. I am also unable to open them with anything (excel, googledocs, R, etc.)
I am a day trader based in INDIA. I am using R to do my research. I want to download the End of the Day(EOD) stock prices for different stocks. I was using Quandl and quantmod but was not satisfied with them ( they are OK for historical data but not for EOD quotes). After much research I found out that the EOD for NSE(national stock exchange of india) can be found in the so called "bhav copy" that can be downloaded daily from its website. The URL, for 30th APRIL, is:
https://www.nseindia.com/content/historical/EQUITIES/2018/APR/cm30APR2018bhav.csv.zip
I have two questions:
1) If I type this in the address box of google chrome and execute, it throws a pop up window that asks where to store the csv file. How do I automate this in R? If I just enter the URL as an argument for read.csv, will it suffice?
2) The bhav copy is updated daily. I want to write a function in R that automates this downloading daily. But the URL changes daily( the above URL is only for 30th APRIL 2018). The function will take the current date as an argument. How can I create a one one map to the date and the URL for that particular date? In other words, the URL for date dt is:
https://www.nseindia.com/content/historical/EQUITIES/2018/APR/cmdtAPR2018bhav.csv.zip
the R function f(dt) should create the URL for that particular date and download the csv file.
Very many thanks for your time and effort....
download.file(url, destfile) should be what you need to download the data from the URL in R. Then you can use read.csv. You may need to use unzip() before processing it, judging by the URL you provided.
If you feel like it, you can use fread from the data.table library to pass the url directly, but if it's a zip file then the first option is probably better for you.
As for the URL and processing dates, the lubridate library will be handy for parsing dates.
Package nser solves your problem.
To download and read today's bhavcopy use bhavtoday
library(nser)
bhavtoday
To download and read historical bhavcopy of Equity segment
bhav("30042018")
bhavcopy of F&O segment
fobhav("30042018")
You can also use RSelenium to download bhavcopy zip file using function bhavs.
Package link https://cloud.r-project.org/web/packages/nser/index.html
I'm trying to download data on margin requirements from the MCX website using R
However, I am unable to recognise the appropriate url to use in order to download this data.
The link is here
files for different dates have seemingly different urls
for instance:
DailyMargin_20170919223427.csv
DailyMargin_20170919223104.csv
DailyMargin_20170919223039.csv
They seem to be of the form
DailyMargin_2017091922****.csv
(20170919 is the date on which I'm trying to download the data)
My code has the line:
myURL = paste("https://www.mcxindia.com/market-operations/clearing-settlement/daily-margin", "DailyMargin_2017091922","****", ".csv", sep = "")
the ****** part seems to be random.
From what I can tell, the remaining **** appears to be a timestamp when the data are created by the webpage using javascript. You will probably not be able to directly download data as it will not exist until it is created. That said, you might be able to utilize a package like rvest to do the scrapping for you.
https://stat4701.github.io/edav/2015/04/02/rvest_tutorial/
I'm trying to load an Excel workbook with a large number of tabs into R, do some analysis, and then export the results back into Excel. I'm using the openxlsx package because of some of the features of that package that are not easily accessible using other packages (such as the ability to create "comments" in the output file, color code the tabs, and work with 64-bit R).
When I try to read in the workbooks, I sometimes get the following error message (or something similar):
Error in unzip(xlsxFile, exdir = xmlDir) :
cannot open file 'C:/Users/MENDEL~1/AppData/Local/Temp/RtmpIb3WOf/_excelXMLRead/xl/worksheets/sheet5.xml': Permission denied
This error message doesn't always show up - but sometimes it will appear and the program crashes.
Does anyone have any ideas how to fix this problem? I don't know why the program sometimes thinks it doesn't have permission to access the sheets.
Thank you in advance!
I can think of two possible scenarios for this error:
Scenario 1:
C:/Users/MENDEL~1/AppData/Local/ (This looks like you are trying to read a temporary file)
Solution:
If that is the case try moving the file to a different location like desktop and make sure that you update your working directory accordingly.
Scenario 2
C:/Users/MENDEL~1/AppData/Local/Temp/RtmpIb3WOf/_excelXMLRead/xl/worksheets/sheet5.xml' (Looks like there is some issue with Sheet5 which is of type .xml and the openxlsx does not allow you to read .xml)
Solution:
Check if there is some issue with the format or contents of sheet5 in the file that you are trying to read.
For additional information check CRAN Documentation