I am working with the EDGAR package in R to download Apple's 2005 Annual Report. This is my code to get that far:
library(edgar)
getMasterIndex(2005)
aapl<-getFilings(2005,320193, '10-K')
This was my output when I did that:
> aapl<-getFilings(2005,320193, '10-K')
Total number of filings to be downloaded=1. Do you want to download (yes/no)? yes
> aapl
Link Status
1 https://www.sec.gov/Archives/edgar/data/320193/0001104659-05-058421.txt
Download success
To me this looks like I just retrieved the URL to this particular document, I did not actually download the text file.
My next step I imagine would be to download the file based on URL. I thought doing a download.file using AAPL as my URL argument would work but I must be missing something.
Thoughts on how to download the full doc based on the URL? Thank you
Related
I'm trying to download all the zipped csv file from the following pg: http://mis.ercot.com/misapp/GetReports.do?reportTypeId=12301&reportTitle=Settlement%20Point%20Prices%20at%20Resource%20Nodes,%20Hubs%20and%20Load%20Zones&showHTMLView=&mimicKey
I've started by trying to download one file as an example before I move on to downloading multiple. This site contains prices from specific locations in Texas - interesting, given recent power outages due to cold weather in Texas.
url <- "http://mis.ercot.com/misapp/GetReports.do?reportTypeId=12301&reportTitle=Settlement%20Point%20Prices%20at%20Resource%20Nodes,%20Hubs%20and%20Load%20Zones&showHTMLView=&mimicKey/cdr.00012301.0000000000000000.20210220.141704636.SPPHLZNP6905_20210220_1415_csv.zip"
temp <- tempfile()
download.file(url,temp, mode = "wb")
data <- read.csv(unzip(temp, "cdr.00012301.0000000000000000.20210220.141704.SPPHLZNP6905_20210220_1415.csv"))
unlink(temp)
Keep receiving the following error message: "error 1 in extracting from zip file."
I'm relatively new to R, so any advice would be helpful.
Edit: If the link above doesn't work, another way to get to the link is following this: http://www.ercot.com/mktinfo/rtm and going to "Real-Time Price Reports" and selecting the last option "Settlement Point Prices at Resource Nodes, Hubs, and Load Zones." Might look a little overwhelming, but my goal for right now is just to download and open the first zipped csv file on there (and ignore all the other files on there)
I'm trying to download a csv file from the Our World in Data website. There are several charts at the end of the post, and each has a "Data" tab that reveals a download link.
When you click the csv downloads directly. The link to the "relative-share-of-deaths-in-usa.csv" button is "https://ourworldindata.org/e9df3be1-29e0-4366-bc14-554bb4ba8be1", but when I use this in RCurl, it downloads an html file. How can I pull into R from the site?
library (RCurl)
download <- getURL("https://ourworldindata.org/e9df3be1-29e0-4366-bc14-554bb4ba8be1")
data <- read.csv(textConnection(download))
I am a day trader based in INDIA. I am using R to do my research. I want to download the End of the Day(EOD) stock prices for different stocks. I was using Quandl and quantmod but was not satisfied with them ( they are OK for historical data but not for EOD quotes). After much research I found out that the EOD for NSE(national stock exchange of india) can be found in the so called "bhav copy" that can be downloaded daily from its website. The URL, for 30th APRIL, is:
https://www.nseindia.com/content/historical/EQUITIES/2018/APR/cm30APR2018bhav.csv.zip
I have two questions:
1) If I type this in the address box of google chrome and execute, it throws a pop up window that asks where to store the csv file. How do I automate this in R? If I just enter the URL as an argument for read.csv, will it suffice?
2) The bhav copy is updated daily. I want to write a function in R that automates this downloading daily. But the URL changes daily( the above URL is only for 30th APRIL 2018). The function will take the current date as an argument. How can I create a one one map to the date and the URL for that particular date? In other words, the URL for date dt is:
https://www.nseindia.com/content/historical/EQUITIES/2018/APR/cmdtAPR2018bhav.csv.zip
the R function f(dt) should create the URL for that particular date and download the csv file.
Very many thanks for your time and effort....
download.file(url, destfile) should be what you need to download the data from the URL in R. Then you can use read.csv. You may need to use unzip() before processing it, judging by the URL you provided.
If you feel like it, you can use fread from the data.table library to pass the url directly, but if it's a zip file then the first option is probably better for you.
As for the URL and processing dates, the lubridate library will be handy for parsing dates.
Package nser solves your problem.
To download and read today's bhavcopy use bhavtoday
library(nser)
bhavtoday
To download and read historical bhavcopy of Equity segment
bhav("30042018")
bhavcopy of F&O segment
fobhav("30042018")
You can also use RSelenium to download bhavcopy zip file using function bhavs.
Package link https://cloud.r-project.org/web/packages/nser/index.html
I am a beginner in R.
I am trying to download NetCDF4 files from the NASA subset wizzard (https://disc.gsfc.nasa.gov/SSW/#keywords=TRMM_3B42RT_Daily) - in my case, I am looking for TRMM 3B42 precipitation data for South Africa. I will need to download thousands of datasets and work with them in raster format in R and therefore want to use the URL's provided by the subset wizzard. e.g.:
http://disc2.gesdisc.eosdis.nasa.gov/opendap/TRMM_RT/TRMM_3B42RT_Daily.
7/2016/10/3B42RT_Daily.20161001.7.nc4.nc4?precipitation[777:867][99:173],
precipitation_cnt[777:867][99:173],uncal_precipitation_cnt[777:867][99:173],
lat[99:173],lon[777:867]
I have tried
url1 <- "http://.."
dest <- "C:\\Users\\User\\Documents\\3_UP\\2016_Masters\\
Dissertation\\Satellite Data\\TRMM Precipitation\\TRMM 3B42 Daily RT\\Try.nc4.nc4"
download.file(url=url1, destfile=name, mode="wb")
And here I receive an error message "cannot open URL" and additional Warning Messages ending with "HTTP status was '401 Unauthorized'.
Which led me to suspect that the browser needed some login details. In Chrome- the URL works as is. in Internet Explorer (R's default) I typed in my username and password once and after that - the URL also works as is.
However, no success with R functions. The errors remain. I have tried to use other R packages and functions - however, this is my first time trying something like this and I am not seeing the light yet. e.g. most RCurl functions see illegal characters found in the URL.
Does anyone have more experience in working with these specific files ?
best regards
Marion
The UK Charity Commission has a site from which you can download charity data for a specified category e.g. dementia
http://beta.charitycommission.gov.uk/charity-search/?q=dementia
When hovering over the Export Button, the link URL is shown at the bottom with the search item and number of charities included
The actual file downloaded is named
"charitydetails_2017_06_14_23_57_17.csv" so no mention of category/number but with a date-stamp instead
I have tried
library(readr)
df <- read_csv("http://beta.charitycommission.gov.uk/charity-search/?q=dementia&exportCSV=317.csv")
but just get 404 error
Is there anyways in R that I can automate this so that entering a different term e.g. blind in, say, a shiny app would download the correct dataset into R for processing?
TIA
You used an incorrect URL. The correct one is:
http://beta.charitycommission.gov.uk/charity-search/?q=dementia&exportCSV=1&p=317