I want to use GET() function from httr package, because this is just an example file and in the original file I need to write in user name and password i.e.
library(httr)
filename<-"filename_in_url.xls"
URL <- "originalurl"
GET(URL, authenticate("usr", "pwd"), write_disk(paste0("C:/Temp/temp/",filename), overwrite = TRUE))
As a test, I tried to import one of the files from I want to import one of the files from https://www.nordpoolgroup.com/historical-market-data/ and do not save it to the disk, but save it to the environment in order to see the data. However, it also does not work.
library(XML)
library(RCurl)
excel <- readHTMLTable(htmlTreeParse(getURL(paste("https://www.nordpoolgroup.com/4a4c6b/globalassets/marketdata-excel-files/elspot-prices_2021_hourly_eur.xls")), useInternalNodes=TRUE))[[1]]
Or if there are other ways how to import data (functions where login information can be as an input)m it will be great to see them
In RStudio I have pressed
File -> Import Dataset -> From text (base)
I selected my dataset "dataset_height_male_female.csv" and I got it working:
Now I created a new R Markdown file and I want to use the dataset that I have imported.
However I get this error:
Error in summary(dataset_height_male_female) : object 'dataset_height_male_female' not found
When you knit the object dataset_height_male_female it is executed in a new environment.
But your object dataset_height_male_female is in your environment at the moment, but not in the new one knit creates.
You have to add the object in the new environment. e.g. put the import sequence (code is on the right lower part when you do it with import dataset) to the first chunk in rmarkdown
for example:
test <- read_csv("test.csv")
summary(test)
I'm having trouble accessing the Energy Information Administration's API through R (https://www.eia.gov/opendata/).
On my office computer, if I try the link in a browser it works, and the data shows up (the full url: https://api.eia.gov/series/?series_id=PET.MCREXUS1.M&api_key=e122a1411ca0ac941eb192ede51feebe&out=json).
I am also successfully connected to Bloomberg's API through R, so R is able to access the network.
Since the API is working and not blocked by my company's firewall, and R is in fact able to connect to the Internet, I have no clue what's going wrong.
The script works fine on my home computer, but at my office computer it is unsuccessful. So I gather it is a network issue, but if somebody could point me in any direction as to what the problem might be I would be grateful (my IT department couldn't help).
library(XML)
api.key = "e122a1411ca0ac941eb192ede51feebe"
series.id = "PET.MCREXUS1.M"
my.url = paste("http://api.eia.gov/series?series_id=", series.id,"&api_key=", api.key, "&out=xml", sep="")
doc = xmlParse(file=my.url, isURL=TRUE) # yields error
Error msg:
No such file or directoryfailed to load external entity "http://api.eia.gov/series?series_id=PET.MCREXUS1.M&api_key=e122a1411ca0ac941eb192ede51feebe&out=json"
Error: 1: No such file or directory2: failed to load external entity "http://api.eia.gov/series?series_id=PET.MCREXUS1.M&api_key=e122a1411ca0ac941eb192ede51feebe&out=json"
I tried some other methods like read_xml() from the xml2 package, but this gives a "could not resolve host" error.
To get XML, you need to change your url to XML:
my.url = paste("http://api.eia.gov/series?series_id=", series.id,"&api_key=",
api.key, "&out=xml", sep="")
res <- httr::GET(my.url)
xml2::read_xml(res)
Or :
res <- httr::GET(my.url)
XML::xmlParse(res)
Otherwise with the post as is(ie &out=json):
res <- httr::GET(my.url)
jsonlite::fromJSON(httr::content(res,"text"))
or this:
xml2::read_xml(httr::content(res,"text"))
Please note that this answer simply provides a way to get the data, whether it is in the desired form is opinion based and up to whoever is processing the data.
If it does not have to be XML output, you can also use the new eia package. (Disclaimer: I'm the author.)
Using your example:
remotes::install_github("leonawicz/eia")
library(eia)
x <- eia_series("PET.MCREXUS1.M")
This assumes your key is set globally (e.g., in .Renviron or previously in your R session with eia_set_key). But you can also pass it directly to the function call above by adding key = "yourkeyhere".
The result returned is a tidyverse-style data frame, one row per series ID and including a data list column that contains the data frame for each time series (can be unnested with tidyr::unnest if desired).
Alternatively, if you set the argument tidy = FALSE, it will return the list result of jsonlite::fromJSON without the "tidy" processing.
Finally, if you set tidy = NA, no processing is done at all and you get the original JSON string output for those who intend to pass the raw output to other canned code or software. The package does not provide XML output, however.
There are more comprehensive examples and vignettes at the eia package website I created.
I am trying to read a simple stock page with the following code. The last line returns an error. I have double checked that the url works and have also tried multiple url's as a check. Any help please?
import urllib.request
url = "https://www.google.com"
data = urllib.request.urlopen(url).read()
I have searched a multiple articles, but unable to get iPython (Python 2.7) to export data to a CSV, and I do not receive an error message to troubleshoot the specific problem, and when I include "print(new_links)" I obtain the desired output; thus, this issue is printing to the csv.
Any suggestions on next steps are much appreciated !
Thanks!
import csv
import requests
import lxml.html as lh
url = 'http://wwwnc.cdc.gov/travel/destinations/list'
page = requests.get(url)
doc = lh.fromstring(page.content)
new_links = []
for link_node in doc.iterdescendants('a'):
try:
new_links.append(link_node.attrib['href'])
except KeyError:
pass
cdc_part1 = open("cdc_part1.csv", 'wb')
wr = csv.writer(cdc_part1, dialect='excel')
wr.writerow(new_links)
Check to make sure the new_links is a list of lists.
If so and wr.writerow(new_links) is still not working, you can try:
for row in new_links:
wr.writerow(row)
I would also check the open statement's file path and mode. Check if you can get it to work with 'w'.