I tried now for some time to simply download a csv file in RStudio(so i am using R) from the web. I have done stuff like this before and never run into the issues i have now. Tried several solutions suggested online. I simply try to download the following file from here https://www.nasdaq.com/market-activity/stocks/aapl/historical. This is the link to the csv -> https://www.nasdaq.com/api/v1/historical/AAPL/stocks/2020-09-09/2020-10-09. I tried httr and RCurl package methods none worked. Hope someone can help me out, thanks in advance.
Edit: What i tried so far:
-Updated ssl
-I can install new packages so internet generally works
-Updated git
-Updated R
-Updated all packages
-Tried what is suggested here: Download.file fails in RStudio
Edit2: A while ago i did webscrape with the package RSelenium and started a rem_session. Is it possible that Rselenium changes some underlying settings?
Edit3: Completely uninstalled Rstudio + R + Rtool manually deleted everything, after a fresh installation still the same problem
Have you tried to load it directly in R through read.csv()?
It worked for me... Check it out:
data = read.csv('https://www.nasdaq.com/api/v1/historical/AAPL/stocks/2020-09-09/2020-10-09')
data
Here is the output:
Date Close.Last Volume Open High Low
1 10/07/2020 $115.08 96848990 $114.62 $115.55 $114.13
2 10/06/2020 $113.16 161498200 $115.7 $116.12 $112.25
3 10/05/2020 $116.5 106243800 $113.91 $116.65 $113.55
4 10/02/2020 $113.02 144712000 $112.89 $115.37 $112.22
5 10/01/2020 $116.79 116120400 $117.64 $117.72 $115.83
6 09/30/2020 $115.81 142675200 $113.79 $117.26 $113.62
7 09/29/2020 $114.09 100060500 $114.55 $115.31 $113.57
8 09/28/2020 $114.96 137672400 $115.01 $115.32 $112.78
9 09/25/2020 $112.28 149981400 $108.43 $112.44 $107.67
10 09/24/2020 $108.22 167743300 $105.17 $110.25 $105
11 09/23/2020 $107.12 150718700 $111.62 $112.11 $106.77
12 09/22/2020 $111.81 183055400 $112.68 $112.86 $109.16
13 09/21/2020 $110.08 195713800 $104.54 $110.19 $103.1
14 09/18/2020 $106.84 287104900 $110.4 $110.88 $106.09
15 09/17/2020 $110.34 178011000 $109.72 $112.2 $108.71
16 09/16/2020 $112.13 155026700 $115.23 $116 $112.04
17 09/15/2020 $115.54 184642000 $118.33 $118.829 $113.61
18 09/14/2020 $115.355 140150100 $114.72 $115.93 $112.8
19 09/11/2020 $112 180860300 $114.57 $115.23 $110
20 09/10/2020 $113.49 182274400 $120.36 $120.5 $112.5
21 09/09/2020 $117.32 176940500 $117.26 $119.14 $115.26
If you want to save it afterwards as a csv file, just run write.csv():
write.csv(data, '~/Downloads/data.csv')
Let me know if it worked for you.
In case you are in a windows OS you need to set the mode=wb argument.
download.file(
"https://www.nasdaq.com/api/v1/historical/AAPL/stocks/2020-09-09/2020-10-09",
"file.csv", mode='wb'
)
Related
I am trying to write code to store a matrix to a variable directly from Matrix Market's website. Below is a sample URL that I'd use:
https://math.nist.gov/pub/MatrixMarket2/Harwell-Boeing/bcsstruc1/bcsstk01.mtx.gz
The example URL will download a bcsstk01.mtx.gz file. I need to extract the bcsstk01.mtx file. Then I need to use MatrixMarket.mmread() so I can save to a variable.
I first tried saving the downloaded file (or URL location) to a variable A = HTTP.get(), but lack of online resources and lack of knowledge led to no results. Then I used HTTP.download() and got the .mtx.gz file, but I can't unzip it. And finally, MatrixMarket.mmread() cannot read .gz files. So I'm stuck with a downloaded file I can't do anything with unless I manually unzip it.
Using the info from link in the comments and some fiddling, I managed to get the following:
using TranscodingStreams, CodecZlib
using Downloads
stream = PipeBuffer()
openstream = TranscodingStream(GzipDecompressor(), stream)
Downloads.download("https://math.nist.gov/pub/MatrixMarket2/Harwell-Boeing/bcsstruc1/bcsstk01.mtx.gz", stream)
for line in eachline(openstream)
println(line)
end
This prints:
%%MatrixMarket matrix coordinate real symmetric
48 48 224
1 1 2.8322685185200e+06
5 1 1.0000000000000e+06
6 1 2.0833333333300e+06
7 1 -3.3333333333300e+03
...
which I suppose is the desired data.
Okay, so I have approximately 2 GB worth of files (images and what not) stored on a server (I'm using Cygwin right now since I'm on Windows) and I was wondering if I was able to get all of this data into R and then eventually translate it onto a website where people can view/download those images?
I currently have installed the ssh package and have logged into my server using:
ssh::ssh_connect("name_and_server_ip_here")
I've been able to successfully connect, however, I am not particular sure how to locate the files on the server through R. I assume I would use something like scp_download to download the files from the server, but as mentioned before, I am not particularly sure how to locate the files from the server, so I wouldn't be able to download them anyways (yet)!
Any sort of feedback and help would be appreciated! Thanks :)
You can use ssh::ssh_exec_internal and some shell commands to "find" commands.
sess <- ssh::ssh_connect("r2#myth", passwd="...")
out <- ssh::ssh_exec_internal(sess, command = "find /home/r2/* -maxdepth 3 -type f -iname '*.log'")
str(out)
# List of 3
# $ status: int 0
# $ stdout: raw [1:70] 2f 68 6f 6d ...
# $ stderr: raw(0)
The stdout/stderr are raw (it's feasible that the remote command did not produce ascii data), so we can use rawToChar to convert. (This may not be console-safe if you have non-ascii data, but it is here, so I'll go with it.)
rawToChar(out$stdout)
# [1] "/home/r2/logs/dns.log\n/home/r2/logs/ping.log\n/home/r2/logs/status.log\n"
remote_files <- strsplit(rawToChar(out$stdout), "\n")[[1]]
remote_files
# [1] "/home/r2/logs/dns.log" "/home/r2/logs/ping.log" "/home/r2/logs/status.log"
For downloading, scp_download is not vectorized, so we can only upload one file at a time.
for (rf in remote_files) ssh::scp_download(sess, files = rf, to = ".")
# 4339331 C:\Users\r2\.../dns.log
# 36741490 C:\Users\r2\.../ping.log
# 17619010 C:\Users\r2\.../status.log
For uploading, scp_upload is vectorized, so we can send all in one shot. I'll create a new directory (just for this example, and to not completely clutter my remote server :-), and then upload them.
ssh::ssh_exec_wait(sess, "mkdir '/home/r2/newlogs'")
# [1] 0
ssh::scp_upload(sess, files = basename(remote_files), to = "/home/r2/newlogs/")
# [100%] C:\Users\r2\...\dns.log
# [100%] C:\Users\r2\...\ping.log
# [100%] C:\Users\r2\...\status.log
# [1] "/home/r2/newlogs/"
(I find it odd that scp_upload is vectorized while scp_download is not. If this were on a shell/terminal, then each call to scp would need to connect, authenticate, copy, then disconnect, a bit inefficient; since we're using a saved session, I believe (unverified) that there is little efficiency lost due to not vectorizing the R function ... though it is still really easy to vectorize it.)
I am new to SyntaxNet and I recently tried to install it step by step from https://github.com/tensorflow/models/blob/master/syntaxnet/README.md#instalation.
Although after running bazel test it was said that "Executed 12 out of 12 tests: 12 tests pass"
when I used this code
ubuntu#ubuntu-VirtualBox:~/Downloads/git-2.7.4/models/syntaxnet$
echo 'Bob brought the pizza to Alice.' |syntaxnet/demo.sh
it gives me this error:
syntaxnet/demo.sh: line 31: bazel-bin/syntaxnet/parser_eval:
No such file or directory
syntaxnet/demo.sh: line 43: bazel-bin/syntaxnet/parser_eval:
No such file or directory
syntaxnet/demo.sh: line 55: bazel-bin/syntaxnet/conll2tree:
No such file or directory
I would really appreciate if anyone could help me.
Thank you so much
I had the same issue.
To fix it, modify the demo.sh file, lines 31 and 55.
The locations it points to find parser_eval and conll2tree are wrong, at least they were in my system.
Do a search for "sudo find / -iname 'parser_eval'".
For me the location of this file was "/home/jesus/.cache/bazel/_bazel_jesus/afbbfe6033ddfb6168467a72894e5682/syntaxnet/bazel-out/local-opt/bin/syntaxnet/parser_eval"
I then proceeded to point line 31 to this location instead of "bazel-bin/syntaxnet/parser_eval".
Then did the same for line 55 and conll2tree.
Saved the file, and got it running.
Hope it helps
I had a similar problem, in case it might be useful to anyone. If you rename or move the path where syntaxnet is installed, you'll break half a dozen symbolic links it creates during installation (it uses absolute paths). In that case, you have to recreate the links with the new path.
I have a CSV with Czech characters in it that looks like this:
id,address,city
660999,Vršovická 10,Praha
676838,Valentova 50,Praha 4
676858,Husova 6740,Pardubice
677971,Lipová 10,Třebíč
678304,Jana Ziky 10/1955,Ostrava
...
When I import into RStudio everything looks fine if I view it using the View() function.
But in the terminal when I view the values everything looks crazy.
xl = read.csv("some_csv.csv")
head(xl)
id address city
1 660999 Vršovická 10 Praha
2 676838 Valentova 50 Praha 4
3 676858 Husova 6740 Pardubice
4 677971 Lipová 10 TÅ™ebÃÄ
5 678304 Jana Ziky 10/1955 Ostrava
When I check the encoding with Encoding(xl[1,2]) for example it says "unknown".
I also have Russian data with the same exact problem.
I've tried switching to Sys.setlocale("LC_CTYPE", "czech") and Sys.setlocale("LC_CTYPE", "russian") and importing under those settings and they behave the same.
I'm using Rstudio Version 0.98.501 with R version 3.0.2 on Windows 7. A colleague on a separate computer is having the same problem.
Anything I can do make these characters work correctly the terminal?
I am begining to use Quandl facilities to import datasets to R with Quandl R API. It appears to be the easiest thing. However I have a problem. The below pasted snipet of code does not work (for me). It returns an error.
library(Quandl)
my_quandl_dtst <- Quandl("DOE/RBRTE")
Error in function (type, msg, asError = TRUE) : couldn't connect to host
What could be the cause of the problem?
I searched this site and found some solutions, also the one below, but it does not work for me.
set_config(use_proxy(url='your.proxy.url',port,username,password))
On the other hand, read.csv with url pasted from quandl website export dataset facility works:
my_quandl_dtst <- read.csv('http://www.quandl.com/api/v1/datasets/DOE/RBRTE.csv?', colClasses = c('Date' = 'Date'))
I would realy like to use the Quandl library, since using it would make my code cleaner. Therefore I would appreciate any help. Thanks in advance.
Ok, I found the solution, I had to set RCurlOptions, because the Quandl function uses getURL() to download data from url. But I had to use options() function as well. So:
options(RCurlOptions = list(proxy = "my.proxy", proxyport = my.proxyport.number))
head(quandldata <- Quandl("NSE/OIL"))
Date Open High Low Last Close Total Trade Quantity Turnover (Lacs)
1 2014-03-03 453.5 460.05 450.10 450.30 451.30 90347 410.08
2 2014-02-28 440.0 460.00 440.00 457.60 455.55 565074 2544.66
3 2014-02-26 446.2 450.95 440.00 440.65 440.60 179055 794.24
4 2014-02-25 445.1 451.75 445.10 446.60 447.20 86858 389.38
5 2014-02-24 443.0 449.50 443.00 446.50 446.30 81197 362.33
6 2014-02-21 447.9 448.65 442.95 445.50 446.80 95791 427.32
I guess you need to check if the domain quand1.com accepts remote Connections to the RBRTE.csv file.