I'm trying to get data from an OPeNDAP server using R and the ncdf4 package. However, the nasa eosdis server requires username / password. How can I pass this info using R?
Here is what I'm trying to do:
require(ncdf4)
f1 <- nc_open('https://disc2.gesdisc.eosdis.nasa.gov/opendap/TRMM_L3/TRMM_3B42.7/2018/020/3B42.20180120.15.7.HDF')
And the error message:
Error in Rsx_nc4_get_vara_double: NetCDF: Authorization failure syntax
error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or
SCAN_ERROR context: HTTP^ Basic: Access denied. Var: nlat Ndims: 1
Start: 0 Count: 400 Error in ncvar_get_inner(d$dimvarid$group_id,
d$dimvarid$id, default_missval_ncdf4(), : C function
R_nc4_get_vara_double returned error
I tried the url https://username:password#disc2.... but that did not work also.
Daniel,
The service you are accessing is using third-party redirection to authenticate users. Therefore the simple way of providing credentials in the URL doesn't work.
You need to create 2 files.
A .dodsrc file (a RC file for the netcdf-c library) with the following content
HTTP.COOKIEFILE=.cookies
HTTP.NETRC=.netrc
A .netrc file, in the location referenced in the .dodsrc, with your credentials:
machine urs.earthdata.nasa.gov
login YOURUSERNAMEHERE
password YOURPASWORDHERE
You can find more details at
https://www.unidata.ucar.edu/software/netcdf/docs/md__Users_wfisher_Desktop_v4_86_81-prep_netcdf-c_docs_auth.html
Regards
Antonio
unfortunately, even after defining the credentials and their location
ncdf4::nc_open("https://gpm1.gesdisc.eosdis.nasa.gov/opendap/GPM_L3/GPM_3IMERGDE.06/2020/08/3B-DAY-E.MS.MRG.3IMERG.20200814-S000000-E235959.V06.nc4")
still returns
Error in Rsx_nc4_get_vara_double: NetCDF: Authorization failure
The same happens when using ncdump from a terminal:
$ ncdump https://gpm1.gesdisc.eosdis.nasa.gov/opendap/GPM_L3/GPM_3IMERGDE.06/2020/08/3B-DAY-E.MS.MRG.3IMERG.20200814-S000000-E235959.V06.nc4
returns
syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or
SCAN_ERROR context: HTTP^ Basic: Access denied. NetCDF: Authorization
failure Location: file
/build/netcdf-KQb2aQ/netcdf-4.6.0/ncdump/vardata.c; line 473
Related
We want to write nginx logs to the TDengine, we tried nginx -> vector -> kafka -> tdengine without success, we can only do the basic database creation completed, but can not write data to the data table, the prompt is as follows.
[2022-05-28 19:25:09,454] ERROR [nginx-tdengine|task-0] WorkerSinkTask{id=nginx-tdengine-0} RetriableException from SinkTask: (org.apache. kafka.connect.runtime.WorkerSinkTask:600)
RetriableException: java.sql.SQLException: Exception chain:
SQLException: TDengine ERROR (80000221): Invalid JSON format
The TDengine version is 2.4.0.14
We refer to the documentation at https://docs.taosdata.com/third-party/kafka.
I wonder what the problem is and if there is a more reasonable solution?
I am trying to fetch the list of folders/files from a specific location in the drive from the server. I authenticated the drive using a service token. (Downloaded the JSON file and passed the location of the file as the parameter).
drive_auth(path = 'servicetoken.json')
I am trying to get the list of files from a specific location
folders_list <- drive_ls(path = "0EIgLgNPOMnJaWVJsdlkey") %>%
as.data.frame()
I am getting an error message
Warning: Error in : Client error: (404) Not Found
* domain: global
* reason: notFound
* message: File not found: 0EIgLgNPOMnJaWVJsdlkey.
* locationType: parameter
* location: fileId
Do we need to generate a JSON file every time we authenticate the drive?
What wrong I am doing here to get the error message in the server?
Not able to test right now, bit you might have to wrap the id in googledrive::as_id('0EIgLgNPOMnJaWVJsdlkey') to declare that you're passing an id and not a path
Probably a usage or settings issue:
I'm trying to use R's googleCloudStorageR package to upload files to my google storage bucket.
Running:
googleCloudStorageR::gcs_upload("test/my_test.csv")
prints these messages:
2020-05-11 18:57:19 -- File size detected as 368 bytes
2020-05-11 18:57:20> Request Status Code: 400
And then this error:
Error: API returned: Cannot insert legacy ACL for an object when uniform bucket-level access is enabled. Read more at https://cloud.google.com/storage/docs/uniform-bucket-level-access.
Is there a different usage to googleCloudStorageR::gcs_upload that will succeed? (not clear from its documentation
If I set predefinedAcl to "default" I get a JSON related error:
Error : lexical error: invalid char in json text
This error message is followed by some html code and following that this message:
> xdg-open: unexpected argument 'minimum-scale=1,'
Try 'xdg-open --help' for more information.`.
I'm not sure what JSON it's referring to but if it's the JSON I set googleCloudStorageR to authenticate access to my bucket than I'm surprised it's complaining at this stage
It looks like in https://github.com/cloudyr/googleCloudStorageR/pull/84 it got support to inherit the bucket level ACL if you set predefinedAcl to default. In your example this would be:
googleCloudStorageR::gcs_upload("test/my_test.csv", predefinedAcl = "default")
The issue has been resolved by the googleCloudStorageR developers. It is not yet on the CRAN distribution but it installing it from github (devtools::install_github("cloudyr/googleCloudStorageR")) should do.
And the usage is:
googleCloudStorageR::gcs_upload("test/my_test.csv", predefinedAcl = "bucketLevel")
here's my problem.
I access Google Analytics API via R with googleAnalyticsR. The whole thing happens in a Markdown script. Local everything runs without problems. But when I load the script on a Shiny Sever the GA token seems to expire after a while (a few hours).
The server log gives the following error message:
Warning in gzfile(file, mode) :
cannot open compressed file '.httr-oauth', probable reason 'Permission denied'
The script to create the token looks like this:
library(googleAuthR)
options(googleAuthR.client_id = "client-id",
googleAuthR.client_secret = "client-secret")
gar_auth()
I read that I should add
googleAuthR.httr_oauth_cache = FALSE
to the options. But when I do this and try to create a new Token I get follwing error:
Error: option('googleAuthR.httr_oauth_cache') must be set to
valid cache file location,
not TRUE or FALSE - (example: '.httr-oauth')
Anybody Ideas?
Thanks a lot
The issue is not that the token expires, the package googleAnalyticsR takes care of refreshing it if needed.
The problem is that the token was not created on the server (I assume you created locally then pushed to server), hence the error:
Warning in gzfile(file, mode) : cannot open compressed file '.httr-oauth', probable reason 'Permission denied'
The shinyuser which runs your app does not have the permission to open that file. I'm not a security expert so take this lightly. You run the following from the root of your application (on your server) there error will go away.
sudo chmod -R 777 .httr-oauth
The above will let shiny access the file.
I am trying to use the linkedin API to access some data for a project. I am following the instructions provided by https://github.com/mpiccirilli/Rlinkedin . But no matter what I do I run into a time out error:
library(pacman)
p_load(rvest,ggplot2,Rlinkedin,httr,curl,devtools,dplyr,devtools)
in.auth <- inOAuth()
The console returns this message:
If you've created you're own application, be sure to copy and paste
the following into 'OAuth 2.0 Redirect URLs' in the LinkedIn
Application Details: http://localhost:1410/ When done, press any key
to continue... Use a local file ('.httr-oauth'), to cache OAuth access
credentials between R sessions?
1: Yes 2: No
I click 1 or 2 and I get the same error:
Adding .httr-oauth to .gitignore Error in curl::curl_fetch_memory(url,
handle = handle) : Timeout was reached: Connection timed out after
10000 milliseconds
No matter what I try I get the same error, any help would be very very very appreciated.