When I use googlesheets4 in R, I use sheets_auth() in the console and it works fine. But when I try to run it in R markdown, and when I try to knit, I cannot seem to get the credentials. Can someone walk me through the process? I've gone to the vignettes for googlesheets4 but cannot seem to understand it.
This is working for me
gs4_auth(path = "xxxxxxxxxxxxxxxx.json")
It doesn't return anything, but after that I'm able to write data in my sheet with sheet_write()
To get the credentials in a json file you have to follow these steps:
From the Developers Console, in the target GCP Project, go to IAM & Admin > Service accounts.
Give it a decent name and description.
For example, the service account used to create the googledrive docs has name “googledrive-docs” and description “Used when generating
googledrive documentation”.
Service account permissions. Whether you need to do anything here depends on the API(s) you are targetting. You can also modify roles
later and iteratively sort this out.
For example, the service account used to create the googledrive docs does not have any explicit roles.
The service account used to test bigrquery has roles BigQuery Admin and Storage Admin.
Grant users access to this service account? So far, I have not done this, so feel free to do nothing here. Or if you know this is useful
to you, then by all means do so.
Do Create key and download as JSON. This file is what we mean when we talk about a “service account token” in the documentation of gargle
and packages that use gargle. gargle::credentials_service_account()
expects the path to this file.
Appreciate that this JSON file holds sensitive information. Treat it like a username & password combo! This file holds credentials that
potentially have a lot of power and that don’t expire.
Consider storing this file in such a way that it will be automatically discovered by the Application Default Credentials search
strategy. See credentials_app_default() for details.
You will notice the downloaded JSON file has an awful name, so sometimes I create a symlink that uses the service account’s name, to
make it easier to tell what this file is.
Remember to grant this service account the necessary permissions on any resources you plan to access, e.g., read or write permission on a
specific Google Sheet. The service account has no formal relationship
to you as a Google user and won’t automatically inherit permissions.
(copied from here https://gargle.r-lib.org/articles/get-api-credentials.html#service-account-token)
Related
We are currently using google sheets for a research project on crowd forecasts for Covid-19 case and death numbers.
Google Sheets is used for convenience, but we are often running into quota limit issues - even though the number of users we have should be well below what Google allows.
I attempted to create a somewhat reproducible example by setting up a new google account and creating a sheet from which to read.
The first thing I tried (without making any changes to the google account) is this:
library(googledrive)
library(googlesheets4)
# Google sheets authentification -----------------------------------------------
options(gargle_oauth_cache = ".secrets")
drive_auth(cache = ".secrets", email = "iamatestotest#gmail.com")
gs4_auth(token = drive_token())
sheet_id <- "1Z2O5Mce_haceWfduLenJQP-hddXF9biY_4Ydob_psyQ"
n_tries <- 50
for (i in 1:n_tries) {
data <- read_sheet(ss = sheet_id)
Sys.sleep(0.5)
print(i)
}
From what I understand I should be able to make around 300 read requests per minute, but I'm usually not be able to get the loop to run beyond 30-34.
As I wasn't sure the 300 requests are readily available I went to https://console.cloud.google.com, created a new test project (not sure why that is needed) and explicitly activated the googlesheets API and created some credentials. I created an API key as well as an OAuth 2.0 Client ID (although I am admittedly somewhat lost what this does and how to use it).
I next tried to login with my api key by running
drive_deauth()
drive_auth_configure(api_key = "thisismyapikey")
gs4_auth(token = drive_api_key())
but that also didn't get me beyond 33ish. I also had a look into the google console, but also couldn't see any traffic - so not sure my API key got actually used?
I assume this is due to my inability to actually use the API in the intended way. Any help in setting this up / increasing the quota would be much appreciated. If that helps I'm happy to give access to the test account - simply write me a message.
With some kind help from very friendly people I think I mostly figured this out and it was indeed my failure to use the API correctly.
Why my approach failed
when you use googlesheets4 and any of its function out of the box, you get asked to authorize the tidyverse API OAuth app (you login with your Google credentials and give the OAuth app access rights). This means that you make all requests through the tidyverse OAuth app, as are all other users in the world who use this functionality. This is very nice as it works out of the box, but runs into limitations if other people are using the package at the same time. Sharing this quota with other people meant that I ran into limitations quite unpredictably.
How to change the setup to make it work
There are a couple of things that help to alleviate / solve the problem.
use the devtools version of googlesheets4 (devtools::install_github("tidyverse/googlesheets4"). This dev version of googlesheets4 in turn relies on the dev version of gargle, the package that manages the google authentification. The dev version of gargle has a retry function, that automatically retries your requests if they fail. This should solve the majority of issues.
Get your own OAuth app / google service account.
this allows you to manage the authentification process all on your own. You therefore don't have to share your quota with other users around the world.
To set up your own OAuth app / google service account, you can do the following (I'm focusing on the google service account here, as that is much easier in practice).
Log into https://console.cloud.google.com/. You will be asked to create a project. You can see your projects on the left next to "Google Cloud Platform".
Type "APIs and Services" into the search bar, press "enable APIs and services" and search for sheets. Enable this API.
Go back to the search bar and type in "Credentials"
Press "Create credentials" and select service account. A service account gives you programmatic access to the APIs. Give it a name and a description. You should be able to skip the optional parts. Create the service account and go back to the credentials overview. You may have to refresh the page or wait a minute.
Click on your service account (it looks like a very cryptic email address) and go to the "KEYS" tab.
Click "ADD KEY" and create a new key. As key type, select JSON.
Download that key and store it somewhere secure. This should be treated as a combination of password and username!
Now to actually use your key with googlesheets4, you can run `gs4_auth(path = "path-to-your-service-account.JSON")
In order to be able to access your google sheets, you need to grant your service account permissions. Go to your google sheet, press share (as you would do to share it with any other user) and type in this cryptic service account email (it should look something like "1234#something.iam.gserviceaccount.com". Everything should work now without you having to log in anywhere. If you have previously tried other things, I would suggest to restart your R session.
profit.
You should now also be able to track the API requests in the google console dashboard.
Note that there is still a limit of 60 requests per user per minute, so you're not getting your full 300 requests, but maybe it is possible to create several service accounts and balance the load between these. But not having other people's request interfere with yours is a big improvement!
Google says that it is a security measure. Try to share through adding their emails
I've been using my Google API credentials to access Googlesheets through R for a while now. Today, I made the mistake of running a script that calls from a different google account (my work account) and now I have this error:
When I clicked through, it says that I only need verification if I'm creating a user-facing app (I'm not). What I'm doing is reading and writing Google Sheet data. I've tried recreating my OAuth key, I've tried changing the project scopes (it's the //auth/drive scope that's throwing the wrench in things).
I'm not actually a developer, I'm a data analyst and use R code for a fairly small scope (Google Sheets, a few data resources like the NOAA, Google Analytics, and social platforms). I use this for my side business and need to get it going again before I get any orders. Since I'm not a developer, I'm really at a bit of a loss here. Help?
I'm not sure if this would help but I had the same problem to share a shiny app to draw numbers and store the value of the pixels in a google sheet. What worked for me was to create a token.rds file which stored the information to authenticate each rstudio session.
How to create the token.rds file
I recommend you to read this tutorial if you have not created the OAuth 2.0 Client ID. If you have created an OAuth 2.0 Client ID for your google account and you have the public and secret key, run the following code:
library(googlesheets)
your_gs_app <- gs_auth(key = "782348718282-bgaocvueexiq9qbackboidne19aaa5v9dg.apps.googleusercontent",
secret = "gFMmSoWPVPLu2EmdBLOBuSZs")
This would require you to verify the app in the same webpage that you have posted in the image. Click on Advanced > Go to application-name (unsafe) and grant the permissions in the next few windows. Once you have granted all the permissions, close the navegator and go back to RStudio to create the token.rds file by running:
saveRDS(your_gs_app, file = "token.rds")
Now that you have created the file to authenticate the R session, you can authenticate any R session in other computers by running:
googlesheets::gs_auth(token = "token.rds")
with the token.rds file in the working directory (obviously)
Hope this helps!
I'm writing some Firebase Cloud Functions but I have need to hide a private key, including from Firebase project admins.
If I embedded this key into my source code and uploaded the code myself, would it be possible for anyone to retrieve the source code and thus the key? Either via Firebase or Google?
Many thanks
The code for your Cloud Functions is never accessible to users of your app.
It is however accessible for the collaborators on your Firebase project. See Get code from firebase console which I deployed earlier
I don't think there's any way to hide such configuration values from collaborators. Since they can see/deploy code, and the code needs access to this private key, they by definition have access to the key too.
Answering precisely to your question: Yes, they can.
The step by step to achieve that is relatively simple
Go into the GCP Functions page
Select the function you want to inspect
Click on source (From there you should be able to see all the files and the code used by that function), or;
Click on variables (From there you should see all environment variables used by your function)
If people being able to see env variables is problematic to you, here's a way to make things more secure:
You can build on what you already and start encrypting those keys before adding them to the codebase or the environment variables. After that, you can use an encryption service such as KMS to decrypt those keys at runtime. In KMS itself you can have a stricter policy in there, only allowing yourself and the function to access that service.
Another great service from GCP is Google's Secret Manager
Maybe setting an environmental variable:
Oficial Doc
I've installed "bigrquery" like this:
devtools::install_github("hadley/bigrquery")
library(bigrquery)
And i get this error, when trying to extract data:
Error: Access Denied: Job triple-xxx-xxx:job_zu6P-qSxxx7DBVICij6_QyDv0: RUN_QUERY_JOB
I've looked here and on the web and everyone says that you just need 2 things to extrac data from Google BigQuery:
1.-Have a Project for it (BigQuery Enabled):
2.-Put a billing address for BigQuery.
I've done that, but still got the problem.
IMPORTAT:
For other packages that interact with Google products (Google Analytics), e.g RGA; you need to create a Client ID (OAUTH), do i need to to this with "bigrquery"???
Someone can update the method to get the data?
Ps. I can get the data in the broswer (with the Web Interface provided by Google). But not in R from "bigrquery" - I'm using the version hosted on CRAN.
Ps2. I don't want that the "authentications" to be stored in the cache, is there a way to make "bigrquery" to ask for authentication everytime it tries to connect to BigQuery?
I found this issue on this post, but with the solution out-of-date:
Google App Engine authorization for Google BigQuery
This error means that the user that was running the query was not authorized to run jobs in the project (triple-xxx-xxx). You'd need to add the user that is running the query to the project via the developers console (https://console.developers.google.com/project).
To answer some of your other questions:
You don't need to create a clientid to use bigquery.
I'm not sure if there is a way to force bigrquery to re-authorize every time. That said, looking at the source code (https://github.com/hadley/bigrquery/blob/master/R/auth.r) you may be able to call set_access_cred with null to clear the authentication.
I'd like to show some map layer on my webpage, so I decided to give a try to this Google service. As the data is collected in a database in my server, I chose to use a service account as explained here and then use the private key generated in my php script.
Everything works fine when creating a table and inserting some test values. I get the table Id and I'm able to play with it from my script. The problem is that I don't know how to access these table from the web browser. In my API console usage stats are shown fine, but when logging with my account to Google Drive I don't see any table in there.
Where am I supposed to access them if at all possible? Do either the apps.googleusercontent.com or developer.gserviceaccount.com accounts play any role to log into some other service to get access through web?
I also got an api key associated, but when trying to query a table I get a 401 error.
Any hint? I'm feeling a bit lost now. Thanks.
You are using a Service Account right?
So when you create a table with this account, this account will be the table owner. No one else has permission to see this table.
When you access the Fusion Tables web interface with Your Personal Account, you will only see tables that you createdwith your Personal Account.
If you wish to inspect the tables created with your Service Account, you have to use the Google Drive API with your Service Account credentials to give access permission to your Personal Account.
Also if you wish to make your table (or any other type of document) public, you need to use this Google Drive API again.
See more about the topic here:
https://developers.google.com/drive/v2/reference/permissions/insert
Tip: if you want to achieve something on behalf of your Service Account that you only need once (so no need to implement a logic for it in your webapp) I'd seriously advise you to consider using the OAuth2 Playground. You can set your Service Account credentials in the "Settings" and issue authorized requests on behalf of your Service Account. Very usefull tool, no coding needed.