We are currently using google sheets for a research project on crowd forecasts for Covid-19 case and death numbers.
Google Sheets is used for convenience, but we are often running into quota limit issues - even though the number of users we have should be well below what Google allows.
I attempted to create a somewhat reproducible example by setting up a new google account and creating a sheet from which to read.
The first thing I tried (without making any changes to the google account) is this:
library(googledrive)
library(googlesheets4)
# Google sheets authentification -----------------------------------------------
options(gargle_oauth_cache = ".secrets")
drive_auth(cache = ".secrets", email = "iamatestotest#gmail.com")
gs4_auth(token = drive_token())
sheet_id <- "1Z2O5Mce_haceWfduLenJQP-hddXF9biY_4Ydob_psyQ"
n_tries <- 50
for (i in 1:n_tries) {
data <- read_sheet(ss = sheet_id)
Sys.sleep(0.5)
print(i)
}
From what I understand I should be able to make around 300 read requests per minute, but I'm usually not be able to get the loop to run beyond 30-34.
As I wasn't sure the 300 requests are readily available I went to https://console.cloud.google.com, created a new test project (not sure why that is needed) and explicitly activated the googlesheets API and created some credentials. I created an API key as well as an OAuth 2.0 Client ID (although I am admittedly somewhat lost what this does and how to use it).
I next tried to login with my api key by running
drive_deauth()
drive_auth_configure(api_key = "thisismyapikey")
gs4_auth(token = drive_api_key())
but that also didn't get me beyond 33ish. I also had a look into the google console, but also couldn't see any traffic - so not sure my API key got actually used?
I assume this is due to my inability to actually use the API in the intended way. Any help in setting this up / increasing the quota would be much appreciated. If that helps I'm happy to give access to the test account - simply write me a message.
With some kind help from very friendly people I think I mostly figured this out and it was indeed my failure to use the API correctly.
Why my approach failed
when you use googlesheets4 and any of its function out of the box, you get asked to authorize the tidyverse API OAuth app (you login with your Google credentials and give the OAuth app access rights). This means that you make all requests through the tidyverse OAuth app, as are all other users in the world who use this functionality. This is very nice as it works out of the box, but runs into limitations if other people are using the package at the same time. Sharing this quota with other people meant that I ran into limitations quite unpredictably.
How to change the setup to make it work
There are a couple of things that help to alleviate / solve the problem.
use the devtools version of googlesheets4 (devtools::install_github("tidyverse/googlesheets4"). This dev version of googlesheets4 in turn relies on the dev version of gargle, the package that manages the google authentification. The dev version of gargle has a retry function, that automatically retries your requests if they fail. This should solve the majority of issues.
Get your own OAuth app / google service account.
this allows you to manage the authentification process all on your own. You therefore don't have to share your quota with other users around the world.
To set up your own OAuth app / google service account, you can do the following (I'm focusing on the google service account here, as that is much easier in practice).
Log into https://console.cloud.google.com/. You will be asked to create a project. You can see your projects on the left next to "Google Cloud Platform".
Type "APIs and Services" into the search bar, press "enable APIs and services" and search for sheets. Enable this API.
Go back to the search bar and type in "Credentials"
Press "Create credentials" and select service account. A service account gives you programmatic access to the APIs. Give it a name and a description. You should be able to skip the optional parts. Create the service account and go back to the credentials overview. You may have to refresh the page or wait a minute.
Click on your service account (it looks like a very cryptic email address) and go to the "KEYS" tab.
Click "ADD KEY" and create a new key. As key type, select JSON.
Download that key and store it somewhere secure. This should be treated as a combination of password and username!
Now to actually use your key with googlesheets4, you can run `gs4_auth(path = "path-to-your-service-account.JSON")
In order to be able to access your google sheets, you need to grant your service account permissions. Go to your google sheet, press share (as you would do to share it with any other user) and type in this cryptic service account email (it should look something like "1234#something.iam.gserviceaccount.com". Everything should work now without you having to log in anywhere. If you have previously tried other things, I would suggest to restart your R session.
profit.
You should now also be able to track the API requests in the google console dashboard.
Note that there is still a limit of 60 requests per user per minute, so you're not getting your full 300 requests, but maybe it is possible to create several service accounts and balance the load between these. But not having other people's request interfere with yours is a big improvement!
Google says that it is a security measure. Try to share through adding their emails
Related
I've been using my Google API credentials to access Googlesheets through R for a while now. Today, I made the mistake of running a script that calls from a different google account (my work account) and now I have this error:
When I clicked through, it says that I only need verification if I'm creating a user-facing app (I'm not). What I'm doing is reading and writing Google Sheet data. I've tried recreating my OAuth key, I've tried changing the project scopes (it's the //auth/drive scope that's throwing the wrench in things).
I'm not actually a developer, I'm a data analyst and use R code for a fairly small scope (Google Sheets, a few data resources like the NOAA, Google Analytics, and social platforms). I use this for my side business and need to get it going again before I get any orders. Since I'm not a developer, I'm really at a bit of a loss here. Help?
I'm not sure if this would help but I had the same problem to share a shiny app to draw numbers and store the value of the pixels in a google sheet. What worked for me was to create a token.rds file which stored the information to authenticate each rstudio session.
How to create the token.rds file
I recommend you to read this tutorial if you have not created the OAuth 2.0 Client ID. If you have created an OAuth 2.0 Client ID for your google account and you have the public and secret key, run the following code:
library(googlesheets)
your_gs_app <- gs_auth(key = "782348718282-bgaocvueexiq9qbackboidne19aaa5v9dg.apps.googleusercontent",
secret = "gFMmSoWPVPLu2EmdBLOBuSZs")
This would require you to verify the app in the same webpage that you have posted in the image. Click on Advanced > Go to application-name (unsafe) and grant the permissions in the next few windows. Once you have granted all the permissions, close the navegator and go back to RStudio to create the token.rds file by running:
saveRDS(your_gs_app, file = "token.rds")
Now that you have created the file to authenticate the R session, you can authenticate any R session in other computers by running:
googlesheets::gs_auth(token = "token.rds")
with the token.rds file in the working directory (obviously)
Hope this helps!
My organization set up Cloud SQL as the default for Google App Maker about one month ago. In the last week, we have been unable to preview or publish apps that use Cloud SQL data sources, including the sample applications which worked perfectly before. The failure occurs during the authorization process. When previewing or publishing an app, Google App Maker displays a dialog stating "Deploying this app requires authorization". Next it prompts the user for their Google account and then requests approval for the necessary authorizations (e.g., "Manage the data in your Google SQL Service instances"). After approving the authorization, the prompts to authorize begin over with the dialog stating "Deploying this app requires authorization".
Observations:
We have repeated this problem on multiple different computers, networks, and four different user accounts.
In the SQL cloud console, our Cloud SQL instance shows new databases being created for each app along with new database-specific user accounts
All of the databases appear as expected when I log directly into the Cloud SQL database using phpMyAdmin
Other apps which don't use a Cloud SQL datasource work fine, including an app that uses a calculated data source which is hosted in the same Cloud SQL instance
The only errors in the Stack driver logs for the Cloud SQL database showed "INFO" level communication errors with the database (aborted connection...Got an error reading communication packets)
I'm unable to find Stack driver logs for the apps because I cannot preview or publish them (either option would provide a link to the Stack driver logs)
There are now approximately 20 databases in our SQL instance (mostly associated with simple app tests) and we have only used 1 GB of 10 GB of space in our SQL instance
I haven't seen any related problems on the Google Issue Tracker for Google App Maker
I'd appreciate any help or suggestions on what to check in order to resolve this issue.
I posted an issue to Google Issue Tracker and Google corrected the problem. They also provided a workaround if this problem happens again.
Here is the response from the Google development team posted on Google Issue Tracker: https://issuetracker.google.com/issues/145345198
It's great to hear your up and working again! We are aware of this issue and are working through a longer term fix. The specific bug appears to be related to some changes made in the Google Cloud session policy control that may have rolled out to your domain recently interacting with AppMaker in a way that was not expected. We've spent time diagnosing the underlying issue and we beleive we know the root cause. I suspect your domain admin did a version of the workaround below.
Without getting too far into the details, the specific bug is that for a Deployer of an AppMaker application, if the Google Cloud Session policy is set with any expiration time, the returned token AppMaker sees is invalid, triggering a loop in AppMaker trying to generate a valid security token. Historically, these session tokens never expired but recently there was beta feature launch that allowed domain admins to set them to expire. We strongly suspect your domain recently set this expiration policy explicitly and that's what is causing the bug.
The good news is that these policies are overridable per Organizational Unit and we have tested that OUs which have the original classic Never Expire setting do, in fact, allow AppMaker to work.
My suspicion is that your domain admin has reverted recent, local changes to your organizational policy under the admin.google.com console, specifically under Security > Google Cloud session control (Beta).
If this happens again, here the workaround we would recommend. Note you don't need to do this if you're currently up and working. You will need the help of someone with admin.gogole.com powers, specifically User and Organizational Unit powers at your organization. It is a slight increase in security risk but it restores some classic behavior that was standard until recently.
The summary of the workaround is to override the Google Cloud session control expiration setting such that individuals who need access to AppMaker deployments can have it. To mitigate systemic security risk, this is best done by creating a limited purpose Organizational Unit with just that setting different than the parent OU settings.
The workaround is to:
Contact someone in your domain with Admin powers for your Google for Business license.
Have your admin proceed to https://admin.google.com. The actions below need to be performed by a domain admin.
Under the Users section, identify the specific user account that needs the ability to deploy AppMaker Apps.
Identify the Organizational Unit of that Appmaker dev user and make a note of it.
Under the Organization Units settings, locate the Organization Unit you identified above.
Create a new Organization Unit underneath that user's current Organizational Unit with some descriptive identifying it as special w.r.t AppMaker. So for Developers, make something like DevelopersWhoAreAlsoAppMakerDevs.
Back under the Users tab, locate the user from step 3. Move this user into the new Organizational Unit you've just created. This change can take a while to propagate.
-Interlude- At this point, you've made a new Organizational Unit for just that individual and added them to it. You can certainly add multiple people to that OU, especially if they're already in the same parent OU. Use your discretion as to what amount of Organizational rework you wish to pursue. You may not be using OUs at all or you may decide to just turn off this control for the whole domain. It's up to you.
Under admin.google.com's Security settings, locate the Google Cloud session control (beta) settings.
Under this panel, from the dropdown menu on the left, locate the Organization Unit you just created.
Be sure to select ONLY the OU you intend to change.
Change the "Google Cloud Console and Google Cloud SDK session control" from expiring to "Session Never Expires".
Save your changes.
The account you selected in step 3 should now be able to deploy AppMaker apps.
It appears this OU change is only necessary for the deployer of an AppMaker app, not an individual user. Note also that if you have multiple AppMaker developers who all have different current OU settings, you may need to create multiple daughter OUs to avoid a sudden radical shift in OU settings for an individual account.
I created a Actions on Google app with the Actions SDK. For this i used as said before the Actions SDK, firebase function for the fulfillment and firestore for storing data. All works fine.
Now i want to implement account linking to provide user specific information. I start to read the full documentation for account linking with the refers to integrate a Oauth 2.0 Server and soon. That is my first time i working with account linking and Oauth servers and now i'm totally confused. I don't understand where my auth server have to sit, how to setup it and what parameters it have to process. After reading more and searching for results i found that firebase provide Account authentication. Is it right that this firebase product is similar a Oauth server?
My next big problem is how to enabling account linking in my Actions app. In the Actions on Google documentation i found a topic how to expand the Action Package for account linking. My problem is to unterstand which information the probiertes need.
So summary, if the firebase authentication is really a Oauth server what i need to do that my app and firebase authentication works together.
Maybe everyone knows a good website for understanding the process of account linking and how it can be implemented.
UPDATE 1:
After getting the first answer for my question i started studying more about account linking and the authentication process.
After this i created following roadmap:
Create an website with a google account sing-in form and host it with firebase hosting
Set up the Oauth2 server
Interact with the linked account. Save account informations in my firestore database
So i started with step one. In the firebase authentication documentation i find a example for a google login form. After modifying and hosting the example i try it. It works fine. After sing in by using the hosted website, my google account linked with my project. I checked this in my google account settings. Also the example response with a lots of data like the profile name, email address and so on. So my question at this point is. Why do i have to set up a OAuth server now? After sign in with the example form i linked my account to my project successful. And so i can start saving the received data in my firebase database and act with them in my Actions app.
UPDATE 2:
Okay . Maybey i have a general problem of understanding the right use of account linking. I try to identify the user who use my action to offer special content when he comes back next time. Or maybe create a question with his name from his google account inside the question. So in my understanding i have to link the users google account with my action and save the account information in a database to identify the use next time. So is account linking for this task the right way?
No, Firebase Authentication is not an OAuth2 server.
Firebase Authentication provides a way for you to manage user accounts for your Firebase-based web or mobile app. With the Auth UI it gives a way for users to log into that account using a variety of means (including their Google account, Facebook account, or phone number). It does not, however, provide components that an OAuth2 server provides.
Most notably, it does not provide any way for a user to log in through another client (like the Google Assistant) to gain authorization for that client. You cannot, with Firebase Authentication, issue a token to the Assistant, nor accept a token from the Assistant and verify if this is a user inside Firebase Authentication.
You need to build these components yourself. Google describes the minimum tasks that it needs to do as part of this authentication. You can use Firebase Authentication as part of this as you build such a server (for example, it is a great way to have people log in to their account and for you to verify that account), and it is reasonable to use a Firebase Database to store user tokens if you go that route, Firebase Functions might be a useful place to implement the token exchange point, and Firebase Hosting would be good to host the login page itself - but you'd need to write code that "puts it all together".
Your auth server can sit anywhere. As I said - you can do it through Firebase Functions, but you don't have it. It just needs to be able to provide some responses through web URLs at HTTPS endpoints.
Once you have done this, you need to configure the endpoints on the actions console and implement a request for account linking in your code or in the action package.
Response to Update 1
After sing in by using the hosted website, my google account linked with my project. I checked this in my google account settings.
From an OAuth perspective - no, the Google Account is not "linked" to your project.
Google has issued a token to you (that is to say, the service that you've written) that gives your service access to certain resources. Those resources include information about a particular user.
This may sound like I'm splitting semantics, but it isn't. It is fundamental to what OAuth is offering and what it means when you get an issue a token. You currently have authorization to do certain things.
Why do i have to set up a OAuth server now? After sign in with the example form i linked my account to my project successful. And so i can start saving the received data in my firebase database and act with them in my Actions app.
You haven't linked your account. You have permission to do certain things.
Furthermore, aside from "that's how they do it", you need to setup an OAuth server because you now need to do the same thing for Google - give them permission to do specific things on your server (like use it). Normally this would be involved with "logging in".
Account Linking is really a fancy term for "logging in". You need a way for users to be able to log into your server. You have an access token, but that is roughly the equivalent of having logged into Google's server.
So why do so many websites, for example, have things like "Log In using Google" or "Log In using Facebook"? Because those sites are willing to trust that if their servers can get permitted to certain information at Google or Facebook, then they can trust you. And you might be willing to accept that when they login to your site (either through the web or through Actions), but the Assistant can't assume that. They need to make sure users actually log into your site - that user's deliberately want to do so and that you deliberately want to let them in.
I've installed "bigrquery" like this:
devtools::install_github("hadley/bigrquery")
library(bigrquery)
And i get this error, when trying to extract data:
Error: Access Denied: Job triple-xxx-xxx:job_zu6P-qSxxx7DBVICij6_QyDv0: RUN_QUERY_JOB
I've looked here and on the web and everyone says that you just need 2 things to extrac data from Google BigQuery:
1.-Have a Project for it (BigQuery Enabled):
2.-Put a billing address for BigQuery.
I've done that, but still got the problem.
IMPORTAT:
For other packages that interact with Google products (Google Analytics), e.g RGA; you need to create a Client ID (OAUTH), do i need to to this with "bigrquery"???
Someone can update the method to get the data?
Ps. I can get the data in the broswer (with the Web Interface provided by Google). But not in R from "bigrquery" - I'm using the version hosted on CRAN.
Ps2. I don't want that the "authentications" to be stored in the cache, is there a way to make "bigrquery" to ask for authentication everytime it tries to connect to BigQuery?
I found this issue on this post, but with the solution out-of-date:
Google App Engine authorization for Google BigQuery
This error means that the user that was running the query was not authorized to run jobs in the project (triple-xxx-xxx). You'd need to add the user that is running the query to the project via the developers console (https://console.developers.google.com/project).
To answer some of your other questions:
You don't need to create a clientid to use bigquery.
I'm not sure if there is a way to force bigrquery to re-authorize every time. That said, looking at the source code (https://github.com/hadley/bigrquery/blob/master/R/auth.r) you may be able to call set_access_cred with null to clear the authentication.
Background: This is my first standalone web development project, and my only experience in Meteor is building the Discover Meteor app over the last summer. I come from about a year of CS experience as a side interest in school, and I am most comfortable with C and C++. I have experience in python and java.
Project so far: I'm creating a calendar management system (for fun). Using accounts-google, I have created user accounts that are authenticated through google. I have requested the necessary permissions that I need for my app, including 'identity' and 'calendar read/write access'. I've spent the last week or so trying to get over this next hurdle, which is actually getting data from google.
Goal: I'd like to be able to make an API call to Calendar.list using a GET request. I've already called meteor add http to add the GET request functionality, my issue comes with the actual implementation.
Problem: I have registered my app on the developer console and set up Accounts using the client ID and secret, but I have not been able to find/generate my 'API key' for use in the request. Here is the google guide for creating the access token by using my (already) downloaded private key. I'm having a hard time wrapping my head around an implementation on the server side using JS because I don't have a lot of experience with what is mentioned in the HTTP/REST portion of the implementation examples. I would appreciate some help on how to implement a handshake and receive an access token for use in my app. If there is a call I can make or some package that will handle the token generation for me, that would be even better than implementation help. I believe an answer to this would also benefit this other question
The SO answer that I've been referring to so far: https://stackoverflow.com/a/14543159/4259653 Some of it is in spanish but it's pretty understandable code. He has an API key for his request, which I asked this question to help me with. The accounts-google documentation isn't really enough to explain this all to me.
Also an unrelated small question: What is the easiest way to deal with 'time' parameters in requests. I'm assuming JS has some sort of built-in functionality that I'm just not aware of yet.
Thanks for your research. I have also asked a very similar question, and right now I am looking into the package you recommend. I have considered this meteor-google-api package, but it looks abandoned.
Regarding your question about time manipulation, I recommend MomentJS. There are many packages out there; I am using meteor add mrt:moment
EDIT: MomentJS now has an official package for Meteor, so use meteor add momentjs:moment instead of the mrt command above
Below is a snippet of what moment can do. More documentation here.
var startTimeUTC = moment.utc(event.startTime, "YYYY-MM-DD HH:mm:ss").format();
//Changes above formatting to "2014-09-08T08:02:17-05:00" (ISO 8601)
//which is acceptable time format for Google API
So I started trying to implement all of this myself on the server side, but was wary of a lot of the hard-coding I was doing and assumptions I was making to fill gaps. My security prof. used to say "never implement encryption yourself", so I decided to take another gander for a helpful package. Revising search criteria to "JWT", I found jagi's meteor-google-oauth-jwt on Atmosphere. The readme is comprehensive and provides everything I need. Following the process used in The Google OAuth Guide, an authorization request can be made and a key generated for making an API call.
Link to Atmosphere: https://atmospherejs.com/jagi/google-oauth-jwt
Link to Repo: https://github.com/jagi/meteor-google-oauth-jwt/
I will update this answer with any additional roadblocks I hit in the Google API process and how I solved them:
Recently, I've been running into problems with the API request result. I get an empty calendarlist back from the API call. I suspect this is becuase I make an API call to my developer account rather than to the subject user. I will investigate the problem and either create a new question or update this solution with the fix I find.
Fix: Wasn't including the 'sub' qualifier to the JWT token. Fixed by modifying JWT package token generation code to include delegationEmail: user.services.google.email after scope. I don't know why he used such a long designation for the option instead of sub: as it is in the google API, but I appreciate his package nontheless.
I'm quickly becoming proficient in this, so if people have meteor-related google auth questions, let me know.
DO NOT USE SERVICE ACCOUNTS AS POSTED ABOVE!
The correct approach is to use standard web access + requesting offline access. The documentation on the api page specifically states this:
Typically, an application uses a service account when the application uses Google APIs to work with its own data rather than a user's data.
The only exception to this is when you are using google apps domain accounts and want to delegate access to your service account for the entire domain:
Authorizing a service account to access data on behalf of users in a domain is sometimes referred to as "delegating domain-wide authority"
This makes logical sense as a user must be allowed to "authorise" your application.
Back to the posters original question the flow is simple:
1) Meteor accounts google package already does most of the work for you to get tokens. You can include the scope for offline access required.
2) if you are building your own flow, you will go through the stock standard process and calls as explained on auth
This will require you to:
1) HTTP call to make the original request or you can piggyback off some of the internal meteor calls : Package.oauth.OAuth.showPopup() -- go look at the source there are more nifty functions around there.
2) Then you need to create an Iron router server side route to accept the oauth response which will contain a code parameter that you will use to exchange for tokens.
3) Next use this code to make a final call to exchange the "code" for the token + refresh_token
4) Store these where ever you want - my requirement was to store them not at the user level but multiple per user
5) Use a package like GoogleAPI this wraps up Google API calls and refreshes when required - it only works when tokens are stored in user accounts so you will need to rip it apart a bit if your tokens are stored somewhere else (like in my case)