My question is, how can I delete a users analytics data from firebase using "Google User Deletion API" and its method: userdeletionRequests:upsert? This is important for me to fully fulfill GDPR.
I tried searching for this, but didn't a solution for using it in combination with "NodeJS" and "firebase-cloud-functions".
My biggest problem is, how I get the access, token, this is what I have for now:
const accessToken = (await admin.credential.applicationDefault().getAccessToken()).access_token;
return ky.post(constants.googleUserDeletionURL, {
body: JSON.stringify({
kind: "analytics#userDeletionRequest",
id: {
type: "USER_ID",
userId: uid,
},
propertyId: constants.googleUserDeletionPropertyId,
}),
headers: {
"Authorization": `Bearer ${accessToken}`,
},
}).catch((err) => {
functions.logger.log(`An Error happened trying to delete user-anayltics ${(err as Error).message}`);
});
But I always get An Error happened trying to delete user-anayltics Request failed with status code 403 Forbidden
Okay, after some painful and long days (literally took me like >20 hours), I've figured out how to achieve this request. Here is a step for step guide:
Step 1 - Needed Dependencies
To send a post-request to google, we need a http-client-library. I've choosen "ky", so we need to install it first with:
npm install ky
Furthermore, we need to create or OWN oAuth2 token, otherwise the post request will be denied with "error 403". To create our own oAuth token, we need another dependency:
npm install #googleapis/docs
Step 2 - Needed Google Property ID
With your request, Google needs to know which property-id / project you are targeting (where it should later delete the user-analytics-data). To get this property-id, we need to log in into GCP via Firebase (not the "real" GCP, the one via Firebase).
For this, go into your "console.firebase.google.com" → Select your project → Analytics Dashboard → "View more in Google Analytics (button at the right corner)"
Write "property-id" into the search field and then save it somewhere (we need it later)
Step 3 - Creating Client-Secret
The third step is to create a service-account, which we will later add into our functions-folder, in order to create our oAuthClient (don't worry, you will see what I mean to a later point)
To create your new service.json, log in to google cloud platform via "https://cloud.google.com/" and then follow the pictures:
FIRST:
SECOND:
THIRD:
FOURTH:
FIFTH
Step 4 - Download JSON
After we created our "oAuth-Deleter service-account", we need to manually download the needed JSON, so we can paste it into our functions-folder.
For this, select "oauth-deleter#your-domain.iam.gserviceaccount.com" under "Service Account"
Then click on "Keys" and "Add key", which will automagically download you a service-json (SELECT Key type → JSON → Create).
Step 5 - Paste JSON file into your functions-folder
To loosen up the mood a bit, here is an easy step. Paste the downloaded JSON-File into your functions-folder.
Step 6 - Grant Access to our new created oAuth-Delelter-Account
Creating the service-account and giving it access in the normal GCP is not enough for Google, so we also have to give it access in our Firebase project. For this, go back into your "GCP via Firebase (see Step 2)" → Click on Setting → "User Access for Account" → Click on the "plus"
Then click on "Add user" and write the email we copied before into the email field (the email from Step 3, Picture FOURTH "Service-Account ID). In our case, it is "oAuth-Deleter#your-domain.iam.gserviceaccount.com". Furthermore, it needs admin-access:
Step 6 - The code
Now, after these million unnecessary but necessary steps, we get to the final part. THE DAMN CODE. I've written this in typescript with "compilerOptions" → "module": "esnext", "target": "esnext". But I am sure that you are smart enough to change the code after completing this many steps :)
import admin from "firebase-admin";
import functions from "firebase-functions";
import ky from "ky";
import docs from "#googleapis/docs";
import { UserRecord } from "firebase-functions/v1/auth";
export const dbUserOnDeleted = functions.
.auth
.user()
.onDelete((user) => doOnDeletedUser(user))
----------------------------
export asnc function doOnDeletedUser/user: UserRecord) {
try {
const googleDeletionURL = "https://www.googleapis.com/analytics/v3/userDeletion/userDeletionRequests:upsert"
// Step 1: Paste propertyID: (See Step 2)
const copiedPropertyID = "12345678"
// Step 2: Create oAuthClient
const oAuthClient = new docs.auth.GoogleAuth({
keyFile: "NAME-OF-THE-FILE-YOU-COPIED-INTO-YOUR-FUNCTIONS-FOLDER",
scopes: ["https://www.googleapis.com/auth/analytics.user.deletion"]
});
// Step 3: Get user uid you want to delete from user-analytics
const uid = user.uid
// Step 4: Generate access token
// (this is the reason why we needed the 5 steps before this)
// yup, a single line of code
const accessToken = await oAuthClient.getAccessToken() as string;
// Step 5: Make the request to google and delete the user
return ky.post(googleDeletionURL, {
body: JSON.stringify({
kind: "analytics#userDeletionRequest",
id: {
type: "USER_ID",
userid: uid
},
propertyId: copiedPropertyID
}),
headers: {
"Authorization": "Bearer " + accessToken,
}
});
} catch (err) {
functions.logger.error(`Something bad happened, ${(err as Error).message)`
}
}
Afterthoughts
This was and probably will be my longest post at stack overflow forever. I have to say that it was a pain in the a** to get this thing to working. The amount of work and setup that is needed to simply delete a data from one endpoint is just ridiculous. Google, please fix.
Related
I have some users signed into my actions-on-google app via Google Sign-In ( https://developers.google.com/actions/identity/google-sign-in )
I want to sent push notifications to one of those users.
For getting push notifications work with actions in the first place, I tried this sample: https://github.com/actions-on-google/dialogflow-updates-nodejs/blob/master/functions/index.js but I only can get this to work without this commit: https://github.com/actions-on-google/dialogflow-updates-nodejs/commit/c655062047b49e372da37af32376bd06d837fc7f#diff-1e53ef2f51bd446c876676ba83d7c888
It works fine, but I think const userID = conv.user.id; returns the deprecated Anonymous User ID. The commit suggests to use const userID = conv.arguments.get('UPDATES_USER_ID'); which returns undefined.
I use this nodejs code to send the push notifications.
const request = require('request');
const {JWT} = require('google-auth-library');
const serviceAccount = require('./service-account.json');
let jwtClient = new JWT(
serviceAccount.client_email, null, serviceAccount.private_key,
['https://www.googleapis.com/auth/actions.fulfillment.conversation'],
null
);
jwtClient.authorize((authErr, tokens) => {
let notification = {
userNotification: {
title: process.argv[2],
},
target: {
userId: USERID,
intent: 'tell_latest_status',
// Expects a IETF BCP-47 language code (i.e. en-US)
locale: 'en-US'
},
};
request.post('https://actions.googleapis.com/v2/conversations:send', {
'auth': {
'bearer': tokens.access_token,
},
'json': true,
'body': {
'customPushMessage': notification, 'isInSandbox': true
},
}, (reqErr, httpResponse, body) => {
console.log(httpResponse.statusCode + ': ' + httpResponse.statusMessage);
});
});
I simply can't get this to work with the const userID = conv.arguments.get('UPDATES_USER_ID'); version, because as I said
When I use conv.user.profile.payload.sub as suggested here: https://developers.google.com/actions/identity/user-info the AoG API returns "SendToConversation response: Invalid user id for target."
Is there any way to make this work with Google Sign-In?
Has anyone made this work? I mean with the UPDATES_USER_ID field?
I already created an issue on the samples repo: https://github.com/actions-on-google/dialogflow-updates-nodejs/issues/15 but I was sent here.
Thanks!
While researching why I sometimes got undefined I found an answer on this question that solved my issue.
I've found solution for this problem. While getting UPDATES_USER_ID
conv.arguments.get() only works for first attempt. So, while building
your action you must save it. If you didn't store or save, you can
reset your profile and try again, you will be able to get.
You can reset your user profile for the action here.
can I generate a custom auth token, for use with a third party, with a resolver in graph.cool? something like this??
type FirebaseTokenPayload {
token: String!
}
extend type Query {
FirebaseToken(userIdentifier: String!): FirebaseTokenPayload
}
const fb = require('myNodeFirebaseAuthLib')
module.exports = event => fb.generateTokenWithPayload({ id: event.data.userId })
Authentication required - restrict who can read data in fields: Permission query:
query ($user_id: ID!, $node_firebaseIdentifier: String) {
SomeUserExists(filter: {
id: $user_id ,
firebaseIdentifier: $node_firebaseIdentifier
})
}
--
I think this question boils down two parts
"is it possible to install node modules in the graph.cool instance -- or for that sort of thing do we need to use a webhook" -- if it must be a webhook, what is the flow of identity verification and how do I pass the payload parameters ?
"can we add permissions queries and authentication to resolvers?"
notes, addendums:
according to this alligator.io blog post, it seems that using the Graphcool framework, you can install node modules! So, I wouldn't need to use a webhook. -- However, that is with an ejected app. I lose auth0 authentication that way -- the template does not produce a createUser and singinUser that works with the same auth0 data that the integration offers.
I forgot to post the answer to this - I had to eject graphcool, I could not use any node_modules I thought to try in my custom functions.
Problem Description
My Android app collects data via Google Analytics for Firebase. For privacy reasons, users must be able to wipe their data off the Firebase servers, should they choose to do so.
The app requests a deletion by forwarding its Firebase APP_INSTANCE_ID to my own server. This server has been prepared in advance with credentials, from my personal Google account (via oauth2), for managing the Firebase project. The server authenticates with www.googleapis.com, and, using the supplied APP_INSTANCE_ID, invokes the upsert.
As noted by the documentation, the generic Google Analytics API is appropriate for this task.
After some initial trouble (b/c I didn't have the correct auth scope, and the Analytics API wasn't properly enabled), googleapis.com now returns HTTP 200 for each upsert request. (As an aside, even if you supply a bogus APP_INSTANCE_ID, it returns 200.)
Here is a sample response from the upsert, which shows nothing amiss:
{ kind: 'analytics#userDeletionRequest',
id:
{ type: 'APP_INSTANCE_ID',
userId: (REDACTED 32-char hexidecimal string) },
firebaseProjectId: (REDACTED),
deletionRequestTime: '2018-08-28T12:46:30.874Z' }
I know the firebaseProjectId is correct, because if I alter it, I get an error. I have verified that the APP_INSTANCE_ID is correct, and stable up until the moment it is reset with resetAnalyticsData().
Test Procedure
To test the deletions, I populated Firebase with several custom events, using the procedure below (Nexus 5X emulator, no Google Play, no Google accounts configured, but that shouldn't make any difference):
Install the app
Fire off some custom events (FirebaseAnalytics.logEvent)
Observe those events appear on the Firebase console
(About a minute later:) Make the upsert call, observe HTTP 200, and note the "deletionRequestTime"
Immediately call FirebaseAnalytics.resetAnalyticsData (to clear any event data cached on the device)
Uninstall the app
Rinse & repeat 7 or 8 times
However, even 24 hours later, 100% of the Firebase events are still present in the events table. No discernable state change has taken place on the Firebase server as a result of the upserts.
Question
So, what am I doing wrong? how do I successfully delete user data from Google Analytics for Firebase?
EDIT
Here's the code I'm using to make a request (from node.js):
const request = require( 'request' );
...
_deletePersonalData( data )
{
return new Promise( (resolve, reject) => {
request.post({
url: 'https://www.googleapis.com/analytics/v3/userDeletion/userDeletionRequests:upsert',
body: {
kind: 'analytics#userDeletionRequest',
id: {
type: 'APP_INSTANCE_ID',
userId: data.firebaseAppInstanceId
},
firebaseProjectId: (REDACTED)
},
headers: {
Authorization: 'Bearer ' + iap.getCurAccessToken()
},
json: true
}, (err, res, body) => {
console.log( 'user-deletion POST complete' );
console.log( 'Error ' + err );
console.log( 'Body ', body );
if( err )
{
reject( err );
return;
}
if( body.error )
{
reject( new Error( 'The Google service returned an error: ' + body.error.message + ' (' + body.error.code + ')' ) );
return;
}
resolve({ deletionRequestTime: body.deletionRequestTime });
});
});
}
Here's a sample request body:
{
kind: 'analytics#userDeletionRequest',
id: {
type: 'APP_INSTANCE_ID',
userId: (REDACTED 32-char hexidecimal string)
},
firebaseProjectId: (REDACTED)
}
And here's the console output for that same request (same userId and everything):
user-deletion POST complete
Error: null
Body: { kind: 'analytics#userDeletionRequest',
id:
{ type: 'APP_INSTANCE_ID',
userId: (REDACTED 32-char hexidecimal string) },
firebaseProjectId: (REDACTED),
deletionRequestTime: '2018-08-29T17:32:06.949Z' }
Firebase support just got back to me, and I quote:
Upsert method deletes any individual user data we have logged, but aggregate metrics are not recomputed. This means that you might not see any changes in the events tab in your Analytics console.
So, basically my mistake was expecting the events to disappear from the console.
This, of course, raises the question of how one determines that the API is actually working... but maybe the HTTP 200 is enough.
Is there a way to get an export of the store state / actions programmatically in Production that could be imported back into dev tools?
For example I can setup middleware to capture the current state and send that to something like (Trackjs,Sentry, Rollbar) but that lacks all the previous state and actions.
I would like to capture in the same format as exporting from the Redux Dev Tools.
Sample export from Dev Tools
{"monitorState":{},"actionsById":{"0":{"type":"PERFORM_ACTION","action":{"type":"##INIT"},"timestamp":1471017239656},"1":{"type":"PERFORM_ACTION","action":{"type":"INCREMENT"},"timestamp":1471017242004}},"nextActionId":2,"stagedActionIds":[0,1],"skippedActionIds":[],"committedState":5,"currentStateIndex":1,"computedStates":[{"state":5},{"state":6}]}
This is currently in development but you can now push action history right in the extension see https://github.com/zalmoxisus/remotedev-server/pull/20
Another option is to save the actions to a JSON file as an array and import them back in.
That's possible as of https://github.com/zalmoxisus/redux-devtools-extension/issues/173
logger.js
let actions = []
export function logActions (stateSanitizer) {
return store => next => action => {
actions.push(action)
return next(action)
}
}
These actions can be saved to a file or database and can be imported back into the dev tools.
Sample actions
[{
"type": "INCREMENT"
}, {
"type": "DECREMENT"
}, {
"type": "DECREMENT"
}, {
"type": "DECREMENT"
}, {
"type": "DECREMENT"
}]
I created this repo which demos this in action https://github.com/timarney/redux-trackjs-logger it uses middleware to log the actions when an error happens.
I maintain a Redux middleware called Raven for Redux which attaches Redux data to Sentry error reports. Currently it adds the following context to each error report:
The complete state object.
The complete last action object.
The type of all actions that lead up to the current state. These are added as "breadcrumbs".
The Sentry blog has a writeup describing it in more detail: https://blog.sentry.io/2016/08/24/redux-middleware-error-logging.html
You can find the middleware as an NPM package here: https://github.com/captbaritone/raven-for-redux
I have a Firebase child node with about 15,000,000 child objects with a total size of about 8 GB of data.
exampele data structure:
firebase.com/childNode/$pushKey
each $pushKey contains a small flat dictionary:
{a: 1.0, b: 2.0, c: 3.0}
I would like to delete this data as efficiently and easy as possible. How?
What i Tried:
My first try was a put request:
PUT firebase.com/childNode.json?auth=FIRE_SECRET
data-raw: null
response: {
"error": "Data requested exceeds the maximum size that can be accessed with a single request. Contact support#firebase.com for help."
}
So that didn't work, let's do a limit request:
PUT firebase.com/childNode.json?auth=FIRE_SECRET&orderBy="$key"&limitToFirst=100
data-raw: null
response: {
"error": "Querying related parameters not supported on this request type"
}
No luck so far :( What about writing a script that will get the first X number of keys and then create a patch request with each value set to null?
GET firebase.com/childNode.json?auth=FIRE_SECRET&shallow=true&orderBy="$key"&limitToLast=100
{
"error" : "Mixing 'shallow' and querying parameters is not supported"
}
It's really not going to be easy this one? I could remove the shallow requirement and get the keys, and finish the script. I was just hoping there would be a easier/more efficient way???
Another thing i tried were to create a node script that listen for childAdded and then directly tries to remove those children?
ref.authWithCustomToken(AUTH_TOKEN, function(error, authData) {
if (error) {console.log("Login Failed!", error)}
if (!error) {console.log("Login Succeeded!", authData)}
ref.child("childNode").on("child_added", function(snap) {
console.log(`found: ${snap.key()}`)
ref.child("childNode").child(snap.key()).remove( function(err) {
if (!err) {console.log(`deleted: ${snap.key()}`)}
})
})
})
This script actually hangs right now, but earlier I did receive somethings like a max stack limit warning from firebase. I know this is not a firebase problem, but I don't see any particular easy way to solve that problem.
Downloading a shallow tree, will download only the keys. So instead of asking the server to order and limit, you can download all keys.
Then you can order and limit it client-side, and send delete requests to Firebase in batches.
You can use this script for inspiration: https://gist.github.com/wilhuff/b78e7391396e09f6c614
Use firebase cli tool for this: firebase database:remove --project .
In Browser Console this is fastest way
database.ref('data').limitToFirst(10000).once('value', snap => {
var updates = {};
snap.forEach(snap => {
updates[snap.key] = null;
});
database.ref('data').update(updates);
});