Related
I am trying to write jest tests using the Firebase Auth emulator and continue to receive the following CORS error.
console.error
Error: Headers X-Client-Version forbidden
at dispatchError (/Users/me/my-project/node_modules/jsdom/lib/jsdom/living/xhr/xhr-utils.js:62:19)
at validCORSPreflightHeaders (/Users/me/my-project/node_modules/jsdom/lib/jsdom/living/xhr/xhr-utils.js:99:5)
at Request.<anonymous> (/Users/me/my-project/node_modules/jsdom/lib/jsdom/living/xhr/xhr-utils.js:367:12)
at Request.emit (events.js:315:20)
at Request.onRequestResponse (/Users/me/my-project/node_modules/request/request.js:1059:10)
at ClientRequest.emit (events.js:315:20)
at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:641:27)
at HTTPParser.parserOnHeadersComplete (_http_common.js:126:17)
at Socket.socketOnData (_http_client.js:509:22)
at Socket.emit (events.js:315:20) undefined
The test is very simple:
import { renderHook, act } from "#testing-library/react-hooks"
import faker from "faker"
import { useAuth, FirebaseProvider, firebase } from "./index"
const wrapper = ({ firebase, children }) => {
return <FirebaseProvider firebase={firebase}>{children}</FirebaseProvider>
}
const createUser = ({ email = faker.internet.email(), password = faker.internet.password({ length: 6 }) } = {}) => {
return firebase
.auth()
.createUserWithEmailAndPassword(email, password)
.then(user => user)
}
const signUserIn = ({ email, password } = {}) => {
return firebase
.auth()
.signInWithEmailAndPassword(email, password)
.then(user => user)
}
describe("useAuth", () => {
it("will return the user", async () => {
const { result } = renderHook(() => useAuth(), { wrapper, initialProps: { firebase } })
const email = faker.internet.email()
const password = faker.internet.password()
await act(async () => {
const user = await createUser({ email, password }) // this fails
await signUserIn({ email, password }) //and so does this
})
expect(result.user).toEqual({ email, password })
})
})
And for reference, the index file:
const FirebaseProvider = ({ children, firebase }) => {
const firestore = firebase.firestore()
const auth = firebase.auth()
if (useEmulator()) {
firestore.useEmulator("localhost", 8080)
auth.useEmulator("http://localhost:9099/")
}
const value = { firestore, auth }
return <FirebaseContext.Provider value={value}>{children}</FirebaseContext.Provider>
}
const throwError = hook => {
throw new Error(`${hook} must be used within a FirebaseProvider`)
}
const useAuth = () => {
const context = useContext(FirebaseContext)
if (context === undefined) throwError("useAuth")
const [user, setUser] = useState()
useEffect(() => {
const cleanup = context.auth.onAuthStateChanged(authUser => {
authUser ? setUser(authUser) : setUser(null)
})
return () => cleanup()
})
return { ...context.auth, user }
}
I have tried using the REST endpoint that the actual emulator uses (below) and it errors in the same way.
http://localhost:9099/identitytoolkit.googleapis.com/v1/projects/<my-project>/accounts
Is there anyway to get this to run when using jest? Or do I need to create the accounts using the emulator UI, export them and re-import when I am running tests?
I have found I can use the REST endpoint below to make a user in the test, however it bypasses the emulator and makes a real user.
https://identitytoolkit.googleapis.com/v1/accounts:signUp?key=<api-key>
Update jsdom version 16.5.2
This new version now supports wildcards for access-control-allow-headers, so updating to this version or using it as resolution, for projects created with Create React App, solves the problem.
Solution for jsdom prior to version 16.5.2
The error is thrown by jsdom because it doesn't support wildcard for access-control-allow-headers, but firebase uses the wildcard (see this issue for jsdom and this pull request related to firebase). There are two open pull requests to fix this issue: https://github.com/jsdom/jsdom/pull/3073 and https://github.com/jsdom/jsdom/pull/2867.
The issue can be fixed by either changing the relevant code manually in the node_modules folder or by using the fork as dependency in the package.json:
"jsdom": "silviot/jsdom#fix/allow-headers"
If jsdom isn't a direct dependency, then you can add the following to the package.json at the top level:
"resolutions": {
"jsdom": "silviot/jsdom#fix/allow-headers"
}
If the fork is used there are some auto-generated files missing in the jsdom folder. These can be generated by running npm install or yarn install in the folder. To automate this you can add a prepare script to the package.json:
"scripts": {
"prepare": "cd node_modules/jsdom && yarn"
},
I also had problems making users programaticly in the firebase auth emulator.
Instead of using
https://identitytoolkit.googleapis.com/v1/accounts:signUp?key=[API_KEY]
You have to use the following format:
http://localhost:9099/identitytoolkit.googleapis.com/v1/accounts:signUp?key=[API_KEY]
Then giving a JSON body like this, and hit post
{
"email": "test#test.com",
"password": "test12"
}
And voila! You have a user in your emulator. Combine this with fetch or axios and you seed your emulator with users. If you need to add custom claims or other info, create function in the functions emulator that triggers on user creation.
functions.auth.user().onCreate
I am trying to test a firebase app locally.
I am running the test with firebase emulators:exec --only firestore 'mocha -r ts-node/register src/**/*.spec.ts
In my spec, I import #firebase/testing and setup my app and followed the directions from
https://firebase.google.com/docs/rules/unit-tests
I have a FirebaseService which is a singleton wrapper for my methods into which I inject my firebase app.
In production, I'll inject the firebase, and it gets initialized in the FirebaseService in testing, I initialize outside of the service.
The wrapper is fairly simple
export const FirebaseService = (function(): FirebaseSrvc {
let firebase;
const fbServiceObj: FirebaseSrvc = {
getInstance: (firebaseConfig, firebaseCore, initialize) => {
firebase = firebaseCore;
if (initialize && firebase.apps.length === 0) {
firebase.initializeApp(firebaseConfig);
}
return fbServiceObj;
},
createActivity: async (title: string) => {
try {
const firebaseUid = firebase.auth().currentuser.uid;
const newActivity: ActivityProps = {
title,
created_at: 123445,
created_by: firebaseUid,
public: false,
available_to: [firebaseUid],
};
console.log(' before create', newActivity);
const createResponse = await firebase
.firestore()
.collection('activities')
.doc(stringToSafeId(title))
.set(newActivity);
console.log('create response', createResponse);
return true;
} catch (e) {
console.log('error creating activity', e);
}
},
getActivity: async (title: string): Promise<ActivityProps> => {
try {
const actResponse: DocumentReferenceTo<ActivityProps> = await firebase
.firestore()
.collection('activities')
.doc(stringToSafeId(title))
.get();
return actResponse as ActivityProps;
} catch (e) {
console.log('error getting activity from firebase', e);
}
},
};
return fbServiceObj;
})();
The test I am attempting to run is
import * as firebase from '#firebase/testing';
import { assert } from 'chai';
import 'mocha';
import * as appConfig from '../../app-dev.json';
import { FirebaseService } from '../services/FirebaseService';
firebase.initializeTestApp({ ...appConfig.expo.extra.firebase, auth: { uid: 'random', email: 'test#test.com' } });
describe('Activity', async () => {
const fb = FirebaseService.getInstance(appConfig.expo.extra.firebase, testApp, false);
const activityData = new Activity(fb);
beforeEach(async () => await firebase.clearFirestoreData({ projectId }));
it('should create a new activity', async () => {
await activityData.set('test-activity'); // this runs FirebaseService.createActivity
const findActivity = await activityData.get('test-activity'); // this run FirebaseService.getActivity
assert(findActivity.title === 'test-activity');
});
});
When I run the test I get an error
Your API key is invalid, please check you have copied it correctly.] {
code: 'auth/invalid-api-key',
message: 'Your API key is invalid, please check you have copied it correctly.'
}
I can confirm that the API key which is passed into firebase.initializeTestApp matches the Web API Key in my firebase console.
I have also downloaded the google-services.json from my firebase console and lists
{
"api_key": [
{ "current_key": different_from_web_key}
]
}
And I have replaced my existing key with this new key, I still get the same error.
I have also tried setting up initializeTestApp({ projectId }) which is how the example from firebase docs sets it up, and I receive the same result.
I am using the same project details to run a project locally in android studio, and I am able to authenticate and write to firestore, so the API key I am using does work, but it appears to have issues being used in the test app.
This usually doesn't have a specific way to solve it. It might be that even a new copy and paste of the API key to the parameters, might make it work and the error to disappear.
I would recommend you to take a look at the following posts from the Community, that have some possible fixes for the error that you are facing.
Firebase Error: auth/invalid-api-key, Your API key is invalid, please check you have copied it correctly
Invalid API Key supplied using Firebase
In addition to that, since Firebase has free support offers, I think you reaching out to the Firebase support would help you fix this quickly. You should be able to contact directly for free.
Let me know if the information helped you!
Is there a way to import CSV or JSON to firebase cloud firestore like in firebase realtime database?
General Solution
I've found many takes on a script allowing to upload a JSON but none of them allowed sub-collections. My script above handles any level of nesting and sub-collections. It also handles the case where a document has its own data and sub-collections. This is based on the assumption that collection is array/object of objects (including an empty object or array).
To run the script make sure you have npm and node installed. Then run your code as node <name of the file>. Note, there is no need to deploy it as a cloud funciton.
const admin = require('../functions/node_modules/firebase-admin');
const serviceAccount = require("./service-key.json");
admin.initializeApp({
credential: admin.credential.cert(serviceAccount),
databaseURL: "https://<your-database-name>.firebaseio.com"
});
const data = require("./fakedb.json");
/**
* Data is a collection if
* - it has a odd depth
* - contains only objects or contains no objects.
*/
function isCollection(data, path, depth) {
if (
typeof data != 'object' ||
data == null ||
data.length === 0 ||
isEmpty(data)
) {
return false;
}
for (const key in data) {
if (typeof data[key] != 'object' || data[key] == null) {
// If there is at least one non-object item in the data then it cannot be collection.
return false;
}
}
return true;
}
// Checks if object is empty.
function isEmpty(obj) {
for(const key in obj) {
if(obj.hasOwnProperty(key)) {
return false;
}
}
return true;
}
async function upload(data, path) {
return await admin.firestore()
.doc(path.join('/'))
.set(data)
.then(() => console.log(`Document ${path.join('/')} uploaded.`))
.catch(() => console.error(`Could not write document ${path.join('/')}.`));
}
/**
*
*/
async function resolve(data, path = []) {
if (path.length > 0 && path.length % 2 == 0) {
// Document's length of path is always even, however, one of keys can actually be a collection.
// Copy an object.
const documentData = Object.assign({}, data);
for (const key in data) {
// Resolve each collection and remove it from document data.
if (isCollection(data[key], [...path, key])) {
// Remove a collection from the document data.
delete documentData[key];
// Resolve a colleciton.
resolve(data[key], [...path, key]);
}
}
// If document is empty then it means it only consisted of collections.
if (!isEmpty(documentData)) {
// Upload a document free of collections.
await upload(documentData, path);
}
} else {
// Collection's length of is always odd.
for (const key in data) {
// Resolve each collection.
await resolve(data[key], [...path, key]);
}
}
}
resolve(data);
You need a custom script to do that.
I wrote one based on the Firebase admin SDK, as long as firebase library does not allow you to import nested arrays of data.
const admin = require('./node_modules/firebase-admin');
const serviceAccount = require("./service-key.json");
const data = require("./data.json");
admin.initializeApp({
credential: admin.credential.cert(serviceAccount),
databaseURL: "https://YOUR_DB.firebaseio.com"
});
data && Object.keys(data).forEach(key => {
const nestedContent = data[key];
if (typeof nestedContent === "object") {
Object.keys(nestedContent).forEach(docTitle => {
admin.firestore()
.collection(key)
.doc(docTitle)
.set(nestedContent[docTitle])
.then((res) => {
console.log("Document successfully written!");
})
.catch((error) => {
console.error("Error writing document: ", error);
});
});
}
});
Update: I wrote an article on this topic - Filling Firestore with data
There is not, you'll need to write your own script at this time.
I used the General Solution provided by Maciej Caputa. Thank you (:
Here are a few hints. Assuming that you have an Ionic Firebase application installed with the required Firebase node modules in the functions folder inside that solution. This is a standard Ionic Firebase install. I created an import folder to hold the script and data at the same level.
Folder Hierarchy
myIonicApp
functions
node_modules
firebase-admin
ImportFolder
script.js
FirebaseIonicTest-a1b2c3d4e5.json
fileToImport.json
Script Parameters
const admin = require('../myIonicApp/functions/node_modules/firebase-admin'); //path to firebase-admin module
const serviceAccount = require("./FirebaseTest-xxxxxxxxxx.json"); //service account key file
admin.initializeApp({
credential: admin.credential.cert(serviceAccount),
databaseURL: "https://fir-test-xxxxxx.firebaseio.com" //Your domain from the hosting tab
});
Creating the Service Account Key File
In the Firebase console for your project, next to the Project
Overwiew item, click on the gear icon and select Users and
permissions
At the bottom of the screen, click Advanced permission
settings
This opens another tab for the Google Cloud Platform Console
On the left select the Service Accounts item
Create a Service Account for an existing Service Account
I simply added a key to the App Engine default service account
The Create key function will offer to download the key to a JSON file
JSON Data Structure
To use the script provided, the data structure must be as follows:
{
"myCollection" : {
"UniqueKey1" : {
"field1" : "foo",
"field2" : "bar"
},{
"UniqueKey2" : {
"field1" : "fog",
"field2" : "buzz"
}...
}
For reference. I wrote a function that helps to import and export data in Firestore.
https://github.com/dalenguyen/firestore-import-export (archived)
Updated 2023:
Use this package instead. Thanks, #Webber
https://github.com/dalenguyen/firestore-backup-restore
https://gist.github.com/JoeRoddy/1c706b77ca676bfc0983880f6e9aa8c8
This should work for an object of objects (generally how old firebase json is set up). You can add that code to an app that's already configured with Firestore.
Just make sure you have it pointing to the correct JSON file.
Good luck!
This workaround in Python may help some people. First convert json or csv to dataframe using Pandas, then convert dataframe to dictionary and upload dictionary to firestore.
import firebase_admin as fb
from firebase_admin import firestore
import pandas as pd
cred = fb.credentials.Certificate('my_credentials_certificate.json')
default_app = fb.initialize_app(cred)
db = firestore.client()
df = pd.read_csv('data.csv')
dict = df.to_dict(orient='records')
my_doc_ref = db.collection('my_collection').document('my_document')
my_doc_ref.set(dict)
There could be similar workarounds in javascript and other languages using libraries similar to Pandas.
No as of now, you can't.. firestore structures data into a different format that is, using collections and each collection has a series of documents which then are stored in JSON format.. in future they might make a tool to convert JSON in to firestore.. for reference check this out
:https://cloud.google.com/firestore/docs/concepts/structure-data
****EDIT :****
You can automate the process up to some extent, that is, write a mock software which only pushes the fields of your CSV or JSON data into your Cloud Firestore DB. I migrated my whole database my making just a simple app which retrieved my DB and pushed it on Firestore
var admin = require('firebase-admin');
var serviceAccount = require('./serviceAccountKey.json');
admin.initializeApp({
credential: admin.credential.cert(serviceAccount),
databaseURL: 'https://csvread-d149c.firebaseio.com'
});
const csv = require('csv-parser');
const fs = require('fs');
const firestore = admin.firestore();
// CSV FILE data.csv
// Name,Surname,Age,Gender
// John,Snow,26,M
// Clair,White,33,F
// Fancy,Brown,78,F
fs.createReadStream('data.csv')
.pipe(csv())
.on('data', (row) => {
console.log(row);
if(row) {
firestore.collection('csv').add({
name: row.Name,
surname: row.Surname,
age: row.Age,
sex: row.Gender
});
}
else {
console.log('No data')
}
})
.on('end', () => {
console.log('CSV file successfully processed');
});
There is now a paid GUI tool that does this, Firefoo. I haven't used it, but it seems quite powerful for managing Firestore - including importing csv files:
This is a small modification that copies the 'id' of the document, if exists, to its path. Otherwise it will use the "for"'s index.
...
...
} else {
// Collection's length of is always odd.
for (const key in data) {
// Resolve each collection.
if (data[key]['id']!==undefined)
await resolve(data[key], [...path,data[key]['id']])
else
await resolve(data[key], [...path, key]);
}
}
}
resolve(data);
1 - Importing only collections
If the names of your collections are not only composed of numbers, then you can define the name of the document as follows.
...
...
} else {
// Collection's length of is always odd.
for (const key in data) {
// // Resolve each collection.
// If key is a number it means that it is not a collection
// The variable id in this case will be the name of the document field or you
// can generate randomly
let id = !isNaN(key) ? data[key].id : key;
await resolve(data[key], [...path, id]);
}
}
}
2 - Import collections and sub-collections
In the same way as in the example above, the name of the sub-collection can not contain only numbers.
...
...
for (const key in data) {
// Resolve each collection and remove it from document data.
if (isCollection(data[key], [...path, key])) {
// Remove a collection from the document data.
delete documentData[key];
// If key is a number it means that it is not a collection
// The variable id in this case will be the name of the document field or you
// can generate randomly
let id = !isNaN(key) ? data[key].id : key;
// Resolve a colleciton.
resolve(data[key], [...path, id]);
}
}
...
...
Note.: Changes in #Maciej Caputa's code
this library you can use https://www.npmjs.com/package/csv-to-firestore
install this library with this command npm i -g csv-to-firestore
put this script inside your ./scripts
module.exports = {
path: "./scripts/palabrasOutput.csv",
firebase: {
credential: "./scripts/serviceAccount.json",
collection: "collectionInFirestore",
},
mapper: (dataFromCSV) => {
return dataFromCSV;
},
};
and run:
csv-to-firestore --config ./scripts/csvToFirestore.js
btw: Generate new private key (it's a file in Service account section in firestore)
put that file in /scripts folder and rename serviceAccount.json
I am exploring Cypress for e2e testing, looks like great software.
The problem is Authentication, the Cypress documentation explains why using the UI is very bad here.
So I tried looking at the network tap of my application, to see if I could create a POST request to the firebase API, and authenticate without using the GUI. But I can see that there at least 2 request fired, and token saved to application storage.
So what approach should I use?
Authenticate with the UI of my application, and instruct Cypress not to touch the local storage
Keep experimenting with a way of sending the correct POST requests, and save the values to local storage.
Make Cypress run custom JS code, and then use the Firebase SDK to login.
I am really looking for some advice here :)
When doing this myself I made custom commands (like cy.login for auth then cy.callRtdb and cy.callFirestore for verifying data). After getting tired of repeating the logic it took to build them, I wrapped it up into a library called cypress-firebase. It includes custom commands and a cli to generate a custom auth token.
Setup mostly just consists of adding the custom commands in cypress/support/commands.js:
import firebase from 'firebase/app';
import 'firebase/auth';
import 'firebase/database';
import 'firebase/firestore';
import { attachCustomCommands } from 'cypress-firebase';
const fbConfig = {
// Your config from Firebase Console
};
window.fbInstance = firebase.initializeApp(fbConfig);
attachCustomCommands({ Cypress, cy, firebase })
And adding the plugin to cypress/plugins/index.js:
const cypressFirebasePlugin = require('cypress-firebase').plugin
module.exports = (on, config) => {
// `on` is used to hook into various events Cypress emits
// `config` is the resolved Cypress config
// Return extended config (with settings from .firebaserc)
return cypressFirebasePlugin(config)
}
But there full details on setup are available in the setup docs.
Disclosure, I am the author of cypress-firebase, which is the whole answer.
I took the approach of using automated UI to obtain the contents of localStorage used by Firebase JS SDK. I also wanted to do this only once per whole Cypress run so I did it before the Cypress start.
Obtain Firebase SDK localStorage entry via pupeteer
Store the contents in the tmp file (problems passing it via env var to Cypress)
Pass the file location to Cypress via env var and let it read the contents and set the localStorage to setup the session
Helper script which obtains contents of localStorage:
const puppeteer = require('puppeteer')
const invokeLogin = async page => {
await page.goto('http://localhost:3000/login')
await page.waitForSelector('.btn-googleplus')
await page.evaluate(() =>
document.querySelector('.btn-googleplus').click())
}
const doLogin = async (page, {username, password}) => {
// Username step
await page.waitForSelector('#identifierId')
await page.evaluate((username) => {
document.querySelector('#identifierId').value = username
document.querySelector('#identifierNext').click()
}, username)
// Password step
await page.waitForSelector('#passwordNext')
await page.evaluate(password =>
setTimeout(() => {
document.querySelector('input[type=password]').value = password
document.querySelector('#passwordNext').click()
}, 3000) // Wait 3 second to next phase to init (couldn't find better way)
, password)
}
const extractStorageEntry = async page =>
page.evaluate(() => {
for (let key in localStorage) {
if (key.startsWith('firebase'))
return {key, value: localStorage[key]}
}
})
const waitForApp = async page => {
await page.waitForSelector('#app')
}
const main = async (credentials, cfg) => {
const browser = await puppeteer.launch(cfg)
const page = await browser.newPage()
await invokeLogin(page)
await doLogin(page, credentials)
await waitForApp(page)
const entry = await extractStorageEntry(page)
console.log(JSON.stringify(entry))
await browser.close()
}
const username = process.argv[2]
const password = process.argv[3]
main({username, password}, {
headless: true // Set to false for debugging
})
Since there were problem with sending JSON as environment variables to Cypress I use tmp file to pass the data between the script and the Cypress process.
node test/getFbAuthEntry ${USER} ${PASSWORD} > test/tmp/fbAuth.json
cypress open --env FB_AUTH_FILE=test/tmp/fbAuth.json
In Cypress I read it from the file system and set it to the localStorage
const setFbAuth = () =>
cy.readFile(Cypress.env('FB_AUTH_FILE'))
.then(fbAuth => {
const {key, value} = fbAuth
localStorage[key] = value
})
describe('an app something', () => {
it('does stuff', () => {
setFbAuth()
cy.viewport(1300, 800)
...
This is certainly a hack but to get around the login part for the app I am working on I use the beforeEach hook to login to the application.
beforeEach(() => {
cy.resetTestDatabase().then(() => {
cy.setupTestDatabase();
});
});
Which is derived from my helper functions.
Cypress.Commands.add('login', () => {
return firebase
.auth()
.signInWithEmailAndPassword(Cypress.env('USER_EMAIL'), Cypress.env('USER_PASSWORD'));
});
Cypress.Commands.add('resetTestDatabase', () => {
return cy.login().then(() => {
firebase
.database()
.ref(DEFAULT_CATEGORIES_PATH)
.once('value')
.then(snapshot => {
const defaultCategories = snapshot.val();
const updates = {};
updates[TEST_CATEGORIES_PATH] = defaultCategories;
updates[TEST_EVENTS_PATH] = null;
updates[TEST_STATE_PATH] = null;
updates[TEST_EXPORT_PATH] = null;
return firebase
.database()
.ref()
.update(updates);
});
});
});
What I would like to know is how the information coming back from firebase ultimately gets saved to localStorage. I don't really have an answer to this but it works. Also, the app uses .signInWithPopup(new firebase.auth.GoogleAuthProvider()) whereas above it signs in with email and password. So I am kind of shortcutting the signin process only because cypress has the CORS limitation.
This is becoming way easier with the upcoming Auth emulator. This has become easier with the Firebase Auth Emulator (firebase-tools >= 8.1.4).
cypress/support/signAs.js:
Cypress.Commands.add('signAs', (uid, opt) => {
cy.visit('/')
cy.window().its('firebase').then( fb => {
cy.wrap( (async _ => {
// Create a user based on the provided token (only '.uid' is used by Firebase)
await fb.auth().signInWithCustomToken( JSON.stringify({ uid }) );
// Set '.displayName', '.photoURL'; for email and password, other functions exist (not implemented)
await fb.auth().currentUser.updateProfile(opt);
})() )
})
})
Use it as:
cy.signAs('joe', { displayName: 'Joe D.', photoURL: 'http://some' });
If you need to set .email or .password, there are similar functions for those, but this was sufficient for my tests. I can now impersonate any user ad-hoc, as part of the test. The approach does not need users to be created in the emulator; you can just claim to be one, with the particular uid. Works well for me.
Note:
Firebase authentication is in IndexedDB (as mentioned in other answers) and Cypress does not clear it, between the tests. There is discussion about this in cypress #1208.
At the time writing, I've examined these approaches
stubbing firebase network requests - really difficult. A bunch of firebase requests is sent continuously. There are so many request params & large payload and they're unreadable.
localStorage injection - as same as request stubbing. It requires an internally thorough understanding of both firebase SDK and data structure.
cypress-firebase plugin - it's not matured enough and lack of documentation. I skipped this option because it needs a service account (admin key). The project I'm working on is opensource and there are many contributors. It's hard to share the key without including it in the source control.
Eventually, I implemented it on my own which is quite simple. Most importantly, it doesn't require any confidential firebase credentials. Basically, it's done by
initialize another firebase instance within Cypress
use that firebase instance to build a Cypress custom command to login
const fbConfig = {
apiKey: `your api key`, // AIzaSyDAxS_7M780mI3_tlwnAvpbaqRsQPlmp64
authDomain: `your auth domain`, // onearmy-test-ci.firebaseapp.com
projectId: `your project id`, // onearmy-test-ci
}
firebase.initializeApp(fbConfig)
const attachCustomCommands = (
Cypress,
{ auth, firestore }: typeof firebase,
) => {
let currentUser: null | firebase.User = null
auth().onAuthStateChanged(user => {
currentUser = user
})
Cypress.Commands.add('login', (email, password) => {
Cypress.log({
displayName: 'login',
consoleProps: () => {
return { email, password }
},
})
return auth().signInWithEmailAndPassword(email, password)
})
Cypress.Commands.add('logout', () => {
const userInfo = currentUser ? currentUser.email : 'Not login yet - Skipped'
Cypress.log({
displayName: 'logout',
consoleProps: () => {
return { currentUser: userInfo }
},
})
return auth().signOut()
})
}
attachCustomCommands(Cypress, firebase)
Here is the commit that has all integration code https://github.com/ONEARMY/community-platform/commit/b441699c856c6aeedb8b73464c05fce542e9ead1
Ok after much trial and error, I tried solution path 2 and it worked.
So my auth flow looks like this:
Send POST request (using cybress.request) to
https://www.googleapis.com/identitytoolkit/v3/relyingparty/verifyPassword,
and parse the response. Create an object: response1 = response.body
Send POST request (using cybress.request) to
https://www.googleapis.com/identitytoolkit/v3/relyingparty/getAccountInfo,
use the idToken from the prev request. Create an object: user = response2.body.users[0];
Combine the response in an object, with the following properties:
const authObject = {
uid: response1.localId,
displayName: response1.displayName,
photoURL: null,
email: response1.email,
phoneNumber: null,
isAnonymous: false,
providerData: [
{
uid: response1.email,
displayName: response1.displayName,
photoURL: null,
email: body.email,
phoneNumber: null,
providerId: 'password'
}
],
'apiKey': apiKey,
'appName': '[DEFAULT]',
'authDomain': '<name of firebase domain>',
'stsTokenManager': {
'apiKey': apiKey,
'refreshToken': response1.refreshToken,
'accessToken': response1.idToken,
'expirationTime': user.lastLoginAt + Number(response1.expiresIn)
},
'redirectEventId': null,
'lastLoginAt': user.lastLoginAt,
'createdAt': user.createdAt
};
Then in cybress, I simply save this object in local storag, in the before hook: localStorage.setItem(firebase:authUser:${apiKey}:[DEFAULT], authObject);
Maybe not perfect, but it solves the problem. Let me know if you interested in the code, and if you have any knowledge about how to build the "authObject", or solve this problem in another way.
Is there a way to import CSV or JSON to firebase cloud firestore like in firebase realtime database?
General Solution
I've found many takes on a script allowing to upload a JSON but none of them allowed sub-collections. My script above handles any level of nesting and sub-collections. It also handles the case where a document has its own data and sub-collections. This is based on the assumption that collection is array/object of objects (including an empty object or array).
To run the script make sure you have npm and node installed. Then run your code as node <name of the file>. Note, there is no need to deploy it as a cloud funciton.
const admin = require('../functions/node_modules/firebase-admin');
const serviceAccount = require("./service-key.json");
admin.initializeApp({
credential: admin.credential.cert(serviceAccount),
databaseURL: "https://<your-database-name>.firebaseio.com"
});
const data = require("./fakedb.json");
/**
* Data is a collection if
* - it has a odd depth
* - contains only objects or contains no objects.
*/
function isCollection(data, path, depth) {
if (
typeof data != 'object' ||
data == null ||
data.length === 0 ||
isEmpty(data)
) {
return false;
}
for (const key in data) {
if (typeof data[key] != 'object' || data[key] == null) {
// If there is at least one non-object item in the data then it cannot be collection.
return false;
}
}
return true;
}
// Checks if object is empty.
function isEmpty(obj) {
for(const key in obj) {
if(obj.hasOwnProperty(key)) {
return false;
}
}
return true;
}
async function upload(data, path) {
return await admin.firestore()
.doc(path.join('/'))
.set(data)
.then(() => console.log(`Document ${path.join('/')} uploaded.`))
.catch(() => console.error(`Could not write document ${path.join('/')}.`));
}
/**
*
*/
async function resolve(data, path = []) {
if (path.length > 0 && path.length % 2 == 0) {
// Document's length of path is always even, however, one of keys can actually be a collection.
// Copy an object.
const documentData = Object.assign({}, data);
for (const key in data) {
// Resolve each collection and remove it from document data.
if (isCollection(data[key], [...path, key])) {
// Remove a collection from the document data.
delete documentData[key];
// Resolve a colleciton.
resolve(data[key], [...path, key]);
}
}
// If document is empty then it means it only consisted of collections.
if (!isEmpty(documentData)) {
// Upload a document free of collections.
await upload(documentData, path);
}
} else {
// Collection's length of is always odd.
for (const key in data) {
// Resolve each collection.
await resolve(data[key], [...path, key]);
}
}
}
resolve(data);
You need a custom script to do that.
I wrote one based on the Firebase admin SDK, as long as firebase library does not allow you to import nested arrays of data.
const admin = require('./node_modules/firebase-admin');
const serviceAccount = require("./service-key.json");
const data = require("./data.json");
admin.initializeApp({
credential: admin.credential.cert(serviceAccount),
databaseURL: "https://YOUR_DB.firebaseio.com"
});
data && Object.keys(data).forEach(key => {
const nestedContent = data[key];
if (typeof nestedContent === "object") {
Object.keys(nestedContent).forEach(docTitle => {
admin.firestore()
.collection(key)
.doc(docTitle)
.set(nestedContent[docTitle])
.then((res) => {
console.log("Document successfully written!");
})
.catch((error) => {
console.error("Error writing document: ", error);
});
});
}
});
Update: I wrote an article on this topic - Filling Firestore with data
There is not, you'll need to write your own script at this time.
I used the General Solution provided by Maciej Caputa. Thank you (:
Here are a few hints. Assuming that you have an Ionic Firebase application installed with the required Firebase node modules in the functions folder inside that solution. This is a standard Ionic Firebase install. I created an import folder to hold the script and data at the same level.
Folder Hierarchy
myIonicApp
functions
node_modules
firebase-admin
ImportFolder
script.js
FirebaseIonicTest-a1b2c3d4e5.json
fileToImport.json
Script Parameters
const admin = require('../myIonicApp/functions/node_modules/firebase-admin'); //path to firebase-admin module
const serviceAccount = require("./FirebaseTest-xxxxxxxxxx.json"); //service account key file
admin.initializeApp({
credential: admin.credential.cert(serviceAccount),
databaseURL: "https://fir-test-xxxxxx.firebaseio.com" //Your domain from the hosting tab
});
Creating the Service Account Key File
In the Firebase console for your project, next to the Project
Overwiew item, click on the gear icon and select Users and
permissions
At the bottom of the screen, click Advanced permission
settings
This opens another tab for the Google Cloud Platform Console
On the left select the Service Accounts item
Create a Service Account for an existing Service Account
I simply added a key to the App Engine default service account
The Create key function will offer to download the key to a JSON file
JSON Data Structure
To use the script provided, the data structure must be as follows:
{
"myCollection" : {
"UniqueKey1" : {
"field1" : "foo",
"field2" : "bar"
},{
"UniqueKey2" : {
"field1" : "fog",
"field2" : "buzz"
}...
}
For reference. I wrote a function that helps to import and export data in Firestore.
https://github.com/dalenguyen/firestore-import-export (archived)
Updated 2023:
Use this package instead. Thanks, #Webber
https://github.com/dalenguyen/firestore-backup-restore
https://gist.github.com/JoeRoddy/1c706b77ca676bfc0983880f6e9aa8c8
This should work for an object of objects (generally how old firebase json is set up). You can add that code to an app that's already configured with Firestore.
Just make sure you have it pointing to the correct JSON file.
Good luck!
This workaround in Python may help some people. First convert json or csv to dataframe using Pandas, then convert dataframe to dictionary and upload dictionary to firestore.
import firebase_admin as fb
from firebase_admin import firestore
import pandas as pd
cred = fb.credentials.Certificate('my_credentials_certificate.json')
default_app = fb.initialize_app(cred)
db = firestore.client()
df = pd.read_csv('data.csv')
dict = df.to_dict(orient='records')
my_doc_ref = db.collection('my_collection').document('my_document')
my_doc_ref.set(dict)
There could be similar workarounds in javascript and other languages using libraries similar to Pandas.
No as of now, you can't.. firestore structures data into a different format that is, using collections and each collection has a series of documents which then are stored in JSON format.. in future they might make a tool to convert JSON in to firestore.. for reference check this out
:https://cloud.google.com/firestore/docs/concepts/structure-data
****EDIT :****
You can automate the process up to some extent, that is, write a mock software which only pushes the fields of your CSV or JSON data into your Cloud Firestore DB. I migrated my whole database my making just a simple app which retrieved my DB and pushed it on Firestore
var admin = require('firebase-admin');
var serviceAccount = require('./serviceAccountKey.json');
admin.initializeApp({
credential: admin.credential.cert(serviceAccount),
databaseURL: 'https://csvread-d149c.firebaseio.com'
});
const csv = require('csv-parser');
const fs = require('fs');
const firestore = admin.firestore();
// CSV FILE data.csv
// Name,Surname,Age,Gender
// John,Snow,26,M
// Clair,White,33,F
// Fancy,Brown,78,F
fs.createReadStream('data.csv')
.pipe(csv())
.on('data', (row) => {
console.log(row);
if(row) {
firestore.collection('csv').add({
name: row.Name,
surname: row.Surname,
age: row.Age,
sex: row.Gender
});
}
else {
console.log('No data')
}
})
.on('end', () => {
console.log('CSV file successfully processed');
});
There is now a paid GUI tool that does this, Firefoo. I haven't used it, but it seems quite powerful for managing Firestore - including importing csv files:
This is a small modification that copies the 'id' of the document, if exists, to its path. Otherwise it will use the "for"'s index.
...
...
} else {
// Collection's length of is always odd.
for (const key in data) {
// Resolve each collection.
if (data[key]['id']!==undefined)
await resolve(data[key], [...path,data[key]['id']])
else
await resolve(data[key], [...path, key]);
}
}
}
resolve(data);
1 - Importing only collections
If the names of your collections are not only composed of numbers, then you can define the name of the document as follows.
...
...
} else {
// Collection's length of is always odd.
for (const key in data) {
// // Resolve each collection.
// If key is a number it means that it is not a collection
// The variable id in this case will be the name of the document field or you
// can generate randomly
let id = !isNaN(key) ? data[key].id : key;
await resolve(data[key], [...path, id]);
}
}
}
2 - Import collections and sub-collections
In the same way as in the example above, the name of the sub-collection can not contain only numbers.
...
...
for (const key in data) {
// Resolve each collection and remove it from document data.
if (isCollection(data[key], [...path, key])) {
// Remove a collection from the document data.
delete documentData[key];
// If key is a number it means that it is not a collection
// The variable id in this case will be the name of the document field or you
// can generate randomly
let id = !isNaN(key) ? data[key].id : key;
// Resolve a colleciton.
resolve(data[key], [...path, id]);
}
}
...
...
Note.: Changes in #Maciej Caputa's code
this library you can use https://www.npmjs.com/package/csv-to-firestore
install this library with this command npm i -g csv-to-firestore
put this script inside your ./scripts
module.exports = {
path: "./scripts/palabrasOutput.csv",
firebase: {
credential: "./scripts/serviceAccount.json",
collection: "collectionInFirestore",
},
mapper: (dataFromCSV) => {
return dataFromCSV;
},
};
and run:
csv-to-firestore --config ./scripts/csvToFirestore.js
btw: Generate new private key (it's a file in Service account section in firestore)
put that file in /scripts folder and rename serviceAccount.json