I’m working on an app with a Firebase backend. During sign up I would like to let new users see which of their contacts are already on the app to add them as friends. So basically, use phone numbers to match users with contacts.
I am having a big performance headache when querying the database to find users.
Since Firestore does not support OR queries, I run two queries per phone number (one to check national format, the other for international format), and if any returns a document, set that document as the found user:
findUserByPhoneNumber = (number, callback) => {
//utility function to, well, sanitize phone numbers
sanitizeNumber = (str) => {
if (str) {
var num = str.match(/\d/g);
num = num.join("");
return num;
} else {
return null
}
}
var foundUser = null
Promise.all([
usersRef.where('phoneNumbers.nationalFormat', '==', sanitizeNumber(number)).get()
.then(snapshot => {
if (snapshot.docs.length > 0 && snapshot.docs[0].data()) {
// console.log('nationalFormat result: ', snapshot.docs[0]);
foundUser = snapshot.docs[0].data()
}
return foundUser
}),
usersRef.where('phoneNumbers.internationalFormat', '==', sanitizeNumber(number)).get()
.then(snapshot => {
if (snapshot.docs.length > 0 && snapshot.docs[0].data()) {
// console.log('internationalFormat result: ', snapshot.docs[0]);
foundUser = snapshot.docs[0].data()
}
return foundUser
})
])
.then(results => {
res = results.filter(el => { return el != null })
if (results.length > 0) {
callback(res[0])
}
})
}
findUserByPhoneNumber runs for each contact in a loop. When testing on my phone with 205 contacts, the whole process takes about 30 seconds, which is about 29 seconds longer than I would like, especially given the test database has only 8 records...
getContacts = () => {
getCs = () => {
// Declare arrays
const contactsWithAccount = []
const contactsWithNoAccount = []
// Get contacts from user's phone
Contacts.getAll((err, contacts) => {
if (err) throw err
// For each contact, iterate
for (var i = 0; i < contacts.length; i++) {
const item = contacts[i]
if (item.phoneNumbers && item.phoneNumbers.length > 0) {
const phone = item.phoneNumbers[0].number
// If the sanitized phone number is different from the current user's phone number (saved in DB), run the following logic
if (this.state.user.phoneNumbers.nationalFormat != sanitizeNumber(phone)
&& this.state.user.phoneNumbers.internationalFormat != sanitizeNumber(phone)
) {
findUserByPhoneNumber(phone, (fu) => {
contactObject = {
key: item.recordID,
name: item.givenName,
normalizedName: item.givenName.toLowerCase(),
phoneNumber: phone,
user: this.state.user,
hasAccount: null,
friendId: null,
isFriend: null
}
const foundUser = fu
// if found user, push in contactsWithAccount, otherwise push in contactsWithNoAccount
if (foundUser && foundUser._id != this.state.user._id) {
contactObject.hasAccount = true
contactObject.friendId = foundUser._id
if (this.state.user.friends && this.state.user.friends.includes(foundUser._id)) {
contactObject.isFriend = true
}
contactsWithAccount.push(contactObject)
}
else {
contactsWithNoAccount.push(contactObject)
}
// if the two arrays are filled up, run the callback
// NOTE_1: we use the two lengths +1 to account for the current
// user's document that we skip and dont add to any of the arrays
// NOTE_2: this bizare method was the only way to handle the results
// coming in asynchronously
if (contactsWithAccount.length + contactsWithNoAccount.length + 1 == contacts.length) {
console.log('finished');
sortCs(contactsWithAccount, contactsWithNoAccount)
}
})
}
}
}
})
}
// sorts the two arrays alphabetically
sortCs = (withAccount, withNoAccount) => {
compare = (a,b) => {
if (a.name < b.name)
return -1;
if (a.name > b.name)
return 1;
return 0;
}
withAccount.sort(compare)
withNoAccount.sort(compare)
this.setState({ withAccount, withNoAccount })
}
// unleash the monster
getCs(sortCs)
}
I am sure the process could be optimized in various ways. Maybe:
different database structure
bundling all queries into one
better use
of async
starting the process at an earlier step in the signup flow
Whatsapp, HouseParty and a bunch of other apps have this feature in place and it loads instantly. I’m not trying to reach that level of perfection yet but there must be some better way…
Any help/suggestions would be greatly appreciated.
Related
I'm trying to do a pagination where the user can see each button's page number in the UI. I'm using Firestore and Buefy for this project.
My problem is that Firestore is returning wrong queries for this case. Sometimes (depending the page that the users clicks on) It works but sometimes don't (It returns the same data of the before page button).
It's really messy I don't understand what's going on. I'll show you the code:
Vue component: (pay attention on the onPageChange method)
<template>
<div>
<b-table
:data="displayData"
:columns="table.columns"
hoverable
scrollable
:loading="isLoading"
paginated
backend-pagination
:total="table.total"
:per-page="table.perPage"
#page-change="onPageChange">
</b-table>
</div>
</template>
<script>
import { fetchBarriosWithLimit, getTotalDocumentBarrios, nextBarrios } from '../../../../firebase/firestore/Barrios/index.js'
import moment from 'moment'
const BARRIOS_PER_PAGE = 5
export default {
data() {
return {
table: {
data: [],
columns: [
{
field: 'name',
label: 'Nombre'
},
{
field: 'dateAddedFormatted',
label: 'Fecha añadido'
},
{
field: 'totalStreets',
label: 'Total de calles'
}
],
perPage: BARRIOS_PER_PAGE,
total: 0
},
isLoading: false,
lastPageChange: 1
}
},
methods: {
onPageChange(pageNumber) {
// This is important. this method gets fired each time a user clicks a new page. I page number that the user clicks.
this.isLoading = true
if(pageNumber === 1) {
console.log('show first 5...')
return;
}
const totalPages = Math.ceil(this.table.total / this.table.perPage)
if(pageNumber === totalPages) {
console.log('show last 5...')
return;
}
/* Here a calculate the next starting point */
const startAfter = (pageNumber - 1) * this.table.perPage
nextBarrios(this.table.perPage, startAfter)
.then((querySnap) => {
this.table.data = []
this.buildBarrios(querySnap)
console.log('Start after: ', startAfter)
})
.catch((err) => {
console.err(err)
})
.finally(() => {
this.isLoading = false
})
},
buildBarrios(querySnap) {
querySnap.docs.forEach((docSnap) => {
this.table.data.push({
id: docSnap.id,
...docSnap.data(),
docSnapshot: docSnap
})
});
}
},
computed: {
displayData() {
let data = []
this.table.data.map((barrioBuieldedObj) => {
barrioBuieldedObj.dateAddedFormatted = moment(Number(barrioBuieldedObj.dateAdded)).format("DD/MM/YYYY")
barrioBuieldedObj.totalStreets ? true : barrioBuieldedObj.totalStreets = 0;
data.push(barrioBuieldedObj)
});
return data;
}
},
mounted() {
// obtener primer paginacion y total de documentos.
this.isLoading = true
getTotalDocumentBarrios()
.then((docSnap) => {
if(!docSnap.exists || !docSnap.data().totalBarrios) {
// mostrar mensaje que no hay barrios...
console.log('No hay barrios agregados...')
this.table.total = 0
return;
}
const totalBarrios = docSnap.data().totalBarrios
this.table.total = totalBarrios
if(totalBarrios <= BARRIOS_PER_PAGE) {
return fetchBarriosWithLimit(totalBarrios)
} else {
return fetchBarriosWithLimit(BARRIOS_PER_PAGE)
}
})
.then((querySnap) => {
if(querySnap.empty) {
// ningun doc. mostrar mensaje q no hay barrios agregados...
return;
}
this.buildBarrios(querySnap)
})
.catch((err) => {
console.error(err)
})
.finally(() => {
this.isLoading = false
})
}
}
</script>
<style lang="scss" scoped>
</style>
The nextBarrios function:
function nextBarrios(limitNum, startAtNum) {
const query = db.collection('Barrios')
.orderBy('dateAdded')
.startAfter(startAtNum)
.limit(limitNum)
return query.get()
}
db is the result object of calling firebase.firestore(). Can I tell a query to start at a certain number where number is the index position of the document within a collection? If not, How could I approach this problem?
Thank you!
Firestore doesn't support offset or index based pagination. It's also not possible to tell how many documents the entire query would return without actually reading them all. So, unfortunately, what you're trying to do isn't possible with Firestore.
It seems also that you're misunderstanding how the pagination APIs actually work. startAfter doesn't take an index - it takes either a DocumentSnapshot of the last document in the prior page, or a value of the ordered field that you used to sort the query, again, the last value you saw in the prior page. You are basically going to use the API to tell it where to start in the next page of results based on what you found in the last page. That's what the documentation means when it says you are working with a "query cursor".
I'm trying to query an empty firebase list. The problem is that the observable method subscribe never finish and I can't show to user that ddbb list is empty.
The function getUserAppointmentsByDate(...) is calling getUserAppointments(...), where this.database.list('/appointment/users/' + user_uid) is an empty firebase list for the input user (user_uid).
how should I manage an empty query to firebase?
thanks in advance!
getUserAppointmentsByDate(user_uid: string, start: string, end: string) {
if (typeof (user_uid) == "undefined" || typeof (start) == "undefined" || typeof (end) == "undefined") {
console.error("invalid argument for getPatientReport");
return;
}
return this.getUserAppointments(user_uid)
.map(
(appointment) => {
return appointment
.filter((appointment) => {
var appointmentStart = new Date(appointment.start);
var startFilter = new Date(start);
var endFilter = new Date(end);
//Filter old, not cancelled and not deleted
return (appointmentStart.getTime() < endFilter.getTime())
&& (appointmentStart.getTime() > startFilter.getTime())
&& (appointment.status != AppointmentStatus.CANCELLED);
});
})
}
getUserAppointments(user_uid: string): any {
return this.database.list('/appointment/users/' + user_uid) //*THIS IS AN EMPTY LIST
.mergeMap((appointments) => {
return Observable.forkJoin(appointments.map(
(appointment) => this.database.object('/appointment/list/' + appointment.$key)
.take(1)))
})
}
As the this.database.list('/appointment/users/' + user_uid) return a empty array. Observable.forkJoin(appointments.map( complete without emit any value (that is the expected way of forkJoin works). In this case, you have two options, handling in the complete function.
.subscribe(
res => console.log('I got values'),
err => console.log('I got errors'),
// do it whatever you want here
() => console.log('I complete with any values')
)
or handle in an if statement:
import { of } from 'rxjs/observable/of';
...
return this.database.list('/appointment/users/' + user_uid)
.mergeMap((appointments) => {
if (appointments.length === 0) return of([]);
return Observable.forkJoin(appointments.map(
(appointment) => this.database.object('/appointment/list/' + appointment.$key)
.take(1)))
})
I am using material autocomplete in a project but unlike their example, I'm pulling from cloud firestore database (beta).
I previously set up a service to handle retrieving the data from json server and this worked fine.
ngOnInit() {
this.branches = this.branchService.get_branches();
...
}
Since moving to the firebase I'm successfully able to display my data but typeahead functionality is not working as expected.
I've tried using both valueChanges() and snapshotChanges() but neither appears to make a difference and I don't know why.
component.ts file:
branches: any = [];
branchCtrl: FormControl = new FormControl();
filteredBranches: Observable<any[]>;
branchCol: AngularFirestoreCollection<Branch>;
branch$: Observable<Branch[]>;
constructor( private afs: AngularFirestore ) {
}
ngOnInit() {
this.branchCol = this.afs.collection('branches');
//this.branch$ = this.branchCol.valueChanges();
this.branch$ = this.branchCol.snapshotChanges().map(actions => { return actions.map(action => { const data = action.payload.doc.data() as Branch; const id = action.payload.doc.id; return { id, ...data }; }); });
this.branches = this.branch$;
this.filteredBranches = this.branchCtrl.valueChanges
.startWith(null)
//.map(b => b && typeof b === 'object' ? b.name : b)
.switchMap(val => {
return this.filterBranches(val || '')
});
}
displayFn(b): string {
return b ? b.name : b;
}
filterBranches(val: string) {
return this.branches
.map(response => response.filter(option => {
return option.name.toLowerCase().indexOf(val.toLowerCase()) === 0
}));
}
getBranch(value){
console.log('branch selected');
}
My assumption is filteredBranches is not able to map the value changes correctly due to collection/document data structure of firebase. Any help would greatly be appreciated.
I'm using Node/Puppeteer in the code below, passing in a large list of URL's for traversal and scraping. It has been difficult to do it asynchronously, though I find that I am getting closer and closer to the answer. I am currently stuck on an issue related to the following error.
UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 17): Error: Protocol error (Target.createTarget): Target closed.
This error occurs once upon every iteration of the while loop. Though I'm not sure what I may be doing incorrectly.
Could someone help me do the following:
1) Diagnose the source of the error.
2) Potentially find a more effective way to traverse a large list of URLs asynchronously.
async function subProc(list, batchSize) {
let subList = null;
let i = 0;
while (list.length > 0) {
let browser = await puppeteer.launch();
subList = list.splice(0, batchSize);
console.log("Master List Size :: " + list.length);
console.log("SubList Size :: " + subList.length);
for (let j = 0; j < subList.length; j++) {
promiseArray.push(new Promise((resolve, reject) => {
resolve(pageScrape(subList[j], browser));
}));
}
Promise.all(promiseArray)
.then(response => {
procArray.concat(response);
});
promiseArray = new Array();
try {
await browser.close();
} catch(ex){
console.log(ex);
}
};
}
async function pageScrape(url, browser) {
let page = await browser.newPage();
await page.goto(url, {
timeout: 0
});
await page.waitFor(1000);
return await page.evaluate(() => {
let appTitle = document.querySelector('').innerText;
let companyName = document.querySelector('').innerText;
let dateListed = document.evaluate("", document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue.innerText;
let category = document.evaluate("']//a//strong", document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue.innerText;
/* */
return {
appTitle,
companyName,
dateListed,
category
}
}).then(response => {
let urlData = {
id: subList[j],
appName: response.appTitle,
companyName: response.companyName,
dateListed: response.dateListed,
category: response.category
}
return urlData;
});
};
I figured out the solution to the problem I was having.
Every computer is limited in its processing ability, so instead of iterating through 1000 urls simultaneously you have to break it down into smaller pieces.
By using a PromiseAll, and iterating and scraping 10 urls at a time and storing these values in an array, I was able to throttle the processing required to iterate through all 1000 urls.
processBatch(subData, 10, procArray).then((processed)=>{
for(let i = 0; i < procArray.length; i++){
for(let j = 0; j < procArray[i].length; j++){
results.push(procArray[i][j]);
}
}
function processBatch(masterList, batchSize, procArray){
return Promise.all(masterList.splice(0, batchSize).map(async url =>
{
return singleScrape(url)
})).then((results) => {
if (masterList.length < batchSize) {
console.log('done');
procArray.push(results);
return procArray;
} else {
console.log('MasterList Size :: ' + masterList.length);
procArray.push(results);
return processBatch(masterList, batchSize, procArray);
}
})
}
I’m having issues getting two dependant types of data from a PouchDB database.
I have a list of cars that I get like so:
localDB.query(function(doc) {
if (doc.type === ‘list’) {
emit(doc);
}
}, {include_docs : true}).then(function(response) {
console.log(“cars”, response);
// Save Cars List to app
for(var i = 0; i < response.rows.length; i++) {
addToCarsList(response.rows[i].id, response.rows[i].carNumber);
}
console.log(“Cars List: " + carsListToString());
return response;
}).then(function(listRecord) {
listRecord.rows.forEach(function(element, index){
console.log(index + ' -> ', element);
localDB.query(function(doc) {
console.log("filtering with carNb = " + element.carNb);
if (doc.type === 'defect' && doc.listId == getCurrentListId() && doc.carNb == element.carNb ) {
emit(doc);
}
}, {include_docs : false}).then(function(result){
console.log("defects", result);
}).catch(function(err){
console.log("an error has occurred", err);
});
});
}).catch(function(err) {
console.log('error', err);
});
Here's what happens. After getting the list of cars, then for each cars I would like to query the defects and store then in some arrays. Then when all that querying is done, I want to build the UI with the data saved.
But what's happening is that the forEach gets processed quickly and does not wait for the inner async'd localDb.query.
How can I query some documents based on an attribute from a parent query? I looked into promises in the PouchDB doc but I can't understand how to do it.
(please forget about curly quotes and possible lint errors, this code was anonymized by hand and ultra simplified)
The method you are looking for is Promise.all() (execute all promises and return when done).
However, your query is already pretty inefficient. It would be better to create a persistent index, otherwise it has to do a full database scan for every query() (!). You can read up on the PouchDB query guide for details.
I would recommend installing the pouchdb-upsert plugin and then doing:
// helper method
function createDesignDoc(name, mapFunction) {
var ddoc = {
_id: '_design/' + name,
views: {}
};
ddoc.views[name] = { map: mapFunction.toString() };
return ddoc;
}
localDB.putIfNotExists(createDesignDoc('my_index', function (doc) {
emit([doc.type, doc.listId, doc.carNb]);
})).then(function () {
// find all docs with type 'list'
return localDB.query('my_index', {
startkey: ['list'],
endkey: ['list', {}],
include_docs: true
});
}).then(function (response) {
console.log("cars", response);
// Save Cars List to app
for(var i = 0; i < response.rows.length; i++) {
addToCarsList(response.rows[i].id, response.rows[i].carNumber);
}
console.log("Cars List: " + carsListToString());
return response;
}).then(function (listRecord) {
return PouchDB.utils.Promise.all(listRecord.rows.map(function (row) {
// find all docs with the given type, listId, carNb
return localDB.query('my_index', {
key: ['defect', getCurrentListId(), row.doc.carNb],
include_docs: true
});
}));
}).then(function (finalResults) {
console.log(finalResults);
}).catch(function(err){
console.log("an error has occurred", err);
});
I'm using a few tricks here:
emit [doc.type, doc.listId, doc.carNb], which allows us to query by type or by type+listId+carNb.
when querying for just the type, we can do {startkey: ['list'], endkey: ['list', {}]}, which matches just those with the type "list" because {} is the "higher" than strings in CouchDB object collation order.
PouchDB.utils.Promise is a "hidden" API, but it's pretty safe to use if you ask me. It's unlikely we'll change it.
Edit Another option is to use the new pouchdb-find plugin, which offers a simplified query API designed to replace the existing map/reduce query() API.
Another approach would be to pull both the list docs and the defect docs down at the same time then merge them together using a reduce like method that will convert them into an array of objects:
{
_id: 1,
type: 'list',
...
defects: [{
type: 'defect'
listId: 1
...
}]
}
By pulling the list and the defects down in one call you save a several calls to the pouchdb query engine, but you do have to iterate through every result to build your collection of lists objects with and embedded array of defects.
// This is untested code so it may not work, but you should get the idea
var _ = require('underscore');
// order documents results by list then defect
var view = function (doc) {
if (doc.type === 'list') {
emit([doc._id, doc.carNumber, 1);
} else if (doc.type === 'defect') {
emit([doc.listId, doc.carNb, 2])
}
}
localDB.query(view, { include_docs: true })
.then(function(response) {
return _(response.rows)
.reduce(function(m, r) {
if (r.key[2] === 1) {
// initialize
r.doc.defects = [];
m.push(r.doc)
return m;
}
if (r.key[2] === 2) {
var list = _(m).last()
if (list._id === r.key[0] && list.carNumber === r.key[1]) {
list.defects.push(r.doc);
}
return m;
}
}, []);
})
.then(function(lists) {
// bind to UI
});
With couch, we found reducing calls to the couch engine to be more performant, but I don't know if this approach is better for PouchDB, but this should work as a solution, especially if you are wanting to embed several collections into one list document.