So right now I am currently storing data with a number after it like EmailSubject1, EmailSubject2, etc which obviously is a stupid way of doing things.
I would instead like to store these in arrays like EmailSubject[1], EmailSubject[2] etc.
I spent some time looking over the firebase documentation and searching online but was struggling to find a solution and I am unfamilar in storing things in firebase this way personally.
let z = 0;
for (z = 0; z < 20; z++) {
db.collection("Users")
.doc("6IzsLbD4r4R5RXdGB5BQy6xq8Dc2")
.set(
{
["EmailSubject" + z]: mail.subject,
["EmailBody" + z]: mail.text,
["EmailFrom" + z]: mail.from,
["EmailDate" + z]: newDate,
},
{ merge: true }
)
.then(() => {
console.log("Doc successful");
return null;
})
.catch((error) => {
console.error("Error writing doc", error);
});
}
Here is my code, any help would be much appreciated =]
If I correctly understand your question, you can use arrayUnion() to add elements to the different arrays, but you should note that this method will only add elements that are not already present.
So if you are sure that you only add new data, you could do as follows:
var promises = [];
var docRef = db
.collection('Users')
.doc('6IzsLbD4r4R5RXdGB5BQy6xq8Dc2');
for (z = 0; z < 20; z++) {
promises.push(
docRef.set(
{
EmailSubject: firebase.firestore.FieldValue.arrayUnion(
mail.subject + z
),
EmailBody: firebase.firestore.FieldValue.arrayUnion('text' + z),
//....
},
{ merge: true }
)
);
}
Promise.all(promises);
If, on the opposite, there are values that are similar in one of the Arrays, you would need to read the Array in your client, push the new value and write back the entire Array.
Ex.: If your EmailFrom array is ['john#gmail.com', 'jack#gmail.com', 'john#gmail.com'], arrayUnion() will not work.
Related
I am using a >= query again a collection. To test the script, I just have 4 entries in my collection.
My query is:
...
.where("Workdesc", ">=", "imple") // no returning as expected
.get()
.then(querySnapshot => {
querySnapshot.forEach(function(doc) {
console.log("Result");
console.log(doc.id, " ===> ", doc.data());
});
});
Workdesc of all 4 docs are -
"kj implementation"
"hb implementation urgent"
"sharu implementation quick response needed"
"cb implementation urgent job"
Result according to me, it should have returned all 4 docs but it is returning only 2. I am attaching screenshot of the console log and Firebase console:
How can I get the result back with partial letter anywhere in the string.
Your query is working as expected. When you perform comparisons with strings, they are sorted lexicographically, or in other words, alphabetically. Here's the actual sort order of each value, and where "impl" sorts among them:
"cb implementation urgent job"
"hb implementation urgent"
"impl"
"kj implementation"
"sharu implementation quick response needed"
Alphabetically, you can see that "k" and "s" come after "i". So, those are the only documents you're going to get from a query where Workdesc values are greater than "impl".
If you're trying to do a substring search to find all the Workdesc strings that contain "impl", that's not possible with Firestore. Firestore doesn't offer substring searches. You'll have to find another way (probably mirroring data to another database that supports it).
To build on Doug's answer, unfortunately Firestore does not support the type of string search you are looking to do. A potential solution that does away with text search is that you could create another field on your todo documents that stores whether you're dealing with an "implementation" or not.
For example, if you had a field isImplementation, which would be true for implementation todos and false for those that are not, you could add this field as part of your where clause to your query. This would ensure that you are fetching implementation todos only.
Once again building on #Doug's answer, Firestore is an indexed document database. To query for data, the query must be performed against an index in a single sweep to keep queries performant in the way the database is designed.
Firebase won't index fields that are strings by default because it isn't efficient and it is quite a taxing operation at scale. A different approach is often the best option.
Take for example the following function that splits an input string into searchable parts that can then be added to an index. As the length of the input string grows, the number of substrings contained within grows rapidly.
function shatter(str, minLength = 1) {
let parts = [str]; // always have full string
let i, subLength = minLength;
let strLength = str.length;
while (subLength < strLength) {
for (i = 0; i < (strLength - subLength + 1); i++) {
parts.push(str.substring(i, i + subLength));
}
subLength++;
}
return parts;
}
Here's an interactive snippet demonstrating this:
function shatter(str, minLength = 1) {
let parts = [str]; // always have full string
let i, subLength = minLength;
let strLength = str.length;
while (subLength < strLength) {
for (i = 0; i < (strLength - subLength + 1); i++) {
parts.push(str.substring(i, i + subLength));
}
subLength++;
}
return parts;
}
let str = prompt('Please type out a string to shatter:', 'This is a test string');
let partsOfMin1 = shatter(str, 1);
console.log('Shattering into pieces of minimum length 1 gives:', partsOfMin1);
let partsOfMin3 = shatter(str, 3);
console.log('Shattering into pieces of minimum length 3 gives:', partsOfMin3);
let partsOfMin5 = shatter(str, 5);
console.log('Shattering into pieces of minimum length 5 gives:', partsOfMin5);
alert('The string "' + str + '" can be shattered into as many as ' + partsOfMin1.length + ' pieces.\r\n\r\nThis can be reduced to only ' + partsOfMin3.length + ' with a minimum length of 3 or ' + partsOfMin5.length + ' with a minimum length of 5.');
However using that above function, we can repurpose it so that it saves the shattered pieces to Firestore at /substringIndex/todos/workDesc with a link back to the document containing the string.
const firebase = require('firebase');
firebase.initializeApp(/* config here */);
const arrayUnion = firebase.firestore.FieldValue.arrayUnion;
const TODOS_COL_REF = firebase.firestore().collection('todos');
const SUBSTRING_INDEX_COL_REF = firebase.firestore().collection('substringIndex');
// splits given string into segments ranging from the given minimum length up to the full length
function shatter(str, minLength = 1) {
let parts = [str];
let i, subLength = minLength;
let strLength = str.length;
while (subLength < strLength) {
for (i = 0; i < (strLength - subLength + 1); i++) {
parts.push(str.substring(i, i + subLength));
}
subLength++;
}
return parts;
}
// upload data
const testData = {
workDesc: 'this is a prolonged string to break code',
assignDate: firebase.firestore.Timestamp.fromDate(new Date()),
assignTo: 'Ddy1QVOAO6SIvB8LfAE8Z0Adj4H3',
followers: ['Ddy1QVOAO6SIvB8LfAE8Z0Adj4H3'],
searchArray: ['v1', 'v2']
}
const todoDocRef = TODOS_COL_REF.doc();
const todoId = todoDocRef.id;
todoDocRef.set(testData)
.then(() => console.log('Uploaded test data!'))
.catch((err) => console.error('Failed to test data!', err));
// Note: in this example, I'm not waiting for the above promise to finish
// Normally, you would integrate it into the batched write operations below
// index each desired string field
const indexDocRef = SUBSTRING_INDEX_COL_REF.doc('todos');
const indexedFields = ["workDesc"];
const indexEntryMinLength = 3;
const indexUpdatePromises = indexedFields.map((fieldName) => {
const indexColRef = indexDocRef.collection(fieldName);
const fieldValue = testData[fieldName];
if (typeof fieldValue !== 'string') return Promise.resolve(undefined); // skip non-string values
const parts = shatter(fieldValue, indexEntryMinLength);
console.log('INFO: Consuming ' + (parts.length * 2) + ' write operations to index ' + fieldName);
// Each batched write can handle up to 500 operations, each arrayUnion counts as two
const partsBatches = [];
if (parts.length > 250) {
for (let i = 0; i < parts.length; i += 250) {
partsBatches.push(parts.slice(i, i + 250));
}
} else {
partsBatches.push(parts);
}
const batchCommitPromises = partsBatches
.map((partsInBatch) => {
const batch = firebase.firestore().batch();
partsInBatch.forEach((part) => {
batch.set(indexColRef.doc(part), {ids: arrayUnion(todoId)}, { merge: true })
})
return batch.commit();
});
return Promise.all(batchCommitPromises);
})
Promise.all(indexUpdatePromises)
.then(() => console.log('Uploaded substring index!'))
.catch((err) => console.error('Failed to upload index!', err));
Then when you want to search for all documents containing "impl" you would use the following to get an array of matching document IDs:
firebase.firestore().doc('substringIndex/todos/workDesc/impl').get()
.then(snap => snap.get('ids'))
.then(console.log, console.error)
While the above code works, you will hit your read/write limits quite quickly as you update the index and you will also likely run into concurrency issues. I also consider it fragile in that non-English characters and punctuation will also trip it up - it is included as a demo only. These issues are why the relevant Firebase documentation recommends making use of a third-party search service like Algolia for full-text search.
TL:DR;
The best solution is to have a human-readable form of your data ("sharu implementation quick response needed") and a indexable form of your data ({implementation: true, urgent: true, pending: true}) as covered by #Luis in their answer.
I have two collections. (applyJobs and Jobs and users). When users apply for a job, I store that record inside applyJobs collection. Like this:
applyId:****,
jobId:*****,
userId:*****
Now, I want to show all apply for jobs by a user.
First: Get logged user id, I store locally logged user id. So, I can get loggged user id.
Second: I filter Apply Jobs by that id. like this, var ref = _db.collection('applyJobs').where('userId',isEqualTo: uid);. I here I didn't call users collection to get uid. because I already store uid on locally. Is it best practice?
Third: I store result here List<ApplyJobsModelClass>. I want to get all jobs by a list of id. How do I filter it?
This is way I tried it. But this is not list of IDs. only one id.
streamApplyJob(List<String> jobId) {
Collection('jobs').document(jobId);
}
And I tried this way too.
Stream<List<JobModel>> streamApplyJob(List<String> jobId) {
var ref = _db.collection('jobs').where('jobId',isEqualTo: jobId);
return ref.snapshots().map((list) =>
list.documents.map((doc) => JobModel.fromFirestore(doc)).toList());
}
tried to get length, but result is 0
db.streamApplyJob(jobIds).listen((v)=>{
print(v.length)
});
Full Code
Database side
///Get a stream of apply jobs
Stream<List<ApplyJobModel>> streamApplyJobs(String uid) {
var ref = _db.collection('applyJobs').where('userId',isEqualTo: uid);
return ref.snapshots().map((list) =>
list.documents.map((doc) => ApplyJobModel.fromFirestore(doc)).toList());
}
///Get a stream of a single document
Stream<List<JobModel>> streamApplyJob(List<String> jobId) {
var ref = _db.collection('jobs').where('jobId',isEqualTo: jobId);
return ref.snapshots().map((list) =>
list.documents.map((doc) => JobModel.fromFirestore(doc)).toList());
}
calling
List<String> jobIds = [];
void getData() {
db.streamApplyJobs(widget.uid).listen((listApplies) => {
for (int i = 0; i < listApplies.length; i++)
{jobIds.add(listApplies[i].jobId)},
});
db.streamApplyJob(jobIds).listen((v)=>{
print(v.length)
});
}
Solution(It's working now)- Is it best practice or are there other best way to do this?
Future<List<JobModel>> getJobs() async {
await db.streamJobs(true).listen((jobs) {
setState(() {
jobModel = jobs;
});
});
return jobModel;
}
Future getData() async {
await getJobs();
db.streamApplyJobs(widget.uid).listen((apply) => {
for (int j = 0; j < jobModel.length; j++)
{
for (int i = 0; i < apply.length; i++)
{
if (apply[i].jobId == jobModel[j].jobId)
{
jobModelNew.add(jobModel[j]),
}
}
}
});
}
I want to get all jobs by a list of id. How do I filter it?
There currently is no way to pass in a list of IDs to a Firestore query and get documents matching all those IDs. See Google Firestore - how to get document by multiple ids in one round trip? (which talks about doing this with document IDs), and Firebase Firestore - OR query (which talks about filtering for multiple values on a single field).
Unless your use-case happens to match the workaround mentioned in that second answer, you'll have to perform a separate query for each value, and merge the results in your application code.
Not sure if it is documented anywhere officially, but this is possible now!
.where(admin.firestore.FieldPath.documentId(), "in", [array, of, ids])
Found here: https://stackoverflow.com/a/52252264/10562805
Please take a look at this example. It binds a CollectionReference to a List.
Let me know if this is helpful.
I'm using AsyncStorage in ReactNative to store some data (large size >2MB) on device, and then read it with the following code
try {
const value = await AsyncStorage.getItem('date_stored_copy');
} catch (e) {
console.log(e);
}
I'm getting the following error:
Row too big to fit into CursorWindow requiredPos=0, totalRows=1...
Is there any way to increase CursorWindow size, or another alternative to AsyncStorage ?
An alternative solution would be to split the data into chunks and then writing it.
I write a wrapper that uses AsyncStorage that does exactly that: https://gist.github.com/bureyburey/2345dfa88a31e00a514479be37848d42
Be aware that it was originally written for using with apollo-cache-persist (a persistence lib for apollo-client).
And since graphql store the data in a very flat structure this solution works pretty well out of the box.
For your case, if your stored object looks like this:
{
data: { a lot of data here }
}
Then it wouldn't matter much and the wrapper won't work
But if your object looks like this:
{
someData: { partial data },
someMoreData: { more partial data },
....
}
Then in theory it should work.
Full disclosure: i haven't tested it thoroughly yet and only used it with apollo-cache-persist
I ran into this problem too, here is how I solved this issue :
Basic description of the algorithm :
The "key" holds the number of parts your data will be divided by. (Example : key is "MyElementToStore", its value is 7 for the number of parts your data needs to be split by to fit each part in a row of the AsyncStorage)
Each part will then be stored as an individual row in the AsyncStorage by having the name of the key followed by the index of the part. (Example : ["MyElementToStore0", "MyElementToStore1", ...]
Retrieving data works the other way around, each row is retrieved and aggregated to the result to return
Final note for clearing the store, it's important to remove each part before removing the key (use the last function "clearStore" to make sure you release memory correctly)
AsyncStorage documentation
import AsyncStorage from "#react-native-async-storage/async-storage";
const getStore = async (key) =>
{
try
{
let store = "";
let numberOfParts = await AsyncStorage.getItem(key);
if(typeof(numberOfParts) === 'undefined' || numberOfParts === null)
return null;
else
numberOfParts = parseInt(numberOfParts);
for (let i = 0; i < numberOfParts; i++) { store += await AsyncStorage.getItem(key + i); }
if(store === "")
return null;
return JSON.parse(store);
}
catch (error)
{
console.log("Could not get [" + key + "] from store.");
console.log(error);
return null;
}
};
const saveStore = async (key, data) =>
{
try
{
const store = JSON.stringify(data).match(/.{1,1000000}/g);
store.forEach((part, index) => { AsyncStorage.setItem((key + index), part); });
AsyncStorage.setItem(key, ("" + store.length));
}
catch (error)
{
console.log("Could not save store : ");
console.log(error.message);
}
};
const clearStore = async (key) =>
{
try
{
console.log("Clearing store for [" + key + "]");
let numberOfParts = await AsyncStorage.getItem(key);
if(typeof(numberOfParts) !== 'undefined' && numberOfParts !== null)
{
numberOfParts = parseInt(numberOfParts);
for (let i = 0; i < numberOfParts; i++) { AsyncStorage.removeItem(key + i); }
AsyncStorage.removeItem(key);
}
}
catch (error)
{
console.log("Could not clear store : ");
console.log(error.message);
}
};
I found another alternative mentioned here
Just install react-native-fs-store
npm i react-native-fs react-native-fs-store
react-native link react-native-fs
And use it like this:
import Store from "react-native-fs-store";
const AsyncStorage = new Store('store1');
it has has exactly same API as that of AsyncStorage, so no code changes are required
** Please notice that react-native-fs-store is slower than AsyncStorage, as each operation is synced to file. So you may notice lag (unresponsive screen) while reading/writing data
android/app/src/main/java/com/tamotam/mainApp/MainApplication.java
import android.database.CursorWindow;
import java.lang.reflect.Field;
...
#Override
public void onCreate() {
...
try {
Field field = CursorWindow.class.getDeclaredField("sCursorWindowSize");
field.setAccessible(true);
field.set(null, 100 * 1024 * 1024); // the 100MB is the new size
} catch (Exception e) {
e.printStackTrace();
}
}
Fixed the issue for me, remember to include the 2 imports!
As per https://github.com/andpor/react-native-sqlite-storage/issues/364#issuecomment-665800433 there might be an addition check if (DEBUG_MODE)... in some solutions, but it caused Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0. in my case.
For example, I can remove document x from collection x, then remove document y from collection y, but if something goes wrong, rollback everything. Based on the documentation, DocumentationReference.delete() is basically the only way to delete a document.
There is a Transaction.delete(DocumentReference) method to delete a document in a transaction.
So to transactionally delete all cities with a population < 100K (modified from the example in the documentation):
var citiesRef = db.collection("cities");
db.runTransaction(function(transaction) {
var count = 0;
return transaction.get(citiesRef).then(function(querySnapshot) {
querySnapshot.forEach(function(doc) {
if (doc.data().population <= 1000000) {
transaction.delete(doc.ref);
count = count + 1;
}
});
});
}).then(function(count) {
console.log("Deleted cities ", count);
}).catch(function(err) {
console.error(err);
});
I use the batch method, as in:
let batch = db.batch()
batch.deleteDocument(documentXref)
batch.deleteDocument(documentYref)
batch.commit() { error in
if let error = error {
print(error.localizedDescription)
}
}
Note that this works even when offline. Thus if you want the operation to fail in that case, you should use the transaction approach.
I use Ionic 2 with Angular 2 in my project. In the root component you can click a "Add" button to add a new Report via a complex form and a lot of preprovided data (there are some selects that are feeded with data fetched from sqlite database)
Now in my "CreateReportComponent" i have the following constructor to load the data and assign it to local array variable:
selectEmployeeOptions: Employee[];
constructor(private dbService: DatabaseService) {
dbService.getAllEmployees().then(employees => {
this.selectEmployeeOptions = employees;
});
// load more data like tasks etc.
});
But when I want to modify this data in my component, the array is empty. I tried to do it in ngOnInit() but this seems to be to early as well.
I want to to something like this, before the component gets displayed:
dbService.getAllEmployees().then(employees => {
this.selectEmployeeOptions = employees;
// modify data
this.selectEmployeeTitleOptions = employees.map((item) => {
return item.title;
});
console.log(JSON.stringify(this.selectEmployeeTitleOptions)) // --> empty
});
But selectEmployeeTitleOptions is empty...
The function in the databaseService looks like this:
getAllEmployees(): Promise<Emplyoee[]> {
let query = "SELECT * FROM employees";
let employeeList = [];
this.database.executeSql(query, []).then((data) => {
if(data.rows.length > 0) {
let e = new Employee();
e.id = data.rows.item(i).id;
e.firstname = data.rows.item(i).firstname;
e.lastname = data.rows.item(i).lastname;
employeeList.push(e);
}
}, (error) => {
// handle error
});
return Promise.resolve(employeeList);
}
I read that there is the Resolve pattern (https://blog.thoughtram.io/angular/2016/10/10/resolving-route-data-in-angular-2.html) But I need to make multiple calls and not only for contacts as in the example.
So the question: How to wait for multiple calls to database?
i think something go wrong here
getAllEmployees(): Promise<Emplyoee[]> {
let query = "SELECT * FROM employees";
let employeeList = [];
this.database.executeSql(query, []).then((data) => {
if(data.rows.length > 0) {
let e = new Employee();
e.id = data.rows.item(i).id;
e.firstname = data.rows.item(i).firstname;
e.lastname = data.rows.item(i).lastname;
employeeList.push(e);
}
}, (error) => {
// handle error
});
return Promise.resolve(employeeList);
}
first return Promise.resolve(employeeList); will return empty array, because it is async process.
you need loop through data.rows, then format return data like this.
getAllEmployees(): Promise<Employee[]> {
let query = "SELECT * FROM employees";
return this.database.executeSql(query, []).then((data) => {
let arr = [];
for(let i = ; i < data.rows.length; ++i) {
let emp = data.rows.item(i);
let e = new Employee();
e.id = emp.id;
e.firstname = emp.firstname;
e.lastname = emp.lastname;
arr.push(e);
}
return arr;
});
}
note that .then() return a promise object.
What you are looking for is forkJoin method that returns Observable that you should switch to instead of using Promises, for reference about why you should do this check here.
Short information about fork join from its GitHub page:
Runs all observable sequences in parallel and collect their last elements.
This way you can safely make parallel requests to your API.
For more information regarding forkJoin go here.
Additionally you should call services using ngOnInit as you mentioned before. For more information about Angular 2 lifecycle hooks see the docs.
You can use Promise.all
https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Global_Objects/Promise/all
You push all promises to an array, and then go
let foo : [Promise<Emplyoee[]>,Promise<void>] = [getAllEmployees(), method2()];
Promise.all(foo).then((results:any[]) => {
let employeearray: any = results[0];
/* and on an on */
});