I am using Firedatabase with Unity3d.
My Issue is the user search for data, and I retrieve only 50 result each time,
It is easy to get first 50 record
Database.RootReference.Child("ID").OrderByChild(AppSettings.UserFullNameKeyLowCase).StartAt(_search.ToLower()).LimitToLast(50);
The problem is getting the next 50 requires to have multiple StartAt, which firebase dose not support.
I did a lot of research and it seems its impossible to do that in a direct way.
My question Is there any way to do that, any alternative or third party tool to do that,
Do you need to get 50 entries at a time from a specific entry?
You could use your own code to get the first 50 results; record the last one and use that variable as the StartAt parameter for the next search.
You could use for loops to download the next 50 data each time.
Related
Let's say i have an multi-restaurant food order app.
I'm storing orders in Firestore as documents.
Each order object/document contains:
total: double
deliveredByUid: str
restaurantId: str
I wanna see anytime during the day, the totals of every Driver to each Restaurant like so:
robert: mcdonalds: 10
kfc: 20
alex: mcdonalds: 35
kfc: 10
What is the best way of calculating the totals of all the orders?
I currently thinking of the following:
The safest and easiest method but expensive: Each time i need to know the totals i just query all the documents in that day and calculate them 1 by 1
Cloud Functions method: Each time an order has been added/removed modify a value in a Realtime database specific child: /totals/driverId/placeId
Manual totals: Each time a driver complete an order and write its id to the order object, make another write to the Realtime database specific child.
Edit: added the whole order object because i was asked to.
What I would most likely do is make sure orders are completely atomic (or as atomic as they can be). Most likely, I'd perform the order on the client within a transaction or batch write (both are atomic) that would not only create this document in question but also update the delivery driver's document by incrementing their running total. Depending on how extensible I wanted to get, I may even create subcollections within the user's document that represented chunks of time if I wanted to be able to record totals by month or year, or whatever. You really want to think this one through now.
The reason I'd advise against your suggested pattern is because it's not atomic. If the operation succeeds on the client, there is no guarantee it will succeed in the cloud. If you make both writes part of the same transaction then they could never be out of sync and you could guarantee that the total will always be accurate.
I am working on a reminder app with FLutter. Is the first time I am working with Firestore. I have checked on google but I could not find the way to do it.
I want to have the option that the reminder can be set on specific date but or multiple days. Like Monday, Saturday and Sunday.
How should I setup the database? I tried with timestamp, but is not what I want. Should be ARRAY?
Cheers
Usually you should write your answer more precise. It' hard to help you because we don't even know what you want to do with your app and the solutions depends on that.
If you need to filter the entries a lot according to those days (Mo, To, Fri) you could save them in an array to filter easy on them. Also if you want to get them together with the document you save in Firestore I would recommended to use an array. That way you won't need to call on a different collection with the days to get them. The bad side of saving in an array is that you can't just update it. You would need to download the whole array, edit it and save the whole array again. But if you just save up to 7 days of a week that won't be a big deal.
On the other side if there is no need to filter on the days and to get them together with the document you could save them in a separate collection or subcollection of your document. With this approach each day of the week would be a document and you could update each of them very easy and even add additional data very easy. The bad side of this approach is that you would need to call that collection of days separately of your event document and cause more reads on Firestore.
As you can see it all depends how you want your app to work and that is the reason you could not find anything on the web. Those kind of questions are to unprecise to be answered as you expect it to be. There is no silver bullet for such kind of database structures and even the same apps with same purpose could have completely different structures and work as expected.
I hope I could at least guide you to a direction so you can go the next step.
I'm trying to use Cosmos DB and I'm having some trouble making a simple count in a collection.
My collection schema is below and I have 80.000 documents in this collection.
{
"_id" : ObjectId("5aca8ea670ed86102488d39d"),
"UserID" : "5ac161d742092040783a4ee1",
"ReferenceID" : 87396,
"ReferenceDate" : ISODate("2018-04-08T21:50:30.167Z"),
"ElapsedTime" : 1694,
"CreatedDate" : ISODate("2018-04-08T21:50:30.168Z")
}
If I run this command below to count all documents in collection, I have the result so quickly:
db.Tests.count()
But when I run this same command but to a specific user, I've got a message "Request rate is large".
db.Tests.find({UserID:"5ac161d742092040783a4ee1"}).count()
In the Cosmos DB documentation I found this cenario and the suggestion is increase RU. Currently I have 400 RU/s, when I increase to 10.000 RU/s I'm capable to run the command with no errors but in 5 seconds.
I already tryed to create index explicity, but it seems Cosmos DB doesn't use the index to make count.
I do not think it is reasonable to have to pay 10,000 RU / s for a simple count in a collection with approximately 100,000 documents, although it takes about 5 seconds.
Count by filter queries ARE using indexes if they are available.
If you try count by filter on a not indexed column the query would not time out, but fail. Try it. You should get error along the lines of:
{"Errors":["An invalid query has been specified with filters against path(s) excluded from indexing. Consider adding allow scan header in the request."]}
So definitely add a suitable index on UserID.
If you don't have index coverage and don't get the above error then you probably have set the enableScanInQuery flag. This is almost always a bad idea, and full scan would not scale. Meaning - it would consume increasingly large amounts of RU as your dataset grows. So make sure it is off and index instead.
When you DO have index on the selected column your query should run. You can verify that index is actually being used by sending the x-ms-documentdb-populatequerymetrics header. Which should return you confirmation with indexLookupTimeInMs and indexUtilizationRatio field. Example output:
"totalExecutionTimeInMs=8.44;queryCompileTimeInMs=8.01;queryLogicalPlanBuildTimeInMs=0.04;queryPhysicalPlanBuildTimeInMs=0.06;queryOptimizationTimeInMs=0.00;VMExecutionTimeInMs=0.14;indexLookupTimeInMs=0.11;documentLoadTimeInMs=0.00;systemFunctionExecuteTimeInMs=0.00;userFunctionExecuteTimeInMs=0.00;retrievedDocumentCount=0;retrievedDocumentSize=0;outputDocumentCount=1;outputDocumentSize=0;writeOutputTimeInMs=0.01;indexUtilizationRatio=0.00"
It also provides you some insight where the effort has gone if you feel like RU charge is too large.
If index lookup time itself is too high, consider if you index is selective enough and if the index settings are suitable. Look at your UserId values and distribution and adjust the index accordingly.
Another wild guess to consider is to check if the API you are using would defer executing find(..) until it knows that count() is really what you are after. It is unclear which API you are using. If it turns out it is fetching all matching documents to client side before doing the counting then that would explain unexpectedly high RU cost, especially if there are large amount of matching documents or large documents involved. Check the API documentation.
I also suggest executing the same query directly in Azure Portal to compare the RU cost and verify if the issue is client-related or not.
I think it just doesn't work.
The index seems to be used when selecting the documents to be counted, but then the count is done by reading each document, so effectively consuming a lot of RU.
This query is cheap and fast:
db.Tests.count({ UserID: { '$eq': '5ac161d742092040783a4ee1' }})
but this one is slow and expensive:
db.Tests.count({ ReferenceID: { '$gt': 10 }})
even though this query is fast:
db.Tests.find({ ReferenceID: { '$gt': 10 }}).sort({ ReferenceID: 1 })
I also found this: https://feedback.azure.com/forums/263030-azure-cosmos-db/suggestions/36142468-make-count-aware-of-indexes. Note the status: "We have started work on this feature. Will update here when this becomes generally available."
Pretty disappointing to be honest, especially since this limitation hasn't been addressed for almost 2 years. Note - I am not an expert in this matter and I'd love to be proven wrong, since I also need this feature.
BTW: I noticed that simple indexes seem to be created automatically for each individual field, so no need to create them manually.
I am building an Ionic application with Firebase Database support. One of my feature of application is polling/voting. I want to refresh my object(i.e make empty to the data) after every 24 hours. I want to achieve this without writing any code to the client side, is it possible that firebase can maintain this routine by it self? Here is the snap, I want to refresh polls object after every 24 hours
After a little search I found Zapier, it can be very easy to operate what you need.
However, I think you should keep your data in the db and change your query to pull data by date.
Also you can use cron-job in any serverside program, to clean the firebase data.
I have posted a question here regarding binding of my gridview - Bind GridView with many records with number of visible records at a moment. The performance hit has come down incredibily to 2sec from 20sec by implementing the answer but still i know it can be lowered to more limit.
Like- When i fetch say 50 records(Customers) at a time from DB, it has one column which fetch number of orders corresponding to that customer from Orders table(just count not complete record). So, it hit 50 queries to a SQL Server to fetch data for 50 customers. Is there a way where i can fetch complete data with a single query??
NOTE- If someone needs to look at a code, do let me know. Hopefuly, the link which i have provided contains enough codebase..
Database Indexing was what i was looking for.