Consider I have an Events collection where it has startTimestamp and endTimestamp indicating when the event starts, ends respectively.
How to query in firestore to find out if the Event is live/finished/upcoming?
If both startTimestamp and endTimestamp properties exist in the database and are of type Date and not String or Number, then you can simply use a query to check if a particular date is within the bounds or not.
For example in Android, if you want to check if a particular date is within the bounds, you might think that a query like the one below will work:
eventsRef.whereGreaterThanOrEqualTo("startTimestamp", yourDate)
.whereLessThanOrEqualTo("endTimestamp", yourDate);
But it won't. You'll get an Exception with the following message:
All where filters other than whereEqualTo() must be on the same field. But you have filters on 'startTimestamp' and 'endTimestamp'
The only solutuin you have is to create three separate queries.
Edit:
According to your comment, one query should check if your yourDate is before startTimestamp
eventsRef.whereLessThanOrEqualTo("startTimestamp", yourDate);
If it is, it means it's an upcoming event.
The second one would be to see if it's grater than the startTimestamp:
eventsRef.whereGreaterThanOrEqualTo("startTimestamp", yourDate);
Where we have two cases. One case, you perform a new (third) query to check if the data is less than endTimestamp:
eventsRef.whereLessThanOrEqualTo("endTimestamp", yourDate);
If it is, it means that the event is within the bounds, so it's a live event otherwise is grater than that which means that the event is finished.
To get that data in realtime, you should use a snapshot listener for every query.
Here are the cases to handle this scenario. I'm pretty sure this is a very common problem but didn't find effective solutions for this anywhere.
Solution 1: Have all documents in a single collection called subscribedEvents
As suggested Alex, We need to do for the following status.
Upcoming : currentTimestamp < startTimestamp
Finished : currentTimestamp > endTimestamp
Live : currentTimestamp > startTimestamp in 1st Query and currentTimestamp < endTimestamp in second query.
Problem : I can have lots of documents (nearly 10,000) in subscribedTimestamp and Live condition is not scalable as I can't limit the results while querying. As it needs to be intersected from the two queries, I need to query with out filters.
Solution 2: This is a bit of hack but scalable. Don't have all the documents in a single subCollection. Separate Upcoming events and put those documents in subscribedEvents/others/Upcoming collection.
When a user subscribes, If its an upcoming event, you can directly store in the subscribedEvents/others/Upcoming collection.
Rest of the documents can go directly into subscribedEvents collection.
Upcoming : Query all the documents with a limit filter from subscribedEvents/others/Upcoming collection.
Finished : currentTimestamp > endTimestamp
Live : currentTimestamp < endTimestamp
The benefit with this structure is we can apply limit filter and lots of documents don't need to be read for your query and there will be only one query required for each status.
Now this step needs additionally a cron job to make sure the upcoming events from the upcoming sub-collection are moved back to subscribedEvents.
However, if you have lesser documents, Solution 1 is the way to go. But not in my case.
Hope it helps someone where they have to scale efficiently.
Related
This is a follow-up/elaboration to a previous question of mine.
In the case of a collection of documents containing a time range represented by two timestamp fields (start and end), how does one go about guaranteeing that two documents don't get added with overlapping time ranges?
Say I had the following JavaScript on form submit:
var bookingsRef = db.collection('bookings')
.where('start', '<', booking.end)
.where('end', '>', booking.start);
bookingsRef.get().then(snapshot => {
// if a booking is found (hence there is an overlap), display error
// if booking is not found (hence there is no overlap), create booking
});
Now if two people were to submit overlapping bookings at the same time, could transactions be used (either on the client or the server) to guarantee that in between the get and add calls no other documents were created that would invalidate the original collection get query where clauses.
Or would my option be using some sort of security create rule that checks for other document time overlaps prior to allowing a new write (if this is at all possible)? One approach to guarantee document uniqueness via security rules seems to be exposing field values in the document ID, but I'm not entirely sure how exposing the start and end timestamp values in the ID would allow a rule to check for overlapping time ranges.
I think transaction is proper approach. According to the documentation:
..., if a transaction reads documents and another client
modifies any of those documents, Cloud Firestore retries the
transaction. This feature ensures that the transaction runs on
up-to-date and consistent data.
This seems to be an answer to your problem. All reads will be retried, if anything will change in the meantime. I think transaction mechanism is exactly for that reason.
I am working on a project that when a user cancels their plan, all their documents should be updated to deactivated except for a pre-defined number of documents that are allowed to stay active. The pre-defined number amount determines the projects allowed to stay active along with the date they were created.
For example, if customer A has 1,000 documents and cancels their plan, all their documents except for the first 100 created should be updated to be deactivated.
My first attempt was to get all document ids with .listDocuments() but I noticed the created date is not part of Firestore's DocumentReference. Therefore I can't exclude the pre-defined number of documents allowed to stay active.
I could use .get() and use the created value, but I'm afraid that getting all the documents at once (which could be a million) would cause my cloud function to run out of memory, even if I have it set to the maximum allowed configuration.
Another option that I thought of, I could use .listDocuments() and write each document id to a temp collection in Firestore, which could kick off a cloud function for each document. This function would only have to work with one document, so it wouldn't have to worry about running out of resources. I am unsure how to determine if the document I'm working on should be marked as deactivated or is allowed to stay active.
I am not that worried about the reads to write as this workflow should not happen very often. Any help would be appreciated.
Thank you
One possible approach would be to mark the documents to be excluded.
I don't know what is your exact algorithm, but if you want to mark the first 100 documents that were created in a collection you can use a Cloud Function that runs for each new document and checks if there are already 100 docs in the collection.
If not, you update a field in this new document with its rank (using a Transaction to get the highest existing rank and increment it). If there are already 100 documents previously created in the collection, you just update the field to 0, for example, in such a way that later on you can query with where("rank", "==", 0).
Then, when you want to delete all the docs but the 100 first ones, just use where("rank", "==", 0) query.
So, concretely:
The first doc is created: you set the rank field to 1.
The Nth doc (N != 1) is created: you fetch all the docs with a query ordered by rank and limited to 1 doc (collecRef.orderBy("rank", "desc").limit(1)) in a Transaction. Since you are in a Cloud Function, you can use a Query in the Transaction (which you cannot with the Client SDKs). Then, still in the Transaction:
If the value of rank for the single doc returned by the Query is < 100 you set the rank value of the newly created do to [single doc value + 1]
If the value of rank for the single doc returned by the Query = 100 you set the rank value to 0
If I didn't make any mistake (I didn't test it! :-)) you end with 100 docs with a value of rank between 1 and 100 (the 100 first created docs) and the rest of the docs with a value of rank equal to 0.
Then, as said above you can use the where("rank", "==", 0) query to select all the docs to be deleted.
At first sight, it's clear what the continuation token does in Cosmos DB: attaching it to the next query gives you the next set of results. But what does "next set of results" mean exactly?
Does it mean:
the next set of results as if the original query had been executed completely without paging at the time of the very first query (skipping the appropriate number of documents)?
the next set of results as if the original query had been executed now (skipping the appropriate number of documents)?
Something completely different?
Answer 1. would seem preferable but unlikely given that the server would need to store unlimited amounts of state. But Answer 2. is also problematic as it may result in inconsistencies, e.g. the same document may be served multiple times across pages, if the underlying data has changed between the page queries.
Cosmos DB query executions are stateless at the server side. The continuation token is used to recreate the state of the index and track progress of the execution.
"Next set of results" means, the query is executed again on from a "bookmark" from the previous execution. This bookmark is provided by the continuation token.
Documents created during continuations
They may or may not be returned depending on the position of insert and query being executed.
Example:
SELECT * FROM c ORDER BY c.someValue ASC
Let us assume the bookmark had someValue = 10, the query engine resumes processing using a continuation token where someValue = 10.
If you were to insert a new document with someValue = 5 in between query executions, it will not show up in the next set of results.
If the new document is inserted in a "page" that is > the bookmark, it will show up in next set of results
Documents updated during continuations
Same logic as above applies to updates as well
(See #4)
Documents deleted during continuations
They will not show up in the next set of results.
Chances of duplicates
In case of the below query,
SELECT * FROM c ORDER BY c.remainingInventory ASC
If the remainingInventory was updated after the first set of results and it now satisfies the ORDER BY criteria for the second page, the document will show up again.
Cosmos DB doesn’t provide snapshot isolation across query pages.
However, as per the product team this is an incredibly uncommon scenario because queries over continuations are very quick and in most cases all query results are returned on the first page.
Based on preliminary experiments, the answer seems to be option #2, or more precisely:
Documents created after serving the first page are observable on subsequent pages
Documents updated after serving the first page are observable on subsequent pages
Documents deleted after serving the first page are omitted on subsequent pages
Documents are never served twice
The first statement above contradicts information from MSFT (cf. Kalyan's answer). It would be great to get a more qualified answer from the Cosmos DB Team specifying precisely the semantics of retrieving pages. This may not be very important for displaying data in the UI, but may be essential for data processing in the backend, given that there doesn't seem to be any way of disabling paging when performing a query (cf. Are transactional queries possible in Cosmos DB?).
Experimental method
I used Sacha Bruttin's Cosmos DB Explorer to query a collection with 5 documents, because this tool allows playing around with the page size and other request options.
The page size was set to 1, and Cross Partition Queries were enabled. Different queries were tried, e.g. SELECT * FROM c or SELECT * FROM c ORDER BY c.name.
After retrieving page 1, new documents were inserted, and some existing documents (including documents that should appear on subsequent pages) were updated and deleted. Then all subsequent pages were retrieved in sequence.
(A quick look at the source code of the tool confirmed that ResponseContinuationTokenLimitInKb is not set.)
I'm struggling to make a (not so) complex query in firebase cloud firestore.
I simply need to get all docs where the id field == a given id.
Then order the results by their date and limit to 10 results.
So this is my current situation
db.firestore().collection('comments')
.where("postId",'==',idOfThePost)
.orderBy('date','asc').limit(10).get().then( snapshot => {
//Nothing happens and the request wasn't even been executed
})
I can get the result only if i don't use the orderBy query but i have to process this sorting for the needs of my application.
Someone has an idea to help me to fix this ?
thanks
You can do this by creating an Index in the firestore.
The first field of the index should be the equality field and the second field of the index should be the order by field.
Given your problem, you would need the following index:
first field: postId, 'asc'
second field: date, 'asc'
Please check the doc. It says
However, if you have a filter with a range comparison (<, <=, >, >=), your first ordering must be on the same field
you can try this code
db.firestore().collection('comments')
.where("postId",'==',idOfThePost)
.orderBy('postId')
.orderBy('date','asc').limit(10).get().then( snapshot => {
.....
})
My Workaround
If you're googling this you've probably realized it can't be done traditionally. Depending on your problem though there may be some viable workarounds, I just finished creating this one.
Scenario
We have an app that has posts that appear in a feed (kind of like Reddit), each post has an algorithmic score 'score' and we needed a way to get the 'top posts' from 12-24 hours ago. Trying to query sorted by 'score' where timestamp uses > and < to build the 12-24 hour ago range fails since Firebase doesn't allow multiple conditional querying or single conditional querying with an descending sort on another field.
Solution
What we ended up doing is using a second field that was an array since you can compound queries for array-contains and descending. At the time a post was made we knew the current hour, suppose it was hour 10000 since the server epoch (i.e. floor(serverTime/60.0/60.0)). We would create an array called arrayOfHoursWhenPostIsTwelveToTwentyFourHoursOld and in that array we would populate the following values:
int hourOffset = 12;
while (hourOffset <= 24) {
[arrayOfHoursWhenPostIsTwelveToTwentyFourHoursOld addObject:#(currentHour+hourOffset)];
hourOffset++;
}
Then, when making the post we would store that array under the field hoursWhenPostIsTwelveToTwentyFourHoursOld
THEN, if it had been, say, 13 hours since the post was made (the post was made at hour 10000) then the current hour would be 10013, so we could use the array-contains query to see if our array contained the value 10013 while also sorting by algorithm score at the same time
Like so:
FIRFirestore *firestore = [Server sharedFirestore];
FIRCollectionReference *collection = [firestore collectionWithPath:#"Posts"];
FIRQuery *query = [collection queryOrderedByField:#"postsAlgorithmScore" descending:YES];
query = [query queryWhereField:#"hoursWhenPostIsTwelveToTwentyFourHoursOld" arrayContains:#(currentHour)];
query = [query queryLimitedTo:numberToLoad];
Almost Done
The above code will not run properly at first since it is using a compound index query, so we had to create a compound index query in firebase, the easiest way to do this is just run the query then look at the error in the logs and firebase SDK will generate a link for you that you can navigate to and it will auto-generate the compound index for your database for you, otherwise you can navigate to firebase>database>index>compound>new and build it yourself using hoursWhenTwelveToTwentyFourHoursOld: Arrays, score: Descending
Enjoy!
same here, it is weird why can't. below is another sample. can't get the result. Hoping firebase can reply about this and update the document.
dbFireStore.collection('room').where('user_id.'+global.obj_user.user_id,'==',true).orderBy('last_update').get().then((qs)=>{
console.log(qs);
});
using other work-around solution is javascript array and array.sort()
I ran into the same issue yesterday on Android. The Callback was just not called. Today I suddenly got an error message. FAILED_PRECONDITION: The query requires an index. It even contains a URL in the error message to generate that index with one click.
It seems that if you want to use orderBy on your data, you need to create an index for that field. The index also needs to be in the correct order (DESC, ASC).
As per firestore document,
If you attempt a compound query with a range clause that doesn't map to an existing index, you receive an error. The error message includes a direct link to create the missing index in the Firebase console.
So just click that link you get in Logcat, it will be redirected to create index page, just create index. It will take some time. after enabling composite index, you will get the result as your requested query.
Stumbled across this looking for help when i found that using the orderBy function didnt work and the documentation still says it does not support it. A bit weird and unclear to be honest, because it does support it so long as you index your Firestore database. For example, this query now works fine for me having set up indexing:
const q = query(docRef, where("category", "==", 'Main'), orderBy('title', 'asc')
Indexing in Firestore
Console Log that even gives you the url to automatically create the index if you try and run with the above command.
Maybe I am missing something, or a later version of Firebase (I am using v9) simply does support it.
From the Transactions doc, second paragraph:
The intention here is for the client to increment the total number of
chat messages sent (ignore for a moment that there are better ways of
implementing this).
What are some standard "better ways" of implementing this?
Specifically, I'm looking at trying to do things like retrieve the most recent 50 records. This requires that I start from the end of the list, so I need a way to determine what the last record is.
The options as I see them:
use a transaction to update a counter each time a record is added, use the counter value with setPriority() for ordering
forEach() the parent and read all records, do my own sorting/filtering at client
write server code to analyze Firebase tables and create indexed lists like "mostRecent Messages" and "totalNumberOfMessages"
Am I missing obvious choices?
To view the last 50 records in a list, simply call "limit()" as shown:
var data = new Firebase(...);
data.limit(50).on(...);
Firebase elements are ordering first by priority, and if priorities match (or none is set), lexigraphically by name. The push() command automatically creates elements that are ordered chronologically, so if you're using push(), then no additional work is needed to use limit().
To count the elements in a list, I would suggest adding a "value" callback and then iterating through the snapshot (or doing the transaction approach we mention). The note in the documentation actually refers to some upcoming features we haven't released yet which will allow you to count elements without loading them first.