Is it possible to filter a Firestore query using the server's timestamp? When trying to perform a query such as:
firebase.app.firestore()
.collection('posts')
.where('timestamp', '<=', firebase.firestore.FieldValue.serverTimestamp())
The following error is thrown.
FirebaseError: Function Query.where() called with invalid data. FieldValue.serverTimestamp() can only be used with update() and set()
I know that I can use new Date() instead of FieldValue.serverTimestamp(), but it would be nice if there was a way to query using the trusted server timestamp. My use case is a collection of discount codes with expiry dates, where the expiry dates are saved as Firestore timestamps. If a user changes their system time this query can potentially return invalid (expired) results if I rely on a client-side date for filtering. I'm using Firebase functions to actually process the discount (where it is easy to validate the expiry using the server timestamp), so users with incorrect system times would still not be able to actually use the discount code. Nonetheless it would be nice to guarantee the time in such a query to never show an expired code in the first place.
FieldValue.serverTimestamp() is a flag, not a real value - it instructs FireeStore to use the current server time as a value during a write operation. It is NOT a true "value" that you can use as a substitute for "now".
What you are thinking of (and want to use) is firebase.firestore.Timestamp.now(), (sp?) https://firebase.google.com/docs/reference/js/firebase.firestore.Timestamp , which uses Epoch time (compensating for timezone). Changing their local clock will NOT defeat this - as you may have noticed elsewhere, most browsers and services (including Firestore) need tot local clock to be fairly accurate to maintain operations. If your user/hacker sufficiently hacks their local clock to try to get around your rules, their Firestore service won't be working anyway.
I've been thinking about a problem similar to yours.
Late reply, but I'll write a solution in hopes that it will help
As you know, "firebase.firestore.FieldValue.serverTimestamp()" cannot be used for querying, and it is very uncertain to query with the device's built-in Timestamp. (e.g. things like currentTimestamp(), Date() )
In this case, the solution may be to use the Security Rule in Firestore.
In my case, I used that method to restrict users.
The "suspesnsions_user" collection is used to limit membership due to fraudulent use of users. (User's membership must be suspended before releasedTimestamp.)
match /suspensions_user/{userId} {
allow read : if request.auth.uid == userId && resource.data.releasedTimestamp > request.time
}
If the Security Rule is used as follows, if the user can access the document located in "suspensions_user/{userId}" through the relevant Security Rule, it can be determined that he is in the status of loss of membership.
Conversely, if the user cannot access the document for reasons such as permission denied , it can be determined that he has normal membership.
As you said in the question, accurate validation is possible if you apply the corresponding Security Rule to a document such as "/coupons/{id}" to validate access to coupons.
Here is what I do to keep my date comparisons accurate:
let timestamp = new Date().toISOString();
// "2021-03-01T11:10:51.392Z"
If you do this in your Firebase Function (or in the client), you'll always get UTC time, in ISO format (ISO 8601).
This is great because you can do simple String comparisons of ISO 8601 date strings.
Related
I have a scheduled function that runs every three minutes.
It is supposed to look on the database (firestore), query the relevant users, send them emails or perform other db actions.
Once it sends an email to a user, it updates the user with a field 'sent_to_today:true'.
If sent_to_today == true, the function won't touch that user for around 24 hours, which is what's intended.
But, because I have many users, and the function is doing a lot of work, by the time it updates the user with sent_to_today:true, another invocation gets to that user beforehand and processes them for sending emails.
This results in some users getting the same email, twice.
What are my options to make sure this doesn't happen?
Data Model (simplified):
users (Collection)
--- userId (document)
--- sent_to_today [Boolean]
--- NextUpdateTime [String representing a Timestamp in ISO String]
When the function runs, if ("Now" >= NextUpdateTime) && (sent_to_today==false), the user is processed, otherwise, they're skipped.
How do I make sure that the user is only processed by one invocation per day, and not many?
As I said, by the time they're processed by one function invocation (which sets "sent_to_today" to true), the next invocation gets to that user and processes them.
Any help in structuring the data better or using any other logical method would be greatly appreciated.
Here is an idea I'm considering:
Each invocation sets a global document's field, "ex: busy_right_now : true" at the start, and when finished it sets it again to false. If a subsequent invocation runs before the current is finished, it does nothing if busy_right_now is still true.
Option 1.
Do you think you the function can be invoked once in ten minutes, rather every three minutes? If yes - just modify the scheduler, and make sure that 'max instances' attribute is '1'. As the function timeout is only 540 seconds, 10 minutes (600 seconds) is more than enough to avoid overlapping.
Option 2.
When a firestore document is chosen for processing, the cloud function modifies some attribute - i.e. __state - and sets its value to IN_PROGRESS for example. When the processing is finished (email is sent), that attribute value is modified again - to DONE for example. Thus, if the function picks up a document, which has the value IN_PROGRESS in the __state attribute - it simply ignores and continues to the next one.
The drawback - if the function crashes - there might be documents with IN_PROGRESS state, and there should be some mechanism to monitor and resolve such cases.
Option 3.
One cloud function runs through the firestore collection, and for each document, which is to be processed - sends a pubsub message which triggers another cloud function. That one works only with one firestore document. Nevertheless the 'state machine' control is required (like in the Option 2 above). The benefit of the option 3 - higher level of specialisation between functions, and there may be many 'second' cloud functions running in parallel.
I am using this to get the timestamp
admin.database.ServerValue.TIMESTAMP
But in log getting this while doing console the variable
{ '.sv': 'timestamp' }
anyone help me out in this.
Actually i want to get the timestamp then compare it with db timestamp
The admin.database.ServerValue.TIMESTAMP does not contain the actual server-side timestamp, but is merely a marker (the { '.sv': 'timestamp' } that you see). The database server recognizes this marker on write operations, and then writes the server-side timestamp in its place.
This means that you can't get the server-side timestamp without writing to the database. A simple way to see how to get this is:
let ref = firebase.database().ref("test");
ref.on("value", function(snapshot) {
console.log(snapshot.val());
})
ref.set(admin.database.ServerValue.TIMESTAMP)
When you run this code, your log will show three values:
null
This is the current value in the database when you attach the listener with on("value". Here I'm assuming the test node didn't exist yet, so the value would be null.
1573659849577
This is an estimate that the client makes when you the ref.set(...) statement executes. So the client estimates what it thinks the server timestamp may be, and fires a value event. You can use this to update the UI immediately, so that the user doesn't have to wait.
1573659859162
This is the value that the server actually wrote to the database, and then sent back to the client. So this is the actual server-side timestamp that you're looking for.
In theory the client-side estimate (2) and server-side value (3) may be the same, in which case you wouldn't get the third event. But I've never seen that in practice, as they're always off by at least a couple of milliseconds.
I'm trying to update the same document which triggered an onUpdate cloud function, with a read value from the same collection.
This is in a kind of chat app made in Flutter, where the previous response to an inquiry is replicated to the document now being updated, for easier showing in the app.
The code does work, however when a user quickly responds to two separate inquiries, they both read the same latest response thus setting the same previousResponse. This must be down to the asynchronous nature of flutter and/or the cloud function, but I can't figure out where to await or if there's a better way to make the function, so it is never triggering the onUpdate for the same user, until a previous trigger is finished.
Last part also sound a bit like a bad idea.
So far I tried sticking the read/update in a transaction, however that only seems to work for the single function call, and not when they're asynchronous.
Also figured I could fix it, by reading the previous response in a transaction on the client, however firebase doesn't allow reading from a collection in a transaction, when not using the server API.
async function setPreviousResponseToInquiry(
senderUid: string,
recipientUid: string,
inquiryId: string) {
return admin.firestore().collection('/inquiries')
.where('recipientUid', '==', recipientUid)
.where('senderUid', '==', senderUid)
.where('responded', '==', true)
.orderBy('modified', 'desc')
.limit(2)
.get().then(snapshot => {
if (!snapshot.empty &&
snapshot.docs.length >= 2) {
return admin.firestore()
.doc(`/inquiries/${inquiryId}`)
.get().then(snap => {
return snap.ref.update({
previousResponse: snapshot.docs[1].data().response
})
})
}
})
}
I see three possible solutions:
Use a transaction on the server, which ensures that the update you write must be based on the version of the data you read. If the value you write depends on the data that trigger the Cloud Function, you may need to re-read that data as part of the transaction.
Don't use Cloud Functions, but run all updates from the client. This allows you to use transactions to prevent the race condition.
If it's no possible to use a transaction, you may have to include a custom version number in both the upstream data (the data that triggers the write), and the fanned out data that you're updating. You can then use security rules to ensure that the downstream data can only be written if its version matches the current upstream data.
I'd consider/try them in the above order, as they gradually get more involved.
In the new Cloud Firestore you can set a server timestamp on my message object using the following line during creation of the object:
showtime: firebase.firestore.FieldValue.serverTimestamp()
This is useful for messages from any user as it does not rely in each users local clock.
The challenge I am having is in querying and setting the showtime field afterwards. What I want to do is look up any message that has a showtime that is early than 'now'. In my application many messages are pushed into the future. I only want the ones with a showtime before today to be returned and displayed.
Here is my query:
var query = firebase.firestore()
.collection('messages')
.where('owner','==',userID)
.where('showtime','<',timenow)
.orderBy('showtime', 'desc')
.limit(25);
The challenge is that I do not know how to get the current time (on the server) to use in the query. There is a now() call on the type Timestamp in Firebase, but I am unsure how to call it AND I am not sure based on some other questions here whether the Firebase timestamp matches the Cloud Firestore timestamp!
So the question is: How do I set a variable called timenow to be the current time on the server and use it in a query to pull 25 messages before now? (in Javascript on the client and also extra credit for in a function on the server)
A quick follow on question is then how I update the showtime timestamp to be one week later than now?
Hope we have some Firebase / Cloud Firestore mavens on Stackoverflow!
** Choosing an answer below with the caveat of a feature request: A call in Firebase to the server to request a current timestamp so that the client code can work off one standard clock. **
Expanding on the excellent answer by Alex Dunlop, I was in a similar situation and it made sense once I realized that Firestore operates according to two design principles (or at least, that is my analysis)
The emphasis is on simple queries that stream live updates to you. To enable this, queries are by design limited in complexity. This takes load of the Firestore database server and allows it to stream (effectively re-run) queries fast.
Because of the limited query language and few 'server side' operators (FieldValue is one of the few), Google can optimize the queries mercilessly.
There are many more complex operations that developers require. Instead of implementing them in Firestore, developers are asked to spin up a cloud function which makes them responsible (in code and cost) for that additional complexity.
If you come from a conventional DB design like Postgres, the lack of query expressiveness and server-side operations is stunning. If you think of the Firestore use case and the principles, it all makes sense.
Try Firestore security rules with data validation:
match /messages/{msgId} {
// optionally: request.time + someBuffer
allow read: if request.time > resource.data.showtime;
}
I don't think you want to trust the client (since you mentioned you want to block client access to future showtimes). Besides changing their clock, they could just edit the javascript.
A buffer might be necessary to not invalidate the query if there is some discrepancy between client provided Date.now() and Firestore rules request.time, sepecifically if the request.time happens to be earlier than client date, then the query would have documents falling outside valid range and fail.
Checking the read timestamp on an empty snapshot seems to work:
const serverTime = (await db.collection('nonexistent').get()).readTime;
Agreed with one of the other comments that this extra call could be expensive, but this should work in cases where an accurate "not before" server time is important for correctness.
You are right serverTimestamp() is exactly for getting a timestamp on the server and not relying on the users local clock. One thing to note is that generally sending a message and getting the timestamp from a users local clock is going to be okay as a message timestamp is not extremely time sensitive. Sure you would like to know when the message is sent but if it is within 1-2 seconds not a problem in most cases.
If you are doing a query on the client side your query should not be based on the server time it should be based on the client side. As it is a client query not a server query.
Here is a client query that you are after.
const currentTime = new Date();
const query = firebase.firestore()
.collection('messages')
.where('owner','==',userID)
.where('showtime','<',currentTime)
.orderBy('showtime', 'desc')
.limit(25);
This query will get 25 messages with a 'showtime' after the current time on the client.
Now if the messages need to be extremely time sensitive and you do absolutely need the messages to based off the server timestamp I recommend that instead of doing a query on the client like above you set up a cloud function api.
Have a look at the firebase docs for calling cloud functions directly if you haven't before.
Here is what you would want your cloud function to look like:
import * as functions from 'firebase-functions';
import * as admin from 'firebase-admin';
admin.initializeApp()
exports.getUserMessages = functions.https.onCall((data, context) => {
const uid = context.auth.uid;
const firestore = admin.firestore();
firestore.collection('messages')
.where('owner', '==', uid)
.where('showtime', '<', new Date())
.orderBy('showtime', 'desc')
.limit(25)
.get()
.then(snapshot => {
return snapshot;
});
});
This will get the messages based of the server timestamp. One thing to note is UNLESS you need this to be extremely time sensitive this is not a good idea. As this call has to do an extra unnecessary call every time you call it. Because you are doing a call to the cloud function and then you are doing a query to the firestore database.
I would recommend that instead of doing it based on the server time you do it based on client timestamp. 99 times out of 100 the time difference between client and server is not worth the extra double calls you are doing, especially when you think about scaling everything up when you get more users.
Hope that answered your question :)
I'm trying to understand how corda oracles work from an example on github. It seems like in every example oracle verification function checks the data in command and data in output state. I don't understand why that should work because we (issuer node) manage that data and put it in command/output state.
// Our contract does not check that the Nth prime is correct. Instead, it checks that the
// information in the command and state match.
override fun verify(tx: LedgerTransaction) = requireThat {
"There are no inputs" using (tx.inputs.isEmpty())
val output = tx.outputsOfType<PrimeState>().single()
val command = tx.commands.requireSingleCommand<Create>().value
"The prime in the output does not match the prime in the command." using
(command.n == output.n && command.nthPrime == output.nthPrime)
}
In this example state gets Nth prime number from oracle but after it's issued the verification function doesn't rerun generateNth prime function to make sure that this number is really the one we needed. I understand that data in this example is deterministic since Nth prime cannot change but what about the case where we have dynamic data like stock values? Shouldn't oracle verification function also send another http request and get current values to check them?
Firstly, note that contracts in Corda are not able to access the outside world in any way (DB reads, HTTP requests, etc.). If they could, transaction validity would be non-deterministic. A transaction that is found to be valid on day n may become invalid on day n+1 (because a database row changed, or a website went down, etc.). This would cause disagreements about whether a given transaction was a valid ledger update.
However, we sometimes need a transaction to include external data for verification (whether a company is bankrupt, whether a natural catastrophe happened, etc.). To do this, we use a trusted oracle that only signs the transaction if a given piece of data is valid.
We could embed the information in the input or output states. However, this would require us to reveal the entire input or output state to the oracle for signing. For privacy reasons, it is therefore preferable to embed the data in a command that only contains the data of interest to the oracle, so that we can filter out all the other parts of the transaction and only present this command to the oracle for signing.
The oracle will usually perform a DB read or make an HTTP request to check the validity of the data before signing.