Firebase Database Rules Timestamp Issue - firebase

Since the time shift at the 29th of October 2017 I'm running in some really strange behaviours while developing with the firebase products.
I'm developing a hybrid app with Ionic (3). While I'm developing and testing in the browser (mobile emulated device) everything works fine. As soon as I switch to my real device (Samsung Galaxy S7, no root, modded or something else), all writes to the database with a timestamp are failing.
In my code I create a timestamp like this: Date.now()
In my firebase rules I validate timestamps usualy like this:
"timestamp": {
".validate": "newData.isNumber() && newData.val() <= now"
}
For me this rule means, that the new data, which wants to be written into the database, must be a number and the value of the new data must be less than or equal to the current server timestamp. If one of these conditions doesn't match, it will throw a warning at the client.
I've spent a whole day debugging my code and finding the bug. When I remove (comment out) the .validate key on the timestamp related rules, everything works fine.
So I played a little bit with the timestamp values in the firebase rules. For example I added a little buffer to the server timestamp like: (now + 10000) (10secs).
Suddenly it worked. I decreased the value until it stopped working on my real device. I stopped at (now + 5000) (5secs).
So now my question is, why this behavior is the way it is.
Before the time shift, everything worked fine. In my understanding it couldn't be possible that a client timestamp is ahead of the server timestamps. (except the local time on the real device was modified by the user itself).
Looking for some help, the workaround with the additional 5 secs seems a bit dirty.
Cheers
Unkn0wn0x
BTW: Each time I modified the firebase rules and deployed them to the server(s), I've waited for about five minutes.

Thanks for Your annotation.
I've played one more time with the timestamps and figured out the unwanted behaviour.
This is the code, I've added to my function with wants to write a timestamp to the database:
const test = Date.now();
const test2 = new Date().getTime();
console.log('server offset: ', snap.val());
console.log('Date.now(): ', test);
console.log('new Date().getTime(): ', test2);
console.log('estimated server timestamp (new Date().getTime() + offset): ', (test2 + snap.val()));
console.log('client timestamp (Date.now() - offset): ', (test - snap.val()));
The output of the above code:
server offset: -2427
Date.now(): 1509730244926
new Date().getTime(): 1509730244926
estimated server timestamp (new Date().getTime() + offset): 1509730242499
client timestamp (Date.now() - offset): 1509730247353
The crux here is the negative offset. I've substracted the server offset from the client timestamp to get a timestamp, which is smaller then the estimated server timestamp.
But: - and - is +. So I accidentally added them both together instead of subtract from one another.
I've executed my function a few times and could determine, that the offset is from execution to execution different. One time +77 ms the other time -2427ms, etc..
So I added a little code snippet, which checks if the returned server offset is a postive or negative number, to be able to calculate the client timestamp correctly.
const serverOffset: number = snap.val();
let clientTimestamp: number = null;
if (Math.sign(serverOffset) === 1){
clientTimestamp = Date.now() - serverOffset;
} else if (Math.sign(serverOffset) === -1){
clientTimestamp = Date.now() + serverOffset;
}
clientTimestamp works now as expected.
This behaviour can also be achieved if the offset is just added up to the client timestamp as mentioned in the Firebase Docs.
Maybe it should be mentioned in the Firebase Docs, that the offset can also be negative instead of just postive. This behaviour can be easily reproduced, if the local device time runs just a second later than the time, which is fetched from the internet.
But why does this suddenly occur and never before?

Related

seemingly simple Firestore query is very slow [duplicate]

I'm having slow performance issues with Firestore while retrieving basic data stored in a document compared to the realtime database with 1/10 ratio.
Using Firestore, it takes an average of 3000 ms on the first call
this.db.collection(‘testCol’)
.doc(‘testDoc’)
.valueChanges().forEach((data) => {
console.log(data);//3000 ms later
});
Using the realtime database, it takes an average of 300 ms on the first call
this.db.database.ref(‘/test’).once(‘value’).then(data => {
console.log(data); //300ms later
});
This is a screenshot of the network console :
I'm running the Javascript SDK v4.50 with AngularFire2 v5.0 rc.2.
Did anyone experience this issue ?
UPDATE: 12th Feb 2018 - iOS Firestore SDK v0.10.0
Similar to some other commenters, I've also noticed a slower response on the first get request (with subsequent requests taking ~100ms). For me it's not as bad as 30s, but maybe around 2-3s when I have good connectivity, which is enough to provide a bad user experience when my app starts up.
Firebase have advised that they're aware of this "cold start" issue and they're working on a long term fix for it - no ETA unfortunately. I think it's a separate issue that when I have poor connectivity, it can take ages (over 30s) before get requests decide to read from cache.
Whilst Firebase fix all these issues, I've started using the new disableNetwork() and enableNetwork() methods (available in Firestore v0.10.0) to manually control the online/offline state of Firebase. Though I've had to be very careful where I use it in my code, as there's a Firestore bug that can cause a crash under certain scenarios.
UPDATE: 15th Nov 2017 - iOS Firestore SDK v0.9.2
It seems the slow performance issue has now been fixed. I've re-run the tests described below and the time it takes for Firestore to return the 100 documents now seems to be consistently around 100ms.
Not sure if this was a fix in the latest SDK v0.9.2 or if it was a backend fix (or both), but I suggest everyone updates their Firebase pods. My app is noticeably more responsive - similar to the way it was on the Realtime DB.
I've also discovered Firestore to be much slower than Realtime DB, especially when reading from lots of documents.
Updated tests (with latest iOS Firestore SDK v0.9.0):
I set up a test project in iOS Swift using both RTDB and Firestore and ran 100 sequential read operations on each. For the RTDB, I tested the observeSingleEvent and observe methods on each of the 100 top level nodes. For Firestore, I used the getDocument and addSnapshotListener methods at each of the 100 documents in the TestCol collection. I ran the tests with disk persistence on and off. Please refer to the attached image, which shows the data structure for each database.
I ran the test 10 times for each database on the same device and a stable wifi network. Existing observers and listeners were destroyed before each new run.
Realtime DB observeSingleEvent method:
func rtdbObserveSingle() {
let start = UInt64(floor(Date().timeIntervalSince1970 * 1000))
print("Started reading from RTDB at: \(start)")
for i in 1...100 {
Database.database().reference().child(String(i)).observeSingleEvent(of: .value) { snapshot in
let time = UInt64(floor(Date().timeIntervalSince1970 * 1000))
let data = snapshot.value as? [String: String] ?? [:]
print("Data: \(data). Returned at: \(time)")
}
}
}
Realtime DB observe method:
func rtdbObserve() {
let start = UInt64(floor(Date().timeIntervalSince1970 * 1000))
print("Started reading from RTDB at: \(start)")
for i in 1...100 {
Database.database().reference().child(String(i)).observe(.value) { snapshot in
let time = UInt64(floor(Date().timeIntervalSince1970 * 1000))
let data = snapshot.value as? [String: String] ?? [:]
print("Data: \(data). Returned at: \(time)")
}
}
}
Firestore getDocument method:
func fsGetDocument() {
let start = UInt64(floor(Date().timeIntervalSince1970 * 1000))
print("Started reading from FS at: \(start)")
for i in 1...100 {
Firestore.firestore().collection("TestCol").document(String(i)).getDocument() { document, error in
let time = UInt64(floor(Date().timeIntervalSince1970 * 1000))
guard let document = document, document.exists && error == nil else {
print("Error: \(error?.localizedDescription ?? "nil"). Returned at: \(time)")
return
}
let data = document.data() as? [String: String] ?? [:]
print("Data: \(data). Returned at: \(time)")
}
}
}
Firestore addSnapshotListener method:
func fsAddSnapshotListener() {
let start = UInt64(floor(Date().timeIntervalSince1970 * 1000))
print("Started reading from FS at: \(start)")
for i in 1...100 {
Firestore.firestore().collection("TestCol").document(String(i)).addSnapshotListener() { document, error in
let time = UInt64(floor(Date().timeIntervalSince1970 * 1000))
guard let document = document, document.exists && error == nil else {
print("Error: \(error?.localizedDescription ?? "nil"). Returned at: \(time)")
return
}
let data = document.data() as? [String: String] ?? [:]
print("Data: \(data). Returned at: \(time)")
}
}
}
Each method essentially prints the unix timestamp in milliseconds when the method starts executing and then prints another unix timestamp when each read operation returns. I took the difference between the initial timestamp and the last timestamp to return.
RESULTS - Disk persistence disabled:
RESULTS - Disk persistence enabled:
Data Structure:
When the Firestore getDocument / addSnapshotListener methods get stuck, it seems to get stuck for durations that are roughly multiples of 30 seconds. Perhaps this could help the Firebase team isolate where in the SDK it's getting stuck?
Update Date March 02, 2018
It looks like this is a known issue and the engineers at Firestore are working on a fix. After a few email exchanges and code sharing with a Firestore engineer on this issue, this was his response as of today.
"You are actually correct. Upon further checking, this slowness on getDocuments() API is a known behavior in Cloud Firestore beta. Our engineers are aware of this performance issue tagged as "cold starts", but don't worry as we are doing our best to improve Firestore query performance.
We are already working on a long-term fix but I can't share any timelines or specifics at the moment. While Firestore is still on beta, expect that there will be more improvements to come."
So hopefully this will get knocked out soon.
Using Swift / iOS
After dealing with this for about 3 days it seems the issue is definitely the get() ie .getDocuments and .getDocument. Things I thought were causing the extreme yet intermittent delays but don't appear to be the case:
Not so great network connectivity
Repeated calls via looping over .getDocument()
Chaining get() calls
Firestore Cold starting
Fetching multiple documents (Fetching 1 small doc caused 20sec delays)
Caching (I disabled offline persistence but this did nothing.)
I was able to rule all of these out as I noticed this issue didn't happen with every Firestore database call I was making. Only retrievals using get(). For kicks I replaced .getDocument with .addSnapshotListener to retrieve my data and voila. Instant retrieval each time including the first call. No cold starts. So far no issues with the .addSnapshotListener, only getDocument(s).
For now, I'm simply dropping the .getDocument() where time is of the essence and replacing it with .addSnapshotListener then using
for document in querySnapshot!.documents{
// do some magical unicorn stuff here with my document.data()
}
... in order to keep moving until this gets worked out by Firestore.
Almost 3 years later, firestore being well out of beta and I can confirm that this horrible problem still persists ;-(
On our mobile app we use the javascript / node.js firebase client. After a lot of testing to find out why our app's startup time is around 10sec we identified what to attribute 70% of that time to... Well, to firebase's and firestore's performance and cold start issues:
firebase.auth().onAuthStateChanged() fires approx. after 1.5 - 2sec, already quite bad.
If it returns a user, we use its ID to get the user document from firestore. This is the first call to firestore and the corresponding get() takes 4 - 5sec. Subsequent get() of the same or other documents take approx. 500ms.
So in total the user initialization takes 6 - 7 sec, completely unacceptable. And we can't do anything about it. We can't test disabling persistence, since in the javascript client there's no such option, persistence is always enabled by default, so not calling enablePersistence() won't change anything.
I had this issue until this morning. My Firestore query via iOS/Swift would take around 20 seconds to complete a simple, fully indexed query - with non-proportional query times for 1 item returned - all the way up to 3,000.
My solution was to disable offline data persistence. In my case, it didn't suit the needs of our Firestore database - which has large portions of its data updated every day.
iOS & Android users have this option enabled by default, whilst web users have it disabled by default. It makes Firestore seem insanely slow if you're querying a huge collection of documents. Basically it caches a copy of whichever data you're querying (and whichever collection you're querying - I believe it caches all documents within) which can lead to high Memory usage.
In my case, it caused a huge wait for every query until the device had cached the data required - hence the non-proportional query times for the increasing numbers of items to return from the exact same collection. This is because it took the same amount of time to cache the collection in each query.
Offline Data - from the Cloud Firestore Docs
I performed some benchmarking to display this effect (with offline persistence enabled) from the same queried collection, but with different amounts of items returned using the .limit parameter:
Now at 100 items returned (with offline persistence disabled), my query takes less than 1 second to complete.
My Firestore query code is below:
let db = Firestore.firestore()
self.date = Date()
let ref = db.collection("collection").whereField("Int", isEqualTo: SomeInt).order(by: "AnotherInt", descending: true).limit(to: 100)
ref.getDocuments() { (querySnapshot, err) in
if let err = err {
print("Error getting documents: \(err)")
} else {
for document in querySnapshot!.documents {
let data = document.data()
//Do things
}
print("QUERY DONE")
let currentTime = Date()
let components = Calendar.current.dateComponents([.second], from: self.date, to: currentTime)
let seconds = components.second!
print("Elapsed time for Firestore query -> \(seconds)s")
// Benchmark result
}
}
well, from what I'm currently doing and research by using nexus 5X in emulator and real android phone Huawei P8,
Firestore and Cloud Storage are both give me a headache of slow response
when I do first document.get() and first storage.getDownloadUrl()
It give me more than 60 seconds response on each request. The slow response only happen in real android phone. Not in emulator. Another strange thing.
After the first encounter, the rest request is smooth.
Here is the simple code where I meet the slow response.
var dbuserref = dbFireStore.collection('user').where('email','==',email);
const querySnapshot = await dbuserref.get();
var url = await defaultStorage.ref(document.data().image_path).getDownloadURL();
I also found link that is researching the same.
https://reformatcode.com/code/android/firestore-document-get-performance

I am having a problem with Firebase Auth-UI user metadata

I am using Firebase Auth UI to register and store users in my app.
When the user is new in the app, I need to register some things, so I need to know when a user is new or not.
According to the documentation I am using the right way to check new users:
Auth-UI Metadata
The way I check to know wheter a user is new or not is comparing the metadata, user creation timestamp and Last Signed Timestamp and it worked perfect.
if (metadata.getCreationTimestamp() == metadata.getLastSignInTimestamp()){
//do some new user stuff
}else{
//log in old user
}
This code is not working right now, because they might have changed something.
The last time I am sure this code worked is the 30th of January.
There is a little difference in miliseconds between the creation timestamp and the sign in timestamp when the user is new:
Creation timestamp: 1549462011000 Last Sign in Timestamp: 1549462011028
I've tried in two different Firebase projects and the problem is the same.
Can anyone help?
I experienced the same problem with my app.
The two timestamps have now few milliseconds difference.
A way to fix it is to check the two timestamps are close enough ( 3 seconds in the code below)
val signUpInterval = 3000L
val isNewUser = Math.abs(metadata.creationTimestamp - metadata.lastSignInTimestamp) < signUpInterval

Get Firestore Server Timestamp from the client side

I'm building an app that tracks time. It calculates the time by differentiating the seconds like so:
serverTimestamp: FieldValue // {seconds: number, milliseconds: number}.
getSeconds() {
const createdTime = new Date(this.serverTimestamp * 1000).getTime()
const currentTime = new Date().getTime();
return Math.abs((currentTime - createdTime) / 1000);
}
The problem is that Date's values equal to the current client's device clock, while firestore's timestamp equal to the firestore server, which may cause inequivalence between these two dates.
The leads me with two alternatives:
1. Save Date instead of serverTimestamp().
This way, the difference between these two dates will be accurate. Although, they won't represent the real date. Plus, if the user will change his clock on his machine, then the seconds would be changed too.
2. Create an HTTPS function that retrieves the current server timestamp.
This way, the difference between these two dates will be accurate either. Although, it feels like I'm going too far only to get the current timestamp of the server.
I am probably going to stick with alternative number two. But I was hoping if there's a better solution to achieve my goal.
PS - I'm using #angular/fire. So, solutions from this package would be welcomed.
You can also write the current timestamp with a serverTimestamp, then read the time back out of the location it was just written. Then you can make changes or calculations with that value.

Firebase security time rule using 'now'

I have a Firebase security rule trying to allow for writes if the field timstamp has not yet passed i.e modifications should be allowed until time has moved past the time set in the timestamp data field.
"tips": {
".read": true,
"$user_id": {
"tips": {
"$game_id": {
".write": "root.child('/games/' + $game_id + '/timestamp/').val() > now"
}
}
}
In trying to get this rule to work I have 2 sample timestamp fields of:
timestamp: 1593840696
timestamp: 1393840696
Although I'm not verifying what 'now' is I understand it to be epoch time and current date/time is around 1493840696. When writing to the Firebase I would expect the first entry to succeed with an update (it's timestamp is after 'now' time) and the second to fail as it is in the past.
What I am seeing though is that neither record is allowed to update with this rule? If I change the greater than '>' to less than '<' they both work. Strange? It would seem the value of 'now' may be greater than the 1593840696.
".write": "root.child('/games/' + $game_id + '/timestamp/').val() < now"
What would the value of 'now' be in this context? Any suggestions on how to work through/debug and achieve the required timestamp protection? Essentially need to protect/allow data changes until after the timestamp field has elapsed.
I can't get this rule to fire and protect the data so any help would be most appreciated.
Thanks for any responses.
The Firebase Database stores timestamps in milliseconds since the epoch. An easy way to get such a value is with JavaScript's Date.now() like in this snippet:
console.log(Date.now());
As I write this the value is:
1493869428756
For comparison, your highest value is:
1593840696
So it seems like you're counting seconds since the epoch, instead of milliseconds.

Google Chrome restores session cookies after a crash, how to avoid?

On Google Chrome (I saw this with version 35 on Windows 8.1, so far I didn't try other versions) when browser crashes (or you simply unplug power cable...) you'll be asked to recover previous session when you'll open it again. Good feature but it will restore session cookies too.
I don't want to discuss here if it's a bug or not anyway IMO it's a moderate security bug because a user with physical access to that machine may "provoke" a crash to stole unclosed sessions with all their content (you won't be asked to login again).
Finally my question is: how a web-site can avoid this? If I'm using plain ASP.NET authentication with session cookies I do not want they survive to a browser crash (even if computer is restarted!).
There is not something similar to a process ID in the User Agent string and JavaScript variables are all restored (so I can't store a random seed, generated - for example - server side). Is there anything else viable? Session timeout will handle this but usually it's pretty long and there will be an unsafe window I would eliminate.
I didn't find anything I can use as process id to be sure Chrome has not been restarted but there is a dirty workaround: if I setup a timer (let's say with an interval of five seconds) I can check how much time elapsed from last tick. If elapsed time is too long then session has been recovered and logout performed. Roughly something like this (for each page):
var lastTickTime = new Date();
setInterval(function () {
var currentTickTime = new Date();
// Difference is arbitrary and shouldn't be too small, here I suppose
// a 5 seconds timer with a maximum delay of 10 seconds.
if ((currentTickTime - lastTickTime) / 1000 > 10) {
// Perform logout
}
lastTickTime = currentTickTime;
}, 5000);
Of course it's not a perfect solution (because a malicious attacker may handle this and/or disable JavaScript) but so far it's better than nothing.
New answers with a better solution are more than welcome.
Adriano's suggestion makes is a good idea but the implementation is flawed. We need to remember the time from before the crash so we can compare it to the time after the crash. The easiest way to do that is to use sessionStorage.
const CRASH_DETECT_THRESHOLD_IN_MILLISECONDS = 10000;
const marker = parseInt(sessionStorage.getItem('crashDetectMarker') || new Date().valueOf());
const diff = new Date().valueOf() - marker;
console.log('diff', diff)
if (diff > CRASH_DETECT_THRESHOLD_IN_MILLISECONDS) {
alert('log out');
} else {
alert ('ok');
}
setInterval(() => {
sessionStorage.setItem('crashDetectMarker', new Date().valueOf());
}, 1000)
To test, you can simulate a Chrome crash by entering chrome://crash in the location bar.
Don't forget to clear out the crashDetectMarker when the user logs out.

Resources