I'm creating a flutter app with firestore ,stuck on data structure scenario. So the scenario is a user can visit multiple cities, I have 3000 cities and 100000 users and I wanna be able to add and query, which users have visited a particular city.
my current implementation is users/userId/userDetails and cities/cityName/cityDetails
Given what little I've learned from the video guides provided by Firebase; it depends on whether you just want to list the users (ie, just some basic information on them) that have visited a city; or, if what you want is to list users by the city/cities that they've visited.
For the first case, where you want to just list each city's visitors, you'd be better off creating a document that contains a list of visitors by their id. This would make it so that when you consult the cities, you don't have to retrieve the massive list of visitors every time you call it.
Alternatively, for the second scenario, you might want to do the reverse, where each user has a (sub) document containing the list of cities they've visited. This would be especially useful if the list of cities is expected to be particularly large, or if the userDetails file contains a lot of information about the user (lots of fields).
If the userDetails file doesn't contain a lot of details, more specifically if the amount of fields is rather small and you don't expect the user to visit more than, say, 150 cities, you could include it as one of the regular fields inside of userDetails, by setting it as an array (if you don't care about the order) or setting it as a Map (if you plan to have each city be a key, with the value pair being when the city was visited)
That being said, I'm advising this under the assumption that the implementation instructions that you've stated have the final entry be a document (cityDetails and userDetails), as it is was I believe makes the most sense (having the userId and the cityName as the collection).
If I've misunderstood your structure and it's actually "cities" as a document, "cityName" as a collection and "cityDetails" as a document; please let me know so I may fix my answer.
Related
I would appreciate some guidance on how to structure data stored within an app. While there are some reasons for the first way, I'm concerned it wouldn't be able to operate efficiently for the second case.
Simplified, the app would contain a list of Places by State. The main use case would be viewing Places within a selected State. The second use case would be that individual users could save specific Places they liked into their profile and view them all at once (showing all state Places in one list).
Option 1- Places saved in one "places" collection, which has a field of "state."
Main use: To show these places by state, the app would query where the "state" field matches the state.
Secondary use: When a user saved the place, the app would save the docID for each place into the user's profile, each of which would need to be retrieved to show the list of places.
Option 2- Have one collection per state.
Main use: To show these places by state, the app would pull all documents within the query and list them out.
Secondary use: When a user saved the place to the user's profile, the app would save the docID for each place into the user's profile, distributed across the different collections, each of which would need to be retrieved to show the list of places.
Goals:
Use the same place document to appear in both the State lists and the user's profile.
Minimize the number of calls/slowness as much as possible in the Secondary use case.
I have been reviewing Firestore data storage guidelines, but I would appreciate any thoughts from experienced developers regarding this data structure.
There is no "perfect", "the best" or "the correct" solution for structuring a Firestore database. We are usually structuring the database according to the queries that we intend to perform.
Regarding storing all the places in a single collection vs. having one collection per state, please note that there is no difference in terms of speed or costs. You'll always have to pay a number of reads that is equal to the number of documents that your query returns. However, if you need to display in your app, for example, all places of all states, then having a collection for each state, will require a separate query for each state.
Furthermore, regarding saving a list of places in a user's profile vs. storing only the IDs, it's a matter of measurement. You should measure how often the details within the places are changed. Remember that if a place is changed, then you should update that data in all places it exists. So if it's not changed so often then you can save the entire place object, otherwise, save only the ID.
I have two Firestore collections, Users and Posts. Below are simplified examples of what the typical document in each contains.
*Note that the document IDs in the friends subcollection are equal to the document ID of the corresponding user documents. Optionally, I could also add a uid field to the friends documents and/or the Users documents. Also, there is a reason not relevant to this question that we have friends as a subcollection to each user, but if need-be we change it into a unified root-level Friends collection.
This setup makes it very easy to query for posts, sorted chronologically, by any given user by simply looking for Posts documents whose owner field is equal to the document reference of that user.
I achieve this in iOS/Swift with the following, though we are building this app for iOS, Android, and web.
guard let uid = Auth.auth().currentUser?.uid else {
print("No UID")
return
}
let firestoreUserRef = firestore.collection("Users").document(uid)
firestorePostsQuery = firestore.collection("Posts").whereField("owner", isEqualTo: firestoreUserRef).order(by: "timestamp", descending: true).limit(to: 25)
My question is how to query Posts documents that have owner values contained in the user's friends subcollection, sorted chronologically. In other words, how to get the posts belonging to the user's friends, sorted chronologically.
For a real-world example, consider Twitter, where a given user's feed is populated by all tweets that have an owner property whose value is contained in the user's following list, sorted chronologically.
Now, I know from the documentation that Firestore does not support logical OR queries, so I can't just chain all of the friends together. Even if I could, that doesn't really seem like an optimal approach for anyone with more than a small handful of friends.
The only option I can think of is to create a separate query for each friend. There are several problems with this, however. The first being the challenges presenting (in a smooth manner) the results from many asynchronous fetches. The second being that I can't merge the data into chronological order without re-sorting the set manually on the client every time one of the query snapshots is updated (i.e., real-time update).
Is it possible to build the query I am describing, or am I going to have to go this less-than optimal approach? This seems like a fairly common query use-case, so I'll be surprised if there is not a way to do this.
The sort chronologically is easy provided you are using a Unix timestamp, e.g. 1547608677790 using the .orderBy method. However, that leaves you with a potential mountain of queries to iterate through (one per friend).
So, I think you want to re-think the data store schema.
Take advantage of Cloud Functions for Firebase Triggers. When a new post is written, have a cloud function calculate who all should see it. Each user could have an array-type property containing all unread-posts, read-posts, etc.
Something like that would be fast and least taxing.
I'm trying to find a way to properly desing my Data Model with Firestore. I'm looking for something similar to what Tinder does, showing you people that you have'nt swiped yet, based on your location.
So I ended up with something like :
A User1 has an array of "met people"
A "Haven't yet met user"/ User2 his also a User with the same document model
They all belong in the same "Users" collection
I want to query all the users that this User1 haven't swiped yet
I know that you can't do something like "array_not_contains" or "!=" because all fields that you query need to be indexed.
So I wonder, is this possible to model data to make it work, or the only solution is to drop Firebase because this kind of query is not possible at all?
One alternative can be to store in a collection all the relationships (with theirs status) between all users. But that also means that whenever a user signup, I have to create as many documents as I have users that's really ugly and make a enormous numbers of documents.
EDIT:
Thanks again for your answer and sorry for my late answer.
There is no need to create a new database call since you already got all the users from that area in the first place.
Not If have a large response set, I will limit to a number. (5 in the example below).
And even If I don't limit the number, in the next db call, how I can know that new peoples has been added and how to retrieve only those.
I will not remove them from Users Collection has they can be show to others users.
P.S: I forget User4 in Users Collection pictures.
For User 1, get 5 first matchs, remove existing ones, show User5.
For User2, get 5 first matchs, remove existing ones, show User4, User5.
After users choices, Users are added to their list. Users Collection stay the same.
For User 1, get 5 first matchs, remove existing ones, nothing to show, even if I have a User 6, 7.
To fix that I launch a second query get the new ones but, more the user use the app more query I may need to do to try to display to him existing user in his area.
Maybe I've misunderstood what you named "initial list", for me it is the list object retrieve from my db containing all users (with limit).
EDIT 2:
You can check the answers of Alex Mamo to know how to query documents that are not exist in an array possible.
Let's me explain my use case and why I think, that won't work.
I want to be able to search all users next to me, for trying to do that in Firebase, I store Geopoint. Geopoint can't be really use for now out of the box with Firebase, so I user Geofirestore in a Cloud Function.
I store and update user Geopoints based on theirs locations, so this means user location change by time.
I limit the numbers of Users return by this function.
In my initial state I retrieve users next to me (User1), I get 3 an 4.
Let's say that I store last checked userId to use it later as a cursor for my query (User 4).
Now my geopoint change, and the users in this area changes too.
I request next bunch of users next to me, and I use my previous userId/document to "startAfter" (more on this
here), see the image below, that's won't work.
If I use the cursor (User4), I'll take 5, but not 2, because in the return list, if I order by Id, 2 will be before 4.
Worse, like below, if the return list may not even have user 4 in it, the cursor will be pointless.
My example is a bit simplified and does not take in account what is described in the first answer and my first edit (limited subset of users, data design).
A possible database structure for your app might be:
Firestore-root
|
--- users (collection)
|
--- uid (document)
|
--- acceptedUsers: ["uidOne", "uidTwo"]
|
--- declinedUsers: ["uidThree", "uidFour"]
|
--- //Other user properties
The mechanism is simple. When you first want to show a user profile to the current (authenticated) user, you have to create a query that will return all users (in user area). According to the user decision, you need to add the corresponding uid in either the acceptedUsers array or in declinedUsers array. Once you want to show another users, use the same query but this time, you need to make an extra operation. Once the query returns the users within user location, add all those users to a list. Compare the list that is coming from the database with your exting arrays and remove all the users from both arrays. In this way you'll have a list that contains only users that the actual user didn't see. This extra step is needed to make sure the id of the user does not exist in one of those arrays. In the end, simply choose a random user from the list and show the details to the user. That's it!
One alternative can be to store in a collection all the relationships (with theirs status) between all users. But that also means that whenever a user signup, I have to create as many documents as I have users.....that's really ugly and make a enormous numbers of documents.
This is not an option. This means that you need to write each time a user joins your app an enormous amount of data, which will be very costly. Since everything in Firestore is about the number of read and writes, I think you should think again about this approach. Please see Firestore usage and limits.
Edit:
Let's consider the initial list of users that has 10 records. With other words, all the users within that area are 10. You say that 7 users are already seen, that makes the list contain only the 3 remaining users.
So I display the 3, (or I do another request to get some more) and he check the 3.
Yes, you should display those 3 users and then remove them one by one from the initial list. There is no need to create a new database call since you already got all the users from that area in the first place. Once the list remains empty, you should display a message to the user that in that particular area are no more users to swipe.
When will create another database call?
Only when needed. Which means that you create another call once new users enter that area. Let's say 3 new users are new, you get a list now of 3 user and use the same algorithm.
More my user use the app more it’s difficult to show people that he haven’t seen, because his list become bigger.
If you think that the arrays will grow more than a document can hold, then you should consider storing the users in a collection and not in an array. So in this case, the problem is that the documents have limits. So there are some limits when it comes to how much data you can put into a document. According to the official documentation regarding usage and limits:
Maximum size for a document: 1 MiB (1,048,576 bytes)
As you can see, you are limited to 1 MiB total of data in a single document. When we are talking about storing text (uids), you can store pretty much but as your array getts bigger, be careful about this limitation.
But if you'll stay within this limits, which I personally think you'll do, you have nothing to worry about.
Edit2:
Not If have a large response set, I will limit to a number. (5 in the example below). And even If I don't limit the number, in the next db call, how I can know that new peoples has been added and how to retrieve only those.
I will not remove them from Users Collection has they can be show to others users.
If you have large amount of data (many users in a single area), yes it's good idea to limit the results, but a much better idea would be to load the data in smaller chunks. In short, get 5 users, remove one by one till the list has zero users, load other 5 users and so on. This can be made using my answer from the following post:
Is there a way to paginate queries by combining query cursors using FirestoreRecyclerAdapter?
The initial list, is the list that you are getting when you first query the database. In this case, the initial list will contain 5 users.
I have a Firestore database set up where I have a Users collection and Animals collection (these animals can be created by users at any time). For a particular user, I want to grab a random animal document that the user hasn't seen yet.
I don't believe it is possible to query for non-existing keys in Firestore, which makes this problem non-trivial for me. Is there a better way to do this than to have a dictionary of all animal ids for each user? My issue with that approach would be scalability since animals can be created by users and thus every user's animal dictionary would have to be updated per new animal.
Thanks for any help in advance!
You won't be able to do this with a single query that returns a single document.
There is no sense of randomness in Firestore queries. If you want something random, you'll have to select that in your code from a set of items in memory. This means, at the very least, you're going to have to first figure out how to query for all the animals a user hasn't seen yet, then select randomly from that set in application code.
You are correct in assuming that you'll need some sort of collection that records who has seen what animal, then query that for a list of unseen animals. Then you can randomly select from that the final animal document.
An in order to do that, you're going to need another collection of documents that record who has seen what animal previously.
I'm building an app that tracks the user's location and updates Firebase. I've read the documentation about structure data but still have a few questions.
I'm considering structuring the data in one of two ways, but can't determine which one.
users
$id
-position
-other attr
vs:
user_position
$id
users
$id
-other attr.
In what scenario would the first design work best, second?
If you only keep one position per user (as seems to be the case by the fact that you use singular user_position), there is no useful difference between the two structures. A user's position in that case is just another attribute, just one that happens to have two value (lat and lon).
But if you want to keep multiple positions per user, then your first structure is mixing entity types: users and user_positions. This is an anti-pattern when it comes to Firebase Database.
The two most common reasons are:
Say you want to show a list of user names (or any specific, single-value attribute). With the first structure you will also need to read the list of all positions of all users, just to get the list of names. With the second structure, you just read the user's attributes. If that is still much more data than you need, consider also keeping a list of /user_names for optimal read performance.
Many developers end up wanting different access rules for the user positions and the other user attributes. In the first structure that is only possible by pushing the read permission from the top /users down to lower in the tree. In the second structure, you can just give separate permissions to /users and /user_positions.