I am currently making an application which stores data for each day of the year. I was thinking that the format would be something similar to:
[
2022: {
january: {
week1: [
{
tasksCompleted: 5,
},
... other weeks
],
... other months
},
}, ... other years
]
I am not too sure how I can store data of this format into firebase. I am totally open to using any other structures that accomplish this goal. How could I update the tasksCompleted property of a single day directly into the dataRef.update() without having to perform an extra database read which gets the data, updates it locally, then sets it in firebase?
I'm sure you're read that you can't directly update a value nested in an array without reading, updating in memory, then writing back out. There are no workarounds for this.
Instead, you can store the items of the array as separate documents (perhaps as a nested subcollection). You will need to know the ID of the document that contains the item to update if you want to avoid an initial read in order to find the value to update. There are really no alternatives to this - you simply cannot update something by its integer index in Firestore, as these operations do not scale massively and efficiently in the way that Firestore requires.
Reading before updating is really not that bad of an option when you consider all of the massive scaling that Firestore gives you with no effort on your part.
Related
I have an IoT Topic receiving data from devices. Each IoT payload includes some properties and an array of objects, which looks something like this.
{
"batchId": "someBatchId",
"prop1": "someProp1",
"objArray": [
{
"arrString1": "someArrString1",
"arrString2": "someArrString2",
"arrNum1": 1,
"arrNum2": 2,
"arrString3": "someArrString3"
},
{
"arrString1": "someArrString4",
"arrString2": "someArrString5",
"arrNum1": 3,
"arrNum2": 4,
"arrString3": "someArrString6"
}
]
}
The array can have hundreds of objects in it. We want to flatten this data out using a Map step and associate the top-level properties with each element in the array and insert that element into DynamoDB. We have the table set-up and the IoT topic working just fine.
The problem we have is that DynamoDB expects strings when inserting numbers. However, since we're receiving this data as a JSON object from IoT and the numbers are inside of the array of objects, we're having a hard time massaging the numbers into strings. So, we want the Step Function to convert the numbers into strings somehow, but I can't see how to do it. The goal here is to build a simple pipeline for storing IoT data into DynamoDB.
We also don't fully control all of the properties that could be sent, so we're also storing copies of the IoT payloads in S3 (which is already wired with the IoT rules engine and works just fine), but this is more of a backup and catch-all. We're mostly interested in the data getting into DynamoDB so that we can actually query it. How can we convince the Step Function to insert the numbers from the JSON payloads into DynamoDB?
You are really asking two questions here.
Can Amazon States Language convert numbers to strings?
How can you get Step Functions to add things to Dynamo DB without specifying the data type.
The answer to the first question is that yes, you can use ASL to convert numbers to strings using the concatenation intrinsic function like so. Given a payload of a single number
{
"key.$": "States.Format('{}',$.Payload)"
}
We can use the States intrinsic function Format to add quotes around the output of this step.
This will not be helpful in your use case however, as you have hundreds of numbers potentially which may not always follow a set format.
In your case, the solution would be to save your data to DynamoDB using a Lambda function with the Document Client.
It would be nice if they had an option to use the document client directly within Step Functions, but as of this writing that is not the case. You simply need to perform the action manually within a Lambda function using the document client. Same result.
In firestore i have a collection called things.
Each thing is owned by a user.
Each thing can be shared by the owner with other specified users.
the structure of thing looks something like
{
id: "thing01",
sharedWith: {
"user1": true,
"user2": true,
},
dtCreated: 3458973948
}
When I want to retrieve all thing objects that are shared with user1, ordered by dtCreated desc,
i can't do this without having to create an index on things.thing.user1
i.e. for every unique userid i have to create an index on the things collection.
Obviously this is not practical. The docs talk about using full text search for this, but this doesn't seem like a problem we would want to use full text search for.
Is there a different way i should be structuring the data to achieve what i want?
Is firestore just the wrong technology choice for this?
It's working very well for storing the thing objects themselves.
---- update ----
this question is not a real duplicate of Firestore: Working with nested single queries because the answer provided there is very specific to the OP's context.
i have two set of entities in my firebase realtime schema. Called Orders and customers.
so far i was not actually relating them in my app but was just showing them related. the current schema looked like:
{
"orders" : [
{"id" : 1, "name": "abc", "price": 200, "customer": "vik"}
],
"customers" : [
{"cust_id" : "10", "name" : "vik", "type": "existing"}
]
}
so i have a orders list page showing all the orders in a table which i get firing /orders.json
But practically, instead of having the customer name directly in the orders i should have cust_id attribute as that is the key.
That naturally makes it a standard relational schema where i will be free to change customer attributes without worrying about mismatch in orders.
However, the downside i see right away is that if i have say 20 orders to show in the order list table then instead of 1 i will end up firing 21 rest calls (1 to get order list and 20 to fetch customer name for each of the order)
What are the recommendations or standards around this ?
Firebase is a NoSQL database. So the rules of normalization that you know from relational databases don't necessarily apply.
For example: having the customer name in each order is actually quite normal. It saves having to do a client-side join for each customer record, significantly simplifying the code and improving the speed of the operation. But of course it comes at the cost of having to store data multiple times (quite normal in NoSQL databases), and having to consider if/how you update the duplicated data in case of updates of the customer record.
I recommend reading NoSQL data modeling, watching Firebase for SQL developers, and reading my answer on keeping denormalized data up to date.
In my app I use Firebase's childByAutoId() (swift) or .push() (web) to insert some data in the following format:
- events
- $autoId
- time:
- name:
- $autoId
- time:
- name:
Where $autoId are the randomly generated keys Firebase makes. time is the epoch time of when the data was pushed.
I want to allow users to modify each inserted entry's time. However, I want to keep the nodes under events sorted by their key and by time which Firebase naturally does when you use .push(). But if they modify the time so that it should actually be in a different order, the entries won't be sorted correctly.
Is there a way to generate an id by the modified time so that if it were inserted into events it would be in the right order? That way I could just delete the old entry and insert the new one while just duplicating the data.
Since the algorithm for Firebase's push IDs is well documented, you could easily modify the function to generate them based on a specific timestamp.
But I'd recommend instead keeping the necessary values as named properties for each child node. If you need to be able to sort by both creation and modification time, keep two separate properties. That way you won't have to depend on the behavior of the push IDs, but instead use more explicitly named properties to accomplish what you need.
I've read almost everywhere about structuring one's Firebase Database for efficient querying, but I am still a little confused between two alternatives that I have.
For example, let's say I want to get all of a user's "maxBenchPressSessions" from the past 7 days or so.
I'm stuck between picking between these two structures:
In the first array, I use the user's id as an attribute to index on whether true or false. In the second, I use userId as the attribute NAME whose value would be the user's id.
Is one faster than the other, or would they be indexed a relatively same manner? I kind of new to database design, so I want to make sure that I'm following correct practices.
PROGRESS
I have come up with a solution that will both flatten my database AND allow me to add a ListenerForSingleValueEvent using orderBy ONLY once, but only when I want to check if a user has a session saved for a specific day.
I can have each maxBenchPressSession object have a key in the format of userId_dateString. However, if I want to get all the user's sessions from the last 7 days, I don't know how to do it in one query.
Any ideas?
I recommend to watch the video. It is told about the structuring of the data very well.
References to the playlist on the firebase 3
Firebase 3.0: Data Modelling
Firebase 3.0: Node Client
As I understand the principle firebase to use it effectively. Should be as small as possible to query the data and it does not matter how many requests.
But you will approach such a request. We'll have to add another field to the database "negativeDate".
This field allows you to get the last seven entries. Here's a video -
https://www.youtube.com/watch?v=nMR_JPfL4qg&feature=youtu.be&t=4m36s
.limitToLast(7) - 7 entries
.orderByChild('negativeDate') - sort by date
Example of a request:
const ref = firebase.database().ref('maxBenchPressSession');
ref.orderByChild('negativeDate').limitToLast(7).on('value', function(snap){ })
Then add the user, and it puts all of its sessions.
const ref = firebase.database().ref('maxBenchPressSession/' + userId);
ref.orderByChild('negativeDate').limitToLast(7).on('value', function(snap){ })