The documentation in google cloud's datastore section is very confusing to me.
I was under the belief datastore may be used as a key-value storage. (Similar to mongoDB) But I guess I misunderstood how its keys work, since I can't use string keys outright, but I need to transform series of strings to a (list) key via some list => dataStore.key(list) transformation.
That's weird, and I don't understand why use a list instead of a string, and I don't understand why I don't use the list outright, and need to use datastore.key first, but I can do that. However, after playing with it a little bit, I discovered that the return value of datastore.key(list) would get me different values for the same list if I run it repeatedly!
So, now I need to somehow remember this key somewhere, but the reason I wanted to use datastore was that I was running in a service with no persistent memory to begin with.
Am I using datastore wrong? Can I use it at all for simple persistent key-value storage?
It appears that the issue was that I used a list that was too short.
The datastore expects collections, which contain documents, each of which is a key-value mapping. So instead of having my key point at a mapping, I needed to set a collection, and in it have keys mapped to mappings. (In other words, I needed to have one more element in the keys list)
results = await this.dataStore.get(this.dataStore.key([collectionId, documentId]));
AND
await this.dataStore.save({ key: this.dataStore.key([collectionId, documentId]), data: someKeyValueObject });
Related
I have a collection where the documents are uniquely identified by a date, and I want to get the n most recent documents. My first thought was to use the date as a document ID, and then my query would sort by ID in descending order. Something like .orderBy(FieldPath.documentId, descending: true).limit(n). This does not work, because it requires an index, which can't be created because __name__ only indexes are not supported.
My next attempt was to use .limitToLast(n) with the default sort, which is documented here.
By default, Cloud Firestore retrieves all documents that satisfy the query in ascending order by document ID
According to that snippet from the docs, .limitToLast(n) should work. However, because I didn't specify a sort, it says I can't limit the results. To fix this, I tried .orderBy(FieldPath.documentId).limitToLast(n), which should be equivalent. This, for some reason, gives me an error saying I need an index. I can't create it for the same reason I couldn't create the previous one, but I don't think I should need to because they must already have an index like that in order to implement the default ordering.
Should I just give up and copy the document ID into the document as a field, so I can sort that way? I know it should be easy from an algorithms perspective to do what I'm trying to do, but I haven't been able to figure out how to do it using the API. Am I missing something?
Edit: I didn't realize this was important, but I'm using the flutterfire firestore library.
A few points. It is ALWAYS a good practice to use random, well distributed documentId's in firestore for scale and efficiency. Related to that, there is effectively NO WAY to query by documentId - and in the few circumstances you can use it (especially for a range, which is possible but VERY tricky, as it requires inequalities, and you can only do inequalities on one field). IF there's a reason to search on an ID, yes it is PERFECTLY appropriate to store in the document as well - in fact, my wrapper library always does this.
the correct notation, btw, would be FieldPath.documentId() (method, not constant) - alternatively, __name__ - but I believe this only works in Queries. The reason it requested a new index is without the () it assumed you had a field named FieldPath with a subfield named documentid.
Further: FieldPath.documentId() does NOT generate the documentId at the server - it generates the FULL PATH to the document - see Firestore collection group query on documentId for a more complete explanation.
So net:
=> documentId's should be as random as possible within a collection; it's generally best to let Firestore generate them for you.
=> a valid exception is when you have ONE AND ONLY ONE sub-document under another - for example, every "user" document might have one and only one "forms of Id" document as a subcollection. It is valid to use the SAME ID as the parent document in this exceptional case.
=> anything you want to query should be a FIELD in a document,and generally simple fields.
=> WORD TO THE WISE: Firestore "arrays" are ABSOLUTELY NOT ARRAYS. They are ORDERED LISTS, generally in the order they were added to the array. The SDK presents them to the CLIENT as arrays, but Firestore it self does not STORE them as ACTUAL ARRAYS - THE NUMBER YOU SEE IN THE CONSOLE is the order, not an index. matching elements in an array (arrayContains, e.g.) requires matching the WHOLE element - if you store an ordered list of objects, you CANNOT query the "array" on sub-elements.
From what I've found:
FieldPath.documentId does not match on the documentId, but on the refPath (which it gets automatically if passed a document reference).
As such, since the documents are to be sorted by timestamp, it would be more ideal to create a timestamp fieldvalue for createdAt rather than a human-readable string which is prone to string length sorting over the value of the string.
From there, you can simply sort by date and limit to last. You can keep the document ID's as you intend.
I have a button which runs the following code when it's clicked:
let dataReference = await db.collection("dog").doc("1").get()
let HashMap = dataReference.data().Annotations
console.log(HashMap)
My firestore database looks like this:
Whenever this function is run, it returns the proper dictionary, however, the ordering of the keys seems to change randomly. Here's a screenshot of my console logs when I pressed the button a bunch of times:
Why does the ordering of the key-value pairs change and is there a way to fix it?
The Firestore SDK does not guarantee an order of iteration of document fields. What you see in the console is always lexically sorted by the code of the console itself. If you require a stable ordering, you should sort them yourself before iteration.
One workaround you can do is to have the order of the keys you want in an array. Because arrays are ordered it will maintain the order you desired. Then you take that key and use it in the dictionary. While the dictionary is out of order, you will be accessing each value in order by key.
I have an IoT Topic receiving data from devices. Each IoT payload includes some properties and an array of objects, which looks something like this.
{
"batchId": "someBatchId",
"prop1": "someProp1",
"objArray": [
{
"arrString1": "someArrString1",
"arrString2": "someArrString2",
"arrNum1": 1,
"arrNum2": 2,
"arrString3": "someArrString3"
},
{
"arrString1": "someArrString4",
"arrString2": "someArrString5",
"arrNum1": 3,
"arrNum2": 4,
"arrString3": "someArrString6"
}
]
}
The array can have hundreds of objects in it. We want to flatten this data out using a Map step and associate the top-level properties with each element in the array and insert that element into DynamoDB. We have the table set-up and the IoT topic working just fine.
The problem we have is that DynamoDB expects strings when inserting numbers. However, since we're receiving this data as a JSON object from IoT and the numbers are inside of the array of objects, we're having a hard time massaging the numbers into strings. So, we want the Step Function to convert the numbers into strings somehow, but I can't see how to do it. The goal here is to build a simple pipeline for storing IoT data into DynamoDB.
We also don't fully control all of the properties that could be sent, so we're also storing copies of the IoT payloads in S3 (which is already wired with the IoT rules engine and works just fine), but this is more of a backup and catch-all. We're mostly interested in the data getting into DynamoDB so that we can actually query it. How can we convince the Step Function to insert the numbers from the JSON payloads into DynamoDB?
You are really asking two questions here.
Can Amazon States Language convert numbers to strings?
How can you get Step Functions to add things to Dynamo DB without specifying the data type.
The answer to the first question is that yes, you can use ASL to convert numbers to strings using the concatenation intrinsic function like so. Given a payload of a single number
{
"key.$": "States.Format('{}',$.Payload)"
}
We can use the States intrinsic function Format to add quotes around the output of this step.
This will not be helpful in your use case however, as you have hundreds of numbers potentially which may not always follow a set format.
In your case, the solution would be to save your data to DynamoDB using a Lambda function with the Document Client.
It would be nice if they had an option to use the document client directly within Step Functions, but as of this writing that is not the case. You simply need to perform the action manually within a Lambda function using the document client. Same result.
Problem: Whenever I add an order to the orders array, an additional nested array element(-KOPWA...) gets added. I wouldn't mind except, I don't know how to access that nested string to access it's child nodes.
Example of database node for users below:
firebase.database().ref('users/'+userIdState+'/orders/'+<<unique numbervariable>>).push({
"order":{"test":"product","quantity":2}
});
I'm using the above code to push new json objects with a unique number to the firebase array. Still the nested array with the weird strings gets generated.
Can anyone help me understand how to either: create my own nested array with my own unique string or how to access the nested string that gets generated from firebase so I can access it's children nodes.
Multiple instances of nest arrays will be generated by users.
Any help is much appreciated.
Thanks,
Moe
You're experiencing this behaviour because Firebase's push is not the same as an array push. I recommend reading this article to understand how it works.
As for a solution, you can simply change push to set in your code. This will create the structure you were (presumably) expecting, that is
1:
order:
...
This is however potentially unsafe, if you allow concurrent writes (i. e. if the "unique number" in your example is not always unique).
Afaik Firebase recommends using push to safely create collections/"arrays". You can retrieve the generated key by calling the key property on the reference returned by push. Like this:
var ref = firebase.database().ref('users/'+userIdState+'/orders/'+<<unique numbervariable>>).push({
"order":{"test":"product","quantity":2}
});
var generatedKey = ref.key; // the value you're looking for
If you decide to use it, you can probably just drop the order number you have right now and just use the generated one.
I want to use angularfirecollection to keep one-way sync with a list of data. The structure is a list of 'things' with various properties (e.g. 'Likes') and users who each hold a subset of 'things' keys (e.g. -jsdzsdrestofkey: true). U
sing angularfirecollection (or firebase native 'on'), I can sync up all things that a particular user has... I can also grab (using firebase native 'once') each 'thing''s properties to display.
In angular, however, I need to use $apply() to inject the property data into scope for each item in the user's 'thing' list. To keep things in sync, I suppose I can use firebase's on change event... But this all requires me to create new references for each thing in a user's list.
What is the best way to approach grabbing relational data in firebase, while keeping both the list and the relational data in sync?
Thanks!
Irfaan
If I understand correctly, it sounds like you should use FirebaseIndex and feed the index directly into an angularFireCollection. Then you wouldn't need to use $apply since the thing data will already be in the $scope, and everything will stay synced:
var index = new FirebaseIndex(fb.child('users/789/thing_list'), fb.child('things'));
$scope.things = angularFireCollection(index);
// $scope.things will contain the user's things with the associated thing data