I've read all the docs, but am struggling with this query in GCP Firestore in Datastore Mode:
SELECT id, title FROM `Book` WHERE author = 'Twain'
I've tried all the permutations of properties and indexes and where clauses that I can think of... What works:
SELECT id, title FROM `Book`
because I have created this index.yaml and built it with gcloud datastore indexes create index.yaml
indexes:
- kind: Book
ancestor: no
properties:
- name: id
- name: title
I can also
SELECT * FROM `Book` WHERE author = 'Twain'
because author is an indexed property of the Book entity.
I've tried adding author to the index (in addition to title and id), but Datastore still complains "GQL Query error: Your Datastore does not have the composite index (developer-supplied) required for this query."
What am I missing? The entity is quite large, and I just want to retrieve a list of the titles to populate the web page, not every property! I also don't want to retrieve the entire set of entities, since obviously only a few of them have been written by a particular author.
If you open up the 'Developer Tools' in your favourite browser, you can see the full error returned to your query is:
"no matching index found. recommended index is:\n- kind: Book\n properties:\n - name: author\n - name: id\n - name: title\n",
Pretty printing that, we get:
- kind: Book
properties:
- name: author
- name: id
- name: title
Which is the expected index because the filtering happens on author, and the id and title fields are pulled from the index for your query.
Related
In my firebase db I have 3 collections:
Users
{user_id}: {name: "John Smith"}
Items
{item_id}: {value: 12345}
Actions
{action_id}: {action: "example", user: {user_id}, items:{item_id}}
Basically, instead of storing the Users and Items under the Actions Collection, I just keep an ID. But now I need a list of all actions and this also needs info from the Users and Items Collection. How can I efficiently query firebase so I can get a result that looks like this:
{
action: "example",
user: {
name: "John Smith"
},
item: {
value: 1234
}
}
Unfortunately, there is no such thing in firebase or a similar database, basically, you are looking for a traditional join, which is no recommended thing to do in a NoSQL database.
If you want to do it in firebase, you will need:
Get the element you are looking for from your main collection Actions in this case.
Then you need to do another call to the Items collections where item_id == action.item_id.
Then assign in the actions["Item"] = item_gotten.
This is not a recommended use as I said, usually, when you are using a NoSQL Database you are expecting a denormalize structure, from your application you need to save the whole Item, in the Action JSON, and also in the Item. Yes, you will have duplicate data but this is fine for this kind of model. also you shouldn't expect too many changes in one specific object within your whole object key If you are managing a big set of changes you could be using the incorrect kind of DB.
For aggregation queries reference, you might check: https://firebase.google.com/docs/firestore/solutions/aggregation
I am trying to migrate a few plugins that use DS to FS and I was wondering on the data structure. In my DS I am utilizing ancestors
so the top level Kind is Users and any other Kinds consist of ancestors from Users. E.g. Kind Products has ancestor Key(Users,'UUID').
In Firestore world would it be structure looks like this:
1. Users(Collections):
{userID:...
...
so on},
{...},
...list of users
2. Products(Collections).User-1(Doc)
Subcollections{...list of product docs belonging to User1}
.User-2(Doc)
Subcollections{...list of product docs belonging to User2}
Users and Products top-level collections.
or
this structure would be better:
+ Users (collection)
* user_1 (document)
- name: "Blah"
- last: "Blah"
+ Product (subcollection)
* product_1 (document)
- title: "blah...."
- vendor: "blah..."
+ Product_variants (subcollection)
* product_1 (document)
- name: "..."
- price: "..."
* product_2 (document)
- name: "..."
- price: "..."
* product_2 (document)
- title: "blah...."
- vendor: "blah..."
+ Product_variants (subcollection)
* product_1 (document)
- name: "..."
- price: "..."
* product_2 (document)
- name: "..."
- price: "..."
Is there a better way to handle this structure? Also concern from an action update perspective which simpler would be? I am trying to understand tradeoffs between update vs. query. For example if I have users that have more than 100K products and getting events on updates/deletes/... is there downside of that structure.
Update: As of May, 2019, Cloud Firestore now supports collection group queries.
You can now structure the data either way and still be able to query across users.
Original Answer
If I'm understanding correctly, you're asking about the trade-offs of having flat collections vs subcollections.
As far as updates are concerned, there aren't any material differences. One thing to look out for is if you have fields that cluster around a single value. For example, with flat collections if products had an update-time field then by default you'd be limited to 500 updates/second across all users. With products nested within users you're limited to 500 updates/second per user. However, with flattened collections you can work around this by disabling the default single-field index on update-time and creating a composite index on (user, update-time). Once you do that, these are equivalent.
The real difference comes down to which queries are possible. In Firestore as it exists today, you can only query within a subcollection tree. So for example, if you wanted to search for products from a specific title or vendor, you'd only be able to search within a single user.
If you flatten the collections such that products is a top-level collection, you can query across users.
Note that collection group queries are a feature we're developing that will remove this restriction. Once that's launched you'll be able to structure the data either way and still be able to query across users.
I understand that enums are not standard type in Dynamo: https://forums.aws.amazon.com/thread.jspa?messageID=836386
However, what is the exact resolution here?
How are we supposed to appropriately represent relations with the generated code?
-- Am I missing something or is the generated code correct and we need to create some custom fields in the dynamo tables and then rewrite the queries?
Example:
type Competition {
id: ID!
name: String!
creator: UserProfile!
startDate: String!
endDate: String!
competitionType: CompetitionType!
competitors: [UserProfile]!
prize: Prize!
}
A competition is created by a user, has a type, a prize, and has competitors. When create resources for this table, the code is clearly missing any information that is derived out of the custom types or enums. Complex schemas will always have this type of structure, so I'm a bit confused on the outputted code and right direction from here.
extend type Mutation {
createCompetition(input: CreateCompetitionInput!): Competition
}
input CreateCompetitionInput {
id: ID!
name: String!
startDate: String!
endDate: String!
## Missing info
}
When AppSync generates the schema automatically it skips these as they are intended to be added manually with additional resolvers. You can define a new query that is attached to each of the custom or enum fields, but the data you are referencing will need to be stamped with something that is unique to the competition so that it can be queried on in relation to this type (as dynamoDB isn't a relational db).
When creating a new Competition you will need to update child fields with something unique to that competition. I.e. each UserProfile that needs to be tracked as a competitor gets stamped with this Competitions ID. Mutations for each of the custom fields need to be handled separately.
This article helped me solve this same question: https://keyholesoftware.com/2018/05/17/go-forth-and-appsync/.
Is it possible to retrieve multiple nodes separated by commas from Firebase?
For example, get the name and image properties of an user using:
/users/$USER_ID/name,image
instead of downloading all the user information
The Firebase SDK always retrieves complete nodes from the database. There is no way to select a subset of the nodes.
If you find you want a subset of the nodes and that downloading all nodes would be overly wasteful, you should model your data to have an additional collection where you only keep the properties you want for each user:
user
uid1
name: "TheUnreal"
image: "https://graph.facebook.com/1214076718653094/picture?type=large"
stackid: 3669981
lastquestionid: 40565750
uid2
name: "Frank van Puffelen"
image: "https://www.gravatar.com/avatar/12d378e6a9788ab9c94bbafe242b82b4?s=48&d=identicon&r=PG"
stackid: 209103
lastquestionid: 39984247
users_names_images
uid1
name: "TheUnreal"
image: "https://graph.facebook.com/1214076718653094/picture?type=large"
uid2
name: "Frank van Puffelen"
image: "https://www.gravatar.com/avatar/12d378e6a9788ab9c94bbafe242b82b4?s=48&d=identicon&r=PG"
With this data structure you can choose whether you get a subset of the data or the complete data by querying the corresponding top-level node. Duplicating data like this is fairly common in NoSQL databases, where you often model the data for how your app consumes it.
You can retrieve the nodes using a class. I do it like that. Why don't you manipulate the returned values so as to make it in a way you'd be able to use.
Coming from a mongodb background, I'd like to set up a document with an embedded collection.
For instance if I have a profile object
Profile
name : string
followers : [
name: string
]
such that it has an embedded collection of followers.
Is there a way that I can create an index on Profile so that I can query for all profiles where Profile.Followers includes myUsername?
In short I can query for profiles I'm following from a dynamoDB table?
In mongo I can easily do this by setting up an index on Profile.followers and doing an $in query. Is there something similar for dynamodb?
This documentation suggests there is nothing like an in clause
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/QueryAndScan.html
Currently DynamoDB does not support indices for non scalar types (i.e. Set, List, or Map data types - see here: http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/SecondaryIndexes.html). If you have a separate users table, you can keep track of all profiles you are following in a Set/List attribute.