Search by array values in Firebase - firebase

I use Firebase via REST API. I have following database structure:
{
"categories" : {
"Cat1" : {},
"Cat2" : {},
"Cat3" : {},
"Cat4" : {}
},
"items" : {
"item1" : {
"categories": ["Cat1", "Cat3"]
},
"item2" : {
"categories": ["Cat1", "Cat3"]
},
"item3" : {
"categories": ["Cat1", "Cat2", "Cat3"]
},
"item4" : {
"categories": ["Cat4"]
}
}
}
As you could see we have relations of type "N <-> N" between categories and items (one category could have several items and one item could be in several categories).
Now I want to get all items of Cat1 via Firebase REST API, but I can not do it.
As we know arrays are stored in the Firebase like map with integral indexes:
"categories": {
"0": "Cat1",
"1": "Cat2",
"2": "Cat3",
}
So, I added ".indexOn": ["categories/*"] to Realtime Database Rules and tried to call this:
curl 'https://...firebaseio.com/...json?orderBy="categories/*"&equalTo="Cat1"'
But I got only this: { }
So, I think that regular expressions do not work in Firebase queries, because this worked:
".indexOn": ["categories/0"] in Realtime Database Rules and
curl 'https://...firebaseio.com/...json?orderBy="categories/0"&equalTo="Cat1"'
Of course, I could change the database model to something like this:
{
"categories" : {
"Cat1" : {},
"Cat2" : {},
"Cat3" : {},
"Cat4" : {}
},
"items" : {
"item1" : {},
"item2" : {},
"item3" : {},
"item4" : {}
},
"category-items": {
"Cat1": ["item1", "item2", "item3"],
"Cat2": ["item3"],
"Cat3": ["item1", "item2", "item3"]
"Cat4": ["item4"]
}
}
And get the category-items and iterate through the Cat1 array, but then I must to call REST API read method too many times (one REST API call for every item in the category). So, it is too expensive.
So, could anybody help me with getting all items in a category in origin database model?
UPDATE
The final model is:
{
"categories" : {
"Cat1" : {},
"Cat2" : {},
"Cat3" : {},
"Cat4" : {}
},
"items" : {
"item1" : {
"Cat1": true,
"Cat3": true,
},
"item2" : {
"Cat1": true,
"Cat3": true,
},
"item3" : {
"Cat1": true,
"Cat2": true,
"Cat3": true,
},
"item4" : {
"Cat4": true
}
}
}
Also I added
{
rules": {
...
"items": {
".indexOn": [ "Cat1", "Cat2", "Cat3", "Cat4" ]
}
}
}
to Realtime Database Rules, and REST API call is
curl 'https://...firebaseio.com/items.json?orderBy="Cat1"&equalTo=tr‌​ue'
Thanks to Vladimir Gabrielyan

Here is the structure which I would suggest to have.
{
"categories" : {
"Cat1" : {
"items": {
"item1":{/*Some item info*/},
"item2":{/*Some item info*/}
}
},
"Cat2" : {
"items": {
"item3":{/*Some item info*/}
}
},
},
"items" : {
"item1" : {
"categories": {
"Cat1": true,
}
},
"item3" : {
"categories": {
"Cat2": true,
"Cat3": true
}
}
}
}
Inside Cat1/Items/{itemId} and items/{itemId} you need to duplicate your item information, but I think that is okay.
Also see this article. https://firebase.googleblog.com/2013/04/denormalizing-your-data-is-normal.html

Wow! Thank you very much! Your suggestion with replace
"item1" : {
"categories": ["Cat1", "Cat3"]
},
to
"item1" : {
"Cat1": true,
"Cat3": true
},
can solve the problem, but then I will have to add every Cat to .indexOn in Realtime Database Rules, but this is not so big problem as origin problem.
But I think that
"categories" : {
"Cat1" : {
"items": {
"item1":{/*Some item info*/},
"item2":{/*Some item info*/}
}
},
"Cat2" : {
"items": {
"item3":{/*Some item info*/}
}
},
}
is not a good idea in my case because then we get many spare data every time we get information about Cat (when we no need list of items, only metadata of Cat). So, I suggest following model:
{
"categories" : {
"Cat1" : {},
"Cat2" : {},
"Cat3" : {},
"Cat4" : {}
},
"items" : {
"item1" : {
"Cat1": true,
"Cat3": true,
},
"item2" : {
"Cat1": true,
"Cat3": true,
},
"item3" : {
"Cat1": true,
"Cat2": true,
"Cat3": true,
},
"item4" : {
"Cat4": true
}
}
}

Related

Updating item in DynamoDB fails for the UpdateExpression syntax

My table data looks like below one
{
"id": {
"S": "alpha-rocket"
},
"images": {
"SS": [
"apple/value:50",
"Mango/aa:284_454_51.0.0",
"Mango/bb:291",
"Mango/cc:4"
]
},
"product": {
"S": "fruit"
}
}
Below is my code to update table. The variables I am passing to function has values product_id has alpha-rocket, image_val has 284_454_53.0.0 and image has Mango/aa:284_454_53.0.0.
I am trying to update value of Mango/aa from 284_454_51.0.0 to 284_454_53.0.0 but getting an error "The document path provided in the update expression is invalid for update"
def update_player_score(product_id, image_val, image):
dynamo = boto3.resource('dynamodb')
tbl = dynamo.Table('<TableName>')
result = tbl.update_item(
expression_attribute_names: {
"#image_name" => "image_name"
},
expression_attribute_values: {
":image_val" => image_val,
},
key: {
"product" => "fruit",
"id" => product_id,
},
return_values: "ALL_NEW",
table_name: "orcus",
update_expression: "SET images.#image_val = :image_val",
}
Is there a way to update the value of Mango/aa or replace full string "Mango/aa:284_454_51.0.0" to "Mango/aa:284_454_53.0.0"
You cannot update a string in a list by matching the string. If you know the index of it you can replace the value of the string by index:
SET images[1] = : image_val
It seems like maybe what you want is not a list of strings, but another map. So instead of your data looking like it does you'd make it look like this, which would allow you to do the update you're looking for:
{
"id": {
"S": "alpha-rocket"
},
"images": {
"M": {
"apple" : {
"M": {
"value": {
"S": "50"
}
},
"Mango" : {
"M": {
"aa": {
"S": "284_454_51.0.0"
},
"bb": {
"S": "291"
},
"cc": {
"S": "4"
}
}
}
},
"product": {
"S": "fruit"
}
}
I would also consider putting the different values in different "rows" in the table and using queries to build the objects.

Strange output query Elastic Search

I just started with using Elastic Search. I've got everything set-up correctly. I'm using Firebase + Flashlight + Elastic Search.
In my front-end I'm building queries based on different search params. I insert them into a node in Firebase /search/requests/. Flashlight will pick this up and putting the response into /search/response, this works like a charm!
However, I'm not sure how to write my queries properly. I'm getting strange results when I'm trying to combine two must match queries. I'm using Query DSL.
My documents in Elastic Search under deliverables/doc are having the following scheme.
...
{
"createdBy" : "admin#xx.org",
"createdOn" : 1501200000000,
"deadLine" : 1508716800000,
"description" : {
"value" : "dummy description"
},
"key" : "<FBKEY>",
"programmes" : [ {
"code" : "95000",
"name" : "Test programme",
"programYear" : 2017
} ],
"projects" : [ {
"projectCode" : "113200",
"projectName" : "Test project",
"projectYear" : 2017
} ],
"reportingYear" : 2017,
"status" : "Open",
"type" : "writing",
"updatedBy" : "admin#xx.org",
"updatedOn" : 1501200000000,
},
...
My query has the following structure.
{
"query": {
"bool": {
"must": [
{
"match": {
"createdBy": "xx#company.org"
},
"match": {
"programmes.code": "95000"
}
}
]
}
}
}
In my output I'm also getting documents that don't have exactly those two fields? They have a very low score as well. Is this normal?
My mapping, automatically created using Flashlight
Update 1
I just tried this query, however it still gives me strange results by not filtering on both fields:
{
"query": {
"bool": {
"filter": {
"bool": {
"must": [
{
"match": {
"programmes.code": "890000"
}
},
{
"match": {
"createdBy": "admin#xx.org"
}
}
]
}
}
}
}
}
The must clause used in bool query is executed in query context(all the documents are returned in decreasing order of score) and contributes to score. see link
If you want it to be executed as a filter, then use the following query:
{
"query": {
"bool": {
"filter": {
"bool": {
"must": [
{
"match": {
"createdBy": "xx#company.org"
}
},
{
"match": {
"programmes.code": "95000"
}
}
]
}
}
}
}
}
NOTE:
By default the string field is analyzed, update the mapping of the string fields as not_analyzed, to use filter query. Refer: mapping-intro

Matching users with similar interest tags using Firebase, Elasticsearch and Flashlight

What is the best way to match documents by tags using elasticsearch using the following setup (or modifying the setup)?
I've got my users in a firebase database, they have associated tags that define their interests:
"users" : {
"bruce" : {
"martial art" : "Jeet Kune Do",
"name" : "Bruce Lee",
"nick" : "Little Phoenix",
"tags" : {
"android" : true,
"ios" : true
}
},
"chan" : {
"account_type" : "contractor",
"martial art" : "Kung Fu",
"name" : "Jackie Chan",
"nick" : "Cannonball",
"tags" : {
"ios" : true
}
},
"chuck" : {
"martial art" : "Chun Kuk Do",
"name" : "Carlos Ray Norris",
"nick" : "Chuck"
}}
Using Flashlight + the Firebase admin SDK I'm keeping an index up to date on Bonsai/heroku, that supposedly will help me to match users with similar interests or related products.
"firebase": {
"aliases": {},
"mappings": {
"user": {
"properties": {
"name": {
"type": "string"
},
"tags": {
"properties": {
"android": {
"type": "boolean"
},
"ios": {
"type": "boolean"
}
}
}
}
}
}...
For now I can query users with certain combination of tags:
{
"query": {
"bool": {
"must" : {
"type" : {
"value" : "user"
}
},
"should": [
{
"term": {
"tags.ios": true
}
},
{
"term": {
"tags.android": true
}
}
],
"minimum_should_match" : 1
}}}
This is great but what I'm looking for is a way to:
Given a user id find other users with similar tags ordered by _score.
There will be other _type's apart from "user" using also tags, for example products so it would also be great to match products to users when they share some tags.
I get the feeling that because I'm absolutely new on elastic search I'm targeting this in the wrong way. Maybe the way the data is modeled?
Problem is that firebase kind of restricts this a lot, for instance I cannot have arrays, so that makes the tag modeling a bit weird ending in even more weird indexed data...maybe an approach could be to manipulate the data before inserting it to the index?

How to structure data in Firebase for filtering potentially large datasets?

I have a node in my realtime database with thousands of children, and want to filter on these, both quick and bandwidth saving (the latter is maybe not that important at this point, but might be when my data grows).
What is the best way of structuring this to avoid fetching all the items and doing the filtering on the client?
Here is what i'm planning on implementing, but as I have no experience with Firebase or other NoSQL databases, I do need some input :)
{
"items" : {
"item1" : {
"name" : "Item 1",
"filters": {
"filter1" : true,
"filter2" : true,
"filter3" : true
}
},
"item2" : {
"name" : "Item 2",
"filters": null
},
"item3" : {
"name" : "Item 3",
"filters": {
"filter2" : true
}
},
"item4" : {
"name" : "Item 4",
"filters": {
"filter2" : true,
"filter3" : true
}
},
"item5" : {
"name" : "Item 5",
"filters": {
"filter3" : true
}
}
// Thousands of items
},
"items_by_filter1" : {
"item1" : true
},
"items_by_filter2" : {
"item1" : true,
"item3" : true
},
"items_by_filter3" : {
"item1" : true,
"item4" : true,
"item5" : true
}
}
Am I overthinking this, or is this a good approach? What if I want to filter on several filters, should I follow the same approach and add something like this for all filter combinations (probably a struggle to maintain)?
"items_by_filter2_and_filter3" : {
"item1" : true,
"item4" : true
}

Elastic Search Date Parsing Error

I'm pretty new at configuring elastic and I am having problems trying to parse a log date - which seems like it should be a trivial thing to do.
Any insight for a newbie?
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "failed to parse [Message.LogTime]"
}
],
"type": "mapper_parsing_exception",
"reason": "failed to parse [Message.LogTime]",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "Invalid format: \"2015-11-12 01:37:35.490\" is malformed at \" 01:37:35.490\""
}
}
My JSON payload
{
"LoggerType": "ErrorAndInfo",
"Message": {
"LogId": 0,
"LogStatus": 0,
"LogTime": "2015-11-12 01:37:35.490",
"VersionInfo": "",
"AdditionalInformation": null
}
}
Elastic Search Template Mapping
"mappings": {
"log_message" : {
"_all" : { "enabled": false },
"properties": {
"LoggerType" : { "type" : "string" },
"Message" : {
"properties": {
"LogId": { "type" : "integer" },
"LogStatus": { "type" : "integer" },
"LogTime": {
"type" : "date",
"format" : "yyyy-MM-dd HH:mm:ss.SSS"
},
"VersionInfo": {
"type" : "string",
"index" : "not_analyzed"
},
}
}
}
}
}
I figured it out. You will have to re-create your index for the changes to be applied.

Resources