I'm attempting to build a recommendation engine for a library system.
This is my db schema:
My starting point is a LoanerCard. The flow is then supposed to look like this: Get all copies -> get the material -> get all copies of the material (including the original) -> get LoanerCard from copy -> get all loaned copies -> return the material name of the copies + an aggregated count to indicate the strength of the recommendation.
My best attempt so far has resulted in this query:
MATCH (L:LoanerCard {Barcode:"10007"})-[:LOANED]->(myLoans)-[:COPY_OF]-
(masterMaterial),
(masterMaterial)<-[:COPY_OF]-(allCopies),
(allCopies)<-[:LOANED]-(coLoaners),
(coLoaners)-[r:LOANED]->(theirCopies),
(theirCopies)-[:COPY_OF]-(materials)
RETURN materials.Title as Recommended, count(*) as Strength ORDER BY Strength DESC
My issue here is that when I traverse the graph it doesn't include the original copy and the adjacent LoanerCards of that so essentially it only traverses the area circled in red and never reaches LoanerCard 10817 and 10558
How can I design my query so it includes these?
A MATCH clause automatically filters out duplicate relationships. Therefore, in order to traverse the same relationships twice, you need to split your MATCH clause in two.
Try this:
MATCH (:LoanerCard {Barcode:"10007"})-[:LOANED]->()-[:COPY_OF]-(masterMaterial)
MATCH (masterMaterial)<-[:COPY_OF]-()<-[:LOANED]-()-[:LOANED]->()-[:COPY_OF]-(materials)
RETURN materials.Title as Recommended, count(*) as Strength ORDER BY Strength DESC
Related
I’m currently working with the Nebula graph database for the first time and I’m running into some issues with a query. In terms of the schema, I have “Person” nodes, which have a “name” property, as well as Location nodes also with a name property. These node types can be connected by a relationship edge, called HAS_LIVED (to signify whether a person has lived in a certain location). Now for the query, I have a list of names (strings). The query looks like:
UNWIND [“Anna”, “Emma”, “Zach”] AS n
MATCH (p:Person {name: n})-[:HAS_LIVED]->(loc)
RETURN loc.Location.name
This should return a list of three places, i.e. [“London”, “Paris”, “Berlin”]. However, I am getting nothing as a result from the query. When I get rid of the UNWIND and write three separate MATCH queries with each name, it works individually. Not sure why.
Try this instead. It is using "where" clause.
UNWIND [“Anna”, “Emma”, “Zach”] AS n
MATCH (p:Person)-[:HAS_LIVED]->(loc)
where p.name = n
RETURN loc.Location.name
I have a large number of nodes representing accounts, which we could label as say (a :Account). Each (:Account) can have potentially tens of thousands of (t :Transaction) nodes connected to it, each representing the data for a transaction that occurred involving that account.
The (:Transaction) nodes have a date property. Given a date to query on what would be the most efficient way to get the latest (:Transaction) node for each (a :Account) that occurs before or on the query date? This could be one way to do it:
// run for all address nodes
match (a :Address)
with distinct a
optional match (a)-->(t :Transaction)
where t.timestamp <= date("2014-03-07")
with a, t
where t.date = max(t.date)
return a, t
However I'm not sure if this method is very efficient when the number of (t) connected to each (a) becomes very large. Is there a way to write the query or to index the database such that the query time scales linearly with the number of accounts, no matter the number of transactions connected to those account nodes?
For disclosure I posted a version of this question on the neo4j community forum, but I'm hoping the greater traffic on this site gives this question more exposure.
In neo4j 3.5, a new "index-backed order by" optimization was added. This means that if you create a "native" index (see here for the details), then the index will be stored in sorted order, and the ORDER BY clause on a property on which the index is used won't actually have to do any sorting.
So, assuming that you have created in index on :Transaction(timestamp), like so:
CREATE INDEX ON :Transaction(timestamp);
then, in neo4j 3.5+, this query (with an optional hint to use that index) should avoid any sorting when finding the Transaction with the maximum timestamp for each Address:
MATCH (a:Address)-->(t:Transaction)
USING INDEX t:Transaction(timestamp)
WHERE t.timestamp <= date("2014-03-07")
WITH a, t
ORDER BY t.timestamp DESC
RETURN a, COLLECT(t)[0] AS transaction
This query should do the following:
Use the index to get all Transaction nodes with an appropriate timestamp (in descending order, without sorting).
Get the Address nodes related to each Transaction.
For each distinct Address node, create a list of all the related Transaction nodes (in descending timestamp order, without sorting), and get the first one from the list.
Return each distinct Address node and its most recent appropriate Transaction node.
This query will scale linearly with the number of appropriate Transactions. If your use case permits it, you could get faster results by reducing the number of appropriate Transactions by also putting a lower bound in your WHERE clause.
I want to create a relationship between nodes that have one or multiple things in common and want to set the count of the common things as a property inside the relationship.
For example: in the movie-tutorial-graph I want to create a relationship between actors that have acted in the same movie(s) together and the set count of the movies they played in together as a property in the relationship.
For the basic counting, the tutorial provides a query:
MATCH (n)-[:ACTED_IN]->(m)<-[:ACTED_IN]-(coActors)
RETURN n.name, coActors.name, count(*) AS Strength ORDER BY Strength DESC
This gives me a list of 2 names and the amount of times they played in movies together (f.e. "Keanu Reeves", "Carrie-Anne Moss", Strength: 3 -> as there are the 3 Matrix movies inside the graph.)
Now I want to create a relationship (ACTED_WITH) between these pairs and set the strength-value as a property inside it.
I can create a relationship like this:
MATCH (a)-[:ACTED_IN]->(p)<-[:ACTED_IN]-(b) MERGE (a)-[r:ACTED_WITH]->(b)
MERGE ensures that there is only one relationship created, but I just can't get the counting-stuff to work with the creation.
I'm not sure to understand what you want but maybe something like that:
MATCH (a)-[:ACTED_IN]->(m)<-[:ACTED_IN]-(b)
WITH a,b, collect(m) AS movies ORDER BY size(movies) DESC
MERGE (a)-[r:ACTED_WITH]-(b)
ON CREATE SET r.Strength=SIZE(movies)
You can use the SET Cypher clause to set properties on a matched node. If you try to set it in the MERGE clause, than merge will treat the key(s) as a unique identifier, and will create a new relationship if one doesn't exist with that specific value yet.
MATCH (a)-[:ACTED_IN]->(p)<-[:ACTED_IN]-(b)
MERGE (a)-[r:ACTED_WITH]->(b)
// reduce matched set to one row of data
WITH DISTINCT a, b, r, COUNT(p) as strength, COLLECT(p) as movies
// set r
SET r.strength = strength
// Return everything to verify above results
RETURN *
SET will overwrite any previous value. If you want to only set it if you created the relationship, you can use ON CREATE or ON MATCH.
Given we have the following Neo4j schema (simplified but it shows the important point). There are two types of nodes NODE and VERSION. VERSIONs are connected to NODEs via a VERSION_OF relationship. VERSION nodes do have two properties from and until that denote the validity timespan - either or both can be NULL (nonexistent in Neo4j terms) to denote unlimited. NODEs can be connected via a HAS_CHILD relationship. Again these relationships have two properties from and until that denote the validity timespan - either or both can be NULL (nonexistent in Neo4j terms) to denote unlimited.
EDIT: The validity dates on VERSION nodes and HAS_CHILD relations are independent (even though the example coincidentally shows them being aligned).
The example shows two NODEs A and B. A has two VERSIONs AV1 until 6/30/17 and AV2 starting from 7/1/17 while B only has one version BV1 that is unlimited. B is connected to A via a HAS_CHILD relationship until 6/30/17.
The challenge now is to query the graph for all nodes that aren't a child (that are root nodes) at one specific moment in time. Given the example above, the query should return just B if the query date is e.g. 6/1/17, but it should return B and A if the query date is e.g. 8/1/17 (because A isn't a child of B as of 7/1/17 any more).
The current query today is roughly similar to that one:
MATCH (n1:NODE)
OPTIONAL MATCH (n1)<-[c]-(n2:NODE), (n2)<-[:VERSION_OF]-(nv2:ITEM_VERSION)
WHERE (c.from <= {date} <= c.until)
AND (nv2.from <= {date} <= nv2.until)
WITH n1 WHERE c IS NULL
MATCH (n1)<-[:VERSION_OF]-(nv1:ITEM_VERSION)
WHERE nv1.from <= {date} <= nv1.until
RETURN n1, nv1
ORDER BY toLower(nv1.title) ASC
SKIP 0 LIMIT 15
This query works relatively fine in general but it starts getting slow as hell when used on large datasets (comparable to real production datasets). With 20-30k NODEs (and about twice the number of VERSIONs) the (real) query takes roughly 500-700 ms on a small docker container running on Mac OS X) which is acceptable. But with 1.5M NODEs (and about twice the number of VERSIONs) the (real) query takes a little more than 1 minute on a bare-metal server (running nothing else than Neo4j). This is not really acceptable.
Do we have any option to tune this query? Are there better ways to handle the versioning of NODEs (which I doubt is the performance problem here) or the validity of relationships? I know that relationship properties cannot be indexed, so there might be a better schema for handling the validity of these relationships.
Any help or even the slightest hint is greatly appreciated.
EDIT after answer from Michael Hunger:
Percentage of root nodes:
With the current example data set (1.5M nodes) the result set contains about 2k rows. That's less than 1%.
ITEM_VERSION node in first MATCH:
We're using the ITEM_VERSION nv2 to filter the result set to ITEM nodes that have no connection other ITEM nodes at the given date. That means that either no relationship must exist that is valid for the given date or the connected item must not have an ITEM_VERSION that is valid for the given date. I'm trying to illustrate this:
// date 6/1/17
// n1 returned because relationship not valid
(nv1 ...)->(n1)-[X_HAS_CHILD ...6/30/17]->(n2)<-(nv2 ...)
// n1 not returned because relationship and connected item n2 valid
(nv1 ...)->(n1)-[X_HAS_CHILD ...]->(n2)<-(nv2 ...)
// n1 returned because connected item n2 not valid even though relationship is valid
(nv1 ...)->(n1)-[X_HAS_CHILD ...]->(n2)<-(nv2 ...6/30/17)
No use of relationship-types:
The problem here is that the software features a user-defined schema and ITEM nodes are connected by custom relationship-types. As we can't have multiple types/labels on a relationship the only common characteristic for these kind of relationships is that they all start with X_. That's been left out of the simplified example here. Would searching with the predicate type(r) STARTS WITH 'X_' help here?
What Neo4j version are you using.
What percentage of your 1.5M nodes will be found as roots at your example date, and if you don't have the limit how much data comes back? Perhaps the issue is not in the match so much as in the sorting at the end?
I'm not sure why you had the VERSION nodes in your first part, at least you don't describe them as relevant for determining a root node.
You didn't use relationship-types.
MATCH (n1:NODE) // matches 1.5M nodes
// has to do 1.5M * degree optional matches
OPTIONAL MATCH (n1)<-[c:HAS_CHILD]-(n2) WHERE (c.from <= {date} <= c.until)
WITH n1 WHERE c IS NULL
// how many root nodes are left?
// # root nodes * version degree (1..2)
MATCH (n1)<-[:VERSION_OF]-(nv1:ITEM_VERSION)
WHERE nv1.from <= {date} <= nv1.until
// has to sort all those
WITH n1, nv1, toLower(nv1.title) as title
RETURN n1, nv1
ORDER BY title ASC
SKIP 0 LIMIT 15
I think a good start for improvement would be to match on nodes using an index so you can quickly get a smaller relevant subset of nodes to search. Your approach right now must inspect all your :NODEs and all their relationships and patterns off of them every single time, which, as you've found, won't scale with your data.
Right now the only nodes in your graph with date/time properties are your :ITEM_VERSION nodes, so let's start with those. You'll need an index on :ITEM_VERSION's from and until properties for fast lookup.
The nulls are going to be problematic for your lookups, as any inequality against a null value returns null, and most workarounds to working with nulls (using COALESCE() or several ANDs/ORs for null cases) seem to prevent usage of index lookups, which is the point of my particular suggestion.
I would encourage you to replace your nulls in from and until with min and max values, which should let you take advantage of finding nodes by index lookup:
MATCH (version:ITEM_VERSION)
WHERE version.from <= {date} <= version.until
MATCH (version)<-[:VERSION_OF]-(node:NODE)
...
That should at least provide quick access to a smaller subset of nodes at the start for continuing your query.
I have recently stumbled upon a problem with selecting relationship details from a 1 table and inserting into another table, i hope someone can help.
I have a table structure as follows:
ID (PK) Name ParentID<br>
1 Myname 0<br>
2 nametwo 1<br>
3 namethree 2
e.g
This is the table i need to select from and get all the relationship data. As there could be unlimited number of sub links (is there a function i can create for this to create the loop ?)
Then once i have all the data i need to insert into another table and the ID's will now have to change as the id's must go in order (e.g. i cannot have id "2" be a sub of 3 for example), i am hoping i can use the same function for selecting to do the inserting.
If you are using SQL Server 2005 or above, you may use recursive queries to get your information. Here is an example:
With tree (id, Name, ParentID, [level])
As (
Select id, Name, ParentID, 1
From [myTable]
Where ParentID = 0
Union All
Select child.id
,child.Name
,child.ParentID
,parent.[level] + 1 As [level]
From [myTable] As [child]
Inner Join [tree] As [parent]
On [child].ParentID = [parent].id)
Select * From [tree];
This query will return the row requested by the first portion (Where ParentID = 0) and all sub-rows recursively. Does this help you?
I'm not sure I understand what you want to have happen with your insert. Can you provide more information in terms of the expected result when you are done?
Good luck!
For the retrieval part, you can take a look at Common Table Expression. This feature can provide recursive operation using SQL.
For the insertion part, you can use the CTE above to regenerate the ID, and insert accordingly.
I hope this URL helps Self-Joins in SQL
This is the problem of finding the transitive closure of a graph in sql. SQL does not support this directly, which leaves you with three common strategies:
use a vendor specific SQL extension
store the Materialized Path from the root to the given node in each row
store the Nested Sets, that is the interval covered by the subtree rooted at a given node when nodes are labeled depth first
The first option is straightforward, and if you don't need database portability is probably the best. The second and third options have the advantage of being plain SQL, but require maintaining some de-normalized state. Updating a table that uses materialized paths is simple, but for fast queries your database must support indexes for prefix queries on string values. Nested sets avoid needing any string indexing features, but can require updating a lot of rows as you insert or remove nodes.
If you're fine with always using MSSQL, I'd use the vendor specific option Adrian mentioned.