Gremlin union query takes long time to execute against Neptune DB - gremlin

Question

The following Gremlin query takes long time when you run using union clause. This is a common search query which is being used from many pages.
It works fine if you use it on entities with fewer nodes. However, if you use it to search for entities with higher number of nodes it times out from UI (after 30 secs).
I have to use containing as it searches as user starts typing (after 4 chars).
I am using AWS Neptune database with Python Gremlin.
It works for entities with 30k nodes.
It is timing out for entities with 200k+ nodes.
When I run these clauses separately it works fine, looks like union is taking long time.
g.V().hasLabel("org").union(
        has(T.id, containing("{searchtext}")),
        has("name", containing("{searchtext}")),
        has("tin", containing("{searchtext}")),
     where(in_("orgcode").has(T.id, containing("{searchtext}".upper())))).limit(10).dedup().project("id","name").by(__.id()).by("name").toList()
*******************************************************
Neptune Gremlin Profile
*******************************************************
Query String
==================
g.V().hasLabel("org").union(
has(T.id, containing("0000804415")),
has("name", containing("0000804415")),
has("tin", containing("0000804415")),
inE("orgcode").has(T.id, containing("0000804415"))).limit(10).dedup().project("id","name").by(__.id()).by("name")
Original Traversal
==================
[GraphStep(vertex,[]), HasStep([~label.eq(org)]), UnionStep([[HasStep([~id.containing(0000804415)]), EndStep], [HasStep([name.containing(0000804415)]), EndStep], [HasStep([tin.containing(0000804415)]), EndStep], [VertexStep(IN,[orgcode],edge), HasStep([~id.containing(0000804415)]), EndStep]]), RangeGlobalStep(0,10), DedupGlobalStep, ProjectStep([id, name],[[IdStep], value(name)])]
Optimized Traversal
===================
Neptune steps:
[
NeptuneGraphQueryStep(Vertex) {
JoinGroupNode {
PatternNode[(?1, <~label>, ?2=<org>, <~>) . project ?1 .], {indexTime=0, joinTime=166, numSearches=1}
}, annotations={path=[Vertex(?1):GraphStep], joinStats=true, optimizationTime=0, maxVarId=15, chunkSize=10, executionTime=16936}
},
NeptuneUnionStep {
NeptuneGraphQueryStep(Vertex) {
JoinGroupNode {
FilterByP(?1: containing(0000804415)) .
}, annotations={initialValues={?1=null}, executionTime=16936, path=[Vertex(?1):GraphStep], chunkSize=10, optimizationTime=0, maxVarId=15, joinStats=true}
},
NeptuneGraphQueryStep(Vertex) {
JoinGroupNode {
PatternNode[(?1, <name>, ?9, ?) . project ask . FilterByP(?9: containing(0000804415)) .], {indexTime=102, joinTime=4304, numSearches=205662}
}, annotations={initialValues={?1=null}, executionTime=16936, path=[Vertex(?1):GraphStep], chunkSize=10, optimizationTime=0, maxVarId=15, joinStats=true}
},
NeptuneGraphQueryStep(Vertex) {
JoinGroupNode {
PatternNode[(?1, <tin>, ?10, ?) . project ask . FilterByP(?10: containing(0000804415)) .], {indexTime=93, joinTime=4215, numSearches=205662}
}, annotations={initialValues={?1=null}, executionTime=16935, path=[Vertex(?1):GraphStep], chunkSize=10, optimizationTime=0, maxVarId=15, joinStats=true}
},
NeptuneGraphQueryStep(Edge) {
JoinGroupNode {
PatternNode[(?11, ?13=<orgcode>, ?1, ?14) . project ?1,?14 . IsEdgeIdFilter(?14) . FilterByP(?14: containing(0000804415)) .], {indexTime=331, joinTime=4847, numSearches=20567}
}, annotations={initialValues={?1=null}, executionTime=16935, path=[Vertex(?1):GraphStep, Edge(?14):VertexStep], chunkSize=10, optimizationTime=0, maxVarId=15, joinStats=true}
}
},
NeptuneTraverserConverterStep
]
+ not converted into Neptune steps: RangeGlobalStep(0,10),
Neptune steps:
[
NeptuneMemoryTrackerStep
]
+ not converted into Neptune steps: DedupGlobalStep,ProjectStep([id, name],[[IdStep], value(name)]),
WARNING: >> [RangeGlobalStep(0,10), DedupGlobalStep] << (or one of the children for each step) is not supported natively yet
Physical Pipeline
=================
NeptuneGraphQueryStep
|-- StartOp
|-- JoinGroupOp
|-- SpoolerOp(10)
|-- DynamicJoinOp(PatternNode[(?1, <~label>, ?2=<org>, <~>) . project ?1 .])
NeptuneUnionStep
|-- BindingSetQueue
|-- JoinGroupOp
|-- FilterOp(FilterByP(?1: containing(0000804415)) .)
|-- BindingSetQueue
|-- JoinGroupOp
|-- SpoolerOp(10)
|-- DynamicJoinOp(PatternNode[(?1, <name>, ?9, ?) . project ask . FilterByP(?9: containing(0000804415)) .])
|-- BindingSetQueue
|-- JoinGroupOp
|-- SpoolerOp(10)
|-- DynamicJoinOp(PatternNode[(?1, <tin>, ?10, ?) . project ask . FilterByP(?10: containing(0000804415)) .])
|-- BindingSetQueue
|-- JoinGroupOp
|-- SpoolerOp(10)
|-- DynamicJoinOp(PatternNode[(?11, ?13=<orgcode>, ?1, ?14) . project ?1,?14 . IsEdgeIdFilter(?14) . FilterByP(?14: containing(0000804415)) .])
Runtime (ms)
============
Query Execution: 16936.285
Serialization: 0.085
Traversal Metrics
=================
Step Count Traversers Time (ms) % Dur
-------------------------------------------------------------------------------------------------------------
NeptuneGraphQueryStep(Vertex) 205662 205662 282.539 1.67
NeptuneUnionStep([[NeptuneGraphQueryStep(Vertex... 16653.466 98.33
NeptuneTraverserConverterStep 0.023 0.00
RangeGlobalStep(0,10) 0.003 0.00
NeptuneMemoryTrackerStep 0.006 0.00
DedupGlobalStep 0.004 0.00
ProjectStep([id, name],[[IdStep], value(name)]) 0.004 0.00
>TOTAL - - 16936.048 -
Predicates
==========
# of predicates: 201
Results
=======
Count: 0
Output: []
Response serializer: application/vnd.gremlin-v3.0+json
Response size (bytes): 216
Index Operations
================
Query execution:
# of statement index ops: 431892
# of unique statement index ops: 431892
Duplication ratio: 1.0
# of terms materialized: 910040
Serialization:
# of statement index ops: 0
# of terms materialized: 0
%%gremlin

Answer 1

First, you can get better details of the execution of the query on Neptune by using Neptune's Gremlin query explainer (versus the default TinkerPop explain() or profile() steps). To do this use %%gremlin profile when executing the query in a cell.
On to the query...
It's likely not the union() that is causing the issues here, but the use of containing(). Neptune does not maintain a substring index of string properties within the database. So using any of the Gremlin Text Predicates [1] will incur some sort of partial scan of all properties with the property key that you're using for those filters.
If you need to do these types of queries on a regular basis, it would beneficial to use Neptune's Full-Text-Search integration with OpenSearch [2].
[1] https://tinkerpop.apache.org/docs/current/reference/#a-note-on-predicates
[2] https://docs.aws.amazon.com/neptune/latest/userguide/full-text-search-cfn-create.html