I have a document:
{
contact: {
id: '123'
},
channels: [
{
... some channel info...
}
],
lastUpdatedEpoch: 1583937675
}
And I have following query which doesn't return the above document:
SELECT p FROM p JOIN c IN p.channels
WHERE (NOT IS_DEFINED(p.lastUpdatedEpoch) OR p.lastUpdatedEpoch < 1585733881)
AND p.contact.id = '123'
But when I remove NOT IS_DEFINED check, it correctly returns the document:
SELECT p FROM p JOIN c IN p.channels
WHERE (p.lastUpdatedEpoch < 1585733881)
AND p.contact.id = '123'
I also tried replacing NOT IS_DEFINED clause with FALSE and it returns the document:
SELECT p FROM p JOIN c IN p.channels
WHERE (FALSE OR p.lastUpdatedEpoch < 1585733881)
AND p.contact.id = '123'
Also, if I remove JOIN, the query works as expected and returns the document:
SELECT p FROM
WHERE (NOT IS_DEFINED(p.lastUpdatedEpoch) OR p.lastUpdatedEpoch < 1585733881)
AND p.contact.id = '123'
To me this behavior is unexpected. When lastUpdatedEpoch is defined, I expect the same result from the first and second query (aside from the fact NOT_ISDEFINED will cause the index to be not used).
Could someone please explain what's going on here?
I try to reproduce your issue on my side but failed.The result is expected for me.
Test sample data:
Sql Output:
It seems that you did not refer any columns in channels.I suggest you create some simple test data to verify whether your sql is right.Then try to compare with your actual data.
I contacted CosmosDB team, and the team was able give some insight about the issue.
There was new optimization that was recently put in to allow inequality and NotIsDefined expressions to utilize the index. There was some issue with this optimization, and the team disabled this feature for now. If you are able to observe this issue with your cluster, please contact their support team.
Related
Having some trouble understanding how to create a Perl hash from a DB select statement.
$sth=$dbh->prepare(qq{select authorid,titleid,title,pubyear from books});
$sth->execute() or die DBI->errstr;
while(#records=$sth->fetchrow_array()) {
%Books = (%Books,AuthorID=> $records[0]);
%Books = (%Books,TitleID=> $records[1]);
%Books = (%Books,Title=> $records[2]);
%Books = (%Books,PubYear=> $records[3]);
print qq{$records[0]\n}
print qq{\t$records[1]\n};
print qq{\t$records[2]\n};
print qq{\t$records[3]\n};
}
$sth->finish();
while(($key,$value) = each(%Books)) {
print qq{$key --> $value\n};
}
The print statements work in the first while loop, but I only get the last result in the second key,value loop.
What am I doing wrong here. I'm sure it's something simple. Many thanks.
OP needs better specify the question and do some reading on DBI module.
DBI module has a call for fetchall_hashref perhaps OP could put it to some use.
In the shown code an assignment of a record to a hash with the same keys overwrites the previous one, row after row, and the last one remains. Instead, they should be accumulated in a suitable data structure.
Since there are a fair number of rows (351 we are told) one option is a top-level array, with hashrefs for each book
my #all_books;
while (my #records = $sth->fetchrow_array()) {
my %book;
#book{qw(AuthorID TitleID Title PubYear)} = #records;
push #all_books, \%book;
}
Now we have an array of books, each indexed by the four parameters.
This uses a hash slice to assign multiple key-value pairs to a hash.
Another option is a top-level hash with keys for the four book-related parameters, each having for a value an arrayref with entries from all records
my %books;
while (my #records = $sth->fetchrow_array()) {
push #{$books{AuthorID}}, $records[0];
push #{$books{TitleID}}, $records[1];
...
}
Now one can go through authors/titles/etc, and readily recover the other parameters for each.
Adding some checks is always a good idea when reading from a database.
I have a table with PK (String) and SK (Integer) - e.g.
PK_id SK_version Data
-------------------------------------------------------
c3d4cfc8-8985-4e5... 1 First version
c3d4cfc8-8985-4e5... 2 Second version
I can do a conditional insert to ensure we don't overwrite the PK/SK pair using ConditionalExpression (in the GoLang SDK):
putWriteItem := dynamodb.Put{
TableName: "example_table",
Item: itemMap,
ConditionExpression: aws.String("attribute_not_exists(PK_id) AND attribute_not_exists(SK_version)"),
}
However I would also like to ensure that the SK_version is always consecutive but don't know how to write the expression. In pseudo-code this is:
putWriteItem := dynamodb.Put{
TableName: "example_table",
Item: itemMap,
ConditionExpression: aws.String("attribute_not_exists(PK_id) AND attribute_not_exists(SK_version) **AND attribute_exists(SK_version = :SK_prev_version)**"),
}
Can someone advise how I can write this?
in SQL I'd do something like:
INSERT INTO example_table (PK_id, SK_version, Data)
SELECT {pk}, {sk}, {data}
WHERE NOT EXISTS (
SELECT 1
FROM example_table
WHERE PK_id = {pk}
AND SK_version = {sk}
)
AND EXISTS (
SELECT 1
FROM example_table
WHERE PK_id = {pk}
AND SK_version = {sk} - 1
)
Thanks
A conditional check is applied to a single item. It cannot be spanned across multiple items. In other words, you simply need multiple conditional checks. DynamoDb has transactWriteItems API which performs multiple conditional checks, along with writes/deletes. The code below is in nodejs.
const previousVersionCheck = {
TableName: 'example_table',
Key: {
PK_id: 'prev_pk_id',
SK_version: 'prev_sk_version'
},
ConditionExpression: 'attribute_exists(PK_id)'
}
const newVersionPut = {
TableName: 'example_table',
Item: {
// your item data
},
ConditionExpression: 'attribute_not_exists(PK_id)'
}
await documentClient.transactWrite({
TransactItems: [
{ ConditionCheck: previousVersionCheck },
{ Put: newVersionPut }
]
}).promise()
The transaction has 2 operations: one is a validation against the previous version, and the other is an conditional write. Any of their conditional checks fails, the transaction fails.
You are hitting your head on some of the differences between a SQL and a no-SQL database. DynamoDB is, of course, a no-SQL database. It does not, out of the box, support optimistic locking. I see two straight forward options:
Use a software layer to give you locking on your DynamoDB table. This may or may not be feasible depending on how often updates are made to your table. How fast 'versions' are generated and the maximum time your application can be gated on the lock will likely tell you if this can work foryou. I am not familiar with Go, but the Java API supports this. Again, this isn't a built-in feature of DynamoDB. If there is no such Go API equivalent, you could use the technique described in the link to 'lock' the table for updates. Generally speaking, locking a no-SQL DB isn't a typical pattern as it isn't exactly what it was created to do (part of which is achieving large scale on unstructured documents to allow fast access to many consumers at once)
Stop using an incrementor to guarantee uniqueness. Typically, incrementors are frowned upon in DynamoDB, in part due to the lack of intrinsic support for it and in part because of how DynamoDB shards you don't want a lot of similarity between records. Using a UUID will solve the uniqueness problem, but if you are porting an existing application that means more changes to the elements that create that ID and updates to reading the ID (perhaps to include a creation-time field so you can tell which is the newest, or the prepending or appending of an epoch time to the UUID to do the same). Here is a pertinent link to a SO question explaining on why to use UUIDs instead of incrementing integers.
Based on Hung Tran's answer, here is a Go example:
checkItem := dynamodb.TransactWriteItem{
ConditionCheck: &dynamodb.ConditionCheck{
TableName: "example_table",
ConditionExpression: aws.String("attribute_exists(pk_id) AND attribute_exists(version)"),
Key: map[string]*dynamodb.AttributeValue{"pk_id": {S: id}, "version": {N: prevVer}},
},
}
putItem := dynamodb.TransactWriteItem{
Put: &dynamodb.Put{
TableName: "example_table",
ConditionExpression: aws.String("attribute_not_exists(pk_id) AND attribute_not_exists(version)"),
Item: data,
},
}
writeItems := []*dynamodb.TransactWriteItem{&checkItem, &putItem}
_, _ = db.TransactWriteItems(&dynamodb.TransactWriteItemsInput{TransactItems: writeItems})
I have the following query
(TYPE:"ecmcndintregst:nd_int_reg_standards" OR TYPE:"ecmcndcountryst:nd_country_standards") AND (=ecmcnddoc:doc_name_ru:"" OR =ecmcnddoc:doc_name_ru:"\-") AND (=ecmcnddoc:doc_kind_cp_ecmcdict_value:"standard_itu")
and it has different results in Alfresco NodeBrowser (fts-alfresco) and in Java code "solr-fts-alfresco" (both results should be < 1000 and they are)
SearchParameters searchParameters = new SearchParameters();
searchParameters.setLanguage(SearchService.LANGUAGE_SOLR_FTS_ALFRESCO);
searchParameters.addStore(StoreRef.STORE_REF_WORKSPACE_SPACESSTORE);
searchParameters.setLimitBy(LimitBy.UNLIMITED);
searchParameters.setLimit(1000);
searchParameters.setPermissionEvaluation(PermissionEvaluationMode.EAGER);
searchParameters.addLocale(new Locale("ru", "RU"));
searchParameters.setQuery(query);
tempResultSet = customSearchService.query(searchParameters);
Also, in the Java code the clause =ecmcnddoc:doc_name_ru:""may fails two different ways: as always FALSE in query:
(TYPE:"ecmcndintregst:nd_int_reg_standards" OR TYPE:"ecmcndcountryst:nd_country_standards") AND (=ecmcnddoc:doc_name_ru:"" OR =ecmcnddoc:doc_name_ru:"\-") AND (=ecmcnddoc:doc_kind_cp_ecmcdict_value:"standard_itu")
And always TRUE in query:
(TYPE:"ecmcndintregst:nd_int_reg_standards" OR TYPE:"ecmcndcountryst:nd_country_standards") AND (=ecmcnddoc:doc_name_ru:"") AND (=ecmcnddoc:doc_kind_cp_ecmcdict_value:"standard_itu")
Could you tell me the proper way to use =ecmcnddoc:doc_name_ru:"" clause?
Thank you!
Assuming you're looking for results that have an empty "doc_name_ru", have you tried:
-EXISTS:"ecmcnddoc:doc_name_ru"
I'm learning LINQ, and I'm trying to figure out how to get all members with the last order failed (each member can have many orders). For efficiency reasons I'd like to do it all in LINQ before putting it into a list, if possible.
So far I believe this is the right way to get all the members with a failed order which joined recently (cutoffDate is current date -10 days).
var failedOrders =
from m in context.Members
from o in context.Orders
where m.DateJoined > cutoffDate
where o.Status == Failed
select m;
I expect I need to use Last or LastOrDefault, or possibly I need to use
orderby o.OrderNumber descending
and then get the First or FirstOrDefault as suggested in this stackoverflow answer.
Note that I want to look at ONLY the last order for a given member and see if that has failed (NOT just find last failed order).
Normally you would write something like:
var failedOrders = from m in context.Members
where m.DateJoined > cutoffDate
select new
{
Member = m,
LastOrder = m.Orders.OrderByDescending(x => x.OrderNumber).FirstOrDefault()
} into mlo
// no need for null checks here, because the query is done db-side
where mlo.LastOrder.Status == Failed
select mlo; // or select mlo.Member to have only the member
This if there is a Members.Orders relationship
EDIT: I accidentally misrepresented the problem when trying to pare-down the example code. A key part of my code is that I am attempting to sort the array after adding elements to it. The hang appears on sort, not insert. The following abstracted code will consistently hang:
<?=
local('a' = array)
#a->insert('test1' = map('a'='1'))
#a->insert('test2' = map('b'='2')) // comment-out to make work
#a->sort
#a
?>
I have a result set for which I want to insert a pair of values into an array for each unique key, as follows:
resultset(2) => {
records => {
if(!$logTypeClasses->contains(field('logTypeClass'))) => {
local(i) = pair(field('logTypeClass'), map('title' = field('logType'), 'class' = field('logTypeClass')))
log_critical(#i)
$logTypeClasses->insert(#i) // Lasso hangs on this line, will return if commented-out
}
}
}
Strangely, I cannot insert the #i local variable into thread variable without Lasso hanging. I never receive an error, and the page never returns. It just hangs indefinitely.
I do see the pairs logged correctly, which leads me to believe that the pair-generating syntax is correct.
I can make the code work as long as the value side of the pair is not a map with values. In other words, it works when the value side of the pair is a string, or even an empty map. As soon as I add key=value parameters to the map, it fails.
I must be missing something obvious. Any pointers? Thanks in advance for your time and consideration.
I can verify the bug with the basic code you sent with sorting. The question does arise how exactly one sorts pairs. I'm betting you want them sorted by the first element in the pair, but I could also see the claim that they should be sorted by last element in the pair (by values instead of by keys)
One thing that might work better is to keep it as a map of maps. If you need the sorted data for some reason, you could do map->keys->asArray->sort
Ex:
local(data) = map('test1' = map('a'=2,'b'=3))
#data->insert('test2' = map('c'=33, 'd'=42))
local(keys) = #data->keys->asArray
#keys->sort
#keys
Even better, if you're going to just iterate through a sorted set, you can just use a query expression:
local(data) = map('test1' = map('a'=2,'b'=3))
#data->insert('test2' = map('c'=33, 'd'=42))
with elm in #data->eachPair
let key = #elm->first
let value = #elm->second
order by #key
do { ... }
I doubt you problem is the pair with map construct per se.
This test code works as expected:
var(testcontainer = array)
inline(-database = 'mysql', -table = 'help_topic', -findall) => {
resultset(1) => {
records => {
if(!$testcontainer->contains(field('name'))) => {
local(i) = pair(field('name'), map('description' = field('description'), 'name' = field('name')))
$testcontainer->insert(#i)
}
}
}
}
$testcontainer
When Lasso hangs like that with no feedback and no immediate crash it is usually trapped in some kind of infinite loop. I'm speculating that it might have to do with Lasso using references whenever possible. Maybe some part of your code is using a reference that references itself. Or something.