We had experienced deadlocks with some standard 'Select' queries provided by Activiti on the following tables:
ACT_RU_TASK, ACT_RU_EXECUTION and ACT_RE_PROCDEF.
Here is an example query:
### SQL: SELECT SUB.* FROM ( select distinct TEMPRES_ID_ as ID_, TEMPP_KEY_ as ProcessDefinitionKey, TEMPP_ID_ as ProcessDefinitionId, TEMPRES_REV_ as REV_, TEMPRES_ACT_ID_ as ACT_ID_, TEMPRES_BUSINESS_KEY_ as BUSINESS_KEY_, TEMPRES_IS_ACTIVE_ as IS_ACTIVE_, TEMPRES_IS_CONCURRENT_ as IS_CONCURRENT_, TEMPRES_IS_SCOPE_ as IS_SCOPE_, TEMPRES_IS_EVENT_SCOPE_ as IS_EVENT_SCOPE_, TEMPRES_PARENT_ID_ as PARENT_ID_, TEMPRES_PROC_INST_ID_ as PROC_INST_ID_, TEMPRES_SUPER_EXEC_ as SUPER_EXEC_, TEMPRES_SUSPENSION_STATE_ as SUSPENSION_STATE_, TEMPRES_CACHED_ENT_STATE_ as CACHED_ENT_STATE_, TEMPVAR_ID_ as VAR_ID_, TEMPVAR_NAME_ as VAR_NAME_, TEMPVAR_TYPE_ as VAR_TYPE_, TEMPVAR_REV_ as VAR_REV_, TEMPVAR_PROC_INST_ID_ as VAR_PROC_INST_ID_, TEMPVAR_EXECUTION_ID_ as VAR_EXECUTION_ID_, TEMPVAR_TASK_ID_ as VAR_TASK_ID_, TEMPVAR_BYTEARRAY_ID_ as VAR_BYTEARRAY_ID_, TEMPVAR_DOUBLE_ as VAR_DOUBLE_, TEMPVAR_TEXT_ as VAR_TEXT_, TEMPVAR_TEXT2_ as VAR_TEXT2_, TEMPVAR_LONG_ as VAR_LONG_ , row_number() over (ORDER BY TEMPRES_ID_ asc) rnk FROM ( select distinct RES.ID_ as TEMPRES_ID_, RES.REV_ as TEMPRES_REV_, P.KEY_ as TEMPP_KEY_, P.ID_ as TEMPP_ID_, RES.ACT_ID_ as TEMPRES_ACT_ID_, RES.PROC_INST_ID_ as TEMPRES_PROC_INST_ID_, RES.BUSINESS_KEY_ as TEMPRES_BUSINESS_KEY_, RES.IS_ACTIVE_ as TEMPRES_IS_ACTIVE_, RES.IS_CONCURRENT_ as TEMPRES_IS_CONCURRENT_, RES.IS_SCOPE_ as TEMPRES_IS_SCOPE_, RES.IS_EVENT_SCOPE_ as TEMPRES_IS_EVENT_SCOPE_, RES.PARENT_ID_ as TEMPRES_PARENT_ID_, RES.SUPER_EXEC_ as TEMPRES_SUPER_EXEC_, RES.SUSPENSION_STATE_ as TEMPRES_SUSPENSION_STATE_, RES.CACHED_ENT_STATE_ as TEMPRES_CACHED_ENT_STATE_, VAR.ID_ as TEMPVAR_ID_, VAR.NAME_ as TEMPVAR_NAME_, VAR.TYPE_ as TEMPVAR_TYPE_, VAR.REV_ as TEMPVAR_REV_, VAR.PROC_INST_ID_ as TEMPVAR_PROC_INST_ID_, VAR.EXECUTION_ID_ as TEMPVAR_EXECUTION_ID_, VAR.TASK_ID_ as TEMPVAR_TASK_ID_, VAR.BYTEARRAY_ID_ as TEMPVAR_BYTEARRAY_ID_, VAR.DOUBLE_ as TEMPVAR_DOUBLE_, VAR.TEXT_ as TEMPVAR_TEXT_, VAR.TEXT2_ as TEMPVAR_TEXT2_, VAR.LONG_ as TEMPVAR_LONG_ from ACT_RU_EXECUTION RES inner join ACT_RE_PROCDEF P on RES.PROC_DEF_ID_ = P.ID_ left outer join ACT_RU_VARIABLE VAR ON RES.PROC_INST_ID_ = VAR.EXECUTION_ID_ and VAR.TASK_ID_ is null WHERE RES.PARENT_ID_ is null and P.KEY_ = ? and RES.BUSINESS_KEY_ = ? )RES ) SUB WHERE SUB.rnk >= ? AND SUB.rnk < ?
Activiti version is 5.15.1, but note that we cannot upgrade it due to a strict project timeline.
Does anyone experienced something similar, or does anyone have an idea to start understanding why this happens and how to prevent it?
When you say "deadlocks" are you referring to the Optimistic locking exceptions or actual deadlocks?
If actual deadlocks, are you using MSSql server DB?
I ask because the causes of the optimistic locking exceptions are relatively well known and can be almost always resolved with good process modeling.
If it is the MSSql deadlock, then it is possibly the same issue reported under the Camunda (an Activiti fork) here:
https://app.camunda.com/jira/si/jira.issueviews:issue-html/CAM-1646/CAM-1646.html
There are instructions for resolution in the Camunda Jira issue if it is this issue.
Cheers,
Greg
Related
I have a table with PK (String) and SK (Integer) - e.g.
PK_id SK_version Data
-------------------------------------------------------
c3d4cfc8-8985-4e5... 1 First version
c3d4cfc8-8985-4e5... 2 Second version
I can do a conditional insert to ensure we don't overwrite the PK/SK pair using ConditionalExpression (in the GoLang SDK):
putWriteItem := dynamodb.Put{
TableName: "example_table",
Item: itemMap,
ConditionExpression: aws.String("attribute_not_exists(PK_id) AND attribute_not_exists(SK_version)"),
}
However I would also like to ensure that the SK_version is always consecutive but don't know how to write the expression. In pseudo-code this is:
putWriteItem := dynamodb.Put{
TableName: "example_table",
Item: itemMap,
ConditionExpression: aws.String("attribute_not_exists(PK_id) AND attribute_not_exists(SK_version) **AND attribute_exists(SK_version = :SK_prev_version)**"),
}
Can someone advise how I can write this?
in SQL I'd do something like:
INSERT INTO example_table (PK_id, SK_version, Data)
SELECT {pk}, {sk}, {data}
WHERE NOT EXISTS (
SELECT 1
FROM example_table
WHERE PK_id = {pk}
AND SK_version = {sk}
)
AND EXISTS (
SELECT 1
FROM example_table
WHERE PK_id = {pk}
AND SK_version = {sk} - 1
)
Thanks
A conditional check is applied to a single item. It cannot be spanned across multiple items. In other words, you simply need multiple conditional checks. DynamoDb has transactWriteItems API which performs multiple conditional checks, along with writes/deletes. The code below is in nodejs.
const previousVersionCheck = {
TableName: 'example_table',
Key: {
PK_id: 'prev_pk_id',
SK_version: 'prev_sk_version'
},
ConditionExpression: 'attribute_exists(PK_id)'
}
const newVersionPut = {
TableName: 'example_table',
Item: {
// your item data
},
ConditionExpression: 'attribute_not_exists(PK_id)'
}
await documentClient.transactWrite({
TransactItems: [
{ ConditionCheck: previousVersionCheck },
{ Put: newVersionPut }
]
}).promise()
The transaction has 2 operations: one is a validation against the previous version, and the other is an conditional write. Any of their conditional checks fails, the transaction fails.
You are hitting your head on some of the differences between a SQL and a no-SQL database. DynamoDB is, of course, a no-SQL database. It does not, out of the box, support optimistic locking. I see two straight forward options:
Use a software layer to give you locking on your DynamoDB table. This may or may not be feasible depending on how often updates are made to your table. How fast 'versions' are generated and the maximum time your application can be gated on the lock will likely tell you if this can work foryou. I am not familiar with Go, but the Java API supports this. Again, this isn't a built-in feature of DynamoDB. If there is no such Go API equivalent, you could use the technique described in the link to 'lock' the table for updates. Generally speaking, locking a no-SQL DB isn't a typical pattern as it isn't exactly what it was created to do (part of which is achieving large scale on unstructured documents to allow fast access to many consumers at once)
Stop using an incrementor to guarantee uniqueness. Typically, incrementors are frowned upon in DynamoDB, in part due to the lack of intrinsic support for it and in part because of how DynamoDB shards you don't want a lot of similarity between records. Using a UUID will solve the uniqueness problem, but if you are porting an existing application that means more changes to the elements that create that ID and updates to reading the ID (perhaps to include a creation-time field so you can tell which is the newest, or the prepending or appending of an epoch time to the UUID to do the same). Here is a pertinent link to a SO question explaining on why to use UUIDs instead of incrementing integers.
Based on Hung Tran's answer, here is a Go example:
checkItem := dynamodb.TransactWriteItem{
ConditionCheck: &dynamodb.ConditionCheck{
TableName: "example_table",
ConditionExpression: aws.String("attribute_exists(pk_id) AND attribute_exists(version)"),
Key: map[string]*dynamodb.AttributeValue{"pk_id": {S: id}, "version": {N: prevVer}},
},
}
putItem := dynamodb.TransactWriteItem{
Put: &dynamodb.Put{
TableName: "example_table",
ConditionExpression: aws.String("attribute_not_exists(pk_id) AND attribute_not_exists(version)"),
Item: data,
},
}
writeItems := []*dynamodb.TransactWriteItem{&checkItem, &putItem}
_, _ = db.TransactWriteItems(&dynamodb.TransactWriteItemsInput{TransactItems: writeItems})
I have the following query
(TYPE:"ecmcndintregst:nd_int_reg_standards" OR TYPE:"ecmcndcountryst:nd_country_standards") AND (=ecmcnddoc:doc_name_ru:"" OR =ecmcnddoc:doc_name_ru:"\-") AND (=ecmcnddoc:doc_kind_cp_ecmcdict_value:"standard_itu")
and it has different results in Alfresco NodeBrowser (fts-alfresco) and in Java code "solr-fts-alfresco" (both results should be < 1000 and they are)
SearchParameters searchParameters = new SearchParameters();
searchParameters.setLanguage(SearchService.LANGUAGE_SOLR_FTS_ALFRESCO);
searchParameters.addStore(StoreRef.STORE_REF_WORKSPACE_SPACESSTORE);
searchParameters.setLimitBy(LimitBy.UNLIMITED);
searchParameters.setLimit(1000);
searchParameters.setPermissionEvaluation(PermissionEvaluationMode.EAGER);
searchParameters.addLocale(new Locale("ru", "RU"));
searchParameters.setQuery(query);
tempResultSet = customSearchService.query(searchParameters);
Also, in the Java code the clause =ecmcnddoc:doc_name_ru:""may fails two different ways: as always FALSE in query:
(TYPE:"ecmcndintregst:nd_int_reg_standards" OR TYPE:"ecmcndcountryst:nd_country_standards") AND (=ecmcnddoc:doc_name_ru:"" OR =ecmcnddoc:doc_name_ru:"\-") AND (=ecmcnddoc:doc_kind_cp_ecmcdict_value:"standard_itu")
And always TRUE in query:
(TYPE:"ecmcndintregst:nd_int_reg_standards" OR TYPE:"ecmcndcountryst:nd_country_standards") AND (=ecmcnddoc:doc_name_ru:"") AND (=ecmcnddoc:doc_kind_cp_ecmcdict_value:"standard_itu")
Could you tell me the proper way to use =ecmcnddoc:doc_name_ru:"" clause?
Thank you!
Assuming you're looking for results that have an empty "doc_name_ru", have you tried:
-EXISTS:"ecmcnddoc:doc_name_ru"
I have a document:
{
contact: {
id: '123'
},
channels: [
{
... some channel info...
}
],
lastUpdatedEpoch: 1583937675
}
And I have following query which doesn't return the above document:
SELECT p FROM p JOIN c IN p.channels
WHERE (NOT IS_DEFINED(p.lastUpdatedEpoch) OR p.lastUpdatedEpoch < 1585733881)
AND p.contact.id = '123'
But when I remove NOT IS_DEFINED check, it correctly returns the document:
SELECT p FROM p JOIN c IN p.channels
WHERE (p.lastUpdatedEpoch < 1585733881)
AND p.contact.id = '123'
I also tried replacing NOT IS_DEFINED clause with FALSE and it returns the document:
SELECT p FROM p JOIN c IN p.channels
WHERE (FALSE OR p.lastUpdatedEpoch < 1585733881)
AND p.contact.id = '123'
Also, if I remove JOIN, the query works as expected and returns the document:
SELECT p FROM
WHERE (NOT IS_DEFINED(p.lastUpdatedEpoch) OR p.lastUpdatedEpoch < 1585733881)
AND p.contact.id = '123'
To me this behavior is unexpected. When lastUpdatedEpoch is defined, I expect the same result from the first and second query (aside from the fact NOT_ISDEFINED will cause the index to be not used).
Could someone please explain what's going on here?
I try to reproduce your issue on my side but failed.The result is expected for me.
Test sample data:
Sql Output:
It seems that you did not refer any columns in channels.I suggest you create some simple test data to verify whether your sql is right.Then try to compare with your actual data.
I contacted CosmosDB team, and the team was able give some insight about the issue.
There was new optimization that was recently put in to allow inequality and NotIsDefined expressions to utilize the index. There was some issue with this optimization, and the team disabled this feature for now. If you are able to observe this issue with your cluster, please contact their support team.
To show related Articles on a website, I need the Cast() function.
My Query looks like:
SELECT
*,
(CAST(a.uploader = ?1 AS UNSIGNED)
+ CAST(a.param2 = ?2 AS UNSIGNED)
...
) AS matches_count
FROM articles AS a
ORDER BY matches_count DESC
It counts the matches and sorts by the highest number of matches_counts.
The problem is, that there's no Cast() function built into doctrine.
After hours of trial and error I found an already available custom DQL Function:
https://github.com/beberlei/DoctrineExtensions/blob/master/src/Query/Mysql/Cast.php
I registered it inside my doctrine.yml.
But it doesn't work, because it expects Cast(X AS Y) and not Cast(Y $comparisonOperator X).
When I'am using this inside my repository, by example:
$this->createQueryBuilder('a, (CAST(author=25 AS UNSIGNED) AS matches_count)')
->getQuery()
->getResult()
;
I get this error, because it doesn't expect a comparison operator:
[Syntax Error] line 0, col 29: Error: Expected Doctrine\ORM\Query\Lexer::T_AS, got '='
Do you know how to maybe extend that class for and not Cast(Y $comparisonOperator X) instead of Cast(X AS Y)?
I didn't find any solution on the internet and tried it for hours.
Thank you in advance for taking the time to write an answer!
Update:
I changed line 37 in the above mentioned custom DQL class for Cast:
//old
//$this->fieldIdentifierExpression = $parser->SimpleArithmeticExpression();
//new
$this->fieldIdentifierExpression = $parser->ComparisonExpression();
and how to create the query:
$this->createQueryBuilder('a')
->select('a, (CAST(a.averageRating=:averageRating AS UNSIGNED) + CAST(a.author=:author AS UNSIGNED)) AS matches_count')
->setParameter('averageRating', $averageRating)
->setParameter('author', $author)
->orderBy('matches_count', 'DESC')
->getQuery()
->getResult();
and that seems to be it!
I hope its the right way of doing it, will help someone and that is the best way for this purpose.
To improve performance later, I plan to cache 10 ids of recommended articles for every single article page into its own table.
So it doesn't need to do the calculation on page load.
This table could get recreated every 24h via a cronjob.
ID | recommended_article_ids | article_id
1 | 10,24,76,88| 5
Feedback and tips are much appreciated!
I am processing 1k record however I get system violation error after 800 record. Could someone please suggest how can this error be resolved?
There are designated methods for using OQL, you should take care to
Use a cursor variable
Declare a size that makes sense for your query
Open the cursor (allocates memory)
Close the cursor (disposes memory)
procedure ShowMoviesInCategory(theCategory : tCategory)
var Curs : aOQLCursor
var curMovie : aMovie
Curs = Motor.OpenOQLCursor
Curs.BatchSize = 50
OQL select * from x in aMovie++ where x.Category = theCategory using Curs
forEach curMovie in Curs
WriteLn(curMovie)
endFor
Motor.CloseOQLCursor(Curs)
endProc
Please also refer to the eWAM Help under OQL and
wTECH 101 (week1 - day 5 "101A - OQL - Search.pptx"
In Wynsure there is a designated variable for this, please refer to the Wynsure Development Rules.docx