DSS boxcarring skipping operation - wso2-data-services-server

Question

I'm using DSS 3.5.0 with PostgreSQL, and a set of operations in a request box is not working in a very peculiar way. I've successfully used request boxes containing thousands of operations in this same project, including operations very similar to those that fail. One of these large request boxes failed, and after spending some time looking for the operations that caused the problem, we were able to reduce it to a set of five operations.
The problem
Looking at the PostgreSQL logs, the query issued by one of the operations is not executed because it never reaches the database.
I'll call the operations O1, O2, O3, O4 and O5 and their queries Q1, Q2, Q3, Q4 and Q5. Playing with the request and checking the resulting database log, we ended up with:
Request box contains O1-O2-O3-O4-O5: database executes Q1-Q2-Q3-Q5
Request box contains O1-O2-O4-O5: database executes Q1-Q2-Q4-Q5
Request box contains O1-O2-O3-O4: database executes Q1-Q2-Q3-Q4
Request box contains O1-O2-O3-O4-O4-O5: database executes Q1-Q2-Q3-Q5
So, it looks weird and it doesn't seem to follow a clearly discernible pattern.
All operations perform correctly if sent separately to the DSS, or in two different request boxes. The exact nature of the operations doesn't seem to be directly linked to the problem because the same operations are used countless times in other scenarios. The queries are not especially long or complex.
Operation 1: updates a record in table A
Operation 2: deletes a record from table B
Operation 3: inserts a record in table B
Operation 4: inserts a record in table A
Operation 5: inserts a record in table B (same as operation 3)
Errors and logs
The actual error message issued by PostgreSQL for operation 5 is
ERROR: null value in column "element_id" violates not-null constraint
This is expected because operation 4 (the one that disappears) inserts a value that is later used to resolve element_id for operation 5.
The PostgreSQL log reports:
LOG: execute <unnamed>: BEGIN
LOG: execute <unnamed>: UPDATE public.project_element SET element_uuid=$1,location_id=$2,from_revit=$3,name=$4,type=$5,model=NULLIF($6,0),parent_element=(SELECT PE.ELEMENT_ID FROM PROJECT_ELEMENT PE WHERE PE.PROJECT_ID = $7 AND (PE.ELEMENT_ID = $8 OR (PE.ELEMENT_UUID = $9 AND PE.ELEMENT_UUID IS NOT NULL))) ,left_border=$10,right_border=$11 WHERE element_id=$12
DETAIL: parameters: $1 = '(element-uuid)', $2 = '85', $3 = '1', $4 = '(some-text)', $5 = '3', $6 = '0', $7 = '22', $8 = NULL, $9 = '(parent-uuid)', $10 = NULL, $11 = NULL, $12 = '9983'
LOG: execute <unnamed>: DELETE FROM ELEMENT_PROPERTY WHERE ELEMENT_ID = (SELECT PE.ELEMENT_ID FROM PROJECT_ELEMENT PE WHERE PE.ELEMENT_ID = $1 AND PE.PROJECT_ID = $2) AND NAME = $3
DETAIL: parameters: $1 = '9983', $2 = '22', $3 = 'num_ports'
LOG: execute <unnamed>: INSERT INTO public.element_property(name,value,type,element_id) VALUES($1,$2,$3,(^M SELECT PE.ELEMENT_ID FROM PROJECT_ELEMENT PE WHERE PE.PROJECT_ID = $4 AND (PE.ELEMENT_ID = $5 OR (PE.ELEMENT_UUID = $6 AND PE.ELEMENT_UUID IS NOT NULL))))
DETAIL: parameters: $1 = 'num_ports', $2 = '48', $3 = '0', $4 = '22', $5 = NULL, $6 = '(element-uuid)'
LOG: execute <unnamed>: INSERT INTO public.element_property(name,value,type,element_id) VALUES($1,$2,$3,(SELECT PE.ELEMENT_ID FROM PROJECT_ELEMENT PE WHERE PE.PROJECT_ID = $4 AND (PE.ELEMENT_ID = $5 OR (PE.ELEMENT_UUID = $6 AND PE.ELEMENT_UUID IS NOT NULL))))
DETAIL: parameters: $1 = 'port_num', $2 = '6', $3 = '0', $4 = '22', $5 = NULL, $6 = '(other-uuid)'
ERROR: null value in column "element_id" violates not-null constraint
DETAIL: Failing row contains (port_num, 6, 0, null).
STATEMENT: INSERT INTO public.element_property(name,value,type,element_id) VALUES($1,$2,$3,(SELECT PE.ELEMENT_ID FROM PROJECT_ELEMENT PE WHERE PE.PROJECT_ID = $4 AND (PE.ELEMENT_ID = $5 OR (PE.ELEMENT_UUID = $6 AND PE.ELEMENT_UUID IS NOT NULL))))
LOG: execute S_2: BEGIN
LOG: execute S_1: ROLLBACK
DSS log starts with an exception, but I'm not sure if it's really related to this problem. The following log goes from the request box start to the first time it complains about the error message returned from PostgreSQL. DSS complains multiple times after that.
DEBUG - {org.apache.axis2.transport.http.AxisServlet}
java.lang.NullPointerException
at javax.servlet.GenericServlet.getServletContext(GenericServlet.java:123)
at org.apache.axis2.transport.http.AxisServlet.createMessageContext(AxisServlet.java:715)
at org.apache.axis2.transport.http.AxisServlet$RestRequestProcessor.<init>(AxisServlet.java:819)
at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:227)
at org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.wso2.carbon.ui.filters.CSRFPreventionFilter.doFilter(CSRFPreventionFilter.java:88)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.wso2.carbon.ui.filters.CRLFPreventionFilter.doFilter(CRLFPreventionFilter.java:59)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:99)
at org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
at org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:57)
at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:62)
at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:159)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1739)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1698)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:744)
DEBUG - Input contentType (application/json) {org.apache.axis2.builder.BuilderUtil}
DEBUG - CharSetEncoding defaulted (UTF-8) {org.apache.axis2.builder.BuilderUtil}
DEBUG - [MessageContext: logID=f9462531f982d008b3e2aacd88bfd07f4a7e4905c354170e] Checking for Service using target endpoint address : /services/iims {org.apache.axis2.dispatchers.RequestURIBasedServiceDispatcher}
DEBUG - org.apache.axis2.i18n.resource::handleGetObject(servicefound) {org.apache.axis2.i18n.ProjectResourceBundle}
DEBUG - [MessageContext: logID=f9462531f982d008b3e2aacd88bfd07f4a7e4905c354170e] Found AxisService : iims {org.apache.axis2.engine.AbstractDispatcher}
DEBUG - Attempt to check for Operation using HTTP Location failed {org.apache.axis2.dispatchers.HTTPLocationBasedDispatcher}
DEBUG - [MessageContext: logID=f9462531f982d008b3e2aacd88bfd07f4a7e4905c354170e] Attempted to check for Operation using target endpoint URI, but the operation fragment was missing {org.apache.axis2.dispatchers.RequestURIBasedOperationDispatcher}
DEBUG - getAction (null) from org.apache.axis2.client.Options#279e70a {org.apache.axis2.client.Options}
DEBUG - SoapAction is (null) {org.apache.axis2.context.MessageContext}
DEBUG - createSOAPEnvelope using Builder (class org.apache.axis2.json.JSONOMBuilder) selected from type (application/json) {org.apache.axis2.transport.TransportUtils}
DEBUG - getAction (null) from org.apache.axis2.client.Options#279e70a {org.apache.axis2.client.Options}
DEBUG - SoapAction is (null) {org.apache.axis2.context.MessageContext}
DEBUG - [MessageContext: logID=f9462531f982d008b3e2aacd88bfd07f4a7e4905c354170e] Checking for Operation using Action : null {org.apache.axis2.dispatchers.ActionBasedOperationDispatcher}
DEBUG - [MessageContext: logID=f9462531f982d008b3e2aacd88bfd07f4a7e4905c354170e] Attempted to check for Operation using target endpoint URI, but the operation fragment was missing {org.apache.axis2.dispatchers.RequestURIBasedOperationDispatcher}
DEBUG - Axis operation is null {org.apache.axis2.json.gson.JSONMessageHandler}
DEBUG - No headers present corresponding to http://www.w3.org/2005/08/addressing {org.apache.axis2.handlers.addressing.AddressingInHandler}
DEBUG - No headers present corresponding to http://schemas.xmlsoap.org/ws/2004/08/addressing {org.apache.axis2.handlers.addressing.AddressingInHandler}
DEBUG - getAction (null) from org.apache.axis2.client.Options#279e70a {org.apache.axis2.client.Options}
DEBUG - SoapAction is (null) {org.apache.axis2.context.MessageContext}
DEBUG - [MessageContext: logID=f9462531f982d008b3e2aacd88bfd07f4a7e4905c354170e] Checking for Operation using Action : null {org.apache.axis2.dispatchers.ActionBasedOperationDispatcher}
DEBUG - getAction (null) from org.apache.axis2.client.Options#279e70a {org.apache.axis2.client.Options}
DEBUG - SoapAction is (null) {org.apache.axis2.context.MessageContext}
DEBUG - [MessageContext: logID=f9462531f982d008b3e2aacd88bfd07f4a7e4905c354170e] Checking for Operation using Action : null {org.apache.axis2.dispatchers.ActionBasedOperationDispatcher}
DEBUG - Get operation for request_box {org.apache.axis2.description.AxisService}
DEBUG - Found axis operation: org.apache.axis2.description.InOutAxisOperation#682d0c2c {org.apache.axis2.description.AxisService}
DEBUG - org.apache.axis2.i18n.resource::handleGetObject(operationfound) {org.apache.axis2.i18n.ProjectResourceBundle}
DEBUG - [MessageContext: logID=f9462531f982d008b3e2aacd88bfd07f4a7e4905c354170e] Found AxisOperation : request_box {org.apache.axis2.engine.AbstractDispatcher}
DEBUG - getAddressingRequirementParemeterValue: value: 'null' {org.apache.axis2.addressing.AddressingHelper}
DEBUG - [MessageContext: logID=f9462531f982d008b3e2aacd88bfd07f4a7e4905c354170e] isReplyRedirected: ReplyTo is null. Returning false {org.apache.axis2.addressing.AddressingHelper}
DEBUG - getAction (null) from org.apache.axis2.client.Options#112f42cb {org.apache.axis2.client.Options}
DEBUG - Old WSAAction is (null) {org.apache.axis2.context.MessageContext}
DEBUG - New WSAAction is (urn:request_boxResponse) {org.apache.axis2.context.MessageContext}
DEBUG - setAction Old action is (null) {org.apache.axis2.client.Options}
DEBUG - setAction New action is (urn:request_boxResponse) {org.apache.axis2.client.Options}
DEBUG - messageID is null. {org.apache.axis2.context.ConfigurationContext}
DEBUG - forceExpand: changing prefix from to {org.apache.axiom.om.impl.llom.OMSourcedElementImpl}
DEBUG - DXXATransactionManager.begin() {org.wso2.carbon.dataservices.core.description.xa.DSSXATransactionManager}
DEBUG - Creating data source connection {org.wso2.carbon.dataservices.core.description.config.SQLConfig}
ERROR - ERROR: null value in column "element_id" violates not-null constraint_ Detalhe: Failing row contains (port_num, 6, 0, null). (Sanitized) {org.wso2.carbon.dataservices.core.description.query.SQLQuery}
org.postgresql.util.PSQLException: ERROR: null value in column "element_id" violates not-null constraint
The implementation
This is the actual request box that fails (some field contents replaced to reduce noise):
{
"request_box":{
"update_project_element_operation":{
"name":"(some-text)",
"element_id":9983,
"element_uuid":"(element-uuid)",
"from_revit":1,
"project_id":22,
"parent_element_uuid":"(parent-uuid)",
"type":3,
"location_id":85,
"model":0
},
"delete_element_property_operation":{
"name":"num_ports",
"element_id":9983,
"project_id":22
},
"insert_element_property_operation":{
"project_id":22,
"element_uuid":"(element-uuid)",
"name":"num_ports",
"value":"48"
},
"insert_project_element_operation":{
"name":"(this operation disappears)",
"element_id":0,
"element_uuid":"(other-uuid)",
"from_revit":1,
"project_id":22,
"parent_element_uuid":"(element-uuid)",
"type":10,
"location_id":85,
"model":0
},
"insert_element_property_operation":{
"project_id":22,
"element_uuid":"(other-uuid)",
"name":"port_num",
"value":"6"
}
}
}
I can provide detailed table, query and operation definitions if necessary. All operations were used before, and each one of them work if issued separately or in two different request boxes. It seems to be a issue directly linked to DSS boxcarring.
Any ideas?

Answer 1

After a few weeks of investigation, including direct contact to the WSO2 support, we concluded that this unusual problem was caused by the JSON to XML conversion inside DSS. This may be related to the fact that the request box representation in JSON format can contain non-unique names (and according to RFC 7159 the behavior in this case is unpredictable and implementation-defined). It should be noted that we also used a request box with thousands of repetitions of the same name without any visible problem, so it isn't a straightforward consequence of all non-unique names not being correctly processed.
When we tried the same request box in XML, all operations were correctly executed. To avoid changing the application, we followed WSO2's advice and had the ESB converting the application-generated JSON to XML. Preliminar tests showed that in this case the XML was correctly generated, however we decided to slightly adjust the JSON generator to issue an array of operation objects instead of an object containing members with non-unique names, to avoid the undefined behavior and the possibility of new, unpredictable problems in JSON parsing.
WSO2 is aware of this problem and it may or may not be fixed by an upcoming release of DSS. Until then, the safer way to avoid request box suprises seems to be to use XML instead of JSON when sending transactions to DSS using request boxes.