Apache Camel: turn/replace synchronous direct into async - asynchronous

My question is very close to this discussion: https://issues.apache.org/jira/browse/CAMEL-4863. I got caught by synchronous behavior of direct as well.
I designed an application like this:
Route1:
from(direct:step1-in).process(...).to(direct:step1-out)
Route2:
from(direct:step2-in).process(...).to(direct:step2-out)
CoreRoute (connects all steps):
from(direct:step1-out).to(direct:step2-in)
After a year I still like this approach. I can implement and test those steps separately and for example switch to seda:step1-out if I need to decouple them.
But seda is rather a heavy way to decouple: you create a queue and a thread pool with fixed number of threads. So even if you most of the time need 1 thread and sometimes 100, you will always have 100 in use.
Sometimes you just want to free the current thread, because it should be doing let's say unzip of the next file already, instead of processing the current.
From documentation I understood you would use threads(1,100) for this:
from(direct:input-list-processor).split(body()).threads(1,100).delay(1000).process(...).to(step-process-unzipped-out)
But this will block the thread routing the exchange to direct-input-list-processor until all threads finish processing.
Instead I would like that the exchange is copied to subsequent threads and current exchange is done.

Thanks to comment from Claus, I came up with the following which seems to do what I want using wireTap:
from("direct:step1-in").to("direct:step1-out");
from("direct:step2-in")
.log("receive async")
.delay(1000).to("log:output");
// core route connecting processing step1 and processing step2
from("direct:step1-out").log("send async").wireTap("direct:step2-in")
.executorService(getContext().getExecutorServiceManager().newThreadPool(this,
"step2-worker", 1, 100));
I think this is the way to go. Though I am confused because
Camel in Action book suggests: 10.4.3 Asynchronous caller using multiple
threads
from("seda:start")
.to("log:A")
.threads(5, 10)
.to("log:B");
This is the most obvious approach, but as stated here: http://camel.apache.org/seda.html it is not the way to go as it can wind up and the suggested alternative to use direct instead of seda does not work for me, hence the question. Also the seda way has an unlimited queue which is a potential "memory leak", so you would need to decide and set an appropriate size everytime you want to use this pattern.
#Claus thank you very much for your great work! Your answers guided me a lot through the project. May be you could clarify if you think it is a good solution or am I still missing something. The only "disadvantages" I see is that a new exchange will be created, also a transaction would not span multiple processing steps.
This log shows the solution with wireTap works asynchronously and scales up in a flexible way
2017-07-30T16:36:15.607 | -1 | main | INFO | --- --- --- route3 | route3 | - send async
2017-07-30T16:36:15.609 | -1 | main | INFO | --- --- --- route3 | route3 | - send async
2017-07-30T16:36:15.610 | -1 | main | INFO | --- --- --- route3 | route3 | - send async
2017-07-30T16:36:15.614 | -1 | Camel Thread #2 - step2-worker | INFO | --- --- --- route2 | route2 | - receive async
2017-07-30T16:36:15.614 | -1 | Camel Thread #1 - step2-worker | INFO | --- --- --- route2 | route2 | - receive async
2017-07-30T16:36:15.614 | -1 | Camel Thread #3 - step2-worker | INFO | --- --- --- route2 | route2 | - receive async
2017-07-30T16:36:15.614 | -1 | main | INFO | --- --- --- route3 | route3 | - send async
2017-07-30T16:36:15.615 | -1 | main | INFO | --- --- --- route3 | route3 | - send async
2017-07-30T16:36:15.616 | -1 | main | INFO | --- --- --- route3 | route3 | - send async
2017-07-30T16:36:15.616 | -1 | main | INFO | --- --- --- route3 | route3 | - send async
2017-07-30T16:36:15.617 | -1 | main | INFO | --- --- --- route3 | route3 | - send async
2017-07-30T16:36:15.618 | -1 | main | INFO | --- --- --- route3 | route3 | - send async
2017-07-30T16:36:15.618 | -1 | main | INFO | --- --- --- route3 | route3 | - send async
2017-07-30T16:36:15.619 | -1 | main | INFO | --- --- --- route3 | route3 | - send async
2017-07-30T16:36:15.620 | -1 | main | INFO | --- --- --- route3 | route3 | - send async
2017-07-30T16:36:15.620 | -1 | main | INFO | --- --- --- route3 | route3 | - send async
2017-07-30T16:36:15.621 | -1 | main | INFO | --- --- --- route3 | route3 | - send async
2017-07-30T16:36:15.622 | -1 | Camel Thread #4 - step2-worker | INFO | --- --- --- route2 | route2 | - receive async
2017-07-30T16:36:15.622 | -1 | main | INFO | --- --- --- route3 | route3 | - send async
2017-07-30T16:36:15.623 | -1 | Camel Thread #5 - step2-worker | INFO | --- --- --- route2 | route2 | - receive async
2017-07-30T16:36:15.623 | -1 | main | INFO | --- --- --- route3 | route3 | - send async
2017-07-30T16:36:15.624 | -1 | main | INFO | --- --- --- route3 | route3 | - send async
2017-07-30T16:36:15.624 | -1 | Camel Thread #6 - step2-worker | INFO | --- --- --- route2 | route2 | - receive async
2017-07-30T16:36:15.625 | -1 | Camel Thread #7 - step2-worker | INFO | --- --- --- route2 | route2 | - receive async
2017-07-30T16:36:15.625 | -1 | main | INFO | --- --- --- route3 | route3 | - send async
2017-07-30T16:36:15.626 | -1 | Camel Thread #8 - step2-worker | INFO | --- --- --- route2 | route2 | - receive async
2017-07-30T16:36:15.626 | -1 | main | INFO | --- --- --- route3 | route3 | - send async
2017-07-30T16:36:15.627 | -1 | Camel Thread #9 - step2-worker | INFO | --- --- --- route2 | route2 | - receive async
2017-07-30T16:36:15.627 | -1 | main | INFO | --- --- --- route3 | route3 | - send async
2017-07-30T16:36:15.628 | -1 | Camel Thread #10 - step2-worker | INFO | --- --- --- route2 | route2 | - receive async
2017-07-30T16:36:15.629 | -1 | main | INFO | --- --- --- route3 | route3 | - send async

Related

How to prevent trimming of sbt dependencyTree output

Is it possible to configure sbt such that output from sbt dependencyTree is not trimmed?
Fragment of trimmed output:
[info] | | | | | | +-com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.12...
[info] | | | | | | +-com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.14.0
[info] | | | | | | | +-com.fasterxml.jackson.core:jackson-core:2.14.0 (evicted by: ..
[info] | | | | | | | +-com.fasterxml.jackson.core:jackson-core:2.14.1
[info] | | | | | | | +-com.fasterxml.jackson.core:jackson-databind:2.14.0 (evicted ..
[info] | | | | | | | +-com.fasterxml.jackson.core:jackson-databind:2.14.1
[info] | | | | | | | +-com.fasterxml.jackson.core:jackson-annotations:2.14.1
[info] | | | | | | | +-com.fasterxml.jackson.core:jackson-core:2.14.1
Trimming is problematic for me, as the output of this command is picked up by snyk cli, which gets confused and reports false positives.
I am using sbt 1.8.0
dependencyTree tasks observe asciiGraphWidth option starting with sbt 1.6.0.
See sbt 1.6.0 release notes
Thus:
add asciiGraphWidth is build.sbt:
asciiGraphWidth := 180
alternatively, this can be passed in via command line:
sbt 'set asciiGraphWidth := 180' dependencyTree

How to handle empty cells in R markdown?

I have an R analysis that I want to present in an R markdown table. The table's layout demands some empty cells:
x <- c(1:14)
Header1 | Header2 | Header3 | Header4
------- | ------- | -------- | -------
Label1 | LabelA | `r x[1]` | `r x[4]`
Label2 | LabelB | `r x[2]` | `r x[5]`
| LabelC | `r x[3]` | `r x[6]`
| LabelD | | `r x[7]`
------- | ------- | -------- | -------
Label3 | LabelA | `r x[8]` | `r x[11]`
Label4 | LabelB | `r x[9]` | `r x[12]`
| LabelC | `r x[10]`| `r x[13]`
| LabelD | | `r x[14]`
However, those empty cells are causing the layout to break, ie, LabelC appears directly under Label2 even though it is supposed to be one column over.
I know, from searching on this issue, that you can't merge table cells in R Markdown. But can I fill those cells with code such that they will present as empty? I tried filling them with r NULL but nothing happened.
The problem with this table is that it does not have the style required by rmarkdown. The table below has the good style and there's no issue with this table:
| Header1 | Header2 | Header3 | Header4 |
|:-------:|:-------:|:--------:|:---------:|
| Label1 | LabelA | `r x[1]` | `r x[4]` |
| Label2 | LabelB | `r x[2]` | `r x[5]` |
| | LabelC | `r x[3]` | `r x[6]` |
| | LabelD | | `r x[7]` |
| ------- | ------- | -------- | --------- |
| Label3 | LabelA | `r x[8]` | `r x[11]` |
| Label4 | LabelB | `r x[9]` | `r x[12]` |
| | LabelC | `r x[10]`| `r x[13]` |
| | LabelD | | `r x[14]` |
You can generate such a table with the pander package, function pandoc.table with the option style="rmarkdown", or with knitr::kable. Both have an option for dealing with the missing values.

openstack ocata glance creating 0 sized image

When I create new image using glance does not matter if using cli or gui I am getting returned code 0 and image is created but its size is zero.
The behavior is slightly different as from GUI my browser crushes but stil image is created from cli I am getting return code 0.
Command:
openstack image create --file cirros-0.4.0-x86_64-disk.img --disk-format qcow2 --container-format bare --public --debug cirros-deb
+------------------+------------------------------------------------------+
| Field | Value |
+------------------+------------------------------------------------------+
| checksum | d41d8cd98f00b204e9800998ecf8427e |
| container_format | bare |
| created_at | 2018-01-20T23:24:47Z |
| disk_format | qcow2 |
| file | /v2/images/c695bc30-731d-4a4f-ab0f-12eb972d8188/file |
| id | c695bc30-731d-4a4f-ab0f-12eb972d8188 |
| min_disk | 0 |
| min_ram | 0 |
| name | cirros-deb |
| owner | a3460a3b0e8f4d0bbdd25bf790fe504c |
| protected | False |
| schema | /v2/schemas/image |
| size | 0 |
| status | active |
| tags | |
| updated_at | 2018-01-20T23:24:47Z |
| virtual_size | None |
| visibility | public |
+------------------+------------------------------------------------------+
clean_up CreateImage:
END return value: 0
I tried with different cirros image and with ubuntu cloud image always behavior is the same.
Under /var/lib/glance/images file is created with size 0:
-rw-r-----. 1 glance glance 0 Jan 21 00:24 c695bc30-731d-4a4f-ab0f-12eb972d8188
grep c695bc30-731d-4a4f-ab0f-12eb972d8188 glance/api.log
2018-01-21 00:24:47.915 1894 INFO eventlet.wsgi.server [req-7246cd30-47c4-41a5-b358-c8e5cc0f4e56 8bd3e4905ffb4f698e2476d9080a7d90 a3460a3b0e8f4d0bbdd25bf790fe504c - default default] 172.19.254.50 - - [21/Jan/2018 00:24:47] "PUT /v2/images/c695bc30-731d-4a4f-ab0f-12eb972d8188/file HTTP/1.1" 204 213 0.111323
2018-01-21 00:24:47.931 1894 INFO eventlet.wsgi.server [req-28e0cda2-c9f7-4543-b19a-d59eccffa47e 8bd3e4905ffb4f698e2476d9080a7d90 a3460a3b0e8f4d0bbdd25bf790fe504c - default default] 172.19.254.50 - - [21/Jan/2018 00:24:47] "GET /v2/images/c695bc30-731d-4a4f-ab0f-12eb972d8188 HTTP/1.1" 200 780 0.015399
Any idea what can be wrong?
Find location of python glance client.
find / -name http.py
vi /usr/lib/python2.7/site-packages/glanceclient/common/http.py
- data = self._chunk_body(data)
+ pass
Referenc:
https://bugs.launchpad.net/python-glanceclient/+bug/1666511
https://ask.openstack.org/en/question/101944/why-does-openstack-image-create-of-cirros-result-in-size-0/?answer=102303#post-id-102303

I want to know if this is possible using firebase or not?

Thinking to Create project using Firebase but need to confirm , it will fullfill my requirement or not . So it will be like there will be numbers of groups and in each group there will be 5-15 devices connected . so if any of the devices from the group does anything that should get updated to every device in that particular group so everyone gets to know what has been done and from which device. ALL the Logs will be generated on the FCM server .
Yes you can achieve this with Firebase. You database should look like this:
Firebase-root
|
--- groups
| |
| --- group1
| | |
| | --- groupName: "G1"
| | |
| | --- Devices
| | |
| | --- deviceId1: true
| | |
| | --- deviceId2: true
| | |
| | --- deviceId3: true
| |
| --- group2
| |
| --- groupName: "G2"
| |
| --- Devices
| |
| --- deviceId4: true
| |
| --- deviceId5: true
| |
| --- deviceId6: true
|
--- devices
|
--- device1
| |
| --- deviceName: D1
| |
| --- groups
| |
| --- group1: true
| |
| --- group2: true
| |
| --- group3: true
|
--- device2
|
--- deviceName: D2
|
--- groups
|
--- group4: true
|
--- group5: true
|
--- group6: true
With this database structure, you can easily display all the groups from groups node and all the devices from the device node. You can also display very easy all the devices that belong to a single group and the groups in which a device exists.
Having the database strcutured like this, you can then use FCM to send messages to all devices from a particular group.
Hope it helps.

Testcase level variables in [Documentation] for robot framework

I cannot get test level variables to appear in documentation.
Let's say I have this testsuite:
| *Variables* |
| ${SystemUnderTest} = | Staging
| *testcase* |
| Device Test |
| | Set Test Variable | ${device} | iPhone
| | [Documentation] | Device is: ${device} |
| | ... | System is: ${SystemUnderTest} |
| | No Operation
That produces this log:
TEST CASE: Device TestExpand All
Full Name: T.Device Test
Documentation:
Device is: ${device} System is: Staging
Notice that the Suite level variable is treated properly, but the test level one is not.
How do I get all variables to be treated equally?
Starting with robotframework 2.7 there is a built-in keyword named Set test documentation, which can be used to replace or append to the existing documentation. This will not affect the output in the console, but the changes will be reflected in the log and report.
For example:
| *Variables* |
| ${SystemUnderTest} = | Staging
| *testcase* |
| Device Test |
| | Set Test Variable | ${device} | iPhone
| | [Documentation] | Device is: ${device} |
| | ... | System is: ${SystemUnderTest} |
| | Substitute vars in documentation
| | No Operation
| *Keywords* |
| Substitute vars in documentation
| | ${doc}= | replace variables | ${test documentation}
| | set test documentation | ${doc}
For more information see http://robotframework.googlecode.com/hg/doc/libraries/BuiltIn.html?r=2.7.7#Set%20Test%20Documentation
This solution feels a little hackerish to me, but it does give you the functionality that you want.
Test.txt
| *Setting* | *Value* |
# This should start as the value for your first test
| Suite Setup | Set Suite Variable | ${device} | foo
| *Test Case* | *Action* | *Argument*
#
| T100 | [Documentation] | Should be foo: ${device}
# Do some stuff
| | No Operation
# This setups the device name for the next test.
| | Set Suite Variable | ${device} | bar
#
| T101 | [Documentation] | Should be bar: ${device}
# Do some stuff
| | No Operation
| | Set Suite Variable | ${device} | bing
#
| T102 | [Documentation] | Should be bing: ${device}
# Do some stuff
| | No Operation
When I run the suite I get this output:
==============================================================================
Test
==============================================================================
T100 :: Should be foo: foo | PASS |
------------------------------------------------------------------------------
T101 :: Should be bar: bar | PASS |
------------------------------------------------------------------------------
T102 :: Should be bing: bing | PASS |
------------------------------------------------------------------------------
Test | PASS |
3 critical tests, 3 passed, 0 failed
3 tests total, 3 passed, 0 failed
==============================================================================
Having the device variable set at the end of the previous test is a little unclean, but as long you leave a comment it shouldn't be unclear at all.

Resources