I´m currently struggling with EJB remote calls against Wildfly 25 or 26.
Same client application works fine with Widfly 10,13,16,20 and to some extent also with Wildfly 25 or 26.
The problem starts wenn the return object size of the EJB call exceeds some limit, which limit seems to change sometime. Example: I made a test EJB method which returns same string what I provide as parameter. Mostly, if the length of the String exceeds ~65.000 chars, the Wildfly EJB client hangs on reading the result. Anyhow, sometimes I have also experienced that client freezes at more than that limit. In my client I have registered an EJB Call Interceptor and I see that the call freezes when the EJB-context.getResult() is invoked. On server side, also based on a server side interceptor, I see that the call was done, but obviuosly something goes wrong on receiving the return value through the EJB client.
This is the stacktrace of the hanging thread:
"main#1" prio=5 tid=0x1 nid=NA waiting
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Object.java:-1)
at java.lang.Object.wait(Object.java:502)
at org.wildfly.httpclient.common.WildflyClientInputStream.read(WildflyClientInputStream.java:147)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at org.jboss.marshalling.SimpleDataInput.read(SimpleDataInput.java:111)
at org.jboss.marshalling.UTFUtils.readUTFBytes(UTFUtils.java:151)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:314)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:231)
at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
at org.wildfly.httpclient.ejb.HttpEJBReceiver$2.getResult(HttpEJBReceiver.java:207)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:620)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:551)
at org.jboss.ejb.protocol.remote.RemotingEJBClientInterceptor.handleInvocationResult(RemotingEJBClientInterceptor.java:57)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:622)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:551)
at org.jboss.ejb.client.TransactionPostDiscoveryInterceptor.handleInvocationResult(TransactionPostDiscoveryInterceptor.java:148)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:622)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:551)
at org.jboss.ejb.client.DiscoveryEJBClientInterceptor.handleInvocationResult(DiscoveryEJBClientInterceptor.java:130)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:622)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:551)
at org.jboss.ejb.client.NamingEJBClientInterceptor.handleInvocationResult(NamingEJBClientInterceptor.java:87)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:622)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:551)
at org.jboss.ejb.client.AuthenticationContextEJBClientInterceptor$$Lambda$94.871790326.get(Unknown Source:-1)
at org.jboss.ejb.client.AuthenticationContextEJBClientInterceptor.call(AuthenticationContextEJBClientInterceptor.java:59)
at org.jboss.ejb.client.AuthenticationContextEJBClientInterceptor.handleInvocationResult(AuthenticationContextEJBClientInterceptor.java:52)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:622)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:551)
at com.ge.hac.ca.common.util.CommonClientInvocationInterceptor.handleInvocationResult(CommonClientInvocationInterceptor.java:196)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:622)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:551)
at org.jboss.ejb.client.TransactionInterceptor.handleInvocationResult(TransactionInterceptor.java:212)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:622)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:551)
at org.jboss.ejb.client.EJBClientInvocationContext.awaitResponse(EJBClientInvocationContext.java:1003)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:182)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:116)
at com.sun.proxy.$Proxy4.loopback(Unknown Source:-1)
at com.ge.hac.ca.perf.connection.TestConnectionToWildfly.checkEJBInvocation_LoggerService(TestConnectionToWildfly.java:225)
at com.ge.hac.ca.perf.connection.TestConnectionToWildfly.executeEJBIterations(TestConnectionToWildfly.java:191)
at com.ge.hac.ca.perf.connection.TestConnectionToWildfly.main(TestConnectionToWildfly.java:401)
I'm using Amazon Corretto jdk1.8.0_292.
Has anyone experienced similar issue, and if yes, how can this be solved?
EJB calls with huge result objects (i.e. String >= 64 Kbytes) doesn't work with Wildfly 25 or 26 because I have to use now the HTTP protocol instead of "remoting" what I have used for years before.
Specifically the HTTP/2 implemenattion doesn't work but the HTTP/1 seems to work.
Explanation:
I have checked / debugged the Wildfly client code and in my opinion there are several bugs in both the server and the client code of the HTTP/2 protocol implementation.
First bug: The server sends suddenly (mostly on bigger objects) in the middle of the data stream a FRAME_TYPE_RST_STREAM (see io.undertow.protocols.http2.Http2FrameHeaderParser:191 --> type = header[3] & 0xff;),
which marks the stream as broken (see io.undertow.server.protocol.framed.AbstractFramedStreamSourceChannel:684 --> state |= STATE_STREAM_BROKEN;)
and afterwards causes "ClosedChannelExceptions" on all stream reads inside the client (see org.wildfly.httpclient.common.WildflyClientInputStream:58 --> int res = streamSourceChannel.read(pooled.getBuffer());)
I didn't debug the server side code, so I can't tell why the server sends suddenly a "reset stream" on some huge objects. Sometimes the same huge object is transferred to the client without issues. On using HTTP/1 this neever happens so it can't be a network issue.
Second bug (initiated by the above reset stream) : The client can't handle correctly this "ClosedChannelExceptions" initiated by the received reset stream, because it freezes the whole communication.
The client keeps reading from the channel and after each read it invokes the wait(0) on its lock Object (see org.wildfly.httpclient.common.WildflyClientInputStream:147 --> lock.wait();)
Normally this wait(0) is continued by invoking the notifyAll() of same lock Object (in case of ClosedChannelExceptions it is the org.wildfly.httpclient.common.WildflyClientInputStream:94 --> lock.notifyAll();
Unluckily after the last read and therefore last invoked wait(0), there is no notifyAll() invoked anymore, so the EJB client call freezes for ever. This should not happen under any circumstances in my opinion.
The solution I found is, but I'm not happy with that, to disable the HTTP/2 protocol on the server and use HTTP/1 instead.
standalone-full.xml --> <https-listener name="https" socket-binding="https" enable-http2="false" />
Making now EJB calls through the HTTP/1 protocol works fine for huge Objects as well.
The mentioned classes with line numbers are all taken from Wildlfy 26.0.0 Client jar:
I hope some Wildfly expert will read this and react.
Related
I want to create an application that works as a "man in the middle" to analyze a protocol (ISO 8583) sent over TCP/IP.
A client connects to the application and sends some binary data (average length 500 bytes).
The application receives the message and then sends it to the real server.
The server responds to the request to the application.
The application sends the response to the original client.
Some context: The main idea is to get the raw binary data and convert it to a string for parsing and decoding the protocol.
There are two parts to the project:
The gateway part (man in the middle).
Parsing and decoding of the data.
I am expending too much time on the first part. So, if there is a mock-up that I can use to get me started, it will be nice. It doesn't have to be with Indy, but I prefer C++Builder.
This is my first time with Indy, and although I have experience working with TCP/IP, I have always used it as something that is already there, never at the low-level implementation.
I am testing with Hercules, and so far I can see the connections.
When I connect to a server in Hercules, I can see that my application is connecting. But, when my application disconnects, I don't see a message that says so, which means (I think) that my app is not disconnecting correctly (but I can reconnect as many times as I want).
I am sending data to my application using Hercules (a "Hello" string). It is working apparently, but I am having a hard time getting the actual data.
The documentation sometimes gets me into dead links, there are no samples or they are available on Delphi.
I am working with the following:
Windows 11 Home
Embarcadero® C++Builder 10.4 Version 27.0.40680.4203
Delphi and C++ Builder 10.4 Update 2
Indy 10.6.2.0
Have a look at Indy's TIdMappedPortTCP component. It is a TCP server that acts as a MITM proxy between clients and a specified server, giving you events when either party sends raw data.
Use the Bindings collection, or DefaultPort property, to specify the local IP/Port(s) that you want the server to listen for clients on.
Use the MappedHost and MappedPort properties to specify the remote server that you want TIdMappedPortTCP to connect to.
The OnBeforeConnect event is fired when a client has connected to TIdMappePortTCP.
The OnConnect event is fired just before TIdMappedPortTCP attempts to connect to the remote server.
The OnOutboundClientConnect event is fired when TIdMappedPortTCP has connected to the remote server.
The OnExecute event is fired when a client send bytes to TIdMappedPortTCP, and before the bytes are sent to the remote server. The event can alter the bytes, if desired.
The OnOutboundData event is fired when the remote server send bytes to TIdMappedPortTCP, and before the bytes are sent to the client. The event can alter the bytes, if desired.
I have a Spring mvc application, and I'm using websockets to communicate a phisical device that sends data with an angular 2 front end.
the architecture is like this
device ----> Spring Mvc <-----angular 2 front end
I have a datasource listener that publish to a websocket topic everytime a new message appears and I consume that topic from angular.
My problem is that this is working properly in my local tomcat install but when I upload it to a faster server it doesn't.
The main problem I'm having is that is buffering the messages and is reaching the limit and closing the websocket session.
What I noticed checking the logs is that in my local server messages come 20 milliseconds after last one is finished but in the other server sometimes are coming at the same time and they are being buffered throwing the session limit exception.
I tried setting a higher buffer size.
Also tried a thread.sleep but it doesn't worked.
Do you have some ideas what can I do ? should I implement some kind of message queue ?
Thanks in advance
Dealing with a Flex client that needs to connect through BlazeDS to a Java backend. Currently, we have a requirement for fifty-plus clients to be attached at any given time. We need to test the load of this requirement against the server to see if we are going to have any performance issues.
So, I have written a client emulator that will act like real clients and connect to the server though Blazeds. The emulator was to make this test require less hardware for I would connect 50 client from one machine (or two 25, basically minimize hardware needs). Instead of having fifty different machines to run clients on. The issue I am running into is a limitation of emulated clients allowed from one session. There seems to be a five client limit. This goes for both IExplorer and FireFox browsers. The problem is with the JMS subscriptions. The JMS topic seem to get connected but never subscribed.
I played around with some settings on the BlazeDS server side to no avail.
- max-streaming-connections-per-session
- max-streaming-clients
After the sixth connection I start getting a
16:18:21.578 [ERROR] com.ray.sv.flex.util.SocketLogTarget SocketLogTarget failed with SecurityError: [SecurityErrorEvent type="securityError" bubbles=false cancelable=false eventPhase=2 text="Error #2048: Security sandbox violation: http://xxx.xxx.xxx.xxx:8080/ClientEmulator/clientemulator.swf cannot load data from 127.0.0.1:1337."]
Very strange, but what is stranger is that client needs to recover from server disconnects. Which was a totally different issue all together with the remote object calls and subscribing timing issues, kept getting Duplicate Session Id errors. But now I have that ironed out, but I see the same issue when the client fails a connection five times.
The sixth time always fails with the same error. And this is with only one client connected.
Has anyone seen this same issue?
Thanks for your time.
The blazeds server-side don't know the client-side has disconnected. But it seems to know the client-side's network has down.
In my case, I use the polling channel, I download the blazeds's source code, and add some log output in the FlexClientOutboundQueueProcessor.flush(MessageClient messageClient, List<Message> outboundQueue) method.
Then I saw this, when a client subscibed, the server-side invoke the FlexClientOutboundQueueProcessor.flush method every 3 seconds, and print what I added in the flush method, then I only shut down the client's network, not close browser(client and server with difference network), I found the server-side don't print anything, it means that the server-side don't invoke the flush method.
And after more than 30 minutes I recover the client's network, the server-side continue to invoke the flush method (the client's session isn't destroyed, if I close the client's browser, after 30 minutes the server-side will destroy the session).
Now, I have two questions,:
How the server-side know the client's network has downed? Is there a listener to monitor the client's network? If so, where is it? If not, how and where the codes?
It seems that the server-side will invoke the FlexClientOutboundQueueProcessor.flush method every 3 seconds, can this interval be configured? And where the code to start or stop this timing task?
Here answer on your first question: Detecting (on the server side) when a Flex client disconnects from BlazeDS destination
About configuration. You can configure in services-config.xml.
Example BlazeDS applications
Configuring channels with servlet-based endpoints
I'm trying to make a Flex-based desktop application consume messages from an ActiveMQ topic with a durable subscription, using the JMS bridge of BlazeDS. The basic scenario is as follows:
Messages are produced by other producers in the topic to which the Flex client is subscribed.
The Flex client may go offline from time to time, but it must receive all the messages it has missed while being offline when it connects to BlazeDS again. (Of course the Flex client connects with the same client ID every time).
It can not be guaranteed that the Flex client is shut down gracefully.
Everything works fine if I explicitly disconnect my consumer on the Flex side by calling disconnect() - I do it in the exit handler of the application. However, due to #3 above, it is not guaranteed that disconnect() is called all the time. When the Flex client shuts down without calling disconnect(), it seems that the subscription of the "proxy JMS client" that BlazeDS creates and associates to the Flex client stays active towards ActiveMQ, so ActiveMQ still thinks that the client is logged in. When the Flex app starts up the next time, it is unable to log in to BlazeDS because ActiveMQ refuses its subscription, claiming that the client ID is already taken. Why is it so and what can I do here to ensure that BlazeDS makes the "proxy JMS client" offline in ActiveMQ when its real Flex counterpart terminates unexpectedly?
More detailed information: some debugging revealed that:
BlazeDS becomes aware of the termination of the Flex client because it prints a few exceptions to the console when in debug mode. The messages are as follows:
[BlazeDS]23:18:13.688 [WARN] Endpoint with id 'my-streaming-amf' is closing the streaming connection to FlexClient with id '71E6466F-D91F-201C-F60A-A6CB52F95D9F' because endpoint encountered a socket write error, possibly due to an unresponsive FlexClient.
ClientAbortException: java.net.SocketException: Broken pipe
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:319)
at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:288)
at org.apache.catalina.connector.Response.flushBuffer(Response.java:542)
at org.apache.catalina.connector.ResponseFacade.flushBuffer(ResponseFacade.java:279)
at flex.messaging.endpoints.BaseStreamingHTTPEndpoint.handleFlexClientStreamingOpenRequest(BaseStreamingHTTPEndpoint.java:818)
at flex.messaging.endpoints.BaseStreamingHTTPEndpoint.serviceStreamingRequest(BaseStreamingHTTPEndpoint.java:1055)
at flex.messaging.endpoints.BaseStreamingHTTPEndpoint.service(BaseStreamingHTTPEndpoint.java:460)
at flex.messaging.MessageBrokerServlet.service(MessageBrokerServlet.java:353)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:263)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:584)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:737)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:434)
at org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:299)
at org.apache.coyote.http11.Http11Processor.action(Http11Processor.java:963)
at org.apache.coyote.Response.action(Response.java:183)
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:314)
... 20 more
[BlazeDS]23:18:13.689 [DEBUG] Streaming thread 'http-8400-1' for endpoint with id 'my-streaming-amf' is releasing connection and returning to the request handler pool.
[BlazeDS]23:18:13.689 [INFO] Number of streaming clients for FlexSession with id '5BC5E8D604A361BCA673B05AC624CCC1' is 0.
[BlazeDS]23:18:13.689 [DEBUG] Number of streaming clients for endpoint with id 'my-streaming-amf' is 0.
At this stage, the subscriptions are still shown on the ActiveMQ web admin interface as being active.
Killing BlazeDS (more precisely, the Tomcat server that hosts it) with kill -9 from the console makes ActiveMQ realize immediately that the "proxy JMS client" is gone and it becomes offline on the ActiveMQ web admin interface. This made me conclude that BlazeDS is keeping the proxy JMS client alive explicitly since kill -9 gives no chance to BlazeDS to unsubscribe the client but it still becomes offline in ActiveMQ.
So, the question once again: What can I do here to ensure that BlazeDS makes the "proxy JMS client" offline in ActiveMQ when its real Flex counterpart terminates unexpectedly? Is this a bug in BlazeDS or am I just missing some hidden configuration setting that would make it work?
Version information: BlazeDS 4.0, ActiveMQ 5.5.0, both freshly downloaded today. I'm using the Tomcat server in the BlazeDS turnkey but ActiveMQ is installed separately because the BlazeDS turnkey ships with ActiveMQ 4.1.1 only. By the way, that version of ActiveMQ has the same issue.
The problem is that there is no way for BlazeDS to detect that your Flex client was shutdown, you will have to implement your own mechanism - my suggestion is to use a heart beat implemented with messaging. If no message is received from the client after a time interval you can assume that the Flex client is gone and do the disconnect (or you can use the session timeout mechanism on the server, and do the disconnect on session expire).
What you have seen (the exception caught when the streaming channel is closed) is not enough to say 100% sure that the Flex client is gone. The streaming is implemented using an HTTP connection kept open forever (used to send server messages) and periodic HTTP post calls (initiated by the client to send messages). In some networks the firewall can decide to kill the HTTP connection after a couple of seconds and you will receive the same error like the one you posted. However, it does not mean that the Flex client is killed - the Flex client can use a fallback strategy and switch to short/long polling in this case. Actually it would be a bug if BlazeDS will automatically do the JMS disconnect in this case.