We are using springboot 1.3.3.RELEASE with embedded tomcat.
The service is running on redhat Linux 2.6.32 (64bit), using java 1.8.0_45.
In our load environment we noticed that the server is up (java VM is still running) and responding to non-HTTP requests, but HTTP requests exposed via SpringMVC Rest do not work, we get timeout.
After comparing thread dump between healthy and not-healthy system, we noticed in a non-healthy system the http-nio-{port}-Acceptor thread is missing.
In particular the dump shown bellow from a healthy machine is missing from bad one.
"http-nio-8080-Acceptor-0" #45 daemon prio=5 os_prio=0
tid=0x00007f13fb9ef800 nid=0x896b runnable [0x00007f146f3f4000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
- locked <0x00000005cd5d0558> (a java.lang.Object)
at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:682)
at java.lang.Thread.run(Thread.java:745)
What could be reasons for the Acceptor thread to be missing?
Is it possible to restart the threat, without restarting the whole application?
Is that the reason we are not able to service HTTP requests?
Related
I am deploying AWS Corda Enterprise Template. The Quick start deployed the stack as per the defined CloudFormation template. I can see 2 AWS instances, up and running as Corda nodes, in Hot-Cold setup with a load balancer.
However the Log for Corda node has following ERROR related to AMQP communication.
[ERROR] 2018-10-18T05:47:55,743Z [Thread-3
(ActiveMQ-scheduled-threads)] core.server.lambda$channelActive$0 -
AMQ224088: Timeout (10 seconds) while handshaking has occurred. {}
What can be possible reason for this error? This error keeps on occurring after a certain time interval. So it looks like some connectivity issue to me.
Note: The load balancer shows the status of this AWS Corda instances as healty (In Service). So I believe the Corda node has booted up successfully.
The ERROR message isn't necessarily tied to AMQP. Perhaps you were confused by the "AMQ" in the error ID (AMQ224088)?
In any event, this error indicates that something on the network is connecting to the ActiveMQ Artemis broker, but it's not completing any protocol handshake. This is commonly seen with, for example, load balancers that do a health check by creating a socket connection without sending any real data just to see if the port is open on the target machine.
I got 2 servers with two equal wcf services hosted on them and one client application server. I can connect to endpoints and send a requests to both services using test wcf client app (.NET Web Service Studio) from my local machine successfully. But when I am trying to connect from client application server using the same test wcf client app I successfully connected only to the one wcf service server, but I have got an error when connecting to another one:
System.Net.WebException: There was an error downloading 'https://XXX/XXX?wsdl'. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it XX.XXX.XX.XXX:443
I performed netstat -an | find "443" command in command prompt on the client server and on my local machine to find out the difference and here what I have got:
1. On my local machine:
2. On the client app server:
What I already tried to do on client application server is:
- turned off firewall;
- stopped windows firewall service
- uninstalled mcafee virusscan enterprise application.
(I tried to set "prevent mass mailing worms from send mail" first, but mcafee was in foreign language that I don't understand, so I just uninstalled it)
after running command netstat -aon | findstr "443" on client application server I have got this result:
but I still got an error.
Does anybody know how to solve this issue?
Could be the problem on the wcf service server side?
The solution was predictable simple one - firewall was blocking the port,
but it's important to notice that the issue was caused by firewall on the wcf service server side, but not on client application server, which is making the request to that service.
I asked the technical support of that server, and they made firewall changes.
After that error was disappeared.
I faced the same issue and tried different ways to fix this. Nothing works. Later i found the issue which is, the application i tried to run is https and in my IIS, https binding was not created. I created binding https with the website and it works.
I started the server and then application on a certain IP and Port from scratch (no dulpicate servers or port already used). The application was crashing due to java-lang-OutOfMemory error, saying that no new thread can be created. I allocated JVM 4G memory, and now that error is silent. But, now afterwards starting the client is crashing again. It runs for about 10-15 seconds and crash with SocketTimeoutException. Here is the trace
org.h2.jdbc.JdbcSQLException: Connection is broken: "java.net.SocketTimeoutException: connect timed out: MYIPADDRESS" [90067-172]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:329)
at org.h2.message.DbException.get(DbException.java:158)
at org.h2.engine.SessionRemote.connectServer(SessionRemote.java:407)
at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:295)
at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:105)
at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:90)
at org.h2.Driver.connect(Driver.java:73)
at java.sql.DriverManager.getConnection(DriverManager.java:579)
at java.sql.DriverManager.getConnection(DriverManager.java:221)
at com.org.tseries.simulator.IPEmailSeriesTestDriver.shootH2(IPEmailSeriesTestDriver.java:201)
at com.org.tseries.simulator.IPEmailSeriesTestDriver.processSimulator(IPEmailSeriesTestDriver.java:160)
at com.org.tseries.simulator.IPEmailSeriesTestDriver.main(IPEmailSeriesTestDriver.java:98)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
at java.net.Socket.connect(Socket.java:579)
at org.h2.util.NetUtils.createSocket(NetUtils.java:119)
at org.h2.util.NetUtils.createSocket(NetUtils.java:100)
at org.h2.engine.SessionRemote.initTransfer(SessionRemote.java:95)
at org.h2.engine.SessionRemote.connectServer(SessionRemote.java:403)
... 14 more
It is crashing when I run the client from terminal but when I ran the same client from STS (eclipse based IDE), it didnt crashed. Any setting or flag I need to add while running the client?
UPDATED: It is solved with appending AUTORECONNECT=TRUE in connection string. More details here, h2database.com/html/features.html#auto_reconnect
But now, I am interested in making a connection pool at Server end that allows different TCP clients to connect. The server shall serve the clients from the connection pool instead of clients reconnecting again. How can it be achieved?
Try open C:\Users\Superuser\db.lock.db. If this database already running you will see IP and port. Then in console type command netstat -ano. It will display all processes. Find PID of proccess that holds IP and port you found in .db file. Than you may kill this proccess using Task Manager or with help of command in console.
Another way: check all configs. You may use different IP in different configs. Synchronize this
Does stopping a BizTalk host instance also stop the Applications that run under it?
Or, what is the difference between stopping a host instance and stopping the applications under it?
No, host instances and applications are completely independent. You can stop a host instance and the application will remain in the started state. You can stop an application and the host instance will remain in the running state.
To understand the difference between stopping a host instance and stopping an application you first need to understand what these things are.
Basically, you need to think of your application as a set of assemblies plus some runtime configuration, and a set of logical subscriptions.
When you "start" an application up there are actually two steps which happen.
The parts of your application which need to receive messages (orchestrations and send ports) are enlisted. This ensures that an internal queue exists to receive the messages. Note that the application is not yet started, but it can receive and queue messages for processing later.
Then when you start the application the various parts of your application are able to process the messages.
The host instance is basically a windows service.
When you stop the host instance, all you are really doing is stopping the underlying windows service which runs the host instance. This means that all the assemblies which contain your application artifacts are unloaded, and the application will obviously stop processing. This is despite the fact that the application is still in the started state.
When you start the host instance again it loads your application assemblies back into memory and will be able to continue processing new messages. Messages which were being processed when the host instance was stopped may be in a state called suspended, but if they are can be manually resumed.
Hope this helps.
Yes, if you application run only on that host instance (meaning: application will stop to process messages). However internals of why it stopped processing is quite different. See explanation below and in hugh jadick's answer.
Stopping host instance for specified host type will stop execution of all artifacts (adapter handlers, receive locations, pipelines, orchestrations, etc.) that run on specified host. Application is a logical group of artifacts which can run on single or multiple host instances. Multiple applications can run on a single host instance, and vice versa. So, stopping an app is just shutdown of execution of artifacts, while stopping host instance is shutdown of physical instance there app artifacts are executing.
I have been trying to set-up my recently reimaged workstation for working with one of our ASP.Net applications that consists of a ASP.Net website and some C# Web Services. I can successfully start debug instances of each the services and web site. However as soon as the site attempts to connect to and use one of the web services I get the following exception.
No connection could be made because
the target machine actively refused it
127.0.0.1:80
Description: An unhandled exception occurred during
the execution of the current web
request. Please review the stack trace
for more information about the error
and where it originated in the code.
Exception Details:
System.Net.Sockets.SocketException: No
connection could be made because the
target machine actively refused it
127.0.0.1:80
So it looks like my PC is refusing the connection to itself. My first thought was to to check the windows firewall is disabled and it is disabled (the service is not running). I do not think IT has any other firewall software running on this PC, so I am a bit stumped as to what else could be causing the connection to be refused.
Silly question, but do you actually have IIS installed on your machine? When you run debug instances of your applications, it's running outside of IIS. If you're running debug instances of both your site and your service, your service probably isn't running on port 80.
Double check your IIS installation...and then set up your service running on port 80 properly.