On occasion Post request sent to our WAS leave the browser (IE) waiting for a response forever (until manually cancelled - e.g. by closing the browser).
Looking at the Trace and SystemOut logs on WAS everything looks fine. in the http_error log we can see the Get and Post requests but not the responses (even for successful requests). Is there a way to see the responses, or at least when they are sent? We need this to ensure that a response is being sent to the browser. Thanks.
I would issue a SIGQUIT signal to the Websphere process in order to produce a javacore, i.e. a threadump, in order to see what my WebContainer threads are actually doing.
Try running a kill -3 <WAS process ID> as root in a unix platform or invoke DumpThreads through wsadmin if running on a Windows platform.
You should enable trace for servlet container. This is done by setting trace string to *=info:com.ibm.ws.webcontainer*=all:com.ibm.wsspi.webcontainer*=all:HTTPChannel=all:GenericBNF=all
Setting up a trace
Related
I am inside a network where I need proxy settings to access the internet.
I have a weird problem.
The internet is working fine.
But it is one particular instance when i get this error:
Network Error (tcp_error)
A communication error occurred: "Operation timed out"
The Web Server may be down, too busy, or experiencing other problems preventing it from responding to requests. You may wish to try again at a later time.
For assistance, contact your network support team.
This happens when I use hadoop in local mode.
I can access the UI interface. I can see the jobs running. but when I try to see the logs of each task.. i am not able to access those logs.
UI--> job-->map--> task--> all <-- this is where the error is..
Any clues?
THanks
Not sure about exactly what your tcp action is, or about Hadoop or your proxy setup, but if you can reliably repeat the error, and the timeout error happens at approximately the same time each time you test, and that time is on the order of minutes, my guess would be that you've got a true processing delay (perhaps caused by blocking somewhere) at the server, but not necessarily.
We have a gaming application in which clients, all running IE make calls to web services using the API generated by the ASP.NET ScriptManager.
After a certain period of time, we start getting failures, with Fiddler showing the following error coming back from the server:
Connection to xxx.yyy.net failed.Exception Text: Only one usage of each socket address (protocol/network address/port) is normally permitted
We are not in control of the server's hardware, so altering the TCP/IP settings is not an easy option. Anyone have thoughts on how to deal with this?
This blog post seems relevant.
"Here is the scoop 1. When you make authenticated calls, the client is closing connections. And when you are making authenticated calls repeatedly to the same server, you are making and closing connections repeatedly 2. The same might happen when you are making regular http [un authenticated] calls but setting keep-alive = false."
http://blogs.msdn.com/b/dgorti/archive/2005/09/18/470766.aspx
I have a server application which runs in the Amazon EC2 cloud. From my client (the browser) I make a HTTP request which uploads a file to the server which then processes the file. If there is a lot of processing (large file
), the server always times out with a 504 backend continuation error always exactly after 120 seconds. Though I get this error, the server continues to process the request and completes it (verified by checking the database) but I cannot see the final result on my client because of the timeout.
I am clueless as to why this is happening. Has anyone faced a similar 504 timeout ? Is there some intermediate proxy server not in my control which is timing out ?
I have a similar problem and in my case I believe it is due to the connection between the Elastic Load Balancer (ELB) and the EC2 instance.
For a long-term solution I will go with the 303 Status response + back-end processing suggested by james.garriss above.
For short-term solution it may be possible for Amazon support to increase the ELB timeout (see their response in https://forums.aws.amazon.com/thread.jspa?messageID=491594񸁊). Unfortunately there doesn't seem to be any way to change the timeout yourself through either API or console.
[Update] AWS now does allow you to update the idle timeout either through console, CLI or .ebextensions configuration. See http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/config-idle-timeout.html (thanks #Daniel Patz for the update)
Assuming that the correct status code is being returned, the problem is that an intermediate proxy is timing out. "The server, while acting as a gateway or proxy, did not receive a timely response from the upstream server specified by the URI." (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.5) It most likely indicates that the origin server is having some sort of issue (i.e., taking a long time to process your request), so it's not responding quickly.
Perhaps the best solution is to re-craft your server app so that it responds with a "303 See Other" status code; then your client can retrieve the data at a later data point, once the server is done processing and creates the final result.
Edit: Another idea is to re-craft your server app so that it responds with a "413 Request Entity Too Large" status code when the request entity size is too large. This will get rid of the error, though it may make your app less useful if it can only process "small" files."
Other possible solutions:
Increase timeout value of the proxy (if it's under your control)
Make your request to a different server (if there's another, faster server with the same app)
Make your request differently (if possible) such that you are sending less data at a time
it is possible that the browser timeouts during the script execution.
I have a Flex client using a Flash binary (TCP) socket for communication with a Java server. I have a localhost (Apache) server providing a crossdomain.xml file which is wide open just while I am testing.
My code successfully loads the policy file on startup.
I then connect the socket to the server without any difficulty and send a message and get a response. All good so far.
However, when I send a second message through the same socket I get a pause of about 12 seconds then a sandbox violation error:
Security Error: Error #2048: Security sandbox violation: file:///C:/apache_root/ttt1/ttt1.swf cannot load data from localhost:45455.
This is the same port and socket through which the first message succeeded.
I tried re-loading the policy file before every send, but I get the same result.
Any idea why this might be happening? I clearly have an open socket at one point. I am flushing the socket after each send and I tried doing that after each read as well, but the same result.
Thanks in advance
EDIT:
If I recreate the socket prior to every call my code works. I am struggling to believe that this is correct, but maybe there is a Socket setting I am missing.
As far as I know if you're doing binary sockets the crossdomain.xml is not loaded via http.
Have you checked your apache's access logs if the crossdomain is even queried?
You might get connection from flash via tcp from flash asking for the file on your java server (not using http. It just sends the string "" or similar). Look out for them. If you don't answer them within 3 seconds (or so) flash throws an sandbox violation.
The first thing you have to do when you want to make a socket connection is to load the policy file. This only has to be done once per load of the SWF.
Security.allowDomain(host);
Security.loadPolicyFile("xmlsocket://"+host+":"+port);
The request will be made on the assigned port(45455 in your case) your server will have to listen on that port for a request "<policy-file-request/>" without the quotes.
When that request is found then you need to return to the client the crossdomain.xml
with node <allow-access-from domain="*" to-ports="*" />
After the cross domain is sent you need to close the socket on the server side
On the client side you need to ignore the domain response as Flex will handle that however at that time you can reconnect to the socket server.
At this time you can do your data send/receive.
I have a feeling the reason it actually worked for you is because you were using the connection for the policy file to transmit your data before it timed out.
I would suggest reading up on the new style of crossdomain policies and also reading up on the protocol you are using for your socket server
I think it depends on the sandbox-policy you used in the compilation process of your swf not on your crossdomain.xml... maybe this docu helps you:Security sandboxes
But I'm not 100% sure
This sort of sounds like a cache problem. Perhaps you're pulling the first socket connection out of cache and the second one gets rejected because it's getting a 200 from the server.
You might want to add localhost to your flash security exceptions list for debugging. that will quiet the sandbox errors until you get your piece to it's production environment.
Any thoughts on why I might be getting tons of "hangs" when trying to download a file via HTTP, based on the following?
Server is IIS 6
File being downloaded is a binary file, rather than a web page
Several clients hang, including TrueUpdate and FlexNet web updating packages, as well as custom .NET app that just does basic HttpWebRequest/HttpWebResponse logic and downloads using a response stream
IIS log file signature when success is 200 0 0 (sc-status sc-substatus sc-win32-status)
For failure, error signature is 200 0 64
sc-win32-status of 64 is "the specified network name is no longer available"
I can point firefox at the URL and download successfully every time (perhaps some retry logic is happening under the hood)
At this point, it seems like either there's something funky with my server that it's throwing these errors, or that this is just normal network behavior and I need to use (or write) a client that is more resilient to the failures.
Any thoughts?
Perhaps your issue was a low level networking issue with the ISP as you speculated in your reply comment. I am experiencing a similar problem with IIS and some mysterious 200 0 64 lines appearing in the log file, which is how I found this post. For the record, this is my understanding of sc-win32-status=64; I hope someone can correct me if I'm wrong.
sc-win32-status 64 means “The specified network name is no longer available.”
After IIS has sent the final response to the client, it waits for an ACK message from the client.
Sometimes clients will reset the connection instead of sending the final ACK back to server. This is not a graceful connection close, so IIS logs the “64” code to indicate an interruption.
Many clients will reset the connection when they are done with it, to free up the socket instead of leaving it in TIME_WAIT/CLOSE_WAIT.
Proxies may have a tendancy to do this more often than individual clients.
I've spent two weeks investigating this issue. For me I had the scenario in which intermittent random requests were being prematurely terminated. This was resulting in IIS logs with status code 200, but with a win32-status of 64.
Our infrastructure includes two Windows IIS servers behind two NetScaler load balancers in HA mode.
In my particular case, the problem was that the NetScaler had a feature called "Intergrated Caching" turned on (http://support.citrix.com/proddocs/topic/ns-optimization-10-5-map/ns-IC-gen-wrapper-10-con.html).
After disabling this feature, the request interruptions ceased. And the site operated normally. I'm not sure how or why this was causing a problem, but there it is.
If you use a proxy or a load balancer, do some investigation of what features they have turned on. For me the cause was something between the client and the server interrupting the requests.
I hope that this explanation will at least save someone else's time.
Check the headers from the server, especially content-type, and content-length, it's possible that your clients don't recognize the format of the binary file and hang while waiting for bytes that never come, or maybe they close the underlying TCP connection, which may cause IIS to log the win32 status 64.
Spent three days on this.
It was the timeout that was set to 4 seconds (curl php request).
Solution was to increase the timeout setting:
//curl_setopt($ch, CURLOPT_TIMEOUT, 4); // times out after 4s
curl_setopt($ch, CURLOPT_TIMEOUT, 60); // times out after 60s
You will have to use wireshare or network monitor to gather more data on this problem. Me think.
I suggest you put Fiddler in between your server and your download client. This should reveal the differences between Firefox and other cients.
Description of all sc-win32-status codes for reference
https://learn.microsoft.com/en-us/windows/win32/debug/system-error-codes--0-499-
ERROR_NETNAME_DELETED
64 (0x40)
The specified network name is no longer available.