I have a spring-boot application which is monitored by monit.
Monit just check the /health endpoint spring is exposing.
Basically, monit registered the following checks :
check host hopsearch_connection with address 127.0.0.1
if failed url http://127.0.0.1:8089/health with timeout 15 seconds then alert
check host hopsearch_health with address 127.0.0.1
if failed url http://127.0.0.1:8089/health
and content != 'DOWN'
with timeout 60 seconds
then alert
and the web application return something like that :
{"status":"UP","jestHealth":{"status":"UP","lastQuerySuccess":true},"diskSpace":{"status":"UP","free":14439550976,"threshold":10485760},"rabbit":{"status":"UP","version":"3.3.2"},"redis":{"status":"UP","version":"3.0.0"},"mongo":{"status":"UP","version":"2.6.1"}}
In this spring application, I have a general #ExceptionHandler to log all unexpected errors and to display an error page :
#ExceptionHandler(Exception.class)
#ResponseStatus(value = HttpStatus.INTERNAL_SERVER_ERROR)
public String handleDefaultError(Exception ex, HttpServletRequest httpRequest) {
logException(httpRequest, ex);
return "error";
}
And this #ExceptionHandler logs every call from monit :
<11>May 1 13:20:39 IP 13:20:39.339 t=http-nio-8089-exec-3 l=ERROR c=c.h.s.w.c.ErrorManager m=Error 500
----------------------------------------------------------------
Request Path=http://127.0.0.1:8089/health
Method=GET
----------------------------------------------------------------
Header :
----------------------------------------------------------------
host = 127.0.0.1:8089
accept = */*
connection = close
user-agent = monit/5.4
----------------------------------------------------------------
<11>May 1 13:20:39 IP 13:20:39.340 t=http-nio-8089- exec-3 l=ERROR c=c.h.s.w.c.ErrorManager m=Unexpected error : java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.8.0_25]
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[na:1.8.0_25]
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[na:1.8.0_25]
at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.8.0_25]
From monit perspective, everything is fine and the check works.
From my perspective, the application works. But I have a lot of errors dumped in the log.
I don't succeed to reproduce on my workstation with curl. This exception only happens on the server with monit.
Is it possible to ignore an error based on a specific user agent in an ExceptionHandler ? (and how ?)
It wouldn't be the best solution but I don't see any other way.
Any thoughts about that ?
Related
I have a flask server with and endpoint that processes some uploaded .csv files and returns a .zip (in a JSON reponse, as a base64 string)
This process can take up to 90 seconds
I've been setting it up for production using gunicorn and nginx and I'm testing the endpoint with smaller .csv s. They get processed fine and in a couple seconds I get the "got blob" log. But nginx doesn't return it to the client and finally it times out. I set up a longer fail-timeout of 10 minutes and the client WILL wait 10 minutes, then time out
the proxy read timeout offered as solution here is set to 3600s
Also the proxy connect timeout is set to 75s according to this
also the timeout for the gunicorn workers according to this
The error log says: "upstream timed out connection timed out while reading upstream"
I also see examples of nginx receiving an OPTIONS request and immediately after the POST request (some CORS weirdness from the client) where nginx passes the OPTIONS request but fails to pass the POST request to gunicorn despite nginx having received it
Question:
What am I doing wrong here?
Many thanks
http {
upstream flask {
server 127.0.0.1:5050 fail_timeout=600;
}
# error log
# 2022/08/18 14:49:11 [error] 1028#1028: *39 upstream timed out (110: Connection timed out) while reading upstream, ...
# ...
server {
# ...
location /api/ {
proxy_pass http://flask/;
proxy_read_timeout 3600;
proxy_connect_timeout 75s;
# ...
}
# ...
}
}
# wsgi.py
from main import app
if __name__ == '__main__':
app.run()
# flask endpoint
#app.route("/process-csv", methods=['POST'])
def process_csv():
def wrapped_run_func():
return blob, export_filename
# ...
try:
blob, export_filename = wrapped_run_func()
b64_file = base64.b64encode(blob.getvalue()).decode()
ret = jsonify(file=b64_file, filename=export_filename)
# return Response(response=ret, status=200, mimetype="application/json")
print("got blob")
return ret
except Exception as e:
app.logger.exception(f"0: Error processing file: {export_filename}")
return Response("Internal server error", status=500)
ps. getting this error from stackoverflow
"Your post appears to contain code that is not properly formatted as code. Please indent all code by 4 spaces using the code toolbar button or the CTRL+K keyboard shortcut. For more editing help, click the [?] toolbar icon."
for having perfectly well formatted code with language syntax, I'm sorry that I had to post it ugly
Sadly I got no response
See last lines for the "solution" finally implemented
CAUSE OF ERROR: I believe the problem is that I'm hosting the Nginx server on wsl1
I tried updating to wsl2 and see if that fixed it but I need to enable some kind of "nested virtualization", as the wsl1 is running already on a VM.
Through conf changes I got it to the point where no error is logged, gunicorn return the file then it just stays in the ether. Nginx never gets/sends the response
"SOLUTION":
I ended up changing the code for the client, the server and the nginx.conf file:
the server saves the resulting file and only returns the file name
the client inserts the filename into an href that then displays a link
on click a request is sent to nginx which in turn just sends the file from a static folder, leaving gunicorn alone
I guess this is the optimal way to do it anyway, though it still bugs me I couldn't (for sure) find the reason of the error
I'm trying to send logs into datadog using rsyslog. Ideally, I'm trying to do this without having the logs stored on the server hosting rsyslog. I've run into an error in my config that I haven't been able to find out much about. The error occurs on startup of rsyslog.
omfwd: could not get addrinfo for hostname '(null)':'(null)': Name or service not known [v8.2001.0 try https://www.rsyslog.com/e/2007 ]
Here's the portion I've added into the default rsyslog.config
module(load="imudp")
input(type="imudp" port="514" ruleset="datadog")
ruleset(name="datadog"){
action(
type="omfwd"
action.resumeRetryCount="-1"
queue.type="linkedList"
queue.saveOnShutdown="on"
queue.maxDiskSpace="1g"
queue.fileName="fwdRule1"
)
$template DatadogFormat,"00000000000000000 <%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% - - - %msg%\n "
$DefaultNetstreamDriverCAFile /etc/ssl/certs/ca-certificates.crt
$ActionSendStreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedPeer *.logs.datadoghq.com
*.* ##intake.logs.datadoghq.com:10516;DatadogFormat
}
First things first.
The module imudp enables log reception over udp.
The module omfwd enables log forwarding over (tcp, udp, ...)
So most probably - or atleast as far as i can tell - with rsyslog you just want to log messages locally and then send them to datadog.
I don't know anything about the $ActionSendStreamDriver tags, so I can't help you there. But what is jumping out is, that in your action you haven't defined where the logs should be sent to.
ruleset(name="datadog"){
action(
type="omfwd"
target="10.100.1.1"
port="514"
protocol="udp"
...
)
...
}
I make a server and request successfully in windows,then I put them to the linux which is the develop machine of company.
And I Got Connect Failed error when I run the clien.py. I have changed the host IP to localhost,0.0.0.0,127.0.0.1 and the IP of develop machine,but it doesn't work.
And I tried the docker inspect command,but the IPAddress is empty like "".
And I changed the port to 8500,8501,4321 and still got the error.
I tried make proxy to the company,the request returns a Name resolution failure error.So I think maybe the cause is the net of company?
Any answer would be very appreciate!
The Error Message:
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "Connect Failed"
debug_error_string = "{"created":"#1551677753.381330483","description":"Failed to create subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":2721,"referenced_errors":[{"created":"#1551677753.381328206","description":"Pick Cancelled","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":241,"referenced_errors":[{"created":"#1551677753.381303114","description":"Connect Failed","file":"src/core/ext/filters/client_channel/subchannel.cc","file_line":689,"grpc_status":14,"referenced_errors":[{"created":"#1551677753.381278484","description":"Failed to connect to remote host: OS Error","errno":111,"file":"src/core/lib/iomgr/tcp_client_posix.cc","file_line":205,"os_error":"Connection refused","syscall":"connect","target_address":"ipv4:0.0.0.0:8501"}]}]}]}"
I am getting below exception while connecting to pipeline debugger. I tried to re start application server / HTTP service / system as well but no luck. Any one have faced the same problem .
com.intershop.beehive.core.rapi.soap.ConnectException:
http://localhost:80/INTERSHOP/servlet/WFS/SOAP/-/SessionMgmtService;
nested exception is:
com.intershop.beehive.core.rapi.soap.SOAPException: SOAP-Stub:
response payload is not wellformed at
com.intershop.beehive.core.rapi.soap.SOAPStub.getPayload(SOAPStub.java:381)
at
com.intershop.beehive.core.internal.bos.session.SessionMgmtService_SOAPStub.setTimeout(SessionMgmtService_SOAPStub.java:130)
at
com.intershop.studio.server.suite6.remote.emc.Session.(Session.java:111)
at
com.intershop.studio.server.suite6.remote.RemoteSuite6ServerInstance.doConnect(RemoteSuite6ServerInstance.java:113)
at
com.intershop.studio.server.remote.services.impl.AbstractRemoteServerInstance$1.run(AbstractRemoteServerInstance.java:52)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:56) Caused
by: com.intershop.beehive.core.rapi.soap.SOAPException: SOAP-Stub:
response payload is not wellformed at
com.intershop.beehive.core.rapi.soap.SOAPStub.getPayload(SOAPStub.java:368)
... 5 more
I am using a PHP script to access the WEB API of gracenote. Though it works sometimes, it gives me a time out error most of the times. Here is the error I am getting:
exception: code=2001, message=Request to a Gracenote WebAPI timed
out., ext= PHP Fatal error: Uncaught exception
'Gracenote\WebAPI\GNException' with message 'Request to a Gracenote
WebAPI timed out.' in
/home/kubuntu/Downloads/php-gracenote-master/php-gracenote-master/php-gracenote/HTTP.class.php:1
Here is my very simple code:
<?php
include("./php-gracenote/Gracenote.class.php");
$clientID = "";
$clientTag = "";
$api = new Gracenote\WebAPI\GracenoteWebAPI($clientID, $clientTag);
$userID = "xxxxxxxxxxxxxxxxxx";
echo "UserID = ".$userID."\n";
$results = $api->searchArtist("Bob Dylan");
var_dump($results);
What am doing wrong?
I had the same issue where every 9 out of 10 requests the connection would time out with the following error:
http: external request POST url=https://1234567.web.cddbp.net/webapi/xml/1.0/, timeout=20000 exception: code=2001, message=Request to a Gracenote WebAPI timed out., ext=0
Fatal error: Uncaught exception 'Gracenote\WebAPI\GNException' with message 'Request to a
Gracenote WebAPI timed out.' in /usr/samba/dev/gracenote/php-gracenote/php-
gracenote/HTTP.class.php:110 Stack trace: #0 /usr/samba/dev/gracenote/php-gracenote/php-
gracenote/HTTP.class.php(94): Gracenote\WebAPI\HTTP->validateResponse(false) #1
/usr/samba/dev/gracenote/php-gracenote/php-gracenote/HTTP.class.php(144): Gracenote\WebAPI\HTTP-
>execute() #2 /usr/samba/dev/gracenote/php-gracenote/php-gracenote/Gracenote.class.php(59):
Gracenote\WebAPI\HTTP->post('<QUERIES>? ...') #3 /usr/samba/dev/gracenote/php-
gracenote/example.php(31): Gracenote\WebAPI\GracenoteWebAPI->register() #4 {main} thrown in
/usr/samba/dev/gracenote/php-gracenote/php-gracenote/HTTP.class.php on line 110
The reason ( i suspect) is the missing SSL option parameter for the curl connection.
I was able to fix permanently by adding this to line 38 in HTTP.class.php.
curl_setopt($this->_ch, CURLOPT_SSLVERSION, 3);
As suggested by Yavor, the problem had to do with SSL settings on the server. But we have now added full support for TLS v1.2, so your original code should work with the default SSL settings.
You should not use SSL v3 because of the POODLE vulnerability.
You should use try/catch to surround gracenote api calls, gracenote HTTP client throws various of exceptions
e.g.
https://github.com/richadams/php-gracenote/blob/25e0346443dd5026a4bc9f0d62a589d44bdc133b/php-gracenote/HTTP.class.php#L110