I deployed ingress controller to my kubernetes cluster. However when i'm trying to access the pod i get HTTP 503. So i've tried to find the problem by accessing ingress controller's logs:
kubectl logs controllername-nginx-ingress-controller-6f486779b5-dnm8k -n kube-system
the piece of the log file that i'm interested in, looks like this:
10.244.0.1 - [10.244.0.1] - - [10/Dec/2018:16:54:12 +0000] "GET /identity HTTP/2.0" 503 599 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" 271 0.000 [] - - - - d7f7ffd22d584e4a5da2b0fa2fae5665
as you can see i see absolutely nothing interesting there. I'm wondering if it's the entire log message as i see a bunch of dashes and a guid value at the end.
Next i decided to enrich ingress' logfiles:
kubectl edit deploy -n kube-system controllername-nginx-ingress-controller
deployment document shows up and i want to add extra line to args section:
spec:
containers:
- args:
- /nginx-ingress-controller
- --v=5 < ------------ this line
- other arguments here
once i close the deployment document i see this message:
error: deployments "misty-marmot-nginx-ingress-controller" is invalid
so what i'm doing wrong? is the log message really complete, in the current form it doesn't describe the error one bit. also why i can't change the logging level?
The problem has appeared due to a formatting issue of editing nginx-ingress-controller Kubernetes resource and was successfully fixed as well. However, even though for the common researches by the community contributors I would recommend to take a look at the general Troubleshooting guideline for any related Kubernetes cluster issues and steps how to resolve them.
Related
I have an ASP.NET Core v.3.0 web application hosted in Amazon AWS Elastic Beanstalk using Docker. The application works fine in Docker running 64bit Amazon Linux (v2.16.11). When I update to Docker running 64bit Amazon Linux 2 (v3.4.12) the requests work fine except for AJAX requests which fail with Status Code Error 400 "Bad request". Nothing else has changed in the source code, dockerfile etc. Only the Linux version has changed from Amazon Linux to Amazon Linux 2. Does anybody have an idea what is different between Amazon Linux 1 and Amazon Linux 2 that may be the cause leading to AJAX requests failing?
More info:
I cannot replicate this error with the official ASP.NET core 3.1 examples. I have not updated my application to v3.1, I will do it soon and I will update this question.
The relevant action inside the controller does not return the partial view in Amazon Linux 2. The controller provides a log just before returning the partial view and this is not triggered in Amazon Linux 2.
The nginx access.log file shows the following response of the load balancer:
Amazon Linux 1:
{IP} - - [10/Apr/2022:07:36:01 +0000] "POST {url} HTTP/1.1" 200 3882 "{url2}" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.75 Safari/537.36" "{IP2}"
Amazon Linux 2:
{IP} - - [10/Apr/2022:07:00:14 +0000] "POST {url} HTTP/1.1" 400 0 "{url2}" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.75 Safari/537.36" "{IP2}"
The call is made with jQuery 3.4.1:
var $form = $("#inputForm");
if ($form.length <= 0) return;
var data = $form.serialize();
$.ajax({
url: "...",
type: "POST",
data: data,
error: function (jqXHR, textStatus, errorThrown) {
alert("There was an error when loading the results (error type = " + errorThrown + ").");
},
success: function (result) {
$("#calculationTarget").html(result)
});
The issue is no longer present if the project is updated from ASP.NET Core 3.0 to ASP.NET Core 3.1.
There is a very simple fix, which is updating to ASP.NET Core 3.1.
In this version, the issue that you have having is fixed.
See the steps below for updating.
If you have a global.json file to target a specific .NET Core SDK version, update the version property.
{
"sdk": {
"version": "3.1.101"
}
}
Update the TFM to netcoreapp3.1, as described below.
<Project Sdk="Microsoft.NET.Sdk.Web">
<PropertyGroup>
<TargetFramework>netcoreapp3.1</TargetFramework>
</PropertyGroup>
</Project>
You need to update the package references. To do this, update every Microsoft.AspNetCore.* (* meaning wildcard) to 3.1.0 (or any version later).
If you are using Docker (which I think you are), then you need to use an ASP.NET Core 3.1 base image. See an example below.
$ docker pull mcr.microsoft.com/dotnet/aspnet:3.1
For extra steps and information, see the official guide from migrating to ASP.NET Core 3.1.
In summary, upgrading your current application to ASP.NET Core 3.1 should fix your issue.
I have a site that have some plugins, and one of those plugins (facebook for woocommerce) is loading until it returns a timeout error (504). I can change some constants in wp-config.php, but none of them works when I need to debug a timeout.
I tried to remove every configuration and file that I found from this plugin and then reinstall it, but the error is still there.
I tried to deactivate every other plugin, but woocommerce and the error is still there.
I looked for some debug plugins, but I only found plugins that change wp-config.php constants and do some logs at files. It is useless, I can do this.
I tried to put some "die" with messages in plugin's code, but nothing changed.
Server log just shows this:
x.x.x.x - - [09/Nov/2020:17:52:56 -0300] "xxxxx.com" "GET /wp-admin/admin.php?page=wc-facebook HTTP/1.1" 504 160 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:82.0) Gecko/20100101 Firefox/82.0" "-"
I don't know what I can to do to debug this timeout, I've tried everything I know with the wordpress.
I solved by asking on plugin's forum: https://wordpress.org/support/topic/plugin-page-giving-timeout-504/#post-13687667
I just needed to activate WP_DEBUG and WP_DEBUG_LOG flags. I discovered the line that was breaking the site and then I could properly debug and find the problem.
I started to learn Python with connexion and flask and got stuck resolving an issue. Even after taking a long break I cannot see where I am wrong and need some advise.
I created a swagger API definition which maps /api/blacklist/{zipcode} to a function. However when I try to access /api/blacklist/12345 I receive a 404. The YAML is located here:
https://github.com/TheHasgarion/pythonflaskrest/blob/master/api.yml
The mapping for /api/blacklist/ works just fine. BTW, even accessing /api/blacklist/ gives me a 404.
The server log says:
127.0.0.1 - - [23/Jun/2019 19:52:09] "GET /api/blacklist/12345 HTTP/1.1" 404 -
127.0.0.1 - - [23/Jun/2019 19:57:16] "GET /api/blacklist HTTP/1.1" 200 -
127.0.0.1 - - [23/Jun/2019 19:57:18] "GET /api/blacklist/ HTTP/1.1" 404 -
Thank you very much in advance for guidance.
Apparently you need to use integer as type, not number. Whyever 'number' means 'float' :-)
Nor do nginx or php-fpm reports a error 500 output, In fact the 500 response is on the access log and not on the nginx error log. The script is running ok in my dev environment.
nginx version: nginx/1.6.2
PHP 5.5.19 (fpm-fcgi)
Tried this
catch_workers_output = 1
Restarted everything, still not working
nginx access log shows:
x.x.x.x - - [12/Dec/2014:19:25:08 -0200] "GET /merchant/customer/mobile/data?sEcho=1&iColumns=3&sColumns=%2C%2C&iDisplayStart=0&iDisplayLength=10&mDataProp_0=0&sSearch_0=&bRegex_0=false&bSearchable_0=true&mDataProp_1=1&sSearch_1=&bRegex_1=false&bSearchable_1=true&mDataProp_2=2&sSearch_2=&bRegex_2=false&bSearchable_2=true&sSearch=&bRegex=false&_=1418418256370 HTTP/1.1" 500 589 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36"
Note the 500 error, which should be on error.log and with the php trace that's is happening on other errors.
Any clues?
When PHP display_errors are disabled, PHP errors can return Nginx 500 error.
Take a look into your php-fpm logs, i'm sure you'll find the error there. With CentOS 7 :
tail -f /var/log/php-fpm/www-error.log
You can finally show PHP errors. In /etc/php.ini, change :
display_errors = Off
to :
display_errors = On
Hope it helps.
In case it helps someone else (Google brought me here), I had a similar problem (although in Apache, not nginx).
An older app I was installing was giving a 500 error with no output anywhere, in spite of every conceivable error logging setting turned to the most verbose level.
It turns out the problem was the controversial error-control operator, "#". As the red warning box in the docs states, no matter how verbose your logging is, if an #-prefixed command causes PHP to stop because of a typo, or because it is not available (say, because you have forgotten to install a critical module like php-mysql), PHP will exit with absolutely no indication as to why.
So, if you find yourself with a 500-error and no logs, check your codebase for "#" symbols.
Verify php-fpm service is running
sudo service php-fpm status
This will display the hostnames that can execute php.
Next edit php.ini to display errors. this will display errors on page..to make things easy to diagnose
sudo vim /etc/php.ini
then hit key [/] and type
display_errors
hit key [i] user arrow keys and backspace to set display_errors = on
then restart php-fpm service
On system type redhat /centos etc..
sudo service php-fpm restart
I am a bit confused about syslog message format. I have to write a program that parses syslog messages. When I read what I get in my syslog-ng instance I get messages like this:
Jan 12 06:30:00 1.2.3.4 apache_server: 1.2.3.4 - - [12/Jan/2011:06:29:59 +0100] "GET /foo/bar.html HTTP/1.1" 301 96 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 ( .NET CLR 3.5.30729)" PID 18904 Time Taken 0
I can clearly determine the real message (which is, in this case an Apache access log message) The rest is metadata about the syslog message itself.
However when I read the RFC 5424 the message examples look like:
without structured data
<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed for lonvick on /dev/pts/8
or with structured data
<165>1 2003-10-11T22:14:15.003Z mymachine.example.com evntslog - ID47 [exampleSDID#32473 iut="3" eventSource="Application" eventID="1011"] BOMAn application event log entry...
So now I am a bit confused. What is the correct syslog message format ? It is a matter of spec version where RFC 5424 obsoleted RFC 3164 ?
The problem in this case is that apache is logging via the standard syslog(3) or via logger. This only supports the old (RFC3164) syslog format, i.e. there is no structured data here.
In order to have the fields from the apache log show up as RFC5424 structured data, apache would need to format the log that way.
The first example is not proper RFC3164 syslog, because the priority value is stripped from the header. Proper RFC3164 format would look like this:
<34>Jan 12 06:30:00 1.2.3.4 apache_server: 1.2.3.4 - - [12/Jan/2011:06:29:59 +0100] "GET /foo/bar.html HTTP/1.1" 301 96 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 ( .NET CLR 3.5.30729)" PID 18904 Time Taken 0
Traditionally rfc3164 syslog messages are saved to files with the priority value removed.
The other two are in RFC5424 format.
If you have access to the installed syslog-daemon on the system you could configure it to write the logs (received both locally or via network) in a different format. rsyslogd for instance allows to configure your own format (just write a template) and also if I remember correctly has a built-in template to store in json format. And there are libraries in almost any language to parse json.
EDIT: You could also make rsyslogd part of your program. rsyslog is very good in reading incoming syslogs in either of the two RFC formats. You can then use rsyslog to output the message in JSON. This way rsyslog does all the decompositioning of the message for you.