I have an nginx used mainly as a reverse proxy for a couple of upstream services. This nginx has a simple endpoint used for health checks:
location /ping { return 200 '{"ping":"successful"}'; }
The problem I'm having is that this ping takes too long to be responded:
$ cat /proc/loadavg; date ; httpstat localhost/ping?foo=bar
2.93 1.98 1.94 8/433 16725
Thu Jul 15 15:25:08 UTC 2021
Connected to 127.0.0.1:80 from 127.0.0.1:42946
HTTP/1.1 200 OK
Date: Thu, 15 Jul 2021 15:26:24 GMT
X-Request-ID: b8d276b0b3828113cfee3bf2daa01293
DNS Lookup TCP Connection Server Processing Content Transfer
[ 4ms | 0ms | 76032ms | 0ms ]
| | | |
namelookup:4ms | | |
connect:4ms | |
starttransfer:76036ms |
total:76036ms
That ^ is telling me that the average load is low at the time of the request (2.93 the 1m average load for an 8-core server is ok)
Curl/httpstat initiated the request at 15:25:08 and response was obtained 15:26:24.
Connection was stablished fast, request sent, then it took 76s for the server to respond.
If I look at the access log for this ping I see "req_time":"0.000" (this is the $request_time variable).
{"t":"2021-07-15T15:26:24+00:00","id":"b8d276b0b3828113cfee3bf2daa01293","cid":"18581172","pid":"13631","host":"localhost","req":"GET /ping?foo=bar HTTP/1.1","scheme":"","status":"200","req_time":"0.000","body_sent":"21","bytes_sent":"373","content_length":"","request_length":"85","stats":"","upstream":{"status":"","sent":"","received":"","addr":"","conn_time":"","resp_time":""},"client":{"id":"#","agent":"curl/7.58.0","addr":",127.0.0.1:42946"},"limit_status":{"conn":"","req":""}}
This is the access log format in case anybody wonders what are the rest of the values:
log_format main escape=json '{"t":"$time_iso8601","id":"$ring_request_id","cid":"$connection","pid":"$pid","host":"$http_host","req":"$request","scheme":"$http_x_forwarded_proto","status":"$status","req_time":"$request_time","body_sent":"$body_bytes_sent","bytes_sent":"$bytes_sent","content_length":"$content_length","request_length":"$request_length","stats":"$location_tag","upstream":{"status":"$upstream_status","sent":"$upstream_bytes_sent","received":"$upstream_bytes_received","addr":"$upstream_addr","conn_time":"$upstream_connect_time","resp_time":"$upstream_response_time"},"client":{"id":"$http_x_auth_appid$http_x_ringdevicetype#$remote_user$http_x_auth_userid","agent":"$http_user_agent","addr":"$http_x_forwarded_for,$remote_addr:$remote_port"},"limit_status":{"conn":"$limit_conn_status","req":"$limit_req_status"}}';
My question is: where could nginx have spent these 76s if the request just took 0s to be processed and responded?
Something special to mention is that the server is timing out a lof of connections with the upstreams at that moment as well: we see a lot of upstream timed out (110: Connection timed out) while reading response header from upstream and upstream server temporarily disabled while reading response header from upstream.
So, these two are related, what I can't see is why upstream timeouts would lead to a /ping taking 76s to be attended and responded when both cpu and load are low/acceptable.
Any idea?
I have this kind of exception, on a production environnement:
Fatal error: Uncaught RuntimeException: Unable to create the cache
directory
(/app/ezplatform/var/cache/prod/a6d61d85393d8924b6a2d32272d325510651f125)
in
/app/ezplatform/vendor/symfony/symfony/src/Symfony/Component/HttpKernel/Kernel.php:765
Stack trace: #0
/app/ezplatform/vendor/symfony/symfony/src/Symfony/Component/HttpKernel/Kernel.php(642):
Symfony\Component\HttpKernel\Kernel->buildContainer() #1
/app/ezplatform/vendor/symfony/symfony/src/Symfony/Component/HttpKernel/Kernel.php(135):
Symfony\Component\HttpKernel\Kernel->initializeContainer() #2
/app/ezplatform/vendor/symfony/symfony/src/Symfony/Component/HttpKernel/Kernel.php(195):
Symfony\Component\HttpKernel\Kernel->boot() #3
/app/ezplatform/web/app.php(64):
Symfony\Component\HttpKernel\Kernel->handle(Object(Symfony\Component\HttpFoundation\Request))
4 {main} thrown in /app/ezplatform/vendor/symfony/symfony/src/Symfony/Component/HttpKernel/Kernel.php
on line 765
I don't understand why the response http code is 200 instead of 500, is it because a server low level exception ? How to customize the error page for this kind of situation.
This the response header:
HTTP/2.0 200 OK
content-type: text/html; charset=UTF-8
date: Wed, 18 Dec 2019 08:32:40 GMT
strict-transport-security: max-age=0
x-debug-info: eyJyZXRyaWVzIjowfQ==
x-request-id: z7m5kvoqa4smukjgb74pxhqd
x-robots-tag: noindex, nofollow
content-length: 982
X-Firefox-Spdy: h2
This error is caused by the Symfony cache generator, not by your application.
The application will try to generate it's cache on a folder, from the directory, but it can't access it. The folder var/cache/ must be allowed to be written by the user that is running the application, maybe your server user.
Changing this with a simple chmod USER_RUNNING_THE_APP +w var/cache -R will solve the issue most likely.
As for the issue about the the response header being 200 instead of 500, this is because the server can respond, and responses using the application. The application will return a response as 200 because will this:
route the response to the application
application does it's thing to generate an output
generates that error message
we have content, so this means it's 200, cause nothing else is specified
return 200 with the error message
When requesting via https it looks like serf is funnelling the request via port 80 instead of 443?
[Mon Jan 16 10:25:48.007386 2017] [error] [pid 350] [mod_pagespeed 1.11.33.4-0 #350] Serf status 120171(APR does not understand this error code) polling for 1 threaded fetches for 0.05 seconds
[Mon Jan 16 10:25:48.007539 2017] [error] [pid 350] [mod_pagespeed 1.11.33.4-0 #350] Serf status 120171(APR does not understand this error code) polling for 1 threaded fetches for 0.05 seconds
[Mon Jan 16 10:25:53.021234 2017] [warn] [pid 350] [mod_pagespeed 1.11.33.4-0 #350] Fetch timed out: https://www.domain.com/assets/76dc6ad2/style.min.css (connecting to:10.33.12.222:80) (1) waiting for 50 ms
SSL termination on the load balancer. SSL is also configured to work from behind the load balancer as well so https can be served from within the network.
ModPagespeedFetchHttps enable
ModPagespeedRespectXForwardedProto on
ModPagespeedEnableFilters prioritize_critical_css
How to have serf request https via port 443?
#dhaupin
I don't seem to notice that error anymore.
This is probably what fixed it, explicitly handling https requests.
ModPagespeedLoadFromFile "https://example.com" "/var/www/example/"
ModPagespeedRespectXForwardedProto on
I currently have a client that is using AEM 6.0. As they have some URL's that are lengthy they have been looking for a solution to shorten them without editing a virtual host file and adding redirects to the virtual host (they do not have the proper knowledge to do that). The only solution I found was to use vanity urls which per my research is supported in AEM 6.0 and as of Dispatcher 4.1.9.
Currently they have a dispatcher version 4.1.10 and I have gone through the AEM documentation https://docs.adobe.com/docs/en/dispatcher/disp-config.html#par_title_21 that discribes how to configure the Dispatcher and Publisher to enable access to vanity urls.
So far:
I have installed the VanityURLS-Components package on the Publisher.
I have added the following configuration to the dispatcher.any on the Dispatcher:
/vanity_urls {
/url "/libs/granite/dispatcher/content/vanityUrls.html"
/file "/tmp/vanity_urls"
/delay 300
}
and checked that the paths are correct.
I have created /tmp/vanity_urls file with ownership of apache:apache (this is on Centos) and permissions of 777.
And I have restarted apache.
Despite these steps it looks like I've overlooked something as /tmp/vanity_urls is not being updated. Maybe there is something I am not understanding here but I thought that the dispatcher updated every x seconds (here 300) /tmp/vanity_urls via the Publisher's /libs/granite/dispatcher/content/vanityUrls.html. Then used /tmp/vanity_urls as a whitelist of vanity urls that are allowed.
I am wondering why this is not working, any thoughts ?
Could it be a permission issue on /tmp/vanity_urls ?
Maybe there is something I erroneously assumed ?
Are there existing bugs out there I am unaware of that impact this dispatcher vanity urls feature ?
Any help is welcome ...
Best,
Nicola
UPDATE:
In my logs found the following:
[Thu Oct 08 16:11:03 2015] [D] [1780(140151407138784)] Vanity URL file (/tmp/vanity_urls) too old (1443478601 < 1444345863) on startup, fetching...
[Thu Oct 08 16:11:03 2015] [D] [1780(140151407138784)] Creating new socket: 127.0.0.1:8080
[Thu Oct 08 16:11:03 2015] [W] [1780(140151407138784)] Unable to connect to 127.0.0.1:8080: Connection refused
[Thu Oct 08 16:11:03 2015] [D] [1780(140151407138784)] incomplete request, no socket reuse
[Thu Oct 08 16:11:03 2015] [E] [1780(140151407138784)] Unable to fetch vanity URLs on farm website: no backend available.
[Thu Oct 08 16:11:03 2015] [D] [1780(140151407138784)] Loaded 0 vanity URLs from file /tmp/vanity_urls
Fairly self explanatory given that my publisher is not on localhost port 8080 ...
Hopefully that should fix my issue will update soon.
Thanks,
Nicola
I figured it out it was a network issue nothing to do with AEM,
/libs/granite/dispatcher/content/vanityUrls.htm was not accessible from my publisher.
I attempted to install Gitolite on a Fedora 17 server with the aim of setting up git and HTTP access along with authorisation. Git access works OK. Can push and pull. But HTTP access falls over with an Error 500. It appears I got something wrong with permissions.
Here's what I did. I followed instructions from here: sitaramc.github.com
I have documented what I have tried to do here if anyone would like to see it down to detail.
HTTP Error:
Internal Server Error - 500
The server encountered an internal error or misconfiguration and was unable to complete your request.
More information about this error may be available in the server error log.
Error Log - /var/log/httpd/error-git.log
[Wed Feb 13 08:26:11 2013] [error] [client 192.168.0.40] suexec failure: could not open log file
[Wed Feb 13 08:26:11 2013] [error] [client 192.168.0.40] fopen: Permission denied
[Wed Feb 13 08:26:11 2013] [error] [client 192.168.0.40] Premature end of script headers: gitolite-suexec-wrapper.sh
[Wed Feb 13 08:30:13 2013] [error] [client 192.168.0.40] Directory index forbidden by Options directive: /var/www/git/
* Update 1 *
- Managed to post the error output here.
* Update 2 *
Relaxed permissions on log directory and gitolite-suexec-wrapper.sh. More details are available at the link above where I have documented in detail.
/var/log/httpd/error-git.log
[Wed Feb 13 21:18:47 2013] [error] [client 192.168.0.40] suexec policy violation: see suexec log for more details
[Wed Feb 13 21:18:47 2013] [error] [client 192.168.0.40] Premature end of script headers: gitolite-suexec-wrapper.sh
[Wed Feb 13 21:18:54 2013] [error] [client 192.168.0.40] Directory index forbidden by Options directive: /var/www/git/
$ sudo more /var/log/httpd/suexec.log
[2013-02-13 21:18:47]: uid: (990/git) gid: (988/git) cmd: gitolite-suexec-wrapper.sh
[2013-02-13 21:18:47]: cannot stat program: (gitolite-suexec-wrapper.sh)
Not sure where next
* Update 3 *
Ok, so I made some progress. I may have fixed the permissions issue. Now facing a PATH issue. Like before, most relevant output is included here. Full details are updated at the link in my original post.
My knowledge of Apache config is very basic. After reading about suEXEC, I realised the permission issue could be arising out of SELinux. So I disabled it for now. (Would like to identify a way of having gitolite working with SELinux active, but that's for later. Suggestions are welcome.)
Now when I access the url: http:// mochapenguin /git/testing.git in browser
001E# service=git-upload-pack
0000003BERR FATAL: unknown git/gitolite command: 'testing.git'
When I test from the client machine, I see:
ssh git#mochapenguin \echo $PATH
FATAL: unknown git/gitolite command: 'echo /usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/home/mochapenguin/.local/bin:/home/mochapenguin/bin'
* Update 4 *
Alright, got it working. No further change was needed since my last update.
I ought to have tried accessing the repo like so, instead of trying the URL in the browser:
git clone http://username:password#mochapenguin/git/testing.git
This setup allows clone and push over http
I can't figure out what I got wrong.
Could someone point me the right way, please?