Wordpress hosted on EC2 overloads the server - wordpress

I have hosted a wordpress website on Amazon EC2 t2.medium server instance.
The OS is 64-bit Ubuntu 16.x.x version
The issue is that, after the server is up for about 2-3 days, the server suddenly overloads and stops responding.
I am not sure what is triggering this overload on the server.
When I stop the apache server, the load gradually decreases, and restarting the server after few minutes fixes the problem.
Here is the load info,
ubuntu#ip-172-31-18-127:~$ uptime
12:42:43 up 6 days, 49 min, 2 users, load average: 32.58, 63.91, 34.88
ubuntu#ip-172-31-18-127:~$ uptime
12:42:47 up 6 days, 49 min, 2 users, load average: 29.97, 62.85, 34.69
ubuntu#ip-172-31-18-127:~$ uptime
12:42:50 up 6 days, 49 min, 2 users, load average: 27.57, 61.80, 34.50
ubuntu#ip-172-31-18-127:~$ uptime
12:42:54 up 6 days, 49 min, 2 users, load average: 25.36, 60.78, 34.32
ubuntu#ip-172-31-18-127:~$ uptime
12:43:27 up 6 days, 49 min, 2 users, load average: 15.37, 54.97, 33.22
ubuntu#ip-172-31-18-127:~$ uptime
12:51:15 up 6 days, 57 min, 2 users, load average: 0.00, 11.38, 20.01
has anyone had the same kind of issue with wordpress on EC2? Is it triggered by some plugin or is my server configuration wrong?
Thank you in advance.
EDIT
I looked into the apache access and error logs,
access logs
191.96.249.54 - - [05/Oct/2016:07:56:10 +0000] "POST /xmlrpc.php HTTP/1.0" 200 560 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.54 - - [05/Oct/2016:07:56:06 +0000] "POST /xmlrpc.php HTTP/1.0" 200 560 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.54 - - [05/Oct/2016:07:56:09 +0000] "POST /xmlrpc.php HTTP/1.0" 200 560 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.53 - - [05/Oct/2016:07:56:05 +0000] "POST /xmlrpc.php HTTP/1.0" 200 560 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.54 - - [05/Oct/2016:07:56:02 +0000] "POST /xmlrpc.php HTTP/1.0" 200 560 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.54 - - [05/Oct/2016:07:56:03 +0000] "POST /xmlrpc.php HTTP/1.0" 200 560 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.53 - - [05/Oct/2016:07:56:07 +0000] "POST /xmlrpc.php HTTP/1.0" 200 560 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
52.58.212.131 - - [05/Oct/2016:07:56:35 +0000] "POST /wp-cron.php?doing_wp_cron=1475.3754720687866210937500 HTTP/1.1" 200 166 "http://52.58.212.131/wp-cron.php?doing_wp_cron=1475.3754720687866210937500" "WordPress/4.6.1; http://52.58.212.131"
191.96.249.54 - - [05/Oct/2016:07:56:06 +0000] "POST /xmlrpc.php HTTP/1.0" 200 560 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.54 - - [05/Oct/2016:07:56:03 +0000] "POST /xmlrpc.php HTTP/1.0" 200 560 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.54 - - [05/Oct/2016:07:56:15 +0000] "POST /xmlrpc.php HTTP/1.0" 200 560 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
191.96.249.53 - - [05/Oct/2016:07:56:11 +0000] "POST /xmlrpc.php HTTP/1.0" 200 560 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
error logs
[Wed Oct 05 06:25:27.206451 2016] [mpm_prefork:notice] [pid 27426] AH00163: Apache/2.4.18 (Ubuntu) configured -- resuming normal operations
[Wed Oct 05 06:25:27.206490 2016] [core:notice] [pid 27426] AH00094: Command line: '/usr/sbin/apache2'
*** Error in `/usr/sbin/apache2': double free or corruption (!prev): 0x00005629347d6910 ***
[Wed Oct 05 07:57:13.804332 2016] [core:notice] [pid 27426] AH00051: child pid 11014 exit signal Segmentation fault (11), possible coredump in /etc/apache2
[Wed Oct 05 07:57:14.806735 2016] [mpm_prefork:notice] [pid 27426] AH00169: caught SIGTERM, shutting down
I suppose this is where the server is overloading, and apache stops responding.

Related

HTTP 301 in Apache logs but 404 in browser/curl

I am dealing with a client who has been blacklisted by Google Adwords because the Googlebot crawlers that are crawling their site is finding tons of weird links that redirect via a 301 code. Some examples:
216.244.66.238 - - [15/Mar/2022:00:22:33 +0000] "GET /ffdd1g/hytera-phone.html HTTP/1.1" 301 - "-" "Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help#moz.com)"
216.244.66.238 - - [15/Mar/2022:00:22:34 +0000] "GET /ffdd1g/od-tools-houdini.html HTTP/1.1" 301 - "-" "Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help#moz.com)"
216.244.66.238 - - [15/Mar/2022:00:22:42 +0000] "GET /7oh5yny/fujifilm-classic-chrome.html HTTP/1.1" 301 - "-" "Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help#moz.com)"
216.244.66.238 - - [15/Mar/2022:00:22:45 +0000] "GET /7oh5yny/fusion-360-join-line-segments.html HTTP/1.1" 301 - "-" "Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help#moz.com)"
But when I recreate any of the requests in my browser or with curl, it 404s correctly. The fact that Google is seeing 301s is what caused them to be blacklisted by Google AdWords. Why could this be happening and how can I make sure that all invalid links always return 404 instead of 301.
This is a WordPress website by the way in-case it makes a difference. Thank you.

How to solve error 503 in Kubernetes NGINX Ingress

I'm trying to configure Kubernetes Dashboard using NGINX INGRESS but for some reason I'm getting a 503 error.
I'm running Kubernetes locally in my macbook with docker desktop.
First thing I did was apply/install NGINX INGRESS CONTROLLER
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.47.0/deploy/static/provider/cloud/deploy.yaml
Second step was to apply/install kubernetes dashboard YML File
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.2.0/aio/deploy/recommended.yaml
Third Step was to apply the ingress service
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: dashboard-ingress
namespace: kubernetes-dashboard
annotations:
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/configuration-snippet: |-
proxy_ssl_server_name on;
proxy_ssl_name $host;
spec:
rules:
- http:
paths:
- pathType: Prefix
path: /
backend:
service:
name: kubernetes-dashboard
port:
number: 433
When I try to access http://localhost and/or https://localhost I get a 503 Service Temporarily Unavailable Error from nginx
Not sure what I'm doing wrong.
Here is part of the log from the NGINX POD
I0630 23:36:42.049398 10 main.go:112] "successfully validated configuration, accepting" ingress="dashboard-ingress/kubernetes-dashboard"
I0630 23:36:42.055306 10 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"kubernetes-dashboard", Name:"dashboard-ingress", UID:"85e7bd9e-308d-4848-8b70-4a3591415464", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"47868", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0630 23:36:42.056435 10 controller.go:146] "Configuration changes detected, backend reload required"
I0630 23:36:42.124850 10 controller.go:163] "Backend successfully reloaded"
I0630 23:36:42.125333 10 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-5b74bc9868-gplcq", UID:"bbd70716-b843-403b-a8f9-2add0f63f63f", APIVersion:"v1", ResourceVersion:"46315", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
192.168.65.3 - - [30/Jun/2021:23:36:44 +0000] "GET / HTTP/1.1" 400 54 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 657 0.003 [kubernetes-dashboard-kubernetes-dashboard-80] [] 10.1.0.25:8443 48 0.002 400 395aec46af3b21e79cd650f2f86722f3
2021/06/30 23:36:44 [error] 1222#1222: *17477 recv() failed (104: Connection reset by peer) while sending to client, client: 192.168.65.3, server: _, request: "GET / HTTP/1.1", upstream: "http://10.1.0.25:8443/", host: "localhost"
2021/06/30 23:36:45 [error] 1222#1222: *17512 recv() failed (104: Connection reset by peer) while sending to client, client: 192.168.65.3, server: _, request: "GET / HTTP/1.1", upstream: "http://10.1.0.25:8443/", host: "localhost"
192.168.65.3 - - [30/Jun/2021:23:36:45 +0000] "GET / HTTP/1.1" 400 54 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 657 0.002 [kubernetes-dashboard-kubernetes-dashboard-80] [] 10.1.0.25:8443 48 0.001 400 a15e1e48987948cb93503b494d188654
2021/07/01 00:09:31 [error] 1224#1224: *49299 recv() failed (104: Connection reset by peer) while reading upstream, client: 192.168.65.3, server: _, request: "GET / HTTP/1.1", upstream: "http://10.1.0.25:8443/", host: "localhost"
192.168.65.3 - - [01/Jul/2021:00:09:31 +0000] "GET / HTTP/1.1" 400 54 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 657 0.002 [kubernetes-dashboard-kubernetes-dashboard-80] [] 10.1.0.25:8443 48 0.001 400 ac6b88ca52b73358c39371cb4422761d
2021/07/01 00:09:32 [error] 1221#1221: *49336 recv() failed (104: Connection reset by peer) while sending to client, client: 192.168.65.3, server: _, request: "GET / HTTP/1.1", upstream: "http://10.1.0.25:8443/", host: "localhost"
192.168.65.3 - - [01/Jul/2021:00:09:32 +0000] "GET / HTTP/1.1" 400 54 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 657 0.001 [kubernetes-dashboard-kubernetes-dashboard-80] [] 10.1.0.25:8443 48 0.001 400 2c5cd2d9403a8e50a77fdc897c694792
2021/07/01 00:09:33 [error] 1221#1221: *49338 recv() failed (104: Connection reset by peer) while sending to client, client: 192.168.65.3, server: _, request: "GET / HTTP/1.1", upstream: "http://10.1.0.25:8443/", host: "localhost"
192.168.65.3 - - [01/Jul/2021:00:09:33 +0000] "GET / HTTP/1.1" 400 54 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 657 0.001 [kubernetes-dashboard-kubernetes-dashboard-80] [] 10.1.0.25:8443 48 0.000 400 f1f630c886d20b9b9c59bd9e0e0e3860
2021/07/01 00:09:33 [error] 1224#1224: *49344 recv() failed (104: Connection reset by peer) while reading upstream, client: 192.168.65.3, server: _, request: "GET / HTTP/1.1", upstream: "http://10.1.0.25:8443/", host: "localhost"
192.168.65.3 - - [01/Jul/2021:00:09:33 +0000] "GET / HTTP/1.1" 400 54 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 657 0.001 [kubernetes-dashboard-kubernetes-dashboard-80] [] 10.1.0.25:8443 48 0.001 400 2ab6774dec6e2a89599c4745d24b9661
192.168.65.3 - - [01/Jul/2021:00:09:33 +0000] "GET / HTTP/1.1" 400 54 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 657 0.001 [kubernetes-dashboard-kubernetes-dashboard-80] [] 10.1.0.25:8443 48 0.000 400 c9147e08203d9ec8e7b0d0debab8d556
2021/07/01 00:09:33 [error] 1222#1222: *49360 recv() failed (104: Connection reset by peer) while sending to client, client: 192.168.65.3, server: _, request: "GET / HTTP/1.1", upstream: "http://10.1.0.25:8443/", host: "localhost"
I0701 00:10:19.024220 10 main.go:112] "successfully validated configuration, accepting" ingress="dashboard-ingress/kubernetes-dashboard"
I0701 00:10:19.026772 10 controller.go:146] "Configuration changes detected, backend reload required"
I0701 00:10:19.027392 10 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"kubernetes-dashboard", Name:"dashboard-ingress", UID:"85e7bd9e-308d-4848-8b70-4a3591415464", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"50637", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0701 00:10:19.102759 10 controller.go:163] "Backend successfully reloaded"
I0701 00:10:19.103246 10 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-5b74bc9868-gplcq", UID:"bbd70716-b843-403b-a8f9-2add0f63f63f", APIVersion:"v1", ResourceVersion:"46315", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
192.168.65.3 - - [01/Jul/2021:00:11:27 +0000] "GET / HTTP/1.1" 503 592 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 657 0.000 [kubernetes-dashboard-kubernetes-dashboard-433] [] - - - - c449f6e8082761ddc3432f956f4701f2
192.168.65.3 - - [01/Jul/2021:00:11:29 +0000] "GET / HTTP/1.1" 503 592 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 657 0.000 [kubernetes-dashboard-kubernetes-dashboard-433] [] - - - - 3a41974b01c5e63e734fce6e37b98e4c
192.168.65.3 - - [01/Jul/2021:00:11:56 +0000] "GET / HTTP/2.0" 503 592 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 408 0.000 [kubernetes-dashboard-kubernetes-dashboard-433] [] - - - - c01f7bec83d3be6b26703b8808f9922a
192.168.65.3 - - [01/Jul/2021:00:11:58 +0000] "GET / HTTP/2.0" 503 592 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 24 0.000 [kubernetes-dashboard-kubernetes-dashboard-433] [] - - - - dc39bcddd4ecfdefe931bf16fe3c1557
192.168.65.3 - - [01/Jul/2021:00:16:36 +0000] "GET / HTTP/1.1" 503 190 "-" "curl/7.64.1" 73 0.000 [kubernetes-dashboard-kubernetes-dashboard-433] [] - - - - 82aad4321afbccb3fc54ac75d96b66ee
192.168.65.3 - - [01/Jul/2021:00:31:47 +0000] "GET / HTTP/2.0" 503 592 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 417 0.000 [kubernetes-dashboard-kubernetes-dashboard-433] [] - - - - c4ab3d2f272be4d38df62c0ffd50bfe9
I0701 00:48:02.059067 10 main.go:112] "successfully validated configuration, accepting" ingress="dashboard-ingress/kubernetes-dashboard"
I0701 00:48:02.062292 10 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"kubernetes-dashboard", Name:"dashboard-ingress", UID:"85e7bd9e-308d-4848-8b70-4a3591415464", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"53737", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0701 00:48:02.062876 10 controller.go:146] "Configuration changes detected, backend reload required"
I0701 00:48:02.131494 10 controller.go:163] "Backend successfully reloaded"
I0701 00:48:02.131787 10 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-5b74bc9868-gplcq", UID:"bbd70716-b843-403b-a8f9-2add0f63f63f", APIVersion:"v1", ResourceVersion:"46315", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
192.168.65.3 - - [01/Jul/2021:00:48:12 +0000] "GET / HTTP/2.0" 503 592 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 417 0.000 [kubernetes-dashboard-kubernetes-dashboard-433] [] - - - - d50e3bb0db3a5fa7581c405b8c50d5c8
192.168.65.3 - - [01/Jul/2021:00:48:14 +0000] "GET / HTTP/2.0" 503 592 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 15 0.000 [kubernetes-dashboard-kubernetes-dashboard-433] [] - - - - c8d8752fb4d79d5bc084839ef9a767b2
I0701 00:49:50.908720 10 main.go:112] "successfully validated configuration, accepting" ingress="dashboard-ingress/kubernetes-dashboard"
I0701 00:49:50.911044 10 controller.go:146] "Configuration changes detected, backend reload required"
I0701 00:49:50.911350 10 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"kubernetes-dashboard", Name:"dashboard-ingress", UID:"85e7bd9e-308d-4848-8b70-4a3591415464", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"53896", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0701 00:49:50.979935 10 controller.go:163] "Backend successfully reloaded"
I0701 00:49:50.980213 10 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-5b74bc9868-gplcq", UID:"bbd70716-b843-403b-a8f9-2add0f63f63f", APIVersion:"v1", ResourceVersion:"46315", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
192.168.65.3 - - [01/Jul/2021:00:50:55 +0000] "GET / HTTP/2.0" 503 592 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 417 0.000 [kubernetes-dashboard-kubernetes-dashboard-433] [] - - - - d62a8012bc23bbc35a47621d54d68a62
192.168.65.3 - - [01/Jul/2021:00:51:00 +0000] "GET / HTTP/2.0" 503 592 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 15 0.000 [kubernetes-dashboard-kubernetes-dashboard-433] [] - - - - 0cbfd2274ad687fc1aaff76dbc483659
Here is the log for the Kubernete Dashboard Pod
kubectl logs kubernetes-dashboard-78c79f97b4-w5pw9 -n kubernetes-dashboard  ✔  docker-desktop ⎈
2021/06/30 23:01:40 Starting overwatch
2021/06/30 23:01:40 Using namespace: kubernetes-dashboard
2021/06/30 23:01:40 Using in-cluster config to connect to apiserver
2021/06/30 23:01:40 Using secret token for csrf signing
2021/06/30 23:01:40 Initializing csrf token from kubernetes-dashboard-csrf secret
2021/06/30 23:01:40 Empty token. Generating and storing in a secret kubernetes-dashboard-csrf
2021/06/30 23:01:40 Successful initial request to the apiserver, version: v1.21.1
2021/06/30 23:01:40 Generating JWE encryption key
2021/06/30 23:01:40 New synchronizer has been registered: kubernetes-dashboard-key-holder-kubernetes-dashboard. Starting
2021/06/30 23:01:40 Starting secret synchronizer for kubernetes-dashboard-key-holder in namespace kubernetes-dashboard
2021/06/30 23:01:41 Initializing JWE encryption key from synchronized object
2021/06/30 23:01:41 Creating in-cluster Sidecar client
2021/06/30 23:01:41 Auto-generating certificates
2021/06/30 23:01:41 Successful request to sidecar
2021/06/30 23:01:41 Successfully created certificates
2021/06/30 23:01:41 Serving securely on HTTPS port: 8443
Here are the endpoints for the kubernetes-dashboard namespace
kubectl get ep -n kubernetes-dashboard
NAME ENDPOINTS AGE
dashboard-metrics-scraper 10.1.0.24:8000 11h
kubernetes-dashboard 10.1.0.25:8443 11h
Any help would be greatly appreciated.

I was able to fix this issue.
In my ingress ymal file I had a typo. Port number was set to 433 instead of 443
As soon as I made and applied that change, I was able to access the dashboard login page with: https://localhost and http://localhost

Strange requests to web server

I have a VPS running Nginx, which currently serves only static content.
Once I was looking at the log and noticed some strange requests:
216.244.66.239 - - [03/Jan/2019:15:04:26 +0100] "GET /en/profile/Souxy HTTP/1.1" 200 4650 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help#moz.com)"
216.244.66.239 - - [03/Jan/2019:15:04:28 +0100] "GET /en/view/8gIi2vad8Y HTTP/1.1" 200 4650 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help#moz.com)"

this is crawler. On this link is descriptin https://moz.com/help/moz-procedures/crawlers/dotbot. Maybe it is indexing your website.
You can block this requests on firewall or add file robots.txt with content
User-agent: dotbot
Disallow: /

NGINX logs awk find bandwidth by IP address

I am trying to find the bandwidth used by the most prevalent ip addresses making requests within nginx access logs. This is what I have started out with:
$ cat /path/to/access.log |awk '{print $1}' |sort |uniq -c |sort -n |tail
($1 is the ip address, while the bytes of request is $10) - which will output:
# of requests | IP Address
1220 xxx.xxx.xxx.xxx
1347 xxx.xxx.xxx.xxx
1420 xxx.xxx.xxx.xxx
2104 xxx.xxx.xxx.xxx
etc...
What I am trying to accomplish is to identify how much bandwidth each one of these addresses is requesting. For example:
# of requests | IP Address | total bytes requested (unique to ip)
1220 xxx.xxx.xxx.xxx 45626026
1347 xxx.xxx.xxx.xxx 49565157
1420 xxx.xxx.xxx.xxx 56689122
2104 xxx.xxx.xxx.xxx 76665299
etc...
My restrictions are not too limited. So, with that said, if the possible solution would be to use more than one command to resolve upon the final query (i.e. find total bandwidth by ip), so be it. Thanks for any help provided!

With single GNU awk solution:
Sample access.log for demonstration purpose:
127.0.0.1 - - [15/Aug/2017:09:38:35 +0300] "GET / HTTP/1.1" 200 111 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"
127.0.0.1 - - [15/Aug/2017:09:38:46 +0300] "GET / HTTP/1.1" 200 171 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"
127.0.0.1 - - [15/Aug/2017:09:59:38 +0300] "GET /favicon.ico HTTP/1.1" 404 152 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"
127.0.0.1 - - [15/Aug/2017:09:59:39 +0300] "GET /favicon.ico HTTP/1.1" 404 1502 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"
127.0.0.1 - - [15/Aug/2017:11:04:45 +0300] "GET / HTTP/1.1" 200 23976 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0"
127.0.0.2 - - [15/Aug/2017:09:38:35 +0300] "GET / HTTP/1.1" 200 14111 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gec$
127.0.0.2 - - [15/Aug/2017:09:38:46 +0300] "GET / HTTP/1.1" 200 1414 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gec$
127.0.0.2 - - [15/Aug/2017:09:59:38 +0300] "GET /favicon.ico HTTP/1.1" 404 1522 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; r$
127.0.0.2 - - [15/Aug/2017:09:59:39 +0300] "GET /favicon.ico HTTP/1.1" 404 1332 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; r$
127.0.0.3 - - [15/Aug/2017:11:04:45 +0300] "GET / HTTP/1.1" 200 23976 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) G$
127.0.0.1 - - [15/Aug/2017:09:38:35 +0300] "GET / HTTP/1.1" 200 141 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gec$
127.0.0.1 - - [15/Aug/2017:09:38:46 +0300] "GET / HTTP/1.1" 200 1041 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gec$
127.0.0.3 - - [15/Aug/2017:09:59:38 +0300] "GET /favicon.ico HTTP/1.1" 404 1529 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; r$
127.0.0.1 - - [15/Aug/2017:09:59:39 +0300] "GET /favicon.ico HTTP/1.1" 404 1026 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; r$
127.0.0.1 - - [15/Aug/2017:11:04:45 +0300] "GET / HTTP/1.1" 200 23976 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) G$
127.0.0.3 - - [15/Aug/2017:09:38:35 +0300] "GET / HTTP/1.1" 200 1414 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gec$
127.0.0.1 - - [15/Aug/2017:09:38:46 +0300] "GET / HTTP/1.1" 200 13341 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gec$
127.0.0.3 - - [15/Aug/2017:09:59:38 +0300] "GET /favicon.ico HTTP/1.1" 404 172 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; r$
127.0.0.3 - - [15/Aug/2017:09:59:39 +0300] "GET /favicon.ico HTTP/1.1" 404 1502 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; r$
127.0.0.3 - - [15/Aug/2017:11:04:45 +0300] "GET / HTTP/1.1" 200 23976 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) G$
The job:
awk 'BEGIN{ PROCINFO["sorted_in"]="#val_num_desc" }
{ a[$1]++; b[$1]+=$10 }
END{
for(i in a) { if(++c>10) break; print i,b[i] }
}' /path/to/access.log
PROCINFO["sorted_in"]="#val_num_desc" - comparison of array values, to sort by IP address frequency in descending order
if(++c>10) - ensures iterating over only first 10 items, which is emulation of tail command (gets the last 10 lines) The loop starts from the most frequent IP address
The output:
127.0.0.1 65437
127.0.0.3 52569
127.0.0.2 18379

how to get real ip in nginx access.log

I have a very problem. My website is spamming through joomla contact form. In nginx access.log I see only:
10.50.0.1 - - [06/Sep/2017:19:57:32 +0200] "GET /index.php/en/kontakt HTTP/1.1" 200 16132 "http://polskaszkolaslough.org/index.php/en/kontakt" "Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1)"
10.50.0.1 - - [06/Sep/2017:19:57:32 +0200] "POST /index.php/en/kontakt HTTP/1.1" 301 193 "http://polskaszkolaslough.org/index.php/en/kontakt" "Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1)"
10.50.0.1 - - [06/Sep/2017:19:57:34 +0200] "POST /index.php/en/kontakt HTTP/1.1" 301 193 "http://polskaszkolaslough.org/index.php/en/kontakt" "Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1)"
10.50.0.1 - - [06/Sep/2017:19:57:34 +0200] "GET /index.php/en/kontakt HTTP/1.1" 301 193 "http://polskaszkolaslough.org/index.php/en/kontakt" "Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1)"
10.50.0.1 - - [06/Sep/2017:19:57:34 +0200] "GET /index.php/en/kontakt HTTP/1.1" 301 193 "http://polskaszkolaslough.org/index.php/en/kontakt" "Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1)"
10.50.0.1 - - [06/Sep/2017:19:57:36 +0200] "GET /index.php/en/kontakt HTTP/1.1" 200 16132 "http://polskaszkolaslough.org/index.php/en/kontakt" "Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1)"
10.50.0.1 - - [06/Sep/2017:19:57:37 +0200] "GET /index.php/en/kontakt HTTP/1.1" 301 193 "http://polskaszkolaslough.org/index.php/en/kontakt" "Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1)"
10.50.0.1 - - [06/Sep/2017:19:57:37 +0200] "GET /index.php/en/kontakt HTTP/1.1" 200 16132 "http://polskaszkolaslough.org/index.php/en/kontakt" "Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1)"
10.50.0.1 - - [06/Sep/2017:19:57:37 +0200] "GET /index.php/en/kontakt HTTP/1.1" 301 193 "http://polskaszkolaslough.org/index.php/en/kontakt" "Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1)"
When I open Linux command tail, new request comes one after another. This is shock! My website is very slow. I have private server with public IP. My local IP is: 10.50.0.6 and a gateway is: 10.50.0.1 DNS is at my domain provider and record A forwards a traffic in to my public IP and then a router forward to my local IP. I would like to block ip range which spamming my domain but I don't see there original address. I see only my gateway IP. I installed fail2ban and I added reCaptha to contact form but it not helped. How can I resolve this problem?

You need access to the router.
The router should be capable of logging the address translations that it makes, and by comparing these logs with your nginx logs you should be able to identify the originating IP address. The router should also be capable of implementing an access list so that you can block the originating IP address.

Resources