Scrapy pauses periodically - web-scraping

I'm trying to do a one-time scraping for a site with 30,000+ pages with Scrapy. However my spider pauses periodically (e.g. at page 148, 285, 425, 558) and resume after several minutes (as seen from the logs below). I'm using scrapy_proxy_pool and scrapy-user-agents for IP and User-Agent rotations. I tried setting the DOWNLOAD_DELAY and AUTOTHROTTLE but the issue still exist. Any help is very much appreciated!
2020-02-25 22:11:08 [scrapy.extensions.logstats] INFO: Crawled 558 pages (at 0 pages/min), scraped 2768 items (at 12 items/min)
2020-02-25 22:12:08 [scrapy.extensions.logstats] INFO: Crawled 558 pages (at 0 pages/min), scraped 2768 items (at 0 items/min)
2020-02-25 22:13:08 [scrapy.extensions.logstats] INFO: Crawled 558 pages (at 0 pages/min), scraped 2768 items (at 0 items/min)
2020-02-25 22:14:08 [scrapy.extensions.logstats] INFO: Crawled 558 pages (at 0 pages/min), scraped 2768 items (at 0 items/min)
2020-02-25 22:15:08 [scrapy.extensions.logstats] INFO: Crawled 558 pages (at 0 pages/min), scraped 2768 items (at 0 items/min)
2020-02-25 22:16:08 [scrapy.extensions.logstats] INFO: Crawled 558 pages (at 0 pages/min), scraped 2768 items (at 0 items/min)
2020-02-25 22:17:08 [scrapy.extensions.logstats] INFO: Crawled 558 pages (at 0 pages/min), scraped 2768 items (at 0 items/min)
2020-02-25 22:18:08 [scrapy.extensions.logstats] INFO: Crawled 558 pages (at 0 pages/min), scraped 2768 items (at 0 items/min)
2020-02-25 22:18:33 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36
2020-02-25 22:18:33 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): free-proxy-list.net:443
2020-02-25 22:18:33 [urllib3.connectionpool] DEBUG: https://free-proxy-list.net:443 "GET /anonymous-proxy.html HTTP/1.1" 200 None
2020-02-25 22:18:34 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): www.us-proxy.org:443
2020-02-25 22:18:34 [urllib3.connectionpool] DEBUG: https://www.us-proxy.org:443 "GET / HTTP/1.1" 200 None
2020-02-25 22:18:34 [urllib3.connectionpool] DEBUG: Starting new HTTP connection (1): www.proxy-daily.com:80
2020-02-25 22:18:34 [urllib3.connectionpool] DEBUG: http://www.proxy-daily.com:80 "GET / HTTP/1.1" 200 185
2020-02-25 22:18:34 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): free-proxy-list.net:443
2020-02-25 22:18:34 [urllib3.connectionpool] DEBUG: https://free-proxy-list.net:443 "GET /uk-proxy.html HTTP/1.1" 200 None
2020-02-25 22:18:34 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): www.sslproxies.org:443
2020-02-25 22:18:35 [urllib3.connectionpool] DEBUG: https://www.sslproxies.org:443 "GET / HTTP/1.1" 200 None
2020-02-25 22:18:35 [urllib3.connectionpool] DEBUG: Starting new HTTP connection (1): www.free-proxy-list.net:80
2020-02-25 22:18:36 [urllib3.connectionpool] DEBUG: http://www.free-proxy-list.net:80 "GET / HTTP/1.1" 301 None
2020-02-25 22:18:36 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): www.free-proxy-list.net:443
2020-02-25 22:18:36 [urllib3.connectionpool] DEBUG: https://www.free-proxy-list.net:443 "GET / HTTP/1.1" 200 None
2020-02-25 22:18:36 [scrapy_proxy_pool.middlewares] WARNING: No proxies available.
2020-02-25 22:18:36 [scrapy_proxy_pool.middlewares] INFO: Try to download with host ip.
2020-02-25 22:18:36 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36
2020-02-25 22:18:36 [urllib3.connectionpool] DEBUG: Starting new HTTP connection (1): www.proxy-daily.com:80
2020-02-25 22:18:36 [urllib3.connectionpool] DEBUG: http://www.proxy-daily.com:80 "GET / HTTP/1.1" 200 185
2020-02-25 22:18:36 [scrapy_proxy_pool.middlewares] WARNING: No proxies available.
2020-02-25 22:18:36 [scrapy_proxy_pool.middlewares] INFO: Try to download with host ip.
2020-02-25 22:18:36 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36
2020-02-25 22:18:36 [urllib3.connectionpool] DEBUG: Starting new HTTP connection (1): www.proxy-daily.com:80
2020-02-25 22:18:36 [urllib3.connectionpool] DEBUG: http://www.proxy-daily.com:80 "GET / HTTP/1.1" 200 185
2020-02-25 22:18:36 [scrapy_proxy_pool.middlewares] WARNING: No proxies available.
2020-02-25 22:18:36 [scrapy_proxy_pool.middlewares] INFO: Try to download with host ip.
2020-02-25 22:18:36 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36
2020-02-25 22:18:36 [urllib3.connectionpool] DEBUG: Starting new HTTP connection (1): www.proxy-daily.com:80
2020-02-25 22:18:36 [scrapy_proxy_pool.middlewares] WARNING: No proxies available.
2020-02-25 22:18:36 [scrapy_proxy_pool.middlewares] INFO: Try to download with host ip.
2020-02-25 22:18:36 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
2020-02-25 22:18:36 [urllib3.connectionpool] DEBUG: Starting new HTTP connection (1): www.proxy-daily.com:80
2020-02-25 22:18:36 [urllib3.connectionpool] DEBUG: http://www.proxy-daily.com:80 "GET / HTTP/1.1" 200 185
2020-02-25 22:18:36 [scrapy_proxy_pool.middlewares] WARNING: No proxies available.
2020-02-25 22:18:36 [scrapy_proxy_pool.middlewares] INFO: Try to download with host ip.
2020-02-25 22:18:36 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36
2020-02-25 22:18:36 [urllib3.connectionpool] DEBUG: Starting new HTTP connection (1): www.proxy-daily.com:80
2020-02-25 22:18:36 [urllib3.connectionpool] DEBUG: http://www.proxy-daily.com:80 "GET / HTTP/1.1" 200 185
2020-02-25 22:18:36 [scrapy_proxy_pool.middlewares] WARNING: No proxies available.
2020-02-25 22:18:36 [scrapy_proxy_pool.middlewares] INFO: Try to download with host ip.
2020-02-25 22:18:36 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31
2020-02-25 22:18:36 [urllib3.connectionpool] DEBUG: Starting new HTTP connection (1): www.proxy-daily.com:80
2020-02-25 22:18:36 [urllib3.connectionpool] DEBUG: http://www.proxy-daily.com:80 "GET / HTTP/1.1" 200 185
2020-02-25 22:18:36 [scrapy_proxy_pool.middlewares] WARNING: No proxies available.
2020-02-25 22:18:36 [scrapy_proxy_pool.middlewares] INFO: Try to download with host ip.
2020-02-25 22:18:36 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36
2020-02-25 22:18:36 [urllib3.connectionpool] DEBUG: Starting new HTTP connection (1): www.proxy-daily.com:80
2020-02-25 22:18:36 [urllib3.connectionpool] DEBUG: http://www.proxy-daily.com:80 "GET / HTTP/1.1" 200 185
2020-02-25 22:18:36 [scrapy_proxy_pool.middlewares] WARNING: No proxies available.
2020-02-25 22:18:36 [scrapy_proxy_pool.middlewares] INFO: Try to download with host ip.
2020-02-25 22:18:36 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36
2020-02-25 22:18:36 [urllib3.connectionpool] DEBUG: Starting new HTTP connection (1): www.proxy-daily.com:80
2020-02-25 22:18:36 [scrapy_proxy_pool.middlewares] WARNING: No proxies available.
2020-02-25 22:18:36 [scrapy_proxy_pool.middlewares] INFO: Try to download with host ip.
2020-02-25 22:18:36 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 5.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.89 Safari/537.36
2020-02-25 22:18:36 [urllib3.connectionpool] DEBUG: Starting new HTTP connection (1): www.proxy-daily.com:80
2020-02-25 22:18:36 [scrapy_proxy_pool.middlewares] WARNING: No proxies available.
2020-02-25 22:18:36 [scrapy_proxy_pool.middlewares] INFO: Try to download with host ip.
2020-02-25 22:18:36 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36
2020-02-25 22:18:36 [urllib3.connectionpool] DEBUG: Starting new HTTP connection (1): www.proxy-daily.com:80
2020-02-25 22:18:36 [scrapy_proxy_pool.middlewares] WARNING: No proxies available.
2020-02-25 22:18:36 [scrapy_proxy_pool.middlewares] INFO: Try to download with host ip.
2020-02-25 22:18:36 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36
2020-02-25 22:18:36 [urllib3.connectionpool] DEBUG: Starting new HTTP connection (1): www.proxy-daily.com:80
2020-02-25 22:18:36 [urllib3.connectionpool] DEBUG: http://www.proxy-daily.com:80 "GET / HTTP/1.1" 200 185
2020-02-25 22:18:36 [scrapy_proxy_pool.middlewares] WARNING: No proxies available.
2020-02-25 22:18:36 [scrapy_proxy_pool.middlewares] INFO: Try to download with host ip.
2020-02-25 22:18:36 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36
2020-02-25 22:18:36 [urllib3.connectionpool] DEBUG: Starting new HTTP connection (1): www.proxy-daily.com:80
2020-02-25 22:18:36 [urllib3.connectionpool] DEBUG: http://www.proxy-daily.com:80 "GET / HTTP/1.1" 200 185
2020-02-25 22:18:36 [scrapy_proxy_pool.middlewares] WARNING: No proxies available.
2020-02-25 22:18:36 [scrapy_proxy_pool.middlewares] INFO: Try to download with host ip.
2020-02-25 22:18:36 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36
2020-02-25 22:18:36 [urllib3.connectionpool] DEBUG: Starting new HTTP connection (1): www.proxy-daily.com:80
2020-02-25 22:18:36 [urllib3.connectionpool] DEBUG: http://www.proxy-daily.com:80 "GET / HTTP/1.1" 200 185
2020-02-25 22:18:36 [scrapy_proxy_pool.middlewares] WARNING: No proxies available.
2020-02-25 22:18:36 [scrapy_proxy_pool.middlewares] INFO: Try to download with host ip.
2020-02-25 22:18:36 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36
2020-02-25 22:18:36 [urllib3.connectionpool] DEBUG: Starting new HTTP connection (1): www.proxy-daily.com:80
2020-02-25 22:18:37 [urllib3.connectionpool] DEBUG: http://www.proxy-daily.com:80 "GET / HTTP/1.1" 200 185
2020-02-25 22:18:37 [scrapy_proxy_pool.middlewares] WARNING: No proxies available.
2020-02-25 22:18:37 [scrapy_proxy_pool.middlewares] INFO: Try to download with host ip.
2020-02-25 22:18:37 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
2020-02-25 22:18:37 [urllib3.connectionpool] DEBUG: Starting new HTTP connection (1): www.proxy-daily.com:80
2020-02-25 22:18:37 [scrapy_proxy_pool.middlewares] WARNING: No proxies available.
2020-02-25 22:18:37 [scrapy_proxy_pool.middlewares] INFO: Try to download with host ip.
2020-02-25 22:18:46 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.myanmartradeportal.gov.mm/commodity-search/view/30989> (referer: https://www.myanmartradeportal.gov.mm/commodity-search/view/1)
2020-02-25 22:18:46 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.89 Safari/537.36
2020-02-25 22:18:46 [urllib3.connectionpool] DEBUG: Starting new HTTP connection (1): www.proxy-daily.com:80
2020-02-25 22:18:46 [urllib3.connectionpool] DEBUG: http://www.proxy-daily.com:80 "GET / HTTP/1.1" 200 185
2020-02-25 22:18:46 [scrapy_proxy_pool.middlewares] WARNING: No proxies available.
2020-02-25 22:18:46 [scrapy_proxy_pool.middlewares] INFO: Try to download with host ip.
2020-02-25 22:18:47 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.myanmartradeportal.gov.mm/commodity-search/view/30988> (referer: https://www.myanmartradeportal.gov.mm/commodity-search/view/1)
2020-02-25 22:18:47 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36
2020-02-25 22:18:47 [urllib3.connectionpool] DEBUG: Starting new HTTP connection (1): www.proxy-daily.com:80
2020-02-25 22:18:47 [urllib3.connectionpool] DEBUG: http://www.proxy-daily.com:80 "GET / HTTP/1.1" 200 185
2020-02-25 22:18:47 [scrapy_proxy_pool.middlewares] WARNING: No proxies available.
2020-02-25 22:18:47 [scrapy_proxy_pool.middlewares] INFO: Try to download with host ip.
2020-02-25 22:18:49 [scrapy.core.engine] DEBUG: Crawled (200) <GET SCRAPING_URL> (referer: REFERER_URL)
Occisonally this message appears while pausing:
2020-02-25 22:53:06 [scrapy_proxy_pool.middlewares] INFO: Blacklist is cleared.

Related

AMP clear cache returning - Public key not found due to ingestion error: 499 error from origin That’s all we know

I am trying to clear the cache of amp page but I am getting this error:
Public key not found due to ingestion error: 499 error from origin
That’s all we know.
What I have checked:
.well-known/amphtml/apikey.pub is publicly available
the file is not roboted(allowed for google bots)
the file has content type plain/text
bypassed cloudflare cache
Suspiction:
There are also 301 requests from google bots. I guess it is because google bot is requesting the file in HTTP initially and redirected to https.
64.233.173.70 - - [13/Jun/2022:16:08:22 +0800] "GET /.well-known/amphtml/apikey.pub HTTP/1.1" 301 193 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Mobile Safari/537.36 (compatible; Google-AMPHTML)"
64.233.173.204 - - [13/Jun/2022:16:08:22 +0800] "GET /.well-known/amphtml/apikey.pub HTTP/1.1" 200 451 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Mobile Safari/537.36 (compatible; Google-AMPHTML)"
66.249.70.56 - - [13/Jun/2022:16:10:55 +0800] "GET /.well-known/amphtml/apikey.pub HTTP/1.1" 301 193 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.64.21 - - [13/Jun/2022:16:11:14 +0800] "GET /.well-known/amphtml/apikey.pub HTTP/1.1" 200 451 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.64.55 - - [13/Jun/2022:17:07:18 +0800] "GET /.well-known/amphtml/apikey.pub HTTP/1.1" 301 193 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.64.114 - - [13/Jun/2022:17:07:19 +0800] "GET /.well-known/amphtml/apikey.pub HTTP/1.1" 200 451 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
Another suspicion:
The .well-known/amphtml/apikey.pub cannot be curled and it returns 403. But from above logs from Google bot, it seems it does not have any problem fetching the file.
I don't understand what I miss here. Please help!

How to solve error 503 in Kubernetes NGINX Ingress

I'm trying to configure Kubernetes Dashboard using NGINX INGRESS but for some reason I'm getting a 503 error.
I'm running Kubernetes locally in my macbook with docker desktop.
First thing I did was apply/install NGINX INGRESS CONTROLLER
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.47.0/deploy/static/provider/cloud/deploy.yaml
Second step was to apply/install kubernetes dashboard YML File
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.2.0/aio/deploy/recommended.yaml
Third Step was to apply the ingress service
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: dashboard-ingress
namespace: kubernetes-dashboard
annotations:
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/configuration-snippet: |-
proxy_ssl_server_name on;
proxy_ssl_name $host;
spec:
rules:
- http:
paths:
- pathType: Prefix
path: /
backend:
service:
name: kubernetes-dashboard
port:
number: 433
When I try to access http://localhost and/or https://localhost I get a 503 Service Temporarily Unavailable Error from nginx
Not sure what I'm doing wrong.
Here is part of the log from the NGINX POD
I0630 23:36:42.049398 10 main.go:112] "successfully validated configuration, accepting" ingress="dashboard-ingress/kubernetes-dashboard"
I0630 23:36:42.055306 10 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"kubernetes-dashboard", Name:"dashboard-ingress", UID:"85e7bd9e-308d-4848-8b70-4a3591415464", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"47868", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0630 23:36:42.056435 10 controller.go:146] "Configuration changes detected, backend reload required"
I0630 23:36:42.124850 10 controller.go:163] "Backend successfully reloaded"
I0630 23:36:42.125333 10 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-5b74bc9868-gplcq", UID:"bbd70716-b843-403b-a8f9-2add0f63f63f", APIVersion:"v1", ResourceVersion:"46315", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
192.168.65.3 - - [30/Jun/2021:23:36:44 +0000] "GET / HTTP/1.1" 400 54 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 657 0.003 [kubernetes-dashboard-kubernetes-dashboard-80] [] 10.1.0.25:8443 48 0.002 400 395aec46af3b21e79cd650f2f86722f3
2021/06/30 23:36:44 [error] 1222#1222: *17477 recv() failed (104: Connection reset by peer) while sending to client, client: 192.168.65.3, server: _, request: "GET / HTTP/1.1", upstream: "http://10.1.0.25:8443/", host: "localhost"
2021/06/30 23:36:45 [error] 1222#1222: *17512 recv() failed (104: Connection reset by peer) while sending to client, client: 192.168.65.3, server: _, request: "GET / HTTP/1.1", upstream: "http://10.1.0.25:8443/", host: "localhost"
192.168.65.3 - - [30/Jun/2021:23:36:45 +0000] "GET / HTTP/1.1" 400 54 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 657 0.002 [kubernetes-dashboard-kubernetes-dashboard-80] [] 10.1.0.25:8443 48 0.001 400 a15e1e48987948cb93503b494d188654
2021/07/01 00:09:31 [error] 1224#1224: *49299 recv() failed (104: Connection reset by peer) while reading upstream, client: 192.168.65.3, server: _, request: "GET / HTTP/1.1", upstream: "http://10.1.0.25:8443/", host: "localhost"
192.168.65.3 - - [01/Jul/2021:00:09:31 +0000] "GET / HTTP/1.1" 400 54 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 657 0.002 [kubernetes-dashboard-kubernetes-dashboard-80] [] 10.1.0.25:8443 48 0.001 400 ac6b88ca52b73358c39371cb4422761d
2021/07/01 00:09:32 [error] 1221#1221: *49336 recv() failed (104: Connection reset by peer) while sending to client, client: 192.168.65.3, server: _, request: "GET / HTTP/1.1", upstream: "http://10.1.0.25:8443/", host: "localhost"
192.168.65.3 - - [01/Jul/2021:00:09:32 +0000] "GET / HTTP/1.1" 400 54 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 657 0.001 [kubernetes-dashboard-kubernetes-dashboard-80] [] 10.1.0.25:8443 48 0.001 400 2c5cd2d9403a8e50a77fdc897c694792
2021/07/01 00:09:33 [error] 1221#1221: *49338 recv() failed (104: Connection reset by peer) while sending to client, client: 192.168.65.3, server: _, request: "GET / HTTP/1.1", upstream: "http://10.1.0.25:8443/", host: "localhost"
192.168.65.3 - - [01/Jul/2021:00:09:33 +0000] "GET / HTTP/1.1" 400 54 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 657 0.001 [kubernetes-dashboard-kubernetes-dashboard-80] [] 10.1.0.25:8443 48 0.000 400 f1f630c886d20b9b9c59bd9e0e0e3860
2021/07/01 00:09:33 [error] 1224#1224: *49344 recv() failed (104: Connection reset by peer) while reading upstream, client: 192.168.65.3, server: _, request: "GET / HTTP/1.1", upstream: "http://10.1.0.25:8443/", host: "localhost"
192.168.65.3 - - [01/Jul/2021:00:09:33 +0000] "GET / HTTP/1.1" 400 54 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 657 0.001 [kubernetes-dashboard-kubernetes-dashboard-80] [] 10.1.0.25:8443 48 0.001 400 2ab6774dec6e2a89599c4745d24b9661
192.168.65.3 - - [01/Jul/2021:00:09:33 +0000] "GET / HTTP/1.1" 400 54 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 657 0.001 [kubernetes-dashboard-kubernetes-dashboard-80] [] 10.1.0.25:8443 48 0.000 400 c9147e08203d9ec8e7b0d0debab8d556
2021/07/01 00:09:33 [error] 1222#1222: *49360 recv() failed (104: Connection reset by peer) while sending to client, client: 192.168.65.3, server: _, request: "GET / HTTP/1.1", upstream: "http://10.1.0.25:8443/", host: "localhost"
I0701 00:10:19.024220 10 main.go:112] "successfully validated configuration, accepting" ingress="dashboard-ingress/kubernetes-dashboard"
I0701 00:10:19.026772 10 controller.go:146] "Configuration changes detected, backend reload required"
I0701 00:10:19.027392 10 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"kubernetes-dashboard", Name:"dashboard-ingress", UID:"85e7bd9e-308d-4848-8b70-4a3591415464", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"50637", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0701 00:10:19.102759 10 controller.go:163] "Backend successfully reloaded"
I0701 00:10:19.103246 10 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-5b74bc9868-gplcq", UID:"bbd70716-b843-403b-a8f9-2add0f63f63f", APIVersion:"v1", ResourceVersion:"46315", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
192.168.65.3 - - [01/Jul/2021:00:11:27 +0000] "GET / HTTP/1.1" 503 592 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 657 0.000 [kubernetes-dashboard-kubernetes-dashboard-433] [] - - - - c449f6e8082761ddc3432f956f4701f2
192.168.65.3 - - [01/Jul/2021:00:11:29 +0000] "GET / HTTP/1.1" 503 592 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 657 0.000 [kubernetes-dashboard-kubernetes-dashboard-433] [] - - - - 3a41974b01c5e63e734fce6e37b98e4c
192.168.65.3 - - [01/Jul/2021:00:11:56 +0000] "GET / HTTP/2.0" 503 592 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 408 0.000 [kubernetes-dashboard-kubernetes-dashboard-433] [] - - - - c01f7bec83d3be6b26703b8808f9922a
192.168.65.3 - - [01/Jul/2021:00:11:58 +0000] "GET / HTTP/2.0" 503 592 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 24 0.000 [kubernetes-dashboard-kubernetes-dashboard-433] [] - - - - dc39bcddd4ecfdefe931bf16fe3c1557
192.168.65.3 - - [01/Jul/2021:00:16:36 +0000] "GET / HTTP/1.1" 503 190 "-" "curl/7.64.1" 73 0.000 [kubernetes-dashboard-kubernetes-dashboard-433] [] - - - - 82aad4321afbccb3fc54ac75d96b66ee
192.168.65.3 - - [01/Jul/2021:00:31:47 +0000] "GET / HTTP/2.0" 503 592 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 417 0.000 [kubernetes-dashboard-kubernetes-dashboard-433] [] - - - - c4ab3d2f272be4d38df62c0ffd50bfe9
I0701 00:48:02.059067 10 main.go:112] "successfully validated configuration, accepting" ingress="dashboard-ingress/kubernetes-dashboard"
I0701 00:48:02.062292 10 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"kubernetes-dashboard", Name:"dashboard-ingress", UID:"85e7bd9e-308d-4848-8b70-4a3591415464", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"53737", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0701 00:48:02.062876 10 controller.go:146] "Configuration changes detected, backend reload required"
I0701 00:48:02.131494 10 controller.go:163] "Backend successfully reloaded"
I0701 00:48:02.131787 10 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-5b74bc9868-gplcq", UID:"bbd70716-b843-403b-a8f9-2add0f63f63f", APIVersion:"v1", ResourceVersion:"46315", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
192.168.65.3 - - [01/Jul/2021:00:48:12 +0000] "GET / HTTP/2.0" 503 592 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 417 0.000 [kubernetes-dashboard-kubernetes-dashboard-433] [] - - - - d50e3bb0db3a5fa7581c405b8c50d5c8
192.168.65.3 - - [01/Jul/2021:00:48:14 +0000] "GET / HTTP/2.0" 503 592 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 15 0.000 [kubernetes-dashboard-kubernetes-dashboard-433] [] - - - - c8d8752fb4d79d5bc084839ef9a767b2
I0701 00:49:50.908720 10 main.go:112] "successfully validated configuration, accepting" ingress="dashboard-ingress/kubernetes-dashboard"
I0701 00:49:50.911044 10 controller.go:146] "Configuration changes detected, backend reload required"
I0701 00:49:50.911350 10 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"kubernetes-dashboard", Name:"dashboard-ingress", UID:"85e7bd9e-308d-4848-8b70-4a3591415464", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"53896", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0701 00:49:50.979935 10 controller.go:163] "Backend successfully reloaded"
I0701 00:49:50.980213 10 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-5b74bc9868-gplcq", UID:"bbd70716-b843-403b-a8f9-2add0f63f63f", APIVersion:"v1", ResourceVersion:"46315", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
192.168.65.3 - - [01/Jul/2021:00:50:55 +0000] "GET / HTTP/2.0" 503 592 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 417 0.000 [kubernetes-dashboard-kubernetes-dashboard-433] [] - - - - d62a8012bc23bbc35a47621d54d68a62
192.168.65.3 - - [01/Jul/2021:00:51:00 +0000] "GET / HTTP/2.0" 503 592 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 15 0.000 [kubernetes-dashboard-kubernetes-dashboard-433] [] - - - - 0cbfd2274ad687fc1aaff76dbc483659
Here is the log for the Kubernete Dashboard Pod
kubectl logs kubernetes-dashboard-78c79f97b4-w5pw9 -n kubernetes-dashboard  ✔  docker-desktop ⎈
2021/06/30 23:01:40 Starting overwatch
2021/06/30 23:01:40 Using namespace: kubernetes-dashboard
2021/06/30 23:01:40 Using in-cluster config to connect to apiserver
2021/06/30 23:01:40 Using secret token for csrf signing
2021/06/30 23:01:40 Initializing csrf token from kubernetes-dashboard-csrf secret
2021/06/30 23:01:40 Empty token. Generating and storing in a secret kubernetes-dashboard-csrf
2021/06/30 23:01:40 Successful initial request to the apiserver, version: v1.21.1
2021/06/30 23:01:40 Generating JWE encryption key
2021/06/30 23:01:40 New synchronizer has been registered: kubernetes-dashboard-key-holder-kubernetes-dashboard. Starting
2021/06/30 23:01:40 Starting secret synchronizer for kubernetes-dashboard-key-holder in namespace kubernetes-dashboard
2021/06/30 23:01:41 Initializing JWE encryption key from synchronized object
2021/06/30 23:01:41 Creating in-cluster Sidecar client
2021/06/30 23:01:41 Auto-generating certificates
2021/06/30 23:01:41 Successful request to sidecar
2021/06/30 23:01:41 Successfully created certificates
2021/06/30 23:01:41 Serving securely on HTTPS port: 8443
Here are the endpoints for the kubernetes-dashboard namespace
kubectl get ep -n kubernetes-dashboard
NAME ENDPOINTS AGE
dashboard-metrics-scraper 10.1.0.24:8000 11h
kubernetes-dashboard 10.1.0.25:8443 11h
Any help would be greatly appreciated.

I was able to fix this issue.
In my ingress ymal file I had a typo. Port number was set to 433 instead of 443
As soon as I made and applied that change, I was able to access the dashboard login page with: https://localhost and http://localhost

Firebase Functions onRequest not triggering

I have a Firebase function host set up to run whenever the user visits a Firebase Hosting URL, but nothing ever gets logged (the function is not being triggered)
index.js:
exports.host = functions.https.onRequest((req, res) => {
console.log(">>>> FUNCTION TRIGGERED!")
res.send(">>>> GOT IT")
})
I have it set up like so in my firebase.json:
{
"hosting": {
"public": "build",
"ignore": [
"firebase.json",
"**/.*",
"**/node_modules/**"
],
"rewrites": [
{
"source": "/**",
"destination": "/index.html",
"function": "host"
}
]
}
}
Here are the emulator function logs when I run it locally (the emulator is running on localhost:4001, and the web app is running on localhost:5000. When I visit localhost:5000, nothing logs, even though the function is said to be initialized.
12:53:42W
ui
Emulator UI unable to start on port 4000, starting on 4001 instead.
12:53:42I
ui
Emulator UI logging to ui-debug.log
12:53:43I
functions
Watching "**firebase-project**/functions" for Cloud Functions...
12:53:43I
functions
http function initialized (http://localhost:5001/****/us-central1/host).
12:53:43I
┌───────────────────────────────────────────────────────────────────────┐
│ ✔ All emulators ready! View status and logs at http://localhost:4001 │
└───────────────────────────────────────────────────────────────────────┘
┌───────────┬────────────────┬─────────────────────────────────┐
│ Emulator │ Host:Port │ View in Emulator UI │
├───────────┼────────────────┼─────────────────────────────────┤
│ Functions │ localhost:5001 │ http://localhost:4001/functions │
├───────────┼────────────────┼─────────────────────────────────┤
│ Hosting │ localhost:5000 │ n/a │
└───────────┴────────────────┴─────────────────────────────────┘
Other reserved ports: 4400, 4500
Issues? Report them at https://github.com/firebase/firebase-tools/issues and attach the *-debug.log files.
12:54:19I
hosting
127.0.0.1 - - [03/Nov/2020:04:54:19 +0000] "GET / HTTP/1.1" 200 2603 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:82.0) Gecko/20100101 Firefox/82.0"
12:54:19I
hosting
127.0.0.1 - - [03/Nov/2020:04:54:19 +0000] "GET /static/css/main.0fec40ca.chunk.css HTTP/1.1" 200 53148 "http://localhost:5000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:82.0) Gecko/20100101 Firefox/82.0"
12:54:19I
hosting
127.0.0.1 - - [03/Nov/2020:04:54:19 +0000] "GET /static/css/2.85b71462.chunk.css HTTP/1.1" 200 43004 "http://localhost:5000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:82.0) Gecko/20100101 Firefox/82.0"
12:54:19I
hosting
127.0.0.1 - - [03/Nov/2020:04:54:19 +0000] "GET /static/js/main.3cf6a16f.chunk.js HTTP/1.1" 200 216613 "http://localhost:5000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:82.0) Gecko/20100101 Firefox/82.0"
12:54:19I
hosting
127.0.0.1 - - [03/Nov/2020:04:54:19 +0000] "GET /static/js/2.8f67d3af.chunk.js HTTP/1.1" 200 3343005 "http://localhost:5000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:82.0) Gecko/20100101 Firefox/82.0"
12:54:19I
hosting
127.0.0.1 - - [03/Nov/2020:04:54:19 +0000] "GET /faceapi/weights/ssd_mobilenetv1_model-weights_manifest.json HTTP/1.1" 200 26548 "http://localhost:5000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:82.0) Gecko/20100101 Firefox/82.0"
12:54:19I
hosting
127.0.0.1 - - [03/Nov/2020:04:54:19 +0000] "GET /faceapi/weights/face_landmark_68_model-weights_manifest.json HTTP/1.1" 200 7889 "http://localhost:5000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:82.0) Gecko/20100101 Firefox/82.0"
12:54:19I
hosting
127.0.0.1 - - [03/Nov/2020:04:54:19 +0000] "GET /static/media/Create%20visuals%20for%20remote%20teaching..9fdce83b.png HTTP/1.1" 200 30230 "http://localhost:5000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:82.0) Gecko/20100101 Firefox/82.0"
12:54:19I
hosting
127.0.0.1 - - [03/Nov/2020:04:54:19 +0000] "GET /static/media/homepage_background.1c84776f.png HTTP/1.1" 200 72508 "http://localhost:5000/static/css/main.0fec40ca.chunk.css" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:82.0) Gecko/20100101 Firefox/82.0"
12:54:19I
hosting
127.0.0.1 - - [03/Nov/2020:04:54:19 +0000] "GET /static/media/homepage_laptop.a18abdeb.png HTTP/1.1" 200 6106290 "http://localhost:5000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:82.0) Gecko/20100101 Firefox/82.0"
12:54:19I
hosting
127.0.0.1 - - [03/Nov/2020:04:54:19 +0000] "GET /faceapi/weights/face_landmark_68_model-shard1 HTTP/1.1" 200 356840 "http://localhost:5000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:82.0) Gecko/20100101 Firefox/82.0"
12:54:19I
hosting
127.0.0.1 - - [03/Nov/2020:04:54:19 +0000] "GET /faceapi/weights/ssd_mobilenetv1_model-shard2 HTTP/1.1" 200 1422653 "http://localhost:5000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:82.0) Gecko/20100101 Firefox/82.0"
12:54:19I
hosting
127.0.0.1 - - [03/Nov/2020:04:54:19 +0000] "GET /faceapi/weights/ssd_mobilenetv1_model-shard1 HTTP/1.1" 200 4194304 "http://localhost:5000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:82.0) Gecko/20100101 Firefox/82.0"
Is this correct way of setting up functions? It's my first time working with them so I'm having a hard time debugging this. For some context, I'm trying to dynamically modify the OpenGraph meta tags depending on the URL, and I'm following this tutorial: https://medium.com/#jalalio/dynamic-og-tags-in-your-statically-firebase-hosted-polymer-app-476f18428b8b

Your hosting rewrite looks incorrect. I suggest taking a look at the documentation for how to configure a rewrite to a function. It shouldn't have a "destination", just "source" and "function". Also, if you intend to serve the URI "/" through the function, you should not have an index.html present at all in your public folder, otherwise it will get served first.

Request to URL defined with ingress.kubernetes.io/auth-url annotation is done with HTTP/1.0

I am using auth-url and auth-signin annotation for authenticating access to app. Problem is that request to URL defined with auth-url is always done with HTTP/1.0 and not with HTTP/1.1 as expected. From logs you can see that all other requests are done with HTTP/1.1.
Version used: nginx-ingress-controller:0.9.0-beta.19
Logs from ELB:
2017-11-30T14:28:30.606436Z dev-sandbox-2cb4 201.137.96.59:58692 10.10.0.101:80 0.000044 0.031215 0.000039 302 302 0 154 "GET https://example.net:443/testing/ HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2
2017-11-30T14:28:30.623944Z dev-sandbox-2cb4 24.134.104.23:40704 10.10.7.144:80 0.000029 0.01263 0.000068 401 401 0 21 "GET https://example.net:443/oauth2/auth HTTP/1.0" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2
2017-11-30T14:28:30.699239Z dev-sandbox-2cb4 201.137.96.59:58692 10.10.3.6:80 0.000028 0.001223 0.000046 302 302 0 395 "GET https://example.net:443/oauth2/start?rd=https://example.net/testing/ HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2
Annotation:
annotations:
ingress.kubernetes.io/auth-url: "https://$host/oauth2/auth"
ingress.kubernetes.io/auth-signin: "https://$host/oauth2/start"
Problem is that in the environment I need to use only 1.1 is allowed.
Is this something to be expected or am I doing something wrong?

Issue can be solved by adding
proxy_http_version 1.1;
under location = {{ $authPath }} block in nginx ingress template.
See https://github.com/kubernetes/ingress-nginx/pull/1787.

Wordpress compromised but I do not know how

today my Wordpress site was compromised so now serve a JS script that redirect to SPAM site.
I looked inside the Apache logs to reconstruct what happened, but i can't know how to interpretate this:
xx.xx.xx.xx - - [09/Jan/2017:10:24:42 +0100] "GET /wp-login.php HTTP/1.1" 200 6111 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:50.0) Gecko/20100101 Firefox/50.0"
xx.xx.xx.xx - - [09/Jan/2017:10:24:42 +0100] "GET /wp-login.php HTTP/1.1" 200 6111 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:50.0) Gecko/20100101 Firefox/50.0"
xx.xx.xx.xx - - [09/Jan/2017:10:24:43 +0100] "GET /wp-login.php HTTP/1.1" 200 6111 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:50.0) Gecko/20100101 Firefox/50.0"
xx.xx.xx.xx - - [09/Jan/2017:10:24:43 +0100] "GET /wp-login.php HTTP/1.1" 200 6111 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:50.0) Gecko/20100101 Firefox/50.0"
xx.xx.xx.xx - - [09/Jan/2017:10:24:43 +0100] "POST /wp-login.php HTTP/1.1" 302 4 "/wp-login.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:50.0) Gecko/20100101 Firefox/50.0"
xx.xx.xx.xx - - [09/Jan/2017:10:24:43 +0100] "POST /wp-login.php HTTP/1.1" 302 4 "/wp-login.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:50.0) Gecko/20100101 Firefox/50.0"
xx.xx.xx.xx - - [09/Jan/2017:10:24:44 +0100] "GET /wp-admin/ HTTP/1.1" 302 4 "http://my.host.name/wp-login.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:50.0) Gecko/20100101 Firefox/50.0"
xx.xx.xx.xx - - [09/Jan/2017:10:24:44 +0100] "GET /wp-admin/ HTTP/1.1" 200 219966 "/wp-login.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:50.0) Gecko/20100101 Firefox/50.0"
xx.xx.xx.xx - - [09/Jan/2017:10:24:47 +0100] "GET /wp-admin/theme-editor.php HTTP/1.1" 200 183974 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:50.0) Gecko/20100101 Firefox/50.0"
xx.xx.xx.xx - - [09/Jan/2017:10:24:48 +0100] "GET /wp-admin/theme-editor.php?file=404.php&theme=twentyfourteen HTTP/1.1" 500 3427 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:50.0) Gecko/20100101 Firefox/50.0"
xx.xx.xx.xx - - [09/Jan/2017:10:24:49 +0100] "GET /wp-admin/theme-install.php?upload HTTP/1.1" 200 161448 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:50.0) Gecko/20100101 Firefox/50.0"
As you can see somebody has tried to login, but he can't. But after two simple GET request to "/wp-admin/" seems to be logged in and he able to modify and install new theme.
At this point I have tried to find differences between the wp-admin directory that is present on the server with the original one downloaded from the official site wordpress.org, but i didn't find any differences.
I compared all file present into wp-admin with "diff" utility and i can't find any difference. For example:
diff /var/www/html/original.wordpress/wp-admin/themes.php /var/www/html.hacked/wp-admin/themes.php didn't output any code
Could you help me to find evidence?

Looks like your exploit happened here:
/wp-admin/theme-install.php?upload
Make sure your themes and plugins are updated. That last request returned a 200 response from your webserver to the attacker, and was likely used to upload a backdoor. I'd audit the contents of /wp-content/uploads/ to see if there are any out of place PHP files (backdoor shells) in there. If so, it's possible they used that to pivot within your site, so there may be other directories containing PHP backdoors elsewhere.

Resources