Kong/Nginx (I'm not sure which) changes upstream to unexpected IP address. I want it to leave it as localhost. The failing request was made by web browser to localhost:8443/cost-recovery.
Log message:
2020/12/15 16:50:09 [error] 88#0: *522 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: kong, request: "GET /login?state=/cost-recovery HTTP/1.1", upstream: "https://192.168.65.2:8444/login?state=/cost-recovery", host: "localhost:8443"
I don't know where it's getting the 192.168.65.2 host, but want it to be localhost.
I'm using pantsel/konga container which uses Kong version 1.2.3. Configuration was done in part via proxy api requests:
Service Request:
{"host":"host.docker.internal","created_at":1608009524,"connect_timeout":60000,"id":"647180a3-6f8c-41ae-9f71-c9fc9db40249","protocol":"http","name":"cost-recovery","read_timeout":60000,"port":8447,"path":null,"updated_at":1608009524,"retries":5,"write_timeout":60000,"tags":null}
Route Request:
{"id":"24100a1d-c679-46b7-93f3-552b055df26b","tags":null,"paths":["\/cost-recovery"],"destinations":null,"protocols":["https"],"created_at":1608009525,"snis":null,"hosts":null,"name":"cost-recovery-route","preserve_host":true,"regex_priority":0,"strip_path":false,"sources":null,"updated_at":1608009525,"https_redirect_status_code":302,"service":{"id":"647180a3-6f8c-41ae-9f71-c9fc9db40249"},"methods":["GET"]}
Plugin Request:
{"name": "access-validator", "protocols": ["https"], "route": { "id": "24100a1d-c679-46b7-93f3-552b055df26b"}, "config": {"redirect_login": true}}
Related
I have a running Scrapyd Instance. This instance has been cloned and is now up and running under another server IP. The cloned server workes just fine, except that I can no longer deploy to the new IP.
I am getting
retry_on_eintr(check_call, [sys.executable, 'setup.py', 'clean', '-a', 'bdist_egg', '-d', d],
Deploying to project "test" in http://myip:6843/addversion.json
Deploy failed (504):
<html>
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.14.0 (Ubuntu)</center>
</body>
</html>
Nginx config looks ok (localhost). UFW as well. Ips are correct. The web interface is reachable and so forth. Just deploy failes.
Nginx error log:
[error] 1180#1180: *62 upstream timed out (110: Connection timed out) while reading response header from upstream, client: myip, server: , request: "POST /addversion.json HTTP/1.1", upstream: "http://127.0.0.1:6800/addversion.json", host: "myip:6843"
What am I missing?
Found the problem. Scrapy was trying to reach a remote MySQL server which was blocked for this IP.
I am trying to expose an mlflow model in a GKE cluster through an ingress-nginx and a google cloud load balancer.
The configuration of service to the respective deployment looks as follows:
apiVersion: v1
kind: Service
metadata:
name: model-inference-service
labels:
app: inference
spec:
ports:
- port: 5555
targetPort: 5555
selector:
app: inference
When forwarding this service to localhost using kubectl port-forward service/model-inference-service 5555:5555 I can successfully query the model by sending a test image to the api endpoint using the following script.
The url the request is sent to is http://127.0.0.1:5555/invocations.
This works as intended so I assume the deployment running the pod exposing the model and the corresponding clusterIP service model-inference-service is configured correctly.
Next, I installed ingress-nxinx into the cluster by doing
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm install my-release ingress-nginx/ingress-nginx
The ingress is configured as follows (I suspect the error has to be here?):
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
# nginx.ingress.kubernetes.io/rewrite-target: /invocations
name: inference-ingress
namespace: default
labels:
app: inference
spec:
rules:
- http:
paths:
- path: /invocations
backend:
serviceName: model-inference-service
servicePort: 5555
The ingress controller pod is running successfully:
my-release-ingress-nginx-controller-6758cc8f45-fwtw7 1/1 Running 0 3h33m
In the GCP console I can see that the load balancer was created successfully as well and I can optain its IP.
When using the same test script I used before to make a request to the Rest api endpoint (previously the service was forwarded to localhost) but now with the ip of the load balancer, I get a 502 Bad Gateway error:
The url is the following now: http://34.90.4.0:80/invocations
Traceback (most recent call last):
File "test_inference.py", line 80, in <module>
run()
File "//anaconda3/lib/python3.7/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "//anaconda3/lib/python3.7/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "//anaconda3/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "//anaconda3/lib/python3.7/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "test_inference.py", line 76, in run
print(score_model(data_path, host, port).text)
File "test_inference.py", line 54, in score_model
status_code=response.status_code, text=response.text
Exception: Status Code 502. <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.19.1</center>
</body>
</html>
When accessing the same url in a browser it says:
502 Bad Gateway
nginx/1.19.1
The logs of the ingress controller state:
2020/08/26 16:06:45 [warn] 86#86: *42282 a client request body is buffered to a temporary file /tmp/client-body/0000000009, client: 10.10.0.30, server: _, request: "POST /invocations HTTP/1.1", host: "34.90.4.0"
2020/08/26 16:06:45 [error] 86#86: *42282 connect() failed (111: Connection refused) while connecting to upstream, client: 10.10.0.30, server: _, request: "POST /invocations HTTP/1.1", upstream: "http://10.52.3.7:5555/invocations", host: "34.90.4.0"
2020/08/26 16:06:45 [error] 86#86: *42282 connect() failed (111: Connection refused) while connecting to upstream, client: 10.10.0.30, server: _, request: "POST /invocations HTTP/1.1", upstream: "http://10.52.3.7:5555/invocations", host: "34.90.4.0"
2020/08/26 16:06:45 [error] 86#86: *42282 connect() failed (111: Connection refused) while connecting to upstream, client: 10.10.0.30, server: _, request: "POST /invocations HTTP/1.1", upstream: "http://10.52.3.7:5555/invocations", host: "34.90.4.0"
10.10.0.30 - - [26/Aug/2020:16:06:45 +0000] "POST /invocations HTTP/1.1" 502 157 "-" "python-requests/2.24.0" 86151 0.738 [default-model-inference-service-5555] [] 10.52.3.7:5555, 10.52.3.7:5555, 10.52.3.7:5555 0, 0, 0 0.000, 0.001, 0.000 502, 502, 502 0d86e360427c0a81c287da4ff5e907bc
To test if the ingress and the load balancer work in principle I replaced the docker image with the real rest api I want to expose with this docker image which returns "hello world" on port 5050 and path /. I changed the port and the path (from /invocations to /) in the service and ingress manifests shown above and could successfully see "hello world" when accessing the ip of the load balancer in the browser.
Does anyone see what I might have done wrong?
Thank you very much!
Best regards,
F
The configuration you have shared is looking fine. There must be something in your cluster environment that is causing this behavior. See if pod-to-pod communication is working. Launch a test pod on the same node as the Nginx ingress controller and do a curl from that pod to the target service. See if you get any DNS or Network issues. Try changing the host header when calling the service and see if it's sensitive to that.
I'm trying to configure rtorrent with xmlrpc using nginx as the web server. I'm running into an issue right now where, when I run this command:
xmlrpc localhost/rpc system.listMethods
I get a 502. nginx logs this error:
connect() to unix:/tmp/scgi.socket failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: localhost, request: "POST /rpc/RPC2 HTTP/1.1", upstream: "scgi://unix:/tmp/scgi.socket:", host: "localhost"
I'm setting the permissions for the socket properly (I think). I've been working on this for a bit, and would appreciate another set of eyes. You can find all the conf files and code here: https://github.com/nVitius/rtorrent-docker
Also, dockerhub link:
https://hub.docker.com/r/nvitius/rtorrent-docker/
After looking at it again this morning, I found that the issue was that rtorrent wasn't picking up the configuration file. I specified the path to it manually, and it works now.
I have deployed my app on EC2 using the software in the title, but I am getting an Internal Server Error. Here is the tutorial I have been following.
Here is the error log for me trying to get on the application via the browser:
2014/02/17 19:48:29 [error] 26513#0: *1 connect() to unix:/tmp/uwsgi.sock failed (111: Connection refused) while connecting to upstream, client: xxx.xxx.xxx.xxx, server: localhost, request: "GET / HTTP/1.1", upstream: "uwsgi://unix:/tm p/uwsgi.sock:", host: "ec2-xx-xxx-xx-xxx.compute-1.amazonaws.com"
If your Ansible playbook is based on Matt Wright's tutorial, then all you need to do is reboot after the installation. The playbook doesn't update supervisor with the new program it installs (which is actually the upstream uWSGI referred to by the log), so the program cannot be started.
I've followed https://www.mercurial-scm.org/wiki/HgWebDirStepByStep to get "hg serve" running over CGI - but it's not quite working.
Here is the command I'm using to spawn the CGI:
spawn-fcgi -a 127.0.0.1 -p 9000 -f /path/to/hgweb.cgi -P /tmp/fcgi.pid 2>&1
The output suggests that the process spawned successfully, but a ps -p reveals that the process has already closed down. Sure enough, when I run the above command with -n, it spits out a load of HTML (the list of repositories) and then quits. Isn't it meant to stick around, listening on port 9000?
Telnetting to port 9000 gives "Connection refused" and this appears to be the problem nginx is having also:
2012/02/15 22:16:20 [error] 13483#0: *13 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: emily, request: "GET /hg/ HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "localhost:8001"
I'm confident my nginx config is correct, although I can post it here if you need to take a look.
Thanks for any help :)