I am having an issue logging into the Pulsar Manager UI running on my k8s cluster in docker-desktop on my M1 Mac.
When I try to login, I am unable to progress past the login page with the default pulsar admin credentials and when I inspect the page I see the following:
Failed to load resource: the server responded with a status of 404 (Not Found)
I think the issue has something to do with me being unable to connect with the backed service over port 7750 but I am honestly not sure how I can resolve that. I have deployed using the helm chart and used the minikube.yaml file values to keep replica counts and such down since it's running on the single docker-destop node.
Has anyone encountered this issue before or know of a solution?
If this issue has already come up here, I would love a link to that thread!
Below I have included some details of what's running in my cluster, the other values are all the same as what's included in the helm chart.
Services:
k get svc -n pulsar
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
pulsar-mini-bookie ClusterIP None <none> 3181/TCP,8000/TCP 21h
pulsar-mini-broker ClusterIP None <none> 8080/TCP,6650/TCP 21h
pulsar-mini-proxy LoadBalancer 10.102.192.239 localhost 80:30132/TCP,6650:30925/TCP 21h
pulsar-mini-pulsar-manager LoadBalancer 10.98.70.14 localhost 9527:30322/TCP 21h
pulsar-mini-toolset ClusterIP None <none> <none> 21h
pulsar-mini-zookeeper ClusterIP None <none> 8000/TCP,2888/TCP,3888/TCP,2181/TCP 21h
Output of csrf command, showing that connection is refused from 7750, even if I try to use kubectl port-forward of the pulsar-mini-pulsar-manager pod (though perhaps this isn't the correct way to do it):
% CSRF_TOKEN=$(curl http://localhost:7750/pulsar-manager/csrf-token)
curl \
-H "X-XSRF-TOKEN: $CSRF_TOKEN" \
-H "Cookie: XSRF-TOKEN=$CSRF_TOKEN;" \
-H 'Content-Type: application/json' \
-X PUT http://localhost:7750/pulsar-manager/users/superuser \
-d '{"name": "admin", "password": "apachepulsar", "description": "test", "email": "username#test.org"}'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (7) Failed to connect to localhost port 7750: Connection refused
curl: (7) Failed to connect to localhost port 7750: Connection refused
When I run the bin/pulsar-admin commands from my local machine, things work just fine. I simply can't access the commands or the UI for some reason.
Output of some commands below:
$ bin/pulsar-admin topics list-partitioned-topics apache/pulsar
"persistent://apache/pulsar/test-topic"
/apache-pulsar-2.9.1
$ bin/pulsar-admin namespaces list apache
"apache/pulsar"
"apache/tester"
$ bin/pulsar-admin topics create-partitioned-topic apache/pulsar/test-topic-2 -p 4
$ bin/pulsar-admin topics list-partitioned-topics apache/pulsar
"persistent://apache/pulsar/test-topic"
"persistent://apache/pulsar/test-topic-2"
Is that the correct port? What is in your PM configuration? Anything in the logs?
https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/
9527:30322/TCP
https://pulsar.apache.org/docs/en/administration-pulsar-manager/
in docker we have to specify the second port.
docker run -it
-p 9527:9527 -p 7750:7750
-e SPRING_CONFIGURATION_FILE=/pulsar-manager/pulsar-manager/application.properties
apachepulsar/pulsar-manager:v0.2.0
You only have the 9527 port specific
Related
I did a helm install of Nginx like this
helm install ingress-nginx ingress-nginx/ingress-nginx \
--namespace ingress-nginx \
--create-namespace \
--version 4.2.0 \
--set controller.ingressClass=nginx \
--set rbac.create=true \
--values /tmp/ingress-nginx.yaml
everythng got deployed fine
[rke#k8gui apps]$ kubectl get all -n ingress-nginx
NAME READY STATUS RESTARTS AGE
pod/ingress-nginx-controller-248rc 1/1 Running 0 3m15s
pod/ingress-nginx-controller-hpv8h 1/1 Running 0 3m15s
pod/ingress-nginx-controller-jwzjn 1/1 Running 0 3m15s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ingress-nginx-controller LoadBalancer 10.43.117.177 192.168.33.100 80:31678/TCP,443:32340/TCP 3m4s
service/ingress-nginx-controller-admission ClusterIP 10.43.119.134 <none> 443/TCP 3m4s
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/ingress-nginx-controller 3 3 3 3 3 kubernetes.io/os=linux 3m4s
[rke#k8gui apps]$ curl http://192.168.33.100
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
but curling on the loadbalancer ext ip show 404 not found
any idea?
You are getting the 404 due to you have maybe not deployed any ingress.
The output you have pasted above is the ingress controller which is a Nginx controller with external IP. The ingress controller manages the ingress or in other words, routing rules.
If you have not created any ingress rules to divert traffic into a cluster or if rules are not matching Nginx will throw the 404 error.
You are getting the 404 itself from Nginx so when you are hitting the External IP of Controller your request reaches till Nginx but no rule matches.
as you see in output
<hr><center>nginx</center>
Try creating one hello app deployment and rule of ingress routing as shown in example : https://kubernetes.io/docs/tasks/access-application-cluster/ingress-minikube/
Seems you are trying to reach external IP from within your cluster.
Could be this is not possible.
Can you try from browser?
Within cluster you could try to reach your cluster IP 10.43.117.177.
I have an nginx pod in the default namespace and a ClusterIP service exposing the pod.
$ kubectl run nginx-pod --image=nginx
$ kubectl expose po nginx-pod --name=nginx-service --port=8080 --target-port=80 --type=ClusterIP
I can access the service via its internal IP from inside the cluster.
$ kubectl get svc nginx-service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-service ClusterIP 10.100.5.218 <none> 8080/TCP 2m49s
$ wget -O- 10.100.5.218:8080
--> 200 OK
I can access the service by name from inside a pod.
$ kubectl run tmp -it --rm --image=busybox --command -- /bin/sh -c 'wget -O- nginx-service:8080'
--> 200 OK
However, why can't I access the service by name from outside a pod?
$ wget -O- nginx-service:8080
or
$ wget -O- nginx-service.default.svc.cluster.local:8080
--> wget: unable to resolve host address ‘nginx-service’
The magic service hostnames (and pod hostnames) are provided by the "cluster DNS" service, usually CoreDNS these days. A resolv.conf aimed at the internal CoreDNS is automatically injected into all pods. But I'm guessing by "outside of a pod" you mean on the underlying host which has no such entries in its resolv.conf.
Please see this.
https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types
You either need NodePort or LoadBalancer service type.
I installed the latest Docker, Minikube, and kubectl into my Mac (Catalina). I also have a recent MySQL, with the command line properly installed in the PATH. I'm using the stock terminal (zsh).
Docker started just fine, tells me of the pods it has installed.
Minikube starts fine, and kubectl get all reports on its artifacts just fine.
Jeromes-MacBook-Pro:cloudnative-statelessness jerome$ kubectl get all
NAME READY STATUS RESTARTS AGE
pod/mysql-7dbfd4dbc4-sz8ps 1/1 Running 0 15m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 20m
service/mysql-svc NodePort 10.111.176.15 <none> 3306:30022/TCP 15m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/mysql 1/1 1 1 15m
NAME DESIRED CURRENT READY AGE
replicaset.apps/mysql-7dbfd4dbc4 1 1 1 15m
When I run minikube service mysql-svc --url I'm expecting to get a URL, like this one from another machine: http://192.168.99.101:31067 . Instead I see something about starting a service in a 'tunnel':
Jeromes-MacBook-Pro:cloudnative-statelessness jerome$ minikube service mysql-svc --url
🏃 Starting tunnel for service mysql-svc.
|-----------|-----------|-------------|------------------------|
| NAMESPACE | NAME | TARGET PORT | URL |
|-----------|-----------|-------------|------------------------|
| default | mysql-svc | | http://127.0.0.1:64966 |
|-----------|-----------|-------------|------------------------|
http://127.0.0.1:64966
❗ Because you are using a Docker driver on darwin, the terminal needs to be open to run it.
At this point the terminal is non-responsive.
I'm believing that minikube service SERVICENAME should try to start a service, and also return that block of text. I'm also believing that the --url suffix should merely returns what is in the URL column, and skip starting a service.
Any good explanations of how I can get the result I want on my Mac?
And BTW, how do I recover control of the terminal session once it states "Because..." ?
Thanks,
Jerome.
UPDATE ON 8/14/2020:
I took Saravanan's advice. I uninstalled Docker from my Mac and used homebrew to install docker + docker-machine + virtualbox (see https://www.robinwieruch.de/docker-macos). When I run "minikube service mysql-svc --url" I no longer get the tunnel problem. Thank you, Saravanan.
My problems have morphed into getting a correct version of my containers (compiled apps, then run thru docker build) from Docker Hub. The YAML file I have points at my account there, but I'm afraid I've an obsolete version. What do I do to overwrite my current version on my Mac, or to delete the Docker containers so that kubectl create can get the updated version?
The reason for this is your minikube image is running in the container.
Try changing the configuration to run it in the virtual box. Then you can reach your sql pod without tunneling.
# first delete the existing minikube image
$ minikube delete
# change the minikube driver to virtualbox
$ minikube config set vm-driver virtualbox
# start minikube again
$ minikube start
Ensure you have virtual box installed before proceeding
Was there a recent change to Nginx Ingress? Out of the blue I'm now getting "Connection refused" errors. I thought it was my own configuration which worked on a previous cluster.
Instead I decided to follow this tutorial GKE NGINX INGRESS and I'm getting the same result.
$ kubectl get deployments --all-namespaces
NAMESPACE NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
default hello-app 1 1 1 1 13m
default nginx-ingress-controller 1 1 1 1 12m
default nginx-ingress-default-backend 1 1 1 0 12m
I see the default-backend isn't running but I don't know enough about Kubernetes to know if that's what's preventing everything from working properly.
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello-app ClusterIP 10.31.255.90 <none> 8080/TCP 14m
kubernetes ClusterIP 10.31.240.1 <none> 443/TCP 19m
nginx-ingress-controller LoadBalancer 10.31.251.198 35.227.50.24 80:31285/TCP,443:30966/TCP 14m
nginx-ingress-default-backend ClusterIP 10.31.242.167 <none> 80/TCP 14m
Finally:
$ kubectl get ing
NAME HOSTS ADDRESS PORTS AGE
ingress-resource * 35.237.184.85 80 10m
According to the tutorial I should just be able to go to here to receive a 200 and here to get a 404.
I've left the links live so you all can see them.
$ curl -I http://35.237.184.85/hello
curl: (7) Failed to connect to 35.237.184.85 port 80: Connection refused
I swear everything worked before and the only thing I can think of is that something from the teller install of nginx-ingress changed.
Please, any help is appreciated! Thank you in advance!
That's because you are trying the request against the IP address created by the Ingress. Your entrypoint is the LoadBalancer type service created IP.
Try curl -I http://35.227.50.24/hello. That's where you will get 200.
OK so 6k views and this is not right. Let's fix this :
I see the default-backend isn't running but I don't know enough about Kubernetes to know if that's what's preventing everything from working properly.
The defaut-backend is what serves thos hello and healthz pages. Not runnig = no pages
Is the LoadBalancer service IP where an A record should point for a domain? I thought traffic was supposed to hit the ingress IP, especially if you want SSL.
Yes you should always point to the ingress' IP. That's the point of ingress they handle named based http(s) request to route to proper services and handling (un the case of https) TLS/SSL stuff. Make sure you have a certificate autority like cert-manager configured on your cluster if you plan on doing this.
Intalled registry in Kubernetes kubectl create -f kube-registry.yaml.
There has docker image in it:
curl http://192.168.99.100:5000/v2/_catalog
{"repositories":["app1"]}
Also did port-forward:
kubectl port-forward --namespace kube-system \
$(kubectl get po -n kube-system | grep kube-registry-v0 | \
awk '{print $1;}') 5000:5000
Set image: 192.168.99.100:5000/app1 in Kubernetes deployment file(deployment.yaml).
After deploy it to cluster:
kubectl create -f deployment.yaml
Check status from Kubernetes dashboard on Deployments menu, got error:
Failed to pull image "192.168.99.100:5000/app1": rpc error: code = 2 desc = Error response from daemon: {"message":"Get https://192.168.99.100:5000/v1/_ping: http: server gave HTTP response to HTTPS client"}
Error syncing pod
The reason is can't get registry content from http protocol. How to resolve it?
Added insecure-registries by (The real content is 192.168.99.100:5000. Here is a different IP/port picture.)
Useful resource
minikube start --insecure-registry "10.0.0.0/24"
https://github.com/kubernetes/minikube/blob/master/docs/insecure_registry.md
Configure docker to pull image using http insecure-registries
create this file in your node and restart docker process, then try to deploy
/etc/docker/daemon.json
{
"insecure-registries" : ["192.168.99.100:5000"]
}