I've successfully installed Prometheus in Google Container Engine and I have these targets up:
kubernetes-apiservers
kubernetes-cadvisor
kubernetes-nodes
Now I would like to scrape Nginx stats from each of the Docker containers inside this Kubernetes cluster (which seems like a sensible thing to do).
But how can I make Prometheus automatically pull the metrics from all the Nginx instances running in all of the Docker containers?
From my research so far, the answer involves kubernetes_sd_config but I simply could not find enough documentation on how to put the pieces together.
Thank you!
Edit: This is not about exposing the Nginx stats. This is just about scraping any stats that are exposed by all Docker containers.
You are correct that you need to use the kubernetes_sd_config directive. Before continuing let me just say that what you should be asking is "Automatically scape all pods from Kubernetes". This is because a pod is considered the lowest unit of scale in Kubernetes. Regardless it is clear what you are trying to do.
So the kubernetes_sd_config can be used to discover all pods with a given tag like so:
- job_name: 'some-app'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app]
regex: python-app
action: keep
The source label [__meta_kubernetes_pod_label_app] is basically using the Kubernetes api to look at app pods that have a label of 'app' and whose value is captured by the regex expression, given on the line below (in this case, matching 'python-app').
Hope that helps. You can follow blog post here for more detail. Also for more information about kubernetes_sd_config check out docs here.
Note: it is worth mentioning that kubernetes_sd_config is still in beta. Thus breaking changes to configuration may occur in future releases.
Related
I have Invoice and Layout services docker images and there is a tutorial on how to deploy it using docker compose here
I would like to deploy the same(Invoice & Layout together) in AKS as on-premise. I've tried multiple ways but couldn't get it worked.
If anyone tried deploying above services in on-premise/disconnected environment using kubernetes then please do help me out.
The layout container hostname has to be same name as this "azure-container-service-layout" and try to deploy in AKS.
In deployment file, make sure below environment variable is exactly same. if you change the hostname of layout container(i.e., IP of layout container/localhost/any other hostname) it doesn't forward the request to layout container and Invoice container endpoints will return the failed status since it is depends on layout service(Details here)
AzureCognitiveServiceLayoutHost=http://azure-cognitive-service-layout:5000
In deployment file under spec, you can explicitly give static hostname as "azure-cognitive-service-layout".
After adding static hostname to the deployment, I'm able to deploy both containers(layout and invoice) together and get it worked as expected.
spec:
**hostname: azure-cognitive-service-layout**
containers:
- name: az-form-recognizer-invoice-deployment
image: "mcr.microsoft.com/azure-cognitive-services/form-recognizer/invoice:latest"
I want to upgrade OKD cluster from 4.5.0-0.okd-2020-10-03-012432 to 4.5.0-0.okd-2020-10-15-235428
version in restricted network.
I could not find any steps on OKD documentation site. However, steps are present on OCP documentation site and looks straight forward.
Queries:
Is this scenario supported in OKD?
In below document at step #7, what could be corresponding step for OKD.
https://docs.openshift.com/container-platform/4.5/updating/updating-restricted-network-cluster.html#update-configuring-image-signature
Where can I get image signature for OKD? Is this step valid for OKD?
I figured it out.
I did not perform steps mentioned in https://docs.openshift.com/container-platform/4.5/updating/updating-restricted-network-cluster.html#update-configuring-image-signature
"--apply-release-image-signature" flag in "oc adm release mirror..." command creates configmap automatically.
I have been using Openshift/Kubernates for some time and this has been the understanding.
For service to service communication
use DNS name of ${service-name} if they are under the same namespace
use DNS name of ${service-name}.${namespace}.svc.cluster.local if they are from different namespaces (network is joined)
Recently i was introduced with the topic of "we should add a dot after the svc.cluster.local to make it FQDN, for better DNS lookup speed". Done some testing and indeed with lookup is much faster with the dot. (~100ms without dot, 10ms with dot)
After some research, it was caused by the default dns setting from the kubernates
sh-4.2$ cat /etc/resolv.conf
search ${namespace}.svc.cluster.local svc.cluster.local cluster.local
nameserver X.X.X.X
options ndots:5
the ndots = 5 will perform a local search (sequential) if the dns name does not contain 5 dots.
In the case of ${service-name}.${namespace}.svc.cluster.local, the local search will be as such
${service-name}.${namespace}.svc.cluster.local + ${namespace}.svc.cluster.local // FAILED LOOKUP
${service-name}.${namespace}.svc.cluster.local + svc.cluster.local // FAILED LOOKUP
${service-name}.${namespace}.svc.cluster.local + cluster.local // FAILED LOOKUP
${service-name}.${namespace}.svc.cluster.local // SUCCESS LOOKUP
And for ${service-name}.${namespace}.svc.cluster.local., the local search will be as such
${service-name}.${namespace}.svc.cluster.local // SUCCESS LOOKUP
References
link
how to debug
Questions:
Since the ndots = 5 is the default setting for kubernetes, why ${service-name}.${namespace}.svc.cluster.local. is not documented on the official side ?
Should we change all service call to ${service-name}.${namespace}.svc.cluster.local. ? any potential downsides ?
Since the ndots = 5 is the default setting for kubernetes, why
${service-name}.${namespace}.svc.cluster.local. is not documented on
the official side ?
Well, it's a really good question. I searched through the official docs and it looks like this is not a documented feature. For this reason much better place for posting your doubts and also request for documentation improvement is the official GitHub site of Kubernetes DNS.
Should we change all service call to
${service-name}.${namespace}.svc.cluster.local. ? any potential
downsides ?
If it works well for you and definitely increases the performance, I would say - Why not ? I can't see any potential downsides here. By adding the last dot you're simply omitting those first 3 lookups that are doomed to failure anyway if you use Service domain name in a form of ${service-name}.${namespace}.svc.cluster.local
Inferring from lookup process you described and your tests, I guess if you use only ${service-name} (of course only within the same namespace), dns lookup should be also much faster and closer to those 10ms you observed when using ${namespace}.svc.cluster.local svc.cluster.local cluster.local. as then it is matched in the very first iteration.
Based on the latest document here, it states that that we should use ${service}.${namespace} to call a service from different namespace and expect to resolve on the second attempt
After upgrading FreePBX, fail2ban does not work. This is based on Asterisk version 13.19.1.
Asterisk-iptables is setup in fail2ban, but misses the login attempts.
After many attempts at changing items, I was finally able to determine the issue was in the "failregex" located in "/etc/fail2ban/filter.d/asterisk.conf.
I was able to add the string below to the bottom of the list:
(?:NOTICE|SECURITY|WARNING).*(FailedACL|InvalidAccountID|ChallengeResponseFailed|InvalidPassword).*RemoteAddress=\"IPV4/UDP/<HOST>/.*
Other steps included:
- ensuring fail2ban was running as a service
- adding an entry for "asterisk-iptables" and pointing to the log files
- ensuring the asterisk logger was creating the log files.
I have not posted explanations on these items, as I was able to follow guides readily available on the internet.
I've gotten pretty far into a deployment of my Meteor application on Distelli. Like, almost there. I've done everything as far as setting up the EC2 box, creating a user group [which didn't even seem necessary as I was able to SSH into the box with full rights without specifying my machine's IP], creating an elastic IP, successful build, and deployment to that box. But, I can't seem to check if Meteor is actually running (note: when I ssh in, there are active instances of Mongo and Node, so SOMETHING is running).
The problem has something to do with associating the elastic IP with my ROOT_URL and domain. I'm just not sure what to do at this step and can't seem to find any directions that are Meteor specific. Been using these guides:
https://www.distelli.com/docs/tutorials/how-to-set-up-aws-ec2
https://www.distelli.com/docs/tutorials/deploying-meteor-applications
http://gregblogs.com/tlt-associate-a-namecheap-domain-with-an-amazon-ec2-instance/
Recap: Distelli deployment is a success, but I get the follow error just before finishing:
Error: $ROOT_URL, if specified, must be an URL
I've set my ROOT_URL to my domain, and associated according to the previous guide. I can run traceroute on the IP, but like port 3000, so my inclination is the Meteor build is silently failing.
My manifest: https://gist.github.com/newswim/c642bd9a1cf136da73c3
I've noticed that when I point the CNAME record to my ec2 public DNS, NameCheap (aptly named) adds a . to the end of the record. Beyond that, I'm pretty much stumped.