I am currently try to load data from Prometheus Pagespeed Exporter (https://github.com/foomo/pagespeed_exporter) directly into ELK using Metricbeat. It seems so, that the Call of Pagespeed Exporter requires more time than Metricbeats offers to scrape the required data. A Client Timeout occures:
unable to decode response from prometheus endpoint: error making http request: Get http://pagespeed-exporter-monitoring:9271/metrics: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Currently the Request is cancelled after about 10s while the Timeout in metricbeat should be defined like Timeout = Period (https://www.elastic.co/guide/en/beats/devguide/current/metricset-details.html). The Period in my case is configured to 3600s.
metricbeat:
deployment:
metricbeatConfig:
metricbeat.yml: |
metricbeat.modules:
- module: prometheus
enabled: true
period: 3600s
hosts: ["pagespeed-exporter-monitoring:9271"]
metrics_path: /metrics
Is there any option to increase the Timeout or is there any other issue?
You can override the timeout on the module config,
https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-metricbeat.html#_timeout
For example:
metricbeat:
deployment:
metricbeatConfig:
metricbeat.yml: |
metricbeat.modules:
- module: prometheus
enabled: true
period: 3600s
hosts: ["pagespeed-exporter-monitoring:9271"]
metrics_path: /metrics
timeout: 60s # adjust here
Related
I have the following configuration of service with route in Kong:
- name: test
host: test-service
port: 80
protocol: http
path: /endpoint/
routes:
- name: test_route
strip_path: true
preserve_host: true
paths:
- /login
I am trying to understand the following behaviour:
when I access http://localhost/login, I will end up with the http://localhost/endpoint in my browser
however, when I try to access http://localhost/login/test/page, nothing will change and I am still in my browser on http://localhost/login/test/page and the upstream server served my request properly
My expectation was that using http://localhost/login/test/page, my browser will have eventually http://localhost/endpoint/test/page. Apparently I misunderstood it.
I understood that in this case, Kong will use the upstream path as /endpoint/test/page, because I have strip_path: true.
However, how it can be changed in case I want to end up with http://localhost/endpoint/test/page in case I will access http://localhost/login/test/page in my browser?
I am using Envoy's LocalRatelimit.
Version Info:
istio 1.11.4
Envoy v1.19.1
Kubenretes 1.12 (bare metal)
In order to make the request Burst resistant, the Token Bucket is specified as follows.
name: envoy.filters.http.local_ratelimit
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit
stat_prefix: http_local_rate_limiter
token_bucket:
max_tokens: 200
tokens_per_fill: 20
fill_interval: 1s
filter_enabled:
runtime_key: local_rate_limit_enabled
default_value:
numerator: 100
denominator: HUNDRED
filter_enforced:
runtime_key: local_rate_limit_enforced
default_value:
numerator: 100
denominator: HUNDRED
response_headers_to_add:
- append: false
header:
key: x-local-rate-limit
value: "true"
https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/local_rate_limit_filter
This means that a maximum of 200 rps can be accepted for one second, and then every second, the Token will recover by 20 to prepare for the next Burst.
However, what I want to achieve now is not to have these 200rps reach the upstream server immediately, but to allow some delay and send the request to the upstream server. Specifically, I'm hoping for something equivalent to the delay option in nginx's limit_req directive.
http://nginx.org/en/docs/http/ngx_http_limit_req_module.html
Is there any richness in Envoy to specify this delay feature? If not, will this be implemented in the future?
This post has been multi-posted to the EnvoyProxy user group (here).
I have a k8s deployment on an azure cluster, connected to a service, exposed via an ingress like below:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
namespace: infrastructure--buildfarm
name: buildfarm-ingress
annotations:
kubernetes.io/ingress.class: nginx-internal
nginx.ingress.kubernetes.io/rewrite-target: /$1
nginx.ingress.kubernetes.io/use-regex: "true"
cert-manager.io/cluster-issuer: selfsigned-cluster-issuer
spec:
rules:
- host: buildfarm.dev.azr.internal.mydomain.com
http:
paths:
- backend:
serviceName: aks-buildfarm
servicePort: 8980
path: /(.*)
I'm trying to execute a simple build using the bazel buildfarm API for bazel, but end up with the following error:
eito#fuji:~/MyRepo$ bazel --client_debug build //examples/... --remote_executor=grpcs://buildfarm.azr.internal.mydomain.com/
[INFO 17:07:22.373 src/main/cpp/option_processor.cc:407] Looking for the following rc files: /etc/bazel.bazelrc,/home/eito/MyRepo/.bazelrc,/home/eito/.bazelrc
[INFO 17:07:22.373 src/main/cpp/rc_file.cc:56] Parsing the RcFile /home/eito/MyRepo/.bazelrc
[INFO 17:07:22.373 src/main/cpp/rc_file.cc:56] Parsing the RcFile user.bazelrc
[INFO 17:07:22.373 src/main/cpp/rc_file.cc:129] Skipped optional import of user.bazelrc, the specified rc file either does not exist or is not readable.
[INFO 17:07:22.373 src/main/cpp/rc_file.cc:56] Parsing the RcFile /home/eito/.bazelrc
[INFO 17:07:22.374 src/main/cpp/blaze.cc:1626] Debug logging requested, sending all client log statements to stderr
[INFO 17:07:22.374 src/main/cpp/blaze.cc:1509] Acquired the client lock, waited 0 milliseconds
[INFO 17:07:22.376 src/main/cpp/blaze.cc:1697] Trying to connect to server (timeout: 30 secs)...
[INFO 17:07:22.381 src/main/cpp/blaze.cc:1264] Connected (server pid=240772).
[INFO 17:07:22.381 src/main/cpp/blaze.cc:1974] Releasing client lock, let the server manage concurrent requests.
INFO: Invocation ID: d00bb07c-440c-4220-a04a-74cb54ef9913
ERROR: Failed to query remote execution capabilities: io.grpc.StatusRuntimeException: UNKNOWN: HTTP status code 0
invalid content-type: null
headers: Metadata(:status=000,date=Fri, 01 Oct 2021 16:07:26 GMT,strict-transport-security=max-age=15724800; includeSubDomains)
DATA-----------------------------
���
DATA-----------------------------
h���Unexpected HTTP/1.x request: POST /build.bazel.remote.execution.v2.Capabilities/GetCapabilities
DATA-----------------------------
[INFO 17:07:26.261 src/main/cpp/blaze.cc:2093] failure_detail: message: "Failed to query remote execution capabilities: io.grpc.StatusRuntimeException: UNKNOWN: HTTP status code 0\ninvalid content-type: null\nheaders: Metadata(:status=000,date=Fri, 01 Oct 2021 16:07:26 GMT,strict-transport-security=max-age=15724800; includeSubDomains)\nDATA-----------------------------\n\000\000\022\004\000\000\000\000\000\000\003\177\357\277\275\357\277\275\357\277\275\000\334\000\020\000\000\000\006\000\000 \000\000\000\004\010\000\000\000\000\000\000\019\000\001\nDATA-----------------------------\n\000\000h\007\000\000\000\000\000\177\357\277\275\357\277\275\357\277\275\000\000\000\001Unexpected HTTP/1.x request: POST /build.bazel.remote.execution.v2.Capabilities/GetCapabilities \nDATA-----------------------------\n"
remote_execution {
code: CAPABILITIES_QUERY_FAILURE
}
I've asked on their slack, and was told I need to use HTTP2 for my ingress, since the error implies I am trying to convert HTTP1->HTTP2. How could I modify my ingress so it uses HTTP2?
I've looked around https://developer-docs.citrix.com/projects/citrix-k8s-ingress-controller/en/latest/how-to/http-use-cases/#http2-upgrade
and saw that it requires re-configuration of our nginx controller?
Is there an easier way of doing this instead of taking a deep dive with the nginx controllers? Or am I missing something?
I am trying to setup OAuth authentication (OpenID Connect) with Azure AD and Spring Cloud Data Flow server. Following below documentation reference but no help.
https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#appendix-identity-provider-azure
https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#configuration-security-oauth2
When I push data flow server in open source cloud foundry, i get below error (if OAuth2 config is removed from application.yml, it gets deployed without issue):
redentials/instance.key and /etc/cf-instance-credentials/instance.crt
18:16:57.512: [HEALTH.0] Failed to make TCP connection to port 8080: connection refused
18:16:57.512: [CELL.0] Timed out after 1m0s: health check never passed.
18:16:57.515: [CELL/SSHD.0] Exit status 0
18:17:07.588: [APP/PROC/WEB.0] Exit status 137 (exceeded 10s graceful shutdown interval)
18:17:07.857: [API.3] Process has crashed with type: "web"
18:17:07.931: [API.3] App instance exited with guid ff60a149-d91f-4d9c-90b9-661c3bb8ad0f payload: {"instance"=>"e35f4a5d-a4f0-433d-6546-82ed", "index"=>0, "cell_id"=>"231ab214-d841-46ba-b20f-243aeac9bbfa", "reason"=>"CRASHED", "exit_description"=>"Instance never healthy after 1m0s: Failed to make TCP connection to port 8080: connection refused", "crash_count"=>3, "crash_timestamp"=>1622845027800626529, "version"=>"3a0686fb-a43a-4528-a425-21a544
From the logs, I do not see any hint of OAuth2.0 setup is taking effect by data flow server. Here is my application.yml entries related to OAuth2 config:
spring:
cloud:
dataflow:
security:
authorization:
provider-role-mappings:
dataflow-server:
map-oauth-scopes: true
role-mappings:
ROLE_VIEW: dataflow.view
ROLE_CREATE: dataflow.create
ROLE_MANAGE: dataflow.manage
ROLE_DEPLOY: dataflow.deploy
ROLE_DESTROY: dataflow.destroy
ROLE_MODIFY: dataflow.modify
ROLE_SCHEDULE: dataflow.schedule
security:
oauth2:
client:
registration:
dataflow-server:
provider: azure
redirect-uri: 'https://data-flow-server/dashboard'
client-id: 977-95bc-4f3645d77f43
client-secret: ~02K-5pf182_E-x-PWn
authorization-grant-type: authorization_code
scope:
- openid
- profile
- email
- offline_access
- api://dataflow-server/dataflow.view
- api://dataflow-server/dataflow.deploy
- api://dataflow-server/dataflow.destroy
- api://dataflow-server/dataflow.manage
- api://dataflow-server/dataflow.modify
- api://dataflow-server/dataflow.schedule
- api://dataflow-server/dataflow.create
provider:
azure:
issuer-uri: https://login.microsoftonline.com/sdf3s-3244f65-b82d-5ec2fd32d5aa/v2.0
user-name-attribute: name
access-token-uri: https://login.microsoftonline.com/sdf3s-3244f65-b82d-5ec2fd32d5aa/oauth2/v2.0/token
token-uri: https://login.microsoftonline.com/sdf3s-3244f65-b82d-5ec2fd32d5aa/oauth2/v2.0/token
user-authorization-uri: https://login.microsoftonline.com/sdf3s-3244f65-b82d-5ec2fd32d5aa/oauth2/v2.0/authorize
authorization-uri: https://login.microsoftonline.com/sdf3s-3244f65-b82d-5ec2fd32d5aa/oauth2/v2.0/authorize
resourceserver:
jwt:
jwk-set-uri: https://login.microsoftonline.com/sdf3s-3244f65-b82d-5ec2fd32d5aa/discovery/v2.0/keys
I've been using elasticsearch, metricbeat and elastalert to watch my server. I have nginx intalled on it that is been used as a reverse proxy and I need to send an to it if nginx drop or return some error, I have already some alerts configured but how can I make a rule to send alert to nginx when it drop or return some error.
Thank a lot
Metricbeat is just for data about the system resources usage. What you need is installing filebeat and activating the nginx module. Then you can use the rule type any of elastalert and filter by fileset.module: nginx and fileset.name: error:
name: your rule name
index: filebeat-*
type: any
filter:
- term:
fileset.module: "nginx"
- term:
fileset.name: "error"
alert:
- "slack"
... # your slack config stuff
realert:
minutes: 1