Airflow doesn't send all the metrics by statsD - airflow
I have enabled the statsD monitoring in airflow.cfg
[metrics]
# StatsD (https://github.com/etsy/statsd) integration settings.
# Enables sending metrics to StatsD.
statsd_on = True
statsd_host = localhost
statsd_port = 8125
statsd_prefix = airflow
Added the mapping file from https://github.com/databand-ai/airflow-dashboards/blob/main/statsd/statsd.conf
But I am getting few warnings -
ts=2022-11-05T09:22:24.814Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.dag_processing.processes
ts=2022-11-05T09:22:24.814Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.scheduler.critical_section_busy
ts=2022-11-05T09:22:24.814Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.dag.callback_exceptions
ts=2022-11-05T09:22:24.814Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.celery.task_timeout_error
ts=2022-11-05T09:22:24.814Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.dag_processing.import_errors
ts=2022-11-05T09:22:24.814Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.dag_processing.total_parse_time
ts=2022-11-05T09:22:24.814Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.dag_processing.processor_timeouts
ts=2022-11-05T09:22:24.814Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.executor.open_slots
ts=2022-11-05T09:22:24.814Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.executor.queued_tasks
ts=2022-11-05T09:22:24.814Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.executor.running_tasks
ts=2022-11-05T09:22:24.815Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.smart_sensor_operator.poked_tasks
ts=2022-11-05T09:22:24.815Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.smart_sensor_operator.poked_success
ts=2022-11-05T09:22:24.815Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.smart_sensor_operator.poked_exception
ts=2022-11-05T09:22:24.815Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.smart_sensor_operator.exception_failures
ts=2022-11-05T09:22:24.815Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.smart_sensor_operator.infra_failures
ts=2022-11-05T09:22:24.815Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.scheduler.critical_section_duration
ts=2022-11-05T09:22:24.815Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.scheduler.tasks.killed_externally
ts=2022-11-05T09:22:24.815Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.scheduler.tasks.running
ts=2022-11-05T09:22:24.815Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.scheduler.tasks.starving
ts=2022-11-05T09:22:24.815Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.scheduler.orphaned_tasks.cleared
ts=2022-11-05T09:22:24.815Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.scheduler.orphaned_tasks.adopted
ts=2022-11-05T09:22:24.815Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.dag_processing.last_runtime.*
ts=2022-11-05T09:22:24.815Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.pool.open_slots.*
ts=2022-11-05T09:22:24.815Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.pool.queued_slots.*
ts=2022-11-05T09:22:24.815Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.pool.running_slots.*
ts=2022-11-05T09:22:24.815Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.pool.starving_tasks.*
ts=2022-11-05T09:22:24.815Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.dagrun.dependency-check.*
ts=2022-11-05T09:22:24.815Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.dag_processing.last_duration.*
ts=2022-11-05T09:22:24.815Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.dagrun.schedule_delay.*
ts=2022-11-05T09:22:24.815Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.dagrun.*.first_task_scheduling_delay
ts=2022-11-05T09:22:24.815Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.ti.start.*.*
ts=2022-11-05T09:22:24.815Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.dag_processing.last_run.seconds_ago.*
ts=2022-11-05T09:22:24.815Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.dag.*.*.duration
ts=2022-11-05T09:22:24.815Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.dagrun.duration.success.*
ts=2022-11-05T09:22:24.815Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.dagrun.duration.failed.*
ts=2022-11-05T09:22:24.815Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.ti_failures
ts=2022-11-05T09:22:24.815Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.ti_successes
ts=2022-11-05T09:22:24.815Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.zombies_killed
ts=2022-11-05T09:22:24.815Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.scheduler_heartbeat
ts=2022-11-05T09:22:24.815Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.sla_email_notification_failure
ts=2022-11-05T09:22:24.815Z caller=fsm.go:314 level=warn msg="backtracking required because of match. Performance may be degraded" match=*.dagbag_size
I am not getting these metrics wherever there is a warning message while I am getting other metrics.
Running airflow with celery executor v. 2.3.0
statsd_exporter, version 0.22.8
Running this command - ./statsd_exporter --statsd.listen-udp=":8125" --statsd.listen-tcp=":8125" --statsd.mapping-config="/tmp/statsd_exporter-0.22.8.linux-arm64/statsd_mapping.yml" and expecting to receive all the metrics once I enable the statsD monitoring.
Related
Julia "Unsatisfiable requirements detected for package" error when installing multiple packages
I'm using Julia v1.5.2 and I got the below error when I tried to install EvalMetrics:
Pkg.add("EvalMetrics")
I even tried using the Pkg manager instead of using import Pkg, but it doesn't seem to make a difference either.
Unsatisfiable requirements detected for package StatsBase [2913bbd2]:
StatsBase [2913bbd2] log:
├─possible versions are: 0.24.0-0.33.19 or uninstalled
├─restricted to versions * by an explicit requirement, leaving only versions 0.24.0-0.33.19
├─restricted by compatibility requirements with JuliaDB [a93385a2] to versions: 0.24.0-0.32.2
│ └─JuliaDB [a93385a2] log:
│ ├─possible versions are: 0.9.0-0.13.1 or uninstalled
│ └─restricted to versions * by an explicit requirement, leaving only versions 0.9.0-0.13.1
└─restricted by compatibility requirements with EvalMetrics [251d5f9e] to versions: 0.33.0-0.33.19 — no versions left
└─EvalMetrics [251d5f9e] log:
├─possible versions are: 0.1.0-0.2.1 or uninstalled
└─restricted to versions 0.1.1 by an explicit requirement, leaving only versions 0.1.1
and when i tried to update StatsBase, it is up to date and i got this:
Updating registry at `~/.julia/registries/General.toml`
No Changes to `~/.julia/environments/v1.7/Project.toml`
No Changes to `~/.julia/environments/v1.7/Manifest.toml`
I'll start by saying don't use Julia 1.5.2 - the current stable release is 1.7.3, and we will soon get 1.8.0, and it's generally adviseable to use the latest stable release (or an LTS if you don't want to make any changes to you environment but still receive bugfixes). That said, the error you're seeing is unrelated to the Julia version. Here's a minimal reproducer in a clean temporary environment (] activate --temp): (jl_0jYGBJ) pkg> add JuliaDB EvalMetrics Resolving package versions... ERROR: Unsatisfiable requirements detected for package StatsBase [2913bbd2]: StatsBase [2913bbd2] log: ├─possible versions are: 0.24.0-0.33.19 or uninstalled ├─restricted by compatibility requirements with JuliaDB [a93385a2] to versions: 0.24.0-0.32.2 │ └─JuliaDB [a93385a2] log: │ ├─possible versions are: 0.9.0-0.13.1 or uninstalled │ └─restricted to versions * by an explicit requirement, leaving only versions 0.9.0-0.13.1 └─restricted by compatibility requirements with EvalMetrics [251d5f9e] to versions: 0.33.0-0.33.19 — no versions left └─EvalMetrics [251d5f9e] log: ├─possible versions are: 0.1.0-0.2.1 or uninstalled └─restricted to versions * by an explicit requirement, leaving only versions 0.1.0-0.2.1 Given that these unsatisfiable requirements errors are reasonably common, I'll try to go through the error message step by step as it can sometimes be daunting for new users to parse, before discussing possible workarounds and commenting on your specific version clash at the end. Understanding the error message What's happening here? I'm trying to add JuliaDB and EvalMetrics to this new environment, so the Pkg resolver tries to determine the most up to date versions of both packages which will work together. In this instance, the attempt fails - there are no compatible versions of JuliaDB and EvalMetrics which can coexist. This is because they both depend on StatsBase, but require non-overlapping version numbers of this package. Here's how to read the error message: StatsBase [2913bbd2] log: ├─possible versions are: 0.24.0-0.33.19 or uninstalled This just tells us that versions 0.24 to 0.33.19 exist in the General registry and can be installed. Next the message tells us which versions are allowed by the different packages we're adding to the environment: ├─restricted by compatibility requirements with JuliaDB [a93385a2] to versions: 0.24.0-0.32.2 so JuliaDB works at most with StatsBase version 0.32.2, and does not admit any newer versions. The next part tells us the restrictions placed on the installation of JuliaDB itself: │ └─JuliaDB [a93385a2] log: │ ├─possible versions are: 0.9.0-0.13.1 or uninstalled │ └─restricted to versions * by an explicit requirement, leaving only versions 0.9.0-0.13.1 This just tells us that JuliaDB has versions 0.9-0.13.1 available, and that we have asked for any JuliaDB version to be installed (restricted to versions * by an explicit requirement means all versions are allowed, but the package has to be installed as we explicitly asked for it). So we have required JuliaDB, and JuliaDB tells us that it only works with StatsBase up to version 0.32.2. Why is this a problem? Look at the next part of the error: └─restricted by compatibility requirements with EvalMetrics [251d5f9e] to versions: 0.33.0-0.33.19 — no versions left EvalMetrics only works with StatsBase versions 0.33.0 and up - which is disallowed by JuliaDB. Workarounds So what's to do? It's not clear from your question, but frequently these errors appear where users dump all packages they are installing into their default environment ((#v1.5) in your case). As the number of packages in the default environment goes up, so does the likelihood that any two packages share some dependency and require incompatible versions of that dependency. Therefore the first "workaround" is to work in project specific environments - there's a good explainer in the Pkg.jl documentation here. In short, make a new folder for whatever analysis you're working on and do ] activate . in that folder to start a new environment to which you only add the specific packages needed for that analysis. (There are other benefits to this, the most important imho being reproducibility of your analysis, but these are unrelated to your question). In this case, if you don't actually need JuliaDB and EvalMetrics at the same time, you can just make two environments and avoid the conflict. What if you do actually need two packages which are incompatible with each other? In this case, a relaxation of the compatibility requirements of one of the packages is needed (often referred to as "bumping the compat bounds" of a package. Two ways to go about this: The easy way (which might take a while though!) is to file an issue on the repo of one of the packages involved. In this case it would likely be JuliaDB, and indeed that issue already exists (more on this later) The harder (but likely faster) way is to just do it yourself - often packages are unaffected by a version change in one of their dependencies, as they didn't actually rely on the functionality of the dependency that broke in the update. To check whether this is the case, you can ]dev the package and update the dependency, then see whether it still works. If it doesn't you can try updating the package as necessary and make a PR to upstream your changes. I'm not going to pretend that this is easy or a "normal" thing that's expected of Julia users, but I will say that developing Julia packages is comparatively easy (as most things are written in all Julia, no hidden C/C++ code like in R/Python), and the community is likely to receive such an attempt of a new user well and help you out where they can - just post on the Julia Discourse if you get stuck. Specific comment on JuliaDB I noted above that the issue asking for a bump of the version bound for StatsBase already exists in the JuliaDB repo. You'll also see that the issue is quite old (almost a year at this point), and if you check the [Project.toml](https://github.com/JuliaData/JuliaDB.jl/blob/main/Project.toml) file, which sets the compat bounds, on the main branch you'll also see that there's been a PR 16 months ago to bump those bounds, but no new version has been released since 2020. That is to say, you happened on a package which has effectively abandoned - see a related issue here which advises users to switch to Dagger.jl to work with large distributed tables. If you're now thinking "wait did I just read through all this only to find out that I shouldn't be using JuliaDB" then I suppose you're right, although I hope that the rest of the answer contained useful information which will help you with future compat issues!
Is the R package AWR.Athena, vulnerable to the security issue in log4j?
AWR.Athena package as per CRAN imports RJDBC, rJava. Hence suspecting if it is vulnerable to the recent Apache Log4j security issue. So information on this would be really helpful.
Implications of not installing packages from source?
Occasionally after install.packages(): Do you want to install from sources the package which needs compilation? (Yes/no/cancel) What are the implications of not installing a package that needs compilation from source(s)? What I know so far I guess it's safer to always select 'Yes', and I suspect this may reinstall (/update) shared libraries or extensions (e.g. c, c++, fortran). Also some very general ideas/advantages listed here
Incompatible packages in meteor (autoform-file and velocity)
This might be a rather general question: How to deal with incompatible packages in meteor? In my case I'm using the testing suite velocity and jasmine, which is not compatible with the current version of autoform-file package. My html-reporter would just continue executing the test without actually getting the result due to the following error: Errors prevented startup: While selecting package versions: error: Potentially incompatible change required to top-level dependency: yogiben:autoform-file 0.3.0, was 0.4.2. Constraints on package "yogiben:autoform-file": To allow potentially incompatible changes to top-level dependencies, you must pass --allow-incompatible-update on the command line. Your application has errors. Waiting for file change. I was thinking of downgrading the autoform-file package to 0.3.0 however, I didn't find any information on how to do it.
I was able to install the package at a specific version using: meteor add yogiben:autoform-file#=0.3.0 That worked for me.
php55w-common conflict when adding php-fpm
So I am trying to add php-fpm to have it work with nginx. But the issue I am having is I already have some other php stuff in there like php-common and now it's having conflicts. So centOS is suggesting I use --skip-broken but i'm not sure if that's the best way to go? Would there be any issues if I do end up using that to install php-fpm? Loaded plugins: fastestmirror, security Loading mirror speeds from cached hostfile * base: dallas.tx.mirror.xygenhosting.com * epel: www.gtlib.gatech.edu * extras: mirror.millry.co * updates: mirror.cs.vt.edu * webtatic: us-east.repo.webtatic.com Setting up Install Process Resolving Dependencies --> Running transaction check ---> Package php-fpm.x86_64 0:5.3.3-27.el6_5.1 will be installed --> Processing Dependency: php-common(x86-64) = 5.3.3-27.el6_5.1 for package: php-fpm-5.3.3-27.el6_5.1.x86_64 --> Running transaction check ---> Package php-common.x86_64 0:5.3.3-27.el6_5.1 will be installed --> Processing Conflict: php55w-common-5.5.16-1.w6.x86_64 conflicts php-common < 5.5.0 --> Finished Dependency Resolution Error: php55w-common conflicts with php-common-5.3.3-27.el6_5.1.x86_64 You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest
If you know what you are doing, you can upgrade PHP by: yum install yum-plugin-replace yum replace php-common --replace-with=php55w-common It will likely give you a message “WARNING: Unable to resolve all providers …”. This is normal, and you can continue by tying “y“. You will be given a chance to see what packages will be installed and removed before again being given a chance to confirm. Then you can continue with your previous installation command