Currently running airflow 1.10.15. Wanted to perform some tests before upgrading to 2+. So installed pip install apache-airflow-upgrade-check in the scheduler pod which installed successfully. So I then run the command airflow upgrade_check but it did not return the results that I expected. It's giving me this output in terminal
[2021-06-15 21:02:38,637] {{settings.py:233}} DEBUG - Setting up DB connection pool (PID 15732)
[2021-06-15 21:02:38,637] {{settings.py:300}} DEBUG - settings.prepare_engine_args(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=15732
[2021-06-15 21:02:38,735] {{sentry.py:179}} DEBUG - Could not configure Sentry: No module named 'blinker', using DummySentry instead.
[2021-06-15 21:02:38,754] {{__init__.py:45}} DEBUG - Cannot import due to doesn't look like a module path
[2021-06-15 21:02:38,916] {{cli_action_loggers.py:42}} DEBUG - Adding <function default_action_log at 0x7f9a637c3a70> to pre execution callback
Please install apache-airflow-upgrade-check distribution from PyPI to perform upgrade checks
[2021-06-15 21:02:39,266] {{settings.py:310}} DEBUG - Disposing DB connection pool (PID 15732)
What am I missing?
Updated 6/16/2021: I verified if the package was installed, I did see the package in list:
...
apache-airflow 1.10.15
apache-airflow-upgrade-check 1.3.0
apispec 1.3.3
argcomplete 1.12.2
...
The problem I had was the container was running as a non-root user which was defined in Dockerfile. If I install the package in the running pod, it will install in some local directory and when executing the airflow upgrade_check command, it cannot find the package. To work-around this issue, I need to add the packages in Dockerfile so it will be included when creating the docker image.
Related
I am trying to setup Apache Atlas on my Windows 10 machine. I have the following prereqs:
Maven Version 3.8.6
JDK Version 1.8.0
Python 2.7.7
I then pull Apache Atlas and I am using tags/release-2.3.0. Once downloaded, I run the following commands in Powershell:
set MAVEN_OPTS="-Xms2g -Xmx2g
mvn clean install
This starts running a set of tests, which fail. I get the following error:
in org.apache.atlas.kafka.KafkaNotificationTest [ERROR]
org.apache.atlas.kafka.KafkaNotificationTest.testReceiveKafkaMessages
Time elapsed: 8.178 s <<< FAILURE! java.lang.NoClassDefFoundError:
org/apache/atlas/kafka/AtlasKafkaMessage
at org.apache.atlas.kafka.KafkaNotificationTest.testReceiveKafkaMessages(KafkaNotificationTest.java:63)
Caused by: java.lang.ClassNotFoundException:
org.apache.atlas.kafka.AtlasKafkaMessage
at org.apache.atlas.kafka.KafkaNotificationTest.testReceiveKafkaMessages(KafkaNotificationTest.java:63)
How can I resolve this issue to install Atlas?
I am testing a deployment of dbt within Cloud composer. On my local machine (Ubuntu 20.04) I have got success in running the dbt models with airflow. When running on Google Cloud composer I get the following error
{subprocess.py:74} INFO - Output:
{subprocess.py:78} INFO - Running with dbt=0.21.0
{subprocess.py:78} INFO - Encountered an error while reading the project:
{subprocess.py:78} INFO - ERROR: Runtime Error
{subprocess.py:78} INFO - at path ['name']: 'jaffle_shop' does not match '^[^\\d\\W]\\w*$'
{subprocess.py:78} INFO -
{subprocess.py:78} INFO - Error encountered in /home/airflow/gcs/dags/dbt_project.yml
{subprocess.py:78} INFO - Encountered an error:
{subprocess.py:78} INFO - Runtime Error
{subprocess.py:78} INFO - Could not run dbt
{subprocess.py:82} INFO - Command exited with return code 2
{taskinstance.py:1503} ERROR - Task failed with exception
We are using a BashOperator to run dbt models in Airflow.
Initially had some problems with dependencies but they were solved.
Using a standard dbt_project.yml file with a single model just to test how this works.
Another way is to use Docker but we need try if this works.
Edit
Versions
dbt: 0.21.0
cloud-composer: 1.17.1
airflow: 2.1.2
Pypi Packages
airflow-dbt: 0.4.0
dbt: 0.21.0
jsonschema: 3.1 (Added this as Pypi was throwing an error about the version
I really appreciate if anyone can help
Pete
The problem here is the jsonschema dependency. Version 3.1.0 does not work, while versions 3.1.1 and 3.2.0 will work--and should also work within Composer's dependency requirements.
There looks to have been an issue with switching to js-regex for the jsonschema folks in 3.1.0, which caused them to revert back to regular re in 3.1.1.
There are some details here, and a couple of related issues described here and here.
In general, it would be much nicer if Cloud Composer supported virtual environments to avoid this entire dependency-collision mess, but apparently Google does not support that approach.
I want to deploy my project using Pivotal from Bamboo. During the deployment of my .NetCore 3.1 project, I'm getting the error "Unable to install dotnet-runtime: could not find a version of dotnet-runtime to install". The deployment stack is set to cflinuxfs3.
I've tried specifying dotnet runtime version (3.1 and 3.1.5) in .csproj file but it got the same error so I removed it.
I've also created a buildpack.yml file and specified the .netcore sdk version as 3.1.x in there.
I'm sharing the property group from csproj and the manifest.yml file.
-- Error Log --
Downloading app package...
Downloaded app package (6.4M)
-----> Dotnet-Core Buildpack version 2.3.12
-----> Supplying Dotnet Core
-----> Installing libunwind 1.4.0
Copy[/tmp/buildpacks/b7bd2a36eb284e204524c677d2dbbfa2/dependencies/a54cfafce3d2a14e4f96777d5fd471f4/libunwind_1.4.0_linux_noarch_cflinuxfs3_05e08b22.tgz]
using the default SDK
-----> Installing dotnet-sdk 3.1.301
Copy[/tmp/buildpacks/b7bd2a36eb284e204524c677d2dbbfa2/dependencies/b422801667458a262ba26a3117b93e4f/dotnet-sdk_3.1.301_linux_x64_any-stack_80a771e4.tar.xz]
-----> Installing dotnet-runtime 3.1.5
Copy[/tmp/buildpacks/b7bd2a36eb284e204524c677d2dbbfa2/dependencies/40b5a460524fe7f1795c876445349a31/dotnet-runtime_3.1.5_linux_x64_any-stack_1aa84612.tar.xz]
-----> Finalizing Dotnet Core
ERROR Unable to install dotnet-runtime: could not find a version of dotnet-runtime to install Failed to compile droplet: Failed to
run finalize script: exit status 12 Exit status 223 Cell
ffe536bc-374c-46b5-a620-e3366dab6b22 stopping instance
b71ecb48-0083-4fd1-8f59-3a77b05de0e0
Error staging application: App staging failed in the buildpack compile
phase
-- .csproj --
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>netcoreapp3.1</TargetFramework>
<ServerGarbageCollection>true</ServerGarbageCollection>
<ConcurrentGarbageCollection>true</ConcurrentGarbageCollection>
<IsTransformWebConfigDisabled>true</IsTransformWebConfigDisabled>
<GenerateDocumentationFile>true</GenerateDocumentationFile>
<NoWarn>$(NoWarn);1591</NoWarn>
</PropertyGroup>
-- manifest.yml --
---
applications:
- name: projectName
memory: 1G
timeout: 240
instances: 1
routes:
- route: routeAddress
services:
- ARCHITECTURE.SERVICES-TST.ServiceRegistry
- ProjectName.Redis
- ProjectName.Dynatrace
- ProjectName.ConfigServer
env:
DT_TAGS: ((appName)) ((dc))-((env))
TZ: country
ASPNETCORE_ENVIRONMENT: environmentName
SPRING_CLOUD_CONFIG_LABEL: configLabel
Ok, here's what is happening.
The code is looking at the RuntimeFrameworkVersion in your .csproj file.
When that is empty, it's going to look at the TargetFramework and attempt to find a substring match using the regex netcoreapp(.*).
It is expecting that to have a single match, but in your case it does not, so you get that error message: could not find a version of dotnet-runtime to install.
Based on your .csproj file, it seems like what you have should be OK though. I did a quick check and the regex works and returns the version.
You could try setting RuntimeFrameworkVersion which would take a slightly different code path and might work for you.
You are trying to perform a source-based deployment. You could try dotnet publish first followed by a Framework Dependent Deployment which again would force a different code path.
It might be that the buildpack isn't finding your file, but it's hard to say without more info.
I installed cloudify3.4 according to the cloudify DOCS. When I install the manager, and executed like this:
# cfy bootstrap --install-plugins -p openstack-manager-blueprint.yaml -i openstack-manager-blueprint-inputs.yaml
an error occurred:
[ERROR] Bootstrap failed! (Workflow failed: Task failed 'neutron_plugin.floatingip.create' -> Expected exactly one object of type network with match {'name': u'178d7438-ca18-4df6-a5d0-dd11a53155a5'} but there are 0)
I have already installed
"cloudify_openstack_plugin-1.4-py27-none-linux_x86_64-centos-Core.wgn"
"cloudify_openstack_plugin-1.4-py27-none-linux_x86_64-redhat-Maipo.wgn"
So, how to solve this error?
Thank you to everyone who helped me!
In Cloudify version 3.4, the Cloudify manager can only be installed (bootstrapped) on either CentOS 7.x or RHEL 7.x.
See more details :
http://docs.getcloudify.org/3.4.0/manager/prerequisites/
FYI: There are no limitations in host agents.
That means that once the Cloudify manager is alive, it will enable you to deploy anything on every operating system.
When install DotNet Core on my windows 10 laptop(DotNetCore.1.0.0.RC2-VS2015Tools.Preview1),the runtime package setup failed.
I found some error information from the install log file:
web developer tool info:
[037C:1EC0][2016-06-19T11:07:26]e000: Error 0x80070643: Failed to install MSI package.
[037C:1EC0][2016-06-19T11:07:26]e000: Error 0x80070643: Failed to execute MSI package.
[2E70:3228][2016-06-19T11:07:26]e000: Error 0x80070643: Failed to configure per-machine MSI package.
[2E70:3228][2016-06-19T11:07:26]i319: Applied execute package: WebToolsExtensionsVS14, result: 0x80070643, restart: None
[2E70:3228][2016-06-19T11:07:26]e000: Error 0x80070643: Failed to execute MSI package.
[037C:1EC0][2016-06-19T11:07:26]i318: Skipped rollback of package: WebToolsExtensionsVS14, action: Uninstall, already: Absent
[2E70:3228][2016-06-19T11:07:26]i319: Applied rollback package: WebToolsExtensionsVS14, result: 0x0, restart: None
[037C:1EC0][2016-06-19T11:07:26]i351: Removing cached package: WebToolsExtensionsVS14, from path: C:\ProgramData\Package Cache\{1F275091-F18D-37F3-8A70-8E6CE66BB1A8}v14.1.20512.0\
[037C:1EC0][2016-06-19T11:07:26]i372: Session end, registration key: SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{22041006-8484-4b8d-a13c-40189695de2f}, resume: ARP, restart: None, disable resume: No
[037C:1EC0][2016-06-19T11:07:26]i371: Updating session, registration key: SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{22041006-8484-4b8d-a13c-40189695de2f}, resume: ARP, restart initiated: No, disable resume: No
[2E70:3228][2016-06-19T11:07:26]i399: Apply complete, result: 0x80070643, restart: None, ba requested restart: No
Another Solution is to:
go to the folder where you have downloaded the installer
open the command prompt by Shift+right click -> Open command window here
Type - dotnetcore.1.0.0-vs2015tools.preview2 SKIP_VSU_CHECK=1 and hit enter
This should skip the VS update check and start the installation normally.
I was able to fix this issue by
turning off Antivirus
clearing the %temp% folder
running the DotNet Core (DotNetCore.1.0.0.RC2-VS2015Tools.Preview1) again in repair mode
After applying the other answers here the install was still failing for me. Periodically during the install, a pop-up warning dialog would open stating "The account already exists". I reviewed the Programs and Features list and saw 3 different entries for the VS2015Tools preview. I right-click Uninstall'ed each of them (which removed each one almost immediately), and tried the install again. This time it completed without errors.