I deployed Airflow 2.0 on ubuntu using pip commands. Have migrated from 1.10.13 to 2.0. But after db inint , I am getting below warnings. Please suggest why am I getting these warnings.
I tried running example dags and they run successfully.
WARNI [airflow.providers_manager] The provider for package 'apache-airflow-providers-http' could not be registered from because providers for that package name have already been registered
WARNI [airflow.providers_manager] The provider for package 'apache-airflow-providers-sqlite' could not be registered from because providers for that package name have already been registered
WARNI [airflow.providers_manager] The provider for package 'apache-airflow-providers-imap' could not be registered from because providers for that package name have already been registered
WARNI [airflow.providers_manager] The provider for package 'apache-airflow-providers-ftp' could not be registered from because providers for that package name have already been registered
WARNI [airflow.providers_manager] The provider for package 'apache-airflow-providers-http' could not be registered from because providers for that package name have already been registered
WARNI [airflow.providers_manager] The provider for package 'apache-airflow-providers-sqlite' could not be registered from because providers for that package name have already been registered
WARNI [airflow.providers_manager] The provider for package 'apache-airflow-providers-imap' could not be registered from because providers for that package name have already been registered
WARNI [airflow.providers_manager] The provider for package 'apache-airflow-providers-ftp' could not be registered from because providers for that package name have already been registered
I think you have a kind of installation where you have both "provider packages" installed and Airflow with source code with providers in the PYTHONPATH. I guess you installed Airflow from sources with -e flag rather than from the PyPI package in which case you will get those warning.
BTW. Those warnings will be gone in 2.0.1 https://github.com/apache/airflow/commit/18df31d5d451299c510a9bf11946f17471f1e573
The warnings are harmless - but then the providers that are installed as packages are not used (the ones from airflow sources are).
The root cause of it that if you want to install Airflow as editable install from sources (basically when you want to contribute to Airflow) you should set INSTALL_PROVIDERS_FROM_SOURCES variable to true before installing Airflow (see https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#provider-packages) - this way you will not install the provider packages from PyPI.
I'm getting these same warnings, and I didn't install from source:
pip --use-deprecated legacy-resolver install "apache-airflow==${AIRFLOW_VERSION}" "apache-airflow[mysql]==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"
Related
I'm trying to install azureml-sdk in a Mac M1 with Python 3.9.15 and facing the following problems:
When I pip install azureml-sdk, it installs an old version (azureml==1.0.23)
If I run:
from azureml.core import Workspace
ws = Workspace.from_config()
I get the following error:
AttributeError: 'BasicTokenAuthentication' object has no attribute 'get_token'
And if I try to install the latest azureml-sdk version with
pip install azureml-sdk==1.48.0
I get the following error:
ERROR: Cannot install azureml-dataset-runtime because these package versions have conflicting dependencies.
The conflict is caused by:
azureml-dataprep 4.8.3 depends on azureml-dataprep-native<39.0.0 and >=38.0.0
azureml-dataprep 4.8.2 depends on azureml-dataprep-native<39.0.0 and >=38.0.0
azureml-dataprep 4.8.1 depends on azureml-dataprep-native<39.0.0 and >=38.0.0
azureml-dataprep 4.8.0 depends on azureml-dataprep-native<39.0.0 and >=38.0.0
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
Any ideas? Thanks in advance
Regarding the first error you cited, this error is typically experienced when the provided login credentials for the Azure Machine Learning workspace are incorrect.
Verify the access rights for the workspace on the service principal you are using for authentication.
Make sure the configuration file you use for authentication is formatted correctly and has all the necessary elements.
Use the newly created service principal and configuration file to authenticate.
The second error is because these package versions have conflicting dependencies, typically experienced when there is a conflict between the dependencies of the installed packages.
You might attempt the following solutions to troubleshoot this problem:
Try downgrading or upgrading the conflicting packages' versions until you find one that works with the Azure Machine Learning SDK.
To avoid installing the dependencies, try installing the Azure Machine Learning SDK using the —no-deps parameter.
If the issue still exists, you can try deleting and then reinstalling every package linked to Azure Machine Learning.
Hi I'm trying to deploy an app on the server, however, I get the following errors
* May be unable to deploy package dependency "rClr" could not
determine a repository URL for the source CRAN.
* May be unable to deploy package dependency "tlf" could not determine
a repository URL for the source CRAN.
Unable to determine the source location for some packages. Packages
should be installed from a package repository like CRAN or a version
control system. Check that options(repos) refers to a package
repository containing the needed package versions.
The backbone packages can not be installed from Cran and have to be done manually, hence the errors. How can this be fixed in order to deploy the app on the server?
Thanks for all help
I'm running Airflow 1.10.6 on Google Cloud Composer with Python 3. To initiate a dataflow job from the composer environment I need to have the PyPi package google-cloud-datastore. When I try to add this package through either the interface or cloud shell, the build fails. It raises the following error:
UPDATE operation on this environment failed 1 hour ago with the following error message:
Failed to install PyPI packages.
If I check the logs of Google Build for the Kubernetes Engine I see the following error:
I 2020-03-04T14:36:23.939075607Z [0mgoogle-cloud-datastore 1.11.0 has requirement google-cloud-core<2.0dev,>=1.0.3, but you have google-cloud-core 0.29.1.
I 2020-03-04T14:36:23.939669242Z google-cloud-datastore 1.11.0 has requirement google-api-core[grpc]<2.0.0dev,>=1.14.0, but you have google-api-core 1.8.1.
So google-cloud-datastore requires google-cloud-core>=1.0.3. I tried to explicitly set google-cloud-core to 1.0.3, but then I get the following error:
I 2020-03-05T08:13:18.539300693Z [0mgoogle-cloud-logging 1.9.1 has requirement google-cloud-core<0.30dev,>=0.29.0, but you have google-cloud-core 1.0.3.
I 2020-03-05T08:13:18.539878367Z google-cloud-bigtable 0.32.0 has requirement google-cloud-core<0.30dev,>=0.29.0, but you have google-cloud-core 1.0.3.
I 2020-03-05T08:13:18.540308937Z google-cloud-bigquery 1.8.1 has requirement google-cloud-core<0.30dev,>=0.29.0, but you have google-cloud-core 1.0.3.
So the ranges of versions required for the packages are not overlapping. Does anyone know if this can be fixed? Please let me know.
There is a compatibility issue with google-cloud-datastore in the main container of Composer, and that the latest version of this package could not be installed. The Composer product team has confirmed the issue and is working on it.
Attempting to install an older version of google-cloud-datastore may be a workaround before the root cause is fixed. I have tried with 1.7.4 version and got positive result:
I hope it helps.
In case you (or someone else reading this) needs to use the latest version of google-cloud-datastore (==1.11.0), I was able to install it in my environment (composer-1.9.1-airflow-1.10.6) by adding recent versions of the following libraries to the dependencies too:
googleapis-common-protos==1.6.0
google-cloud-spanner==1.14.0
google-cloud-bigtable==1.2.1
google-cloud-logging==1.14.0
google-cloud-bigquery==1.22
I'm trying to connect RStudio on a compute instance to the Oracle autonomous warehouse.
I've covered the following steps:
Provisioned Autonomous Data Warehouse
(https://docs.oracle.com/en/cloud/paas/autonomous-data-warehouse-cloud/tutorial-provision-adw/)
Stored the DB connection Wallet;
Installed Oracle Instant client basic package: https://www.oracle.com/technetwork/topics/winx64soft-089540.html
Downloaded and placed in the same directory as the instant client: SDK package & ODBC package
included Instant package path in environment veriable PATH ("C:/oracle_client/instantclient_18_5") and create OCI_LIB64 system variable with the same path
installed GTools checking the PATH options
Extracted the wallet to: C:\oracle_client\instantclient_18_5\network\admin
Now I'm experience an inssue installing the ROracle package within RStudio.
I'm using the following script:
Sys.setenv(OCI_LIB64="C:/oracle_client/instantclient_18_5")
install.packages("ROracle")
The error I got is the include headers have not been found:
Warning in strptime(xx, f, tz = tz) :
unable to identify current timezone 'C':
please set environment variable 'TZ'
* installing *source* package 'ROracle' ...
** package 'ROracle' successfully unpacked and MD5 sums checked
** using staged installation
Oracle Client Shared Library 64-bit - 18.5.0.0.0 Operating in Instant Client mode.
found Instant Client C:/oracle_client/instantclient_18_5
found Instant Client SDK C:/oracle_client/instantclient_18_5/sdk/include
ERROR: cannot find Oracle Client include headers in C:/oracle_client/instantclient_18_5/sdk/include.
Please set OCI_INC to correct location.
ERROR: configuration failed for package 'ROracle'
* removing 'C:/Users/opc/Documents/R/win-library/3.6/ROracle'
Warning in install.packages :
installation of package ‘ROracle’ had non-zero exit status
Everybody is saying to also set OCI_INC like this (but I do not have a directory in the manner "...client_1/oci/include/:
Sys.setenv(OCI_INC="C:/Oracle/product/11.2.0/client_1/oci/include")
Any idea what am I missing for the ROracle or in general?
R version: 3.6
Operating system: Windows Server on a compute instance
According to the documentation (https://docs.oracle.com/en/database/oracle/r-enterprise/1.5.1/oread/installing-oracle-database-instant-client.html#GUID-8C502BDD-C1A1-4EA0-9E88-D2F8452D4AC2) you need to download "Instant Client Package - SDK". When I checked the webpage for instant client the links for the SDK version were absent.
However,if you go here: https://www.oracle.com/database/technologies/instant-client/downloads.html and select you O/S on the following page there should be a link for the SDK version. For example, I use a Mac and the correct download is here:https://download.oracle.com/otn_software/mac/instantclient/193000/instantclient-sdk-macos.x64-19.3.0.0.0dbru.zip
Once I unzipped that file, the directory structure had an .../include folder with .h files that I think the installer needs to complete. Therefore, the correct settings on my Mac would be:
OCI_LIB64 set to /Users/keithlaker/Downloads/instantclient_19_3
OCI_INC set to /Users/keithlaker/Downloads/instantclient_19_3/sdk/include
One thing to note...I am not familiar with R but in broad general terms Oracle's in-database implementation of R (in-database R script repository and R object Datastore) that is part of the now free EE "Advanced Analytics" option is not part of Autonomous Database.
Therefore, your R project should be able to connect but you will not be able to use any of the usual in-database R packages because they don't exist.
Hope this helps.
Just found a thread on our internal forum which is worth sharing. The above might allow you to install the R client but ADB (ADW/ATP) does not support the use of ore.connect() and ore.disconnect(). Support for R is on our product map, see this presentation from OpenWorld 2019 - https://static.rainfocus.com/oracle/oow19/sess/1553766982670001yXnT/PF/CON4719%20OOW%20SF%202019%20-%20Hornick%20-%20OML%20Overview%20and%20Roadmap%20v1.4_1568740954169001k3Df.pdf
I'm trying to experiment with SignalR by adding it to an existing (but small) ASP.Net (.Net 4.6.1, VS 2017) web application. However, I'm having real problems getting the Nuget package installed; more specifically the Microsoft.Owin component (which appears to be a dependency).
I've followed the instructions here, but as soon as I try to install Owin - either explicitly, or indirectly by trying to install the main SignalR package, and regardless of whether it's via Package Manager or Package Manager Console - I get the error:
PM> Install-Package Microsoft.Owin
Attempting to gather dependency information for package 'Microsoft.Owin.4.0.0' with respect to project
'SignalRProject', targeting '.NETFramework,Version=v4.6.1'
Gathering dependency information took 43.64 ms Attempting to resolve
dependencies for package 'Microsoft.Owin.4.0.0' with
DependencyBehavior 'Lowest'
Resolving dependency information took 0 ms
Resolving actions to install package 'Microsoft.Owin.4.0.0'
Resolved
actions to install package 'Microsoft.Owin.4.0.0'
Retrieving package
'Microsoft.Owin 4.0.0' from 'nuget.org'.
Retrieving package 'Owin 1.0.0' from 'nuget.org'.
Adding package 'Owin.1.0.0' to folder 'c:\code\SignalRProject\packages'
Install failed. Rolling back...
Package 'Owin.1.0.0' does not exist in project
'SignalRProject'
Removing package 'Owin.1.0.0' from folder
'c:\code\SignalRProject\packages'
Install-Package : TF400024: The
change on c:\code\SignalRProject\packages\Owin.1.0\Owin.1.0.nupkg
cannot be undone because a file already exists at
c:\code\SignalRProject\packages\Owin.1.0\Owin.1.0.nupkg. The file
must be deleted from disk for the undo to succeed.
I've tried the Owin.1.0.nupkg file, and the parent directory itself, but after restarting VS I continue to get the error. I've tried upgrading to .Net 4.7.1, but to no effect. How can I find out what's causing the Owin install error?
You could donwload a running microsoft example and then compare with yours https://code.msdn.microsoft.com/Getting-Started-with-c366b2f3
Install-Package : TF400024: The change on
c:\code\SignalRProject\packages\Owin.1.0\Owin.1.0.nupkg cannot be
undone because a file already exists at
c:\code\SignalRProject\packages\Owin.1.0\Owin.1.0.nupkg. The file must
be deleted from disk for the undo to succeed.
This error is caused by TFVC source control. You should first check-in your changes and then install nuget packages
I've managed to work around my issue, and get Owin (and SignalR) installed by using the 3rd-party Paket Nuget alternative.
>paket.exe add Microsoft.Owin --project SignalRProject
Paket version 5.148.0
Dependencies files saved to c:\code\SignalRProject\paket.dependencies
Adding package Microsoft.Owin to c:\code\SignalRProject\paket.references into group Main
References file saved to c:\code\SignalRProject\paket.references
- Creating model and downloading packages.
- paket.references -> SignalRProject.csproj
Performance:
- Disk IO: 63 milliseconds
- Runtime: 6 seconds