I need execute Data Fusion pipelines from Composer, using de operatos for this:
from airflow.providers.google.cloud.operators.datafusion import (
CloudDataFusionCreateInstanceOperator,
CloudDataFusionCreatePipelineOperator,
CloudDataFusionDeleteInstanceOperator,
CloudDataFusionDeletePipelineOperator,
CloudDataFusionGetInstanceOperator,
CloudDataFusionListPipelinesOperator,
CloudDataFusionRestartInstanceOperator,
CloudDataFusionStartPipelineOperator,
CloudDataFusionStopPipelineOperator,
CloudDataFusionUpdateInstanceOperator,
)
The issue I have is about modulo "apache-airflow-backport-providers-google", with the support of this links i knew what I need to use this modulo:
reference to install the modulo in airflow instance (answered by #Gonzalo Pérez Fernández): https://airflow.apache.org/docs/apache-airflow-providers-google/stable/operators/cloud/datafusion.html
when i tried to install python dependency on Composer like PyPi Package i get this error:
UPDATE operation on this environment failed 7 minutes ago with the following error message:
Failed to install PyPI packages.
apache-airflow-providers-google 5.0.0 has requirement google-ads>=12.0.0, but you have google-ads 7.0.0. Check the Cloud Build log at https://console.cloud.google.com/cloud-build/builds/a2ecf37a-4c47-4770-9489-6fb65e87d82f?project=341768372632 for details. For detailed instructions see https://cloud.google.com/composer/docs/troubleshooting-package-installation
the log deail is:
apache-airflow-providers-google 5.0.0 has requirement google-ads>=12.0.0, but you have google-ads 7.0.0.
apache-airflow-backport-providers-google 2021.3.3 has requirement apache-airflow~=1.10, but you have apache-airflow 2.1.2+composer.
The command '/bin/sh -c bash installer.sh $COMPOSER_PYTHON_VERSION fail' returned a non-zero code: 1
ERROR
ERROR: build step 0 "gcr.io/cloud-builders/docker" failed: step exited with non-zero status: 1
is there any way to use de module "apache-airflow-backport-providers-google" without depedency issues on composer instance?, or What would be the best way to use data fusion operators no need to change or parse package versions in python?.
Composer Image version used:
composer-1.17.0-airflow-2.1.2
Thanks.
There is no need to install apache-airflow-backport-providers-google in Airflow 2.0+. This package actually backports Airflow 2 operators into Airflow 1.10.*. In addition, in Composer version composer-1.17.0-airflow-2.1.2 the apache-airflow-providers-google==5.0.0 package is already installed according to the documentation. You should be able to import the Data Fusion operators with the code snippet you posted as is.
However, if this is not the case, you should probably handle the conflict shown in the logs when trying to reinstall apache-airflow-providers-google==5.0.0:
apache-airflow-providers-google 5.0.0 has requirement google-ads>=12.0.0, but you have google-ads 7.0.0.
You can add the requirement for google-ads=12.0.0 in your PyPi dependencies and see if it works.
Related
I am installing mpi4py using pip using the command pip3 install mpi4py, on a centos7 system.
I get this error:
/glade/u/apps/ch/opt/openmpi/4.0.5/intel/19.0.5/bin/mpicc _configtest.o -L/glade/u/apps/ch/opt/python/3.7.9/gnu/9.1.0/lib -Lbuild/temp.linux-x86_64-3.7 -Wl,--enable-new-dtags,-R/glade/u/apps/ch/opt/python/3.7.9/gnu/9.1.0/lib -o _configtest
/usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../x86_64-suse-linux/bin/ld: /opt/pbs/lib/libpbs.so.0: undefined reference to `sec_set_context'
failure.
removing: _configtest.c _configtest.o
error: Cannot link MPI programs. Check your configuration!!!
----------------------------------------
Does anybody have any tips?
In this case on Cheyenne's HPC cluster, you will likely need to add these arguments to your compilation/link commands -L/opt/pbs/lib -Wl,-rpath,/opt/pbs/lib -lsec or reload the ncarcompilers module.
Nevertheless, I recommend you reach out to CISL's helpdesk at NCAR for any similar problems you encounter on our system as there are dedicated teams of people to help users resolve issues like this.
On Azure ML Workspace Notebook, I'm trying to get my workspace instance, as seen at
https://learn.microsoft.com/en-us/azure/machine-learning/tutorial-auto-train-models#configure-workspace.
I have a config file and I am running the notebook in an Azure compute instance.
I tried to execute Workspace.from_config().
As a result, I'm getting the 'MSIAuthentication' object has no attribute 'get_token' error.
I tried to submit both MsiAuthentication and InteractiveLoginAuthentication, as suggested in
https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/manage-azureml-service/authentication-in-azureml/authentication-in-azureml.ipynb.
There are 2 solutions I've found:
1.- Use the kernel "Python 3.6 - AzureML"
2.- pip install azureml-core --upgrade
This will upgrade
azureml-core to 1.32.0
But will downgrade:
azure-mgmt-resource to 13.0.0 (was 18.0.0)
azure-mgmt-storage down to 11.2.0 (was 18.0.0)
urllib3 to 1.26.5 (was 1.26.6)
This upgrade / downgrade allows the same package versions as in the python 3.6 anaconda install
I am trying to build and deploy this bookdown project with GitHub Actions. One of the chapters uses the keras R package, which means I need to install Conda (or set up a virtual environment). At the end of the Miniconda installation command, there is an error when trying to collect metadata.
2020-06-24T04:47:59.7495480Z * Miniconda has been successfully installed at '/Users/runner/Library/r-miniconda'.
2020-06-24T04:47:59.7496060Z [1] "/Users/runner/Library/r-miniconda"
2020-06-24T04:48:00.3909040Z * Project '~/runners/2.263.0/work/drake/drake' loaded. [renv 0.10.0]
2020-06-24T04:48:00.7964920Z * The project and lockfile are out of sync -- use `renv::status()` for more details.
2020-06-24T04:48:00.7968340Z Warning message:
2020-06-24T04:48:00.7969190Z Project requested R version '3.6.0' but '4.0.1' is currently being used
2020-06-24T04:48:05.2408080Z Collecting package metadata (current_repodata.json): ...working... failed
2020-06-24T04:48:05.2410390Z
2020-06-24T04:48:05.2410820Z NotWritableError: The current user does not have write permissions to a required path.
2020-06-24T04:48:05.2411080Z path: /usr/local/miniconda/pkgs/cache/b89cf7bf.json
2020-06-24T04:48:05.2411230Z uid: 501
2020-06-24T04:48:05.2411350Z gid: 20
2020-06-24T04:48:05.2411430Z
2020-06-24T04:48:05.2411690Z If you feel that permissions on this path are set incorrectly, you can manually
2020-06-24T04:48:05.2411940Z change them by executing
2020-06-24T04:48:05.2412010Z
2020-06-24T04:48:05.2412260Z $ sudo chown 501:20 /usr/local/miniconda/pkgs/cache/b89cf7bf.json
2020-06-24T04:48:05.2412330Z
2020-06-24T04:48:05.2413470Z In general, it's not advisable to use 'sudo conda'.
2020-06-24T04:48:05.2413570Z
2020-06-24T04:48:05.2414250Z
2020-06-24T04:48:05.2886400Z ##[error]Error: Error 1 occurred creating conda environment r-reticulate
2020-06-24T04:48:05.2890770Z Execution halted
2020-06-24T04:48:05.3050700Z ##[error]Process completed with exit code 1.
The full job log is here.
Depending on how R is set up, this post might be helpful for you. You might need to configure the .Renviron file.
Unable to change python path in reticulate (R)
I'm having issues with Azure Machine Learning SDK for R: "module 'azureml' has no attribute 'core'"...
For reasons that aren't my own, I have to use azureml to apply machine learning (my own stuff, written in R) to data from our data warehouse that is put in the blob storage. The modelled output should be put back into the blob storage so it can be accessed from the data warehouse.
I've written the code in R on my local machine (stored in a git repo). Preferably, I'd find some method to pull my code from git into a pipeline in the azureml environment so that it can be directly run whenever new data is available in the blob storage.
I've embarked on a tutorial-spree and found this seemingly relevant walkthrough: Train and deploy your first model with Azure ML (and this one).
But... after trying all I could think of, I'm stuck on the first steps. After installing all (or at least.. that's what I think) packages, modules, apps etc, and running the following code in RStudio:
library(azuremlsdk)
existing_ws <- get_workspace(name = name,
subscription_id = subscription_id,
resource_group)
I run into an error that I haven't been able to fix:
AttributeError: module 'azureml' has no attribute 'core'
It seems that the azuerml is supposed to have an attribute "core", but when looking at it more closely, there is indeed no such attribute.
The function "get_workspace()" is trying to access: "azureml$core$Workspace$get".
I found that "azuerML$Workspace" does exist, but then I can't figure out how to make that work.
Can anyone explain to me why I'm encountering this error?
Does anyone know of a better tutorial on how to connect my R code the azureml's cloud service?
Any pointers in the right direction are much appreciated!
EDITS - still not solved:
After advice from others, I double, triple and quadruple checked the installation.
I updated R and I'm now running:
R.version
platform x86_64-w64-mingw32
arch x86_64
os mingw32
system x86_64, mingw32
status
major 3
minor 6.2
year 2019
month 12
day 12
svn rev 77560
language R
version.string R version 3.6.2 (2019-12-12)
nickname Dark and Stormy Night
I installed Conda with Python 3.6.10.
I installed the azuremlsdk R package (I tried both provided options).
I then realized that there are some inconsistencies with the versions of the azure-modules, so I also tried installing it with the keyword 'multi-arch':
remotes::install_cran('azuremlsdk', repos = 'http://cran.us.r-project.org', INSTALL_opts=c("--no-multiarch"))
Then, I installed the azureml python sdk.
I had a look at all the versions again (using python -m pip freeze):
azure-common==1.1.24
azure-graphrbac==0.61.1
azure-mgmt-authorization==0.60.0
azure-mgmt-containerregistry==2.8.0
azure-mgmt-keyvault==2.0.0
azure-mgmt-resource==7.0.0
azure-mgmt-storage==7.1.0
azureml==0.2.7
azureml-automl-core==1.0.83.1
azureml-core==1.0.69
azureml-dataprep==1.1.36
azureml-dataprep-native==13.2.0
azureml-pipeline==1.0.69
azureml-pipeline-core==1.0.69
azureml-pipeline-steps==1.0.69
azureml-sdk==1.0.69
azureml-telemetry==1.0.69
azureml-train==1.0.69
azureml-train-automl-client==1.0.83
azureml-train-core==1.0.69
azureml-train-restclients-hyperdrive==1.0.69
As I was surprised to see all the 1.0.69 versions, instead of the 1.0.83 versions, I re-installed the azureml python sdk using:
azuremlsdk::install_azureml(version = "1.0.83")
This worked, in the sense that indeed all versions are now 1.0.83:
azure-common==1.1.24
azure-graphrbac==0.61.1
azure-mgmt-authorization==0.60.0
azure-mgmt-containerregistry==2.8.0
azure-mgmt-keyvault==2.0.0
azure-mgmt-resource==7.0.0
azure-mgmt-storage==7.1.0
azureml==0.2.7
azureml-automl-core==1.0.83.1
azureml-core==1.0.83
azureml-dataprep==1.1.36
azureml-dataprep-native==13.2.0
azureml-pipeline==1.0.83
azureml-pipeline-core==1.0.83
azureml-pipeline-steps==1.0.83
azureml-sdk==1.0.83
azureml-telemetry==1.0.83
azureml-train==1.0.83
azureml-train-automl-client==1.0.83
azureml-train-core==1.0.83
azureml-train-restclients-hyperdrive==1.0.83
But still... I get the error with the missing core. I get it both when running:
library(azuremlsdk)
get_current_run()
and also when running:
library(azuremlsdk)
existing_ws <- get_workspace(name = name,
subscription_id = subscription_id,
resource_group)
Note that the first time running this code after starting up RStudio, I get the error:
Error in py_get_attr_impl(x, name, silent) :
AttributeError: module 'azureml' has no attribute '_base_sdk_common'
And every time after that I get this error:
Error in py_get_attr_impl(x, name, silent) :
AttributeError: module 'azureml' has no attribute 'core'
Any help would be much appreciated!
This issue was introduced by the latest reticulate 1.14 release, in which reticulate would create a default r-reticulate conda environment. Since Azure ML was installing the python SDK in an environment named r-azureml, the r-reticulate environment used by reticulate was missing the python SDK. A fix for this issue was addressed in a PR and has been merged into master. Please install from GitHub for now if you have reticulate version 1.14 and are running into this issue. We will be releasing an update to CRAN shortly.
I seemed to have fixed the issue by specifically installing the python package azureml AND azureml.core:
python -m pip install azureml
and then...
python -m pip install azureml.core
I did this for the Conda version that was called by R (r-reticulate). It's a bit odd to not be able to use the Conda environment 'r-azureml' without R switching back to 'r-reticulate', but ah well... at least I don't get my 'azureml' has no attribute 'core' anymore.
I want to use mysql form ploneformgen, but I
Can't buildout plone.
buildout log http://pastie.org/5345272.js
Getting required 'MySQL-python>=1.2.1'
required by Products.ZMySQLDA 3.1.1.
We have no distributions for MySQL-python that satisfies 'MySQL-python>=1.2.1'.
Getting distribution for 'MySQL-python>=1.2.1'.
Running easy_install:
/usr/local/Plone/Python-2.6/bin/python "-c" "from setuptools.command.easy_install import main; main()" "-mUNxd" "/usr/local/Plone/zeocluster/../buildout-cache/eggs/tmpDSODu0" "-Z" "/usr/local/Plone/buildout-cache/downloads/dist/MySQL-python-1.2.4c1.zip"
path=/usr/local/Plone/buildout-cache/eggs/distribute-0.6.21-py2.6.egg
Processing MySQL-python-1.2.4c1.zip
Running MySQL-python-1.2.4c1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-gFbWLf/MySQL-python-1.2.4c1/egg-dist-tmp-16g1TE
The required version of distribute (>=0.6.28) is not available,
and can't be installed while this script is running. Please
install a more recent version first, using
'easy_install -U distribute'.
(Currently using distribute 0.6.19 (/usr/local/Plone/Python-2.6/lib/python2.6/site-packages/distribute-0.6.19-py2.6.egg))
error: Setup script exited with 2
An error occured when trying to install MySQL-python 1.2.4c1. Look above this message for any errors that were output by easy_install.
While:
Installing client1.
Getting distribution for 'MySQL-python>=1.2.1'.
Error: Couldn't install: MySQL-python 1.2.4c1
*************** PICKED VERSIONS ****************
[versions]
Products.PloneFormGen = 1.7.1
Products.ZMySQLDA = 3.1.1
collective.classifieds = 1.6
plone.app.ldap = 1.2.8
quintagroup.dropdownmenu = 1.2.5
quintagroup.pfg.captcha = 1.0.5
zettwerk.ui = 1.1.1
buildout conf http://pastie.org/5345300
some links:
http://blog.mysqlboy.com/2010/08/installing-mysqldb-python-module.html
http://plone.293351.n2.nabble.com/Plone-amp-MySQL-No-quot-Z-MYSQL-Database-Connection-quot-from-ZMI-td5487160.html
It appears the MySQLdb egg requires a newer version of distribute:
The required version of distribute (>=0.6.28) is not available,
and
(Currently using distribute 0.6.19 (/usr/local/Plone/Python-2.6/lib/python2.6/site-packages/distribute-0.6.19-py2.6.egg))
Upgrade your distribute egg first; if you are using the unified installer, for example, versions.cfg pins the version. If so, edit versions.cfg to correct the version number there:
[versions]
...
# Buildout infrastructure
...
distribute = 0.6.28
While you have a perfectly good answer for the specific problem, I highly recommend forgetting about ZMySQLDA and use SQLAlchemyDA which gives you access to any database supported by SQLAlchemy (I've used all of MySQL, PostGreSQL, Oracle, SQLServer) with a single product, and is better supported.