my project structure looks like :
my_project
-dags
-config
however on airflow dashboard I see an error Broken DAG pointing to this line : from config.setup_configs import somemethod
and yields this err:
Broken DAG: [/usr/local/airflow/dags/airflow_foo.py] No module named 'config'
although the directory exists
According to documentation Airflow has, by default, three directories to path
AIRFLOW_HOME/dags
AIRFLOW_HOME/config
AIRFLOW_HOME/plugins
Any other path has to be added to system path, as described in airflow module management
For sake of simplicity, I added my module mymodule.py to AIRFLOW_HOME/plugins and I can import them successfully.
from mymodule import my_method
So, in your case, if you rename configurations to plugins and update import statement into DAG,
from setup_configs import somemethod
it should work.
You need to move the config directory into the dags folder and create an empty __init__.py file within the config folder. Then, it should work.
Related
I'm using airflow, and I want to import some plugins for my dags
The problem get into the play: when I'm trying to use them, follow by:
from dags_folder.plugins import fetchingData
I get this error
Module name "dags_folder" not found!
(here is my layout)
I've setting up with the airflow.cfg, but it doesn't work!
notes: I'm using conda-env
You need to add the folder dags_folder to python path:
export PYTHONPATH=${PYTHONPATH}:/path/to/dags_folder
then you could import fetchingData from plugins:
from plugins import fetchingData
(if you want to import it from dags_folder.plugins, you need to add the parent folder of dags_folder to the python path).
If you need this plugin in one module, you can add the plugins package to python path pragmatically:
import sys
sys.path.append("/path/to/dags_folder")
from plugins import fetchingData
I have the following folder-structure
airflow/
|_dag/
as far as I understand, airflow uses the "airflow" folder as root i.e I assume that everything placed in "airflow" would be able to be imported.
Say I have different projects with tasks placed in the following structure
airflow/
|_dag/
| |_ mydag.py
|
|_myprojects/
|_projectone/
| |_tasks/
| |_ mytask.py
|_projecttwo/
|_tasks/
|_ mytask.py
then I would assume that I in mydag.py should be able to import mytask from a given project like
#mydag.py
from myprojects.projectone import tasks
but I get a DAG import error; ModuleNotFoundError: No module named 'myprojects'.
Is this doable, or should I (somehow) change the airflows PYTHONPATH (and in that case, where is that done?)
Note, I have created __init__.py files in the folders.
One option is to set the path just before importing myproject:
#mydag.py
import sys
# it's important for path being inserted before importing `myproject`
sys.path.insert(0, "..")
from myproject import tasks
The second option is to move myproject under the dag folder:
airflow
+-- dag
+--- myproject
The third option is to move some logic into plugins/ folder.
https://airflow.apache.org/docs/apache-airflow/stable/modules_management.html
As far as I know the import command needs the destination file name everytime when we try to import the files.
svn import file1.txt https:\\server\path\file1.txt
My question is is there a way similar to Linux where we can copy/import multiple files at once,something like
svn import *.* https:\\server\path\
Is this possible? I don't want to use a looping process to achieve this.
The description of the command should give you a clue: "svn import — Commit an unversioned file or tree into the repository".
Give svn import a directory instead of a single file, and all files inside will be imported, recursively by default. svn:ignore and related properties and settings apply, for filtering out files you don't want added.
Following this tutorial on windows: https://github.com/divio/django-cms-tutorial/blob/master/Step%201%20-%20Initial%20Setup.md
typing this line:
djangocms -p . my_site
I have this error:
OSError: Could not import settings 'ira_site.settings'
(Is it on sys.path? Is there an import error in the settings file?):
No module named 'ira_site'
ira_site is the name of an unrelated django (no-cms) project.
I've grepped (using ransack) the whole "c:\python34" and "virtual_env" folders to look for "ira_site". Not a trace of it.
I've opened every settings.py file I could open. To check of installed apps.
Not trace of it. Why would djangocms installer try to import the settings of a totally unrelated django project ? What am I missing ?
Allright, it was quite stupid. In case someone has the same problem:
your environment variables:
DJANGO_SETTINGS_MODULE
Must be set so django-admin.py can find it ( as the installer is also creating the project ). What I did is:
reset it to global-settings.py
run the installer.
And once installation was over, put it back to the settings.py file of the newly created site.
I'm new to jython and failing utterly at importing a java class within a jar.
What I am trying to do is write a wrapper shell script which calls a jython script. I can not allowed to edit the jython at all, so adding jars to sys.path within that jython script is not possible.
Error
y", line 17, in
from com.polarland.testModule.cache import CacheInterface
ImportError: No module named polarland
I've added the jar which contains the above package with name of TestModule.jar to PATH, ClASSPATH and JYTHONPATH with no avail. I'm worried this is due to the name of the jar but am not sure.
Any advice would be greatly appreciated!!
In your shell script use:
export CLASSPATH=TestModule.jar:$CLASSPATH
jython ...
In my case setting CLASSPATH is enough. Remember to use full path name and remember to use good .jar name (testmodule.jar and TestModul.jar are different). Maybe you use wrong file rights. Try file command to check if you can read that file. Example for one of jars I use:
mn$ file junit-4.1.jar
junit-4.1.jar: Zip archive data, at least v2.0 to extract
I was getting same issue.
I tried below function and worked well!
>>> import sys
>>> sys.path.append('/path/to/helloworld.jar')
>>> from com.leosoto import HelloWorld
For more info:
http://blog.leosoto.com/2008/07/jython-import-logic.html