Is there any way to import custom python modules into dag file without mixing dag environs and sys.path? Can't use something like
environ["PROJECT_HOME"] = "/path/to/some/project/files"
# import certain project files
sys.path.append(environ["PROJECT_HOME"])
import mymodule
because it the sys.path is shared among all dags and this causes problems (eg. sharing of values between dag definitions) if want to import modules from different places that have the same name for different dag definitions (and if there are many dags, this is hard to keep track of).
The docs for using packaged dags (which seemed like a solution) do not seem to avoid the problem
the zip file will be inserted at the beginning of module search list (sys.path) and as such it will be available to any other code that resides within the same interpreter.
Anyone with more airflow knowledge know how to handle this kind of situation?
* Differs from linked-to question in that is less specific about implementation
Ended up doing something like this:
if os.path.isfile("%s/path/to/specific/module/%s.py" % (PROJECT_HOME, file_name)):
import imp
f = imp.load_source("custom_module", "%s/path/to/specific/module/%s.py" % (PROJECT_HOME, file_name))
df = f.myfunc(sparkSession, df)
To get the needed module file explicitly from known paths, based on the SO post here.
Related
I am currently starting my first Vue3 project (I have done many Vue2 projects) and am currently facing errors when importing components.
In vue2/webpack, I was used to doing imports like this (in fact, phpstorm/webstorm is importing them for me like this)
import PageBase from "./components/PageBase";
This however yields
[vite] Internal server error: Failed to resolve import "./components/PageBase" from "src/App.vue". Does the file exist?
At first, I thought that it was due to the # vs ./ notation.
But later I noticed that it is actually about the .vue extension at the end.
import PageBase from "#/components/PageBase.vue";
and
import PageBase from "./components/PageBase.vue";
work just fine.
Is this the desired behaviour?
Its a bit confusing and inconvenient, as my IDE is importing it by default like this.
Ps: I am using an out of the box Vue3 setup like here: https://vuejs.org/guide/quick-start.html#with-build-tools
Thanks for any clarifications.
Im looking to separate out my complex API structure so that I have the following structure. I am wondering. Is there a way to mount all files under the users/ folder to the same ./api/v1/users route? And the same for projects/ ? One key point of consideration is the fact that I will have dynamic routes defined within these files too (e.g. ./projects/<project_id>)
In shiny, to accomplish something like this Id use source('file.R', local=TRUE) but Plumber doesn't work in the same way.
The reason I am structuring it this way is to reduce complexity during development (as opposed to adding multiple verbs to the same endpoint).
+-- v1/
|+-- users/
|+----- GET.R
|+----- POST.R
|+-- projects/
|+----- GET.R
|+----- POST.R
Ive tested mounting but unfortunately cannot mount multiple files from each folder to the same route name. See the example code
v2 <- plumber::Plumber$new("api/v1/projects/GET.R")
root$mount(paste0(ROOT_URI,"/v1"), v2)
v1 <- plumber::Plumber$new("api/v1/projects/POST.R")
root$mount(paste0(ROOT_URI,"/v1"), v1)
(within the GET.R and POST.R files are each one function named "projects" that handle one of two verbs)
The answer is sort of. Using the 'here' package allows you to import functions defined within files relative to your plumber file. Then in your plumber file you can fill in your decorator and place your function after it.
I am trying to import my JSON files into the same directory but once I import the first one, the latter one overrides the former one:
The first import:
After the second one:
As you can see above, the first file was placed inside the latter one. How can I import multiple JSON files in the same directory level?
Assuming that you import the data using the console, any data in the JSON file replaces the existing location where you run the import. There is no way to change this behavior.
What you can do is import the data to a different location in the console. So if you open the recipes node to import the first JSON, and open the searches node for the second JSON, the two imports won't overwrite each other.
If you want to import them into the root of the database, you'll have to merge the two JSON files yourself and then import them in one go.
I have a DAG with one DataflowTemplateOperator that can deal with different json files. When I trigger the dag I pass some parameters via {{dag_run.conf['param1']}} and works fine.
The issue I have is trying to rename the task_id based on param1.
i.e. task_id="df_operator_read_object_json_file_{{dag_run.conf['param1']}}",
it complains about only alphanumeric characters
or
task_id="df_operator_read_object_json_file_{}".format(dag_run.conf['param1']),
it does not recognise dag_run plus the alpha issue.
The whole idea behind this is that when I see at the dataflow jobs console and job has failed I know who the offender is based on param1. Dataflow Job names are based on task_id like this:
df-operator-read-object-json-file-8b9eecec
and what I need is this:
df-operator-read-object-param1-json-file-8b9eecec
Any ideas if this is possible?
There is no need to generate new operator per file.
DataflowTemplatedJobStartOperator has job_name parameter which is also templated so can be used with Jinja.
I didn't test it but this should work:
from airflow.providers.google.cloud.operators.dataflow import DataflowTemplatedJobStartOperator
op = DataflowTemplatedJobStartOperator(
task_id="df_operator_read_object_json_file",
job_name= "df_operator_read_object_json_file_{{dag_run.conf['param1']}}"
template='gs://dataflow-templates/your_template',
location='europe-west3',
)
I've created a simple module and needed to add some translations messages.
Next step was to create a "translations" folder with a po file named "pt-pt.po" inside (my language defined is pt-pt) and programmatically didn't worry to import those messages.
I thought that those messages were imported when module activation but that didn't happen.
I thought wrongly? :) Or what I'm missing?
I have to import manually per-environment?
Thanks.
Does the pt-pt.po file contain strings (I don't understand from that programmatically part)? If not, you'll need something like this http://drupal.org/project/potx to extract the strings from your module.
If the translation file has strings but those were not imported keep in mind that the translations are read and imported only when a module is installed, not activated. So if the module was activated after it was deactivated at some point then the translations will not be re-read.
In this case this could solve the issue: http://drupal.org/project/po_re_importer.