I'm setting up an AWS MWAA instance and I have a problem with import custom plugins.
My local project structure looks like this:
airflow-project
├── dags
│ └── dag1.py
└── plugins
├── __init__.py
└── operators
├── __init__.py
└── customopertaor.py
I tried to match this structure in the s3 bucket:
s3://{my-bucket-name}
└── DAGS
├── dags
│ └── dag1.py
└── plugins
├── __init__.py
└── operators
├── __init__.py
└── customopertaor.py
However when I use the custom operator on the local project the import works like this -
from operators import customOperators
and on the MWAA it only recognize imports like this -
from plugins.operators import customOperators
Is there a way to get the MWAA recognize the import as the local (from operators)?
should I upload the files in certain way to the s3?
I also tried to upload a plugins.zip file but it didn't work:
s3://{my-bucket-name}
├── DAGS
│ └── dags
│ └── dag1.py
└── plugins.zip
I believe the proper way is to place your custom python modules in the plugin.zip file. This file will be uploaded to MWAA and gets extracted to /usr/local/airflow/plugins/. I believe the DAGs are placed in the very same folder.
AWS has published a User Guide that gives some good explanation and examples.
I had the same problem and i solve it looking inside my .zip file. In my case the structure inside .zip file creates an extra folder called plugins. Review this using unzip -l plugins.zip and look the tree generated. This is my working structure:
Archive: plugins.zip
Length Date Time Name
0 10-18-2021 11:39 hooks/
125 10-18-2021 11:40 hooks/my_airflow_hook.py
0 10-18-2021 11:40 sensors/
359 10-18-2021 11:40 sensors/my_airflow_sensor.py
395 10-18-2021 13:28 my_airflow_plugin.py
0 10-18-2021 11:42 operators/
437 10-18-2021 11:42 operators/hello_operator.py
480 10-18-2021 11:42 operators/my_airflow_operator.py
you can import the plugin as a python-module like below
import imp
customopertaor = imp.load_source('customopertaor','/usr/local/airflow/plugins/operators/customopertaor.py')
Your plugin folder tree looks good.
You need to restart your airflow environment to take the new plugin into account.
Alternatively you can use the config reload_on_plugin_change.
Related
I am currently building a backend using FastAPI and I am facing some issues to run the backend using poetry scripts. This is my project structure:
├── backend
└── src
└── asgi.py
└── Dockerfile
└── poetry.lock
└── pyproject.toml
pyproject.toml
[tool.poetry]
name = "backend"
version = "0.1.0"
description = ""
authors = ["Pierre-Alexandre35 <46579114+pamousset75#users.noreply.github.com>"]
readme = "README.md"
[tool.poetry.dependencies]
python = "^3.9"
uvicorn = "^0.17.6"
fastapi = "^0.78.0"
psycopg2 = "^2.9.3"
jwt = "^1.3.1"
python-multipart = "^0.0.5"
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
[tool.poetry.scripts]
foo='asgi:__main__'
If I am running poetry run python asgi.py, it is working perfectly but if I am using poetry foo script, I am getting No file/folder found for package backend. Those are all combinaisons I tried and I have the same error for every poetry run foo:
foo='asgi:main'
foo='backend.asgi:__main__'
foo='backend.asgi:main'
foo='backend.asgi:.'
Your project structure does not seem to be correct. Assuming backend is the package u are trying to create.
Use this structure
└── pyproject.toml
└── poetry.lock
└── README.md
├── backend
└── src
└── asgi.py
└── Dockerfile
└── __init__.py
Also in scripts use. (Assuming you are trying to run main with foo)
[tool.poetry.scripts]
foo='backend.asgi:__main__'
I have a project with a proto files in a:
$ tree proto/
proto/
├── common
│ └── request.proto
├── file
│ ├── file.proto
│ └── file_service.proto
├── job
│ ├── job.proto
│ └── job_service.proto
├── pool
│ ├── pool.proto
│ └── pool_service.proto
└── worker
├── worker.proto
└── worker_service.proto
5 directories, 9 files
I want to generate a one single file from worker_service.proto but these file has imports from common.
Is there a option in grpc_tools.protoc to generate one single python file?
Or is there a tool to generate one proto file?
Based on the information, I guess by generate one Python file means: instead of generate one Python file for messages (*_pb2.py) and one Python file for services (*_pb2_grpc.py), you hope to concatenate both of them into one Python file. To take a look at the generated file content, here is the Helloworld example.
Combining the two output file is currently not supported by the gRPC Python ProtoBuf plugin (unlike Java/Go). You can post a feature request and add more detail about your use case: https://github.com/grpc/grpc/issues
Is there a way to have a Firebase/Google cloud function with this kind of architecture with cli command (firebase deploy --only functions) ?
Expected:
.
└── functions/
├── function_using_axios/
│ ├── node_modules/
│ ├── package.json
│ └── index.js
└── function_using_moment/
├── node_modules/
├── package.json
└── index.js
Currently, my archi look like this:
.
└── functions/
├── node_modules/
├── package.json
├── index.js
├── function_using_axios.js
└── function_using_moment.js
The fact is, i have a lot of useless packages dependencies for some functions.
And it increase cold start time.
I know this is possible with the web UI.
WEB UI Exemple:
List
One package for one Function
My Current Archi see on WEB UI, one Package for all functions:
Any idea ?
Thanks.
When deploying through Firebase there can only be a single index.js file, although gcloud may any different in this respect.
To ensure you only load the dependencies that each function needs, move the require for each dependency into the functions that need it:
exports.usageStats = functions.https.onRequest((request, response) => {
const module = require('your-dependency');
// ...
});
Also see:
the Firebase documentation on organizing functions, which shows a way to have the functions over multiple files (although you'll still need to import/export them all in index.js).
I'm having this rather annoying trouble that I'm having some issues understanding when my Sass compiles.
I have a SCSS folder that compiles to a CSS folder in the root of my site, but in my SCSS folder I have individual folders for layout, utilities, etc.
The issue I am having is within my SCSS structure I have a vendors folder that houses everything from Bootstrap Grid, Font Awesome, etc. To keep the structure of the vendors folder neat and tidy I like to keep each vendor add on in a seperate folder. I use the following watch command:
sass --watch scss:css
Here's the file structure where ... is the files within the folder.
project-name/
├── scss/
│ └── style.scss
└── vendors/
├── bootstrap-grid
│ └── ...
└── fontawesome
└── ...
The issue I have when compiling to the CSS folder, the vendors folder and its contents are being compiled to the CSS folder:
project-name/
├── css/
│ └── style.css
└── vendors/
├── bootstrap-grid
│ └── ...
└── fontawesome
└── ...
Thanks in advance :)
Try using this command:
sass --watch scss/style.scss:css/style.css
I really like eslint for es6 projects. Previously I've used it for new projects. Now I want to add it to a legacy project.
Fixing all pre-existing lint issues in one go is too much effort. Can I configure eslint (in .eslintrc.js) to only check files where I've explicitly enabled it with /* eslint-enable */ or similar?
ESLint has no default-disabled state that can be toggled by a file comment. You might be able to use .eslintignore for this purpose, however. You can ignore everything and then gradually whitelist files as you migrate them by using ! to un-ignore individual files. For example:
.
├── .eslintignore
├── .eslintrc.js
├── package.json
├── node_modules
│ └── ...
├── src
│ ├── index.js
│ └── module
│ └── foo.js
└── yarn.lock
Then your .eslintignore could look something like this:
# Start by ignoring everything by default
src/**/*.js
# Enable linting just for some files
!src/module/foo.js
In this case, src/index.js would be ignored, but it would lint src/module/foo.js.