My Airflow's Git repository has the following structure:
├── dags/
│ └── example-dag1.py
│ └── example-dag2.py
├── include/
│ ├── operators/
│ │ └── my_operator.py
│ └── sensors/
│ └── my_sensor.py
│ └── dag_utils.py
└── tests/
I know I can sync DAGs from a Git repository as described here in the following way:
dags:
gitSync:
enabled: true
repo: git#github.com/<username>/<private-repo-name>.git
branch: <branch-name>
subPath: ""
sshKeySecret: airflow-ssh-secret
extraSecrets:
airflow-ssh-secret:
data: |
gitSshKey: '<base64-converted-ssh-private-key>'
What if I also want to sync custom operators, sensors and generally any other custom, reusable code from the include folder? How can I do it? Seems to be impossible right now but that's hard to believe as writing custom operators is standard in Airflow.
Related
I'm developing a micro frontend, for this I use Webpack 5 Module Federation and ngx-build-plus builder on an Angular 14 application.
When I build my project I get the following files:
dist
├── my-webco
│ ├── 106.js
│ ├── 136.js
│ ├── 497.js
│ ├── .
│ ├── .
│ ├── .
│ ├── icon-font.woff
│ ├── icon-font.woff2
│ ├── index.html
│ ├── main.js
│ ├── remoteEntry.js
│ ├── style.css
So I want to import the main.js file into another application this way:
<script src="https://my-host/my-webco/main.js" type="text/javascript"></script>
<my-web-component></my-web-component>
The webcomponent works well but I have a 404 error on the font because my font is on the path "https://my-host/my-webco/font-icons.woff" and my application will load it on the path "https://my-host/font-icons.woff".
do you know where I can set up this path?
Let's assume a config group foo and config files organized in the following directory structure:
conf
├── foo
│ ├── bar
│ │ ├── a.yaml
│ │ ├── b.yaml
│ │ ├── c.yaml
│ └── baz
│ ├── d.yaml
│ ├── e.yaml
│ └── f.yaml
Each of the yaml files sets the package to foo using # #package foo. When running the corresponding application, I can simply override foo by specifying something like foo=bar/a or foo=baz/f. Thereby, the sub-directories bar and baz indicate a certain category withing a larger set of possible configurations.
While this works fine for standard use in hydra, some more advanced features of hydra appear to be not compatible with this structure. For instance, I would like to use glob in conjunction with the directory structure like this foo=glob(bar/*) to sweep over all configs of a certain category. However, this does not appear to work as glob does not find any configs in this example. Also if I assign an invalid config to foo and hydra lists the available options, the list is empty.
This makes me wonder if structuring within a config group is a generally supported feature in hydra, and just some corner cases are not covered yet, or if I am using hydra wrong and directories should not be used for organizing configs in a group?
This is not recommended, but not explicitly prohibited.
There are scenarios where this can help, but as you have discovered it does not play well with some other features. A config group contains other config groups/configs.
Hydra 1.1 is adding support for recursive default lists which will make this kind of scenario more common.
See The Defaults List documentation page:
├── server
│ ├── db
│ │ ├── mysql.yaml
│ │ └── sqlite.yaml
│ └── apache.yaml
└── config.yaml
In the scenario from the example there, the entities under server/db are different than the entities under server, so such globing would not make sense.
I have this project currently running on ASP.NET MVC and using AngularJS for the front end.
I am using bower and gulp but want to migrate away to use new tooling, if possible webpack.
I have started the migration process, where all bower packages have been moved to package.json.
I need help figuring out how to set up webpack. And how the changes will affect the .NET Bundle.Config.cs file in App_Start.
Basic Project structure:
MyApp/
├── App_Start/
│ └── BundleConfig.cs
│
├── Controllers/
│ └── HomeController.cs
│
├── Content/
│ └── node_modules/
│ └── package.json
│
├── ng-app/
│ └── app.js
│ └── run.js
│ └── routes.js
│ └── controllers/
│ └── homeCtrl.js
│
├── ng-views/
│ └── home.html
│
└── Views/
├── Home/
│ └── Index.cshtml
│
└── Shared/
└── _Layout.cshtml
If anyone has some pointers/ articles/ guides, whatever help is welcomed.
If you need more info on my setup please ask.
Suppose I have a file structure as follows:
code
├── configure
├── configure.in
├── Demo
│ ├── experiments
│ │ ├── 1.txt
│ │ ├── 2.txt
│ │ ├── 3.txt
│ │ ├── 4.txt
│ │ ├── 5.txt
│ │ └── 6.txt
| |___test.cpp
I am trying to create a rule that uses each one of the files in txt and creates the following files:
1.test
2.test
3.test
4.test
5.test
If I had the rule:
%.test: %.txt
do something
How do I run it on every file in experiments eg:
make *.test
and then run them individually eg:
.\*.test?
I really need help with this as I have thousands of files to process in this way.
You can use $(wildcard):
TEST_SOURCES = $(wildcard *.txt)
TEST_OUTPUT = $(patsubst %.txt,%.test,${TEST_SOURCES})
all: ${TEST_OUTPUT}
Question:
How do I set up Typescript with Symfony with the minimum amount of configurations changes to Symphony’s configuration files?
Here are the points that this solution should solve:
Typescript MVC Pattern in a private typescript directory:
src > XBundle > Resources > private > typescript
Javascript bundles compiled in :
src > XBundle > Resources > public > js
the private directory should consist of multiple apps for different pages. (If an app requires it's own tsconfig.json file, that's fine)
an app is simply (for example) home.app.ts that imports (for example) a search.component.ts and a chat.component.ts
Compiled "apps" should be located in the public > js repository mentioned in point (2) and should be named (example taken from point (4)) home.bundle.js
In the public > js folder, there should only be x.bundle.js files
Adding the bundles to my twig files, and calling my view should immediately run the bundle. I should not have to add an extra script to call the "module" (this is the reason why I want to avoid AMD / System )
What I'm not looking for:
I'm not looking for a solution with react and angular but a general solution using the /web directory (or even the Resources directory in a bundle) at the root of a symfony project.
Most articles regarding this talk about symfony2 and try integrating react and angular.
I'm not looking for an installation tutorial for npm and tsc.
I don't need an automatic compile. I use Phpstorm so it does it automatically anyway.
I ended up using webpack for this to work. Props to #zerkms.
This is a work in progress, and could be better optimized.
Installation:
Install webpack, I personally installed it globally. (Unsure how to configure Phpstorm to use webpack, there doesn't seem to be a direct built in system for it)
Go to src > XBundle > Resources > private > typescript
npm init -y
Install the dev dependencies: npm install --save-dev awesome-typescript-loader and npm install --save-dev typescript
Side note:
#Morgan Touverey Quilling, recommends installing webpack locally, your choice:
npm install --save-dev webpack
Configuration:
Here is my folder structure:
├── XBundle/
│ ├── Controller
│ ├── Resources
│ │ ├── config
│ │ ├── private
│ │ │ ├── typescript
│ │ │ │ ├── components
│ │ │ │ │ ├── auth
│ │ │ │ │ │ ├── auth.component.ts
│ │ │ │ │ ├── search
│ │ │ │ │ │ ├── search.component.ts
│ │ │ │ ├── services
│ │ │ │ │ ├── http.service.ts
│ │ │ │ ├── node_modules
│ │ │ │ ├── package.json
│ │ │ │ ├── webpack.config.js
│ │ │ │ ├── tsconfig.json
│ │ ├── public
│ │ │ ├── js
│ │ │ │ ├── build
│ │ │ │ │ ├── auth.bundle.js
webpack.config.js
This config could probably be simplified much further.
For every bundle, a new config should be created pointing to the main file. Remember to rename the output bundle.
There shouldn't be more than one bundle per page. If you need (for example) the auth.bundle.js and the search.bundle.js on the home page, you should probably create a home.component.ts that uses auth.component.ts and search.component.ts and create a config in webpack.config.js to create the home.bundle.js
const path = require('path');
//Common configurations
let config = {
module: {
loaders: [
{ test: /\.ts$/, loader: 'awesome-typescript-loader' }
]
},
resolve: {
extensions: ['.ts']
}
};
//Copy and paste this for as many different bundles
//as required
let authConfig = Object.assign({}, config, {
entry: path.join(__dirname, 'components/auth','auth.component.ts'),
output: {
path: path.join(__dirname, '../../public/js/build'),
filename: 'auth.bundle.js',
library: 'XApp'
}
});
//Add each config here.
module.exports = [
authConfig
];
tsconfig.json
{
"compileOnSave": true,
"compilerOptions": {
"sourceMap": true,
"moduleResolution": "node",
"lib": ["dom", "es2015", "es2016"]
},
"exclude": [
"node_modules"
]
}
Run
Type webpack in the directory where webpack.config.js resides.
Adding JS to your Twig file
Somewhere in one of your templates.
{% block javascripts %}
{# More stuff here #}
{% javascripts
'#XBundle/Resources/public/js/build/auth.bundle.js'
%}
{# More stuff here #}
{% endjavascripts %}
{% endblock %}
And run the following commands for any new bundles.js created for the first time in the root directory of your symfony project:
php bin/console assets:install
php bin/console assetic:dump