I'm trying to understand how to migrate to Gatsby/NextJS our structure.
src
├── scripts
├── components
├── domains
│ ├── ca.cdn.domain-1.com
│ │ ├── global
│ │ └── pages
│ │ │ ├── page.html
│ │ │ └── page-2.html
│ ├── m.cdn.domain-2.de
│ │ ├── global
│ │ └── pages
│ │ │ ├── page.html
│ │ │ └── page-2.html
page.html files can have different html templates, but sharing some common components from src.
Currently, we generate static pages per domain with Gulp. Tried Gatsby today and pages can be only in src folder. Any suggestions on how to use Gatsby/NextJS with multiple domains in 1 repo?
Also more detailed I described the question here for NextJS.
Related
I'm developing a micro frontend, for this I use Webpack 5 Module Federation and ngx-build-plus builder on an Angular 14 application.
When I build my project I get the following files:
dist
├── my-webco
│ ├── 106.js
│ ├── 136.js
│ ├── 497.js
│ ├── .
│ ├── .
│ ├── .
│ ├── icon-font.woff
│ ├── icon-font.woff2
│ ├── index.html
│ ├── main.js
│ ├── remoteEntry.js
│ ├── style.css
So I want to import the main.js file into another application this way:
<script src="https://my-host/my-webco/main.js" type="text/javascript"></script>
<my-web-component></my-web-component>
The webcomponent works well but I have a 404 error on the font because my font is on the path "https://my-host/my-webco/font-icons.woff" and my application will load it on the path "https://my-host/font-icons.woff".
do you know where I can set up this path?
R blogdown is super convenient to publish Rmarkdown report using netlify.
In order to build a private blog and control access, I found netlify identity widget could be a solution. Useful reference are:
Authenticate users with Netlify Identity
netlify identity widget
From the README file in netlify identity widget
"You can add controls for the widget with HTML:
<!DOCTYPE html>
<html>
<head>
<title>A static website</title>
<!-- include the widget -->
<script type="text/javascript" src="https://identity.netlify.com/v1/netlify-identity-widget.js"></script>
</head>
<body>
<!-- Add a menu:
Log in / Sign up - when the user is not logged in
Username / Log out - when the user is logged in
-->
<div data-netlify-identity-menu></div>
<!-- Add a simpler button:
Simple button that will open the modal.
-->
<div data-netlify-identity-button>Login with Netlify Identity</div>
</body>
</html>
You can use this global object like this:
// Open the modal
netlifyIdentity.open();
// Get the current user:
// Available after on('init') is invoked
const user = netlifyIdentity.currentUser();
// Bind to events
netlifyIdentity.on('init', user => console.log('init', user));
netlifyIdentity.on('login', user => console.log('login', user));
netlifyIdentity.on('logout', () => console.log('Logged out'));
netlifyIdentity.on('error', err => console.error('Error', err));
netlifyIdentity.on('open', () => console.log('Widget opened'));
netlifyIdentity.on('close', () => console.log('Widget closed'));
// Unbind from events
netlifyIdentity.off('login'); // to unbind all registered handlers
netlifyIdentity.off('login', handler); // to unbind a single handler
// Close the modal
netlifyIdentity.close();
// Log out the user
netlifyIdentity.logout();
// Refresh the user's JWT
// Call in on('login') handler to ensure token refreshed after it expires (1hr)
// Note: this method returns a promise.
netlifyIdentity.refresh().then((jwt)=>console.log(jwt))
// Change language
netlifyIdentity.setLocale('en');
Which blogdown file should I add above code into?
How to integrate netlify identity widget into R blogdown?
Thanks a lot!
UPDATE:
I found two approaches might be helpful:
netlify-plugin-password-protection, which applies staticrypt.
Another tutorial: Password-protected pages on Netlify.
For reproducible example, the Hugo Theme I used in blogdown is pulp theme, the file structure is the following:
.
└── pulp
├── .editorconfig
├── .eslintrc.json
├── .gitignore
├── LICENSE.md
├── README.md
├── archetypes
│ └── default.md
├── assets
│ ├── css
│ │ ├── markdown-dark.css
│ │ ├── markdown.css
│ │ ├── style-dark.css
│ │ ├── style.css
│ │ └── syntax-highlight.css
│ └── js
│ ├── jquery-3.3.1.min.js
│ ├── jquery.mark.es6.min.js
│ ├── lunr.js
│ └── search.js
├── exampleSite
│ ├── .gitignore
│ ├── config.toml
│ ├── content
│ │ └── blog
│ │ ├── emoji-support.md
│ │ ├── markdown-syntax.md
│ │ ├── math-typesetting.mmark
│ │ ├── placeholder-text.md
│ │ └── rich-content.md
│ └── static
│ ├── css
│ │ └── custom.css
│ ├── img
│ │ ├── avatar.jpg
│ │ └── favicon.ico
│ └── js
│ └── custom.js
├── images
│ ├── logo.png
│ ├── screenshot.png
│ └── tn.png
├── layouts
│ ├── 404.html
│ ├── _default
│ │ ├── baseof.html
│ │ ├── list.html
│ │ ├── list.json
│ │ ├── single.html
│ │ └── terms.html
│ ├── index.html
│ └── partials
│ ├── footer.html
│ ├── head.html
│ └── header.html
├── package.json
├── resources
│ └── _gen
│ └── assets
│ ├── css
│ │ ├── style0.css_d11fe7b62c27961c87ecd0f2490357b9.content
│ │ └── style0.css_d11fe7b62c27961c87ecd0f2490357b9.json
│ └── js
│ ├── bundle0.js_d11fe7b62c27961c87ecd0f2490357b9.content
│ └── bundle0.js_d11fe7b62c27961c87ecd0f2490357b9.json
├── static
│ ├── .DS_Store
│ ├── categories
│ │ └── img
│ │ ├── avatar.png
│ │ ├── clear.png
│ │ ├── favicon.ico
│ │ └── search.png
│ └── img
│ ├── avatar-border.svg
│ ├── avatar.jpg
│ ├── bu.png
│ ├── clear.png
│ ├── favicon.ico
│ └── search.png
├── theme.toml
└── yarn.lock
I have this project currently running on ASP.NET MVC and using AngularJS for the front end.
I am using bower and gulp but want to migrate away to use new tooling, if possible webpack.
I have started the migration process, where all bower packages have been moved to package.json.
I need help figuring out how to set up webpack. And how the changes will affect the .NET Bundle.Config.cs file in App_Start.
Basic Project structure:
MyApp/
├── App_Start/
│ └── BundleConfig.cs
│
├── Controllers/
│ └── HomeController.cs
│
├── Content/
│ └── node_modules/
│ └── package.json
│
├── ng-app/
│ └── app.js
│ └── run.js
│ └── routes.js
│ └── controllers/
│ └── homeCtrl.js
│
├── ng-views/
│ └── home.html
│
└── Views/
├── Home/
│ └── Index.cshtml
│
└── Shared/
└── _Layout.cshtml
If anyone has some pointers/ articles/ guides, whatever help is welcomed.
If you need more info on my setup please ask.
I am trying to add qtcharts to meta-toolchain-qt5. My question is to which
recipes/files should I add qtcharts to have qtcharts on meta-toolchain-qt5
sdk?
I need to compile my program with qmake at
/opt/myimage/2.1.2/sysroots/x86_64-pokysdk-linux/usr/bin/qt5/qmake
I am not sure where should I put the qtcharts. Under meta-qt5 or
meta-myapplication/recipes-qt or somewhere else
here is poky directory:
poky$ tree -L 2
├── bitbake
│ └── ...
├── build
│ └── ...
├── meta
│ └── ...
├── meta-freescale-distro
│ └── ...
├── meta-fsl-arm
│ └── ...
├── meta-myqtapplication
│ ├── classes
│ ├── conf
│ ├── DOC
│ ├── licenses
│ ├── meta-patch
│ ├── README.md
│ ├── recipes-apps
│ ├── recipes-bsp
│ ├── recipes-connectivity
│ ├── recipes-core
│ ├── recipes-kernel
│ └── recipes-qt
├── meta-openembedded
│ └── ...
├── meta-poky
│ └── ...
├── meta-qt5
│ ├── classes
│ ├── conf
│ ├── COPYING.MIT
│ ├── files
│ ├── lib
│ ├── licenses
│ ├── README
│ ├── recipes-connectivity
│ ├── recipes-devtools
│ └── recipes-qt
├── meta-selftest
├── meta-skeleton
├── meta-yocto
├── meta-yocto-bsp
First of all, In Yocto/Bitbake while You are using meta-toolchain-qt5 recipe for generating SDK, You need to apply qtcharts as nativesdk package into nativesdk-packagegroup-qt5-toolchain-host.bb recipe, moving recipe to another localisation like poky/ will not apply them as rdepends package.
I see that qtcharts recipe is not available as nativesdk package, so You can extend BBCLASSEXTEND variable for this purpose, or use nativesdk class file.
My questions :
What is a good directory structure in order to organize your dags and tasks? (the dags examples show only couple of tasks)
I currently have my dags at the root of the dags folder and my tasks in separate directories, not sure is the way to do it ?
Should we use zip files ? https://github.com/apache/incubator-airflow/blob/a1f4227bee1a70531cfa90769149322513cb6f92/airflow/models.py#L280
I use something like this.
A project is normally something completely separate or unique. Perhaps DAGs to process files that we receive from a certain client which will be completely unrelated to everything else (almost certainly a separate database schema)
I have my operators, hooks, and some helper scripts (delete all Airflow data for a certain DAG, etc.) in a common folder
I used to have a single git repository for the entire Airflow folder, but now I have a separate git per project (makes it more organized and easier to grant permissions on Gitlab since projects are so unrelated). This means that each project folder also as a .git and .gitignore, etc as well
I tend to save the raw data and then 'rest' a modified copy of the data which is exactly what gets copied into the database. I have to heavily modify some of the raw data due to different formats from different clients (Excel, web scraping, HTML email scraping, flat files, queries from SalesForce or other database sources...)
Example tree:
├───dags
│ ├───common
│ │ ├───hooks
│ │ │ pysftp_hook.py
│ │ │
│ │ ├───operators
│ │ │ docker_sftp.py
│ │ │ postgres_templated_operator.py
│ │ │
│ │ └───scripts
│ │ delete.py
│ │
│ ├───project_1
│ │ │ dag_1.py
│ │ │ dag_2.py
│ │ │
│ │ └───sql
│ │ dim.sql
│ │ fact.sql
│ │ select.sql
│ │ update.sql
│ │ view.sql
│ │
│ └───project_2
│ │ dag_1.py
│ │ dag_2.py
│ │
│ └───sql
│ dim.sql
│ fact.sql
│ select.sql
│ update.sql
│ view.sql
│
└───data
├───project_1
│ ├───modified
│ │ file_20180101.csv
│ │ file_20180102.csv
│ │
│ └───raw
│ file_20180101.csv
│ file_20180102.csv
│
└───project_2
├───modified
│ file_20180101.csv
│ file_20180102.csv
│
└───raw
file_20180101.csv
file_20180102.csv
Update October 2021. I have a single repository for all projects now. All of my transformation scripts are in the plugins folder (which also contains hooks and operators - basically any code which I import into my DAGs). DAG code I try to keep pretty bare so it basically just dictates the schedules and where data is loaded to and from.
├───dags
│ │
│ ├───project_1
│ │ dag_1.py
│ │ dag_2.py
│ │
│ └───project_2
│ dag_1.py
│ dag_2.py
│
├───plugins
│ ├───hooks
│ │ pysftp_hook.py
| | servicenow_hook.py
│ │
│ ├───sensors
│ │ ftp_sensor.py
| | sql_sensor.py
| |
│ ├───operators
│ │ servicenow_to_azure_blob_operator.py
│ │ postgres_templated_operator.py
│ |
│ ├───scripts
│ ├───project_1
| | transform_cases.py
| | common.py
│ ├───project_2
| | transform_surveys.py
| | common.py
│ ├───common
| helper.py
| dataset_writer.py
| .airflowignore
| Dockerfile
| docker-stack-airflow.yml
I would love to benchmark folder structure with other people as well. Maybe it will depend on what you are using Airflow to but I will share my case. I am doing data pipelines to build a data warehouse so in high level I basically have two steps:
Dump a lot of data into a data-lake (directly accessible only to a few people)
Load data from data lake into a analytic database where the data will be modeled and exposed to dashboard applications (many sql queries to model the data)
Today I organize the files into three main folders that try to reflect the logic above:
├── dags
│ ├── dag_1.py
│ └── dag_2.py
├── data-lake
│ ├── data-source-1
│ └── data-source-2
└── dw
├── cubes
│ ├── cube_1.sql
│ └── cube_2.sql
├── dims
│ ├── dim_1.sql
│ └── dim_2.sql
└── facts
├── fact_1.sql
└── fact_2.sql
This is more or less my basic folder structure.
I am using Google Cloud Composer. I have to manage multiple projects with some additional SQL scripts and I want to sync everything via gsutil rsync Hence I use the following structure:
├───dags
│ │
│ ├───project_1
│ │
│ ├───dag_bag.py
│ │
│ ├───.airflowignore
│ │
│ ├───dag_1
│ │ dag.py
│ │ script.sql
│
├───plugins
│ │
│ ├───hooks
│ │ hook_1.py
│ │
│ ├───sensors
│ │ sensor_1.py
│ │
│ ├───operators
│ │ operator_1.py
And the file dag_bag.py containes these lines
from airflow.models import DagBag
dag_bag = DagBag(dag_folder="/home/airflow/gcs/dags/project_1", include_examples=False)