I do store my software releases in Artifactory.
As a minimal example my repository structure looks like this:
my_repo/
├─ release-1/
├─ release-2/
├─ release-3/
│ ├─ reports/
│ │ ├─ trace.pzip
│ │ ├─ cov_report.html
│ ├─ release/
│ │ ├─ app.exe
│ ├─ debug/
│ │ ├─ app_debug.exe
Now I want to create different zipped deliveries. E.g. so that we do have an internal release package (release-3_woDebug.zip), one for the end customer (release-3_woreports.zip), etc...
my_repo/release-3/ contains all files of the release, to create the packages I want to exclude specific files.
my_repo/
├─ packages/
│ ├─ release-3-packages/
│ │ ├─ release-3_woDebug.zip
│ │ ├─ release-3_woreports.zip
├─ release-1/
├─ release-2/
├─ release-3/
Is there a simple REST or jFrog CLI command which I can use to create such zip files.
When looking at the documentation, the jf rt download (ref) command has an --exclusions option which looks already promising.
But is there a way to create such packages (zip files) without downloading the content first?
Related
i have this following structure under pages
├─ pages
│ ├─ index.tsx
│ ├─ betting
│ │ ├─ [bet].tsx
│ │ ├─ ippica live.tsx
into index.tsx i have a next/link that redirects to ippica live, in the url i have localhost:3000/betting/ippica%20live
but ippica live.tsx is not triggered but [bet].tsx does
i noticed that if i rename the file ippica live.tsx into ippica%20live.tsx all starts working but is very ugly and unconvenient
is there any other clean solution other than applying %20 into the file name?
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 7 months ago.
Improve this question
I have seen that new symfony projects have a diffrent directory stucture. Mainly the ressources-folder is not used anymore. I personally found that old structuring very nice, since all ressources were where they were used. When I start using bundles. it feels wrong to have all in the root folder. This is clearly an opinion, but I got a question from that, and the following question is not about opinion, but in practicality of the old structure in Symfony > 6:
What is the benefit of the new folder structure compared to the old approach with src/Ressources? Is there an advantage to the new structure which I don't see, or is it just a matter of taste and i can go the old way?
Your inputs are highly appreciated
Old approach
Root/
├─ src/
│ ├─ Ressources/
│ │ ├─ config/
│ │ ├─ public/
│ │ ├─ js/
│ │ ├─ styles/
│ │ ├─ views/
│ │ ├─ translations/
New approach
Root/
├─ src/
├─ public/
├─ templates/
├─ translations/
├─ config/
First of all, you can use old structure in new Symfony versions too! It all depends on few configurations.
To me, new structure is better, it makes more sense to move most things out of Resources directory into assets.
So it should really be:
├─ root/
│ ├─ config/
│ ├─ src/
│ ├─ public/
│ ├─ views/
│ ├─ translations/
│ ├─ assets/
│ │ ├─ js/
│ │ ├─ styles/
I learned of a way to run julia, so I can use the file structure of a package for my project.
Namely julia --project inside my developement directory. With this I can load all projects structured like projectName/src/projectName.jl inside the same folder.
An Example:
all my julia projects/
├─ project 1/
│ ├─ working with files in julia.jl
│ ├─ data.csv
├─ project 2/
│ ├─ project.toml
│ ├─ src/
│ │ ├─ project 2.jl
├─ project 3/
│ ├─ draft.uxf
│ ├─ .gitignore
│ ├─ project.toml
│ ├─ auto_compile.jl
│ ├─ src/
│ │ ├─ project 3.jl
With this file structure I want to call auto_compile.jl that does the following:
using Pkg
cd("..")
Pkg.activate(".")
Pkg.instatiate()
Pkg.add("PackageCompiler")
using PackageCompiler
create_app("Project 3", "Project 3 Compiled")
However, PackageCompiler.jl only works with the --project command. The --project argument doesnt seem to modify LOAD_PATH, what does it do exactly? Can I edit my julia session with --project later on? I figured julia does the same when loading packages from the ~user/.julia/packages/ dir, but how do I do that, and are there more usefull arguments I should know about?
after some quick "testing" I can confirm:
the --project/--project=. flags when starting julia from the terminal do the same as Pkg.activate(".")
My questions :
What is a good directory structure in order to organize your dags and tasks? (the dags examples show only couple of tasks)
I currently have my dags at the root of the dags folder and my tasks in separate directories, not sure is the way to do it ?
Should we use zip files ? https://github.com/apache/incubator-airflow/blob/a1f4227bee1a70531cfa90769149322513cb6f92/airflow/models.py#L280
I use something like this.
A project is normally something completely separate or unique. Perhaps DAGs to process files that we receive from a certain client which will be completely unrelated to everything else (almost certainly a separate database schema)
I have my operators, hooks, and some helper scripts (delete all Airflow data for a certain DAG, etc.) in a common folder
I used to have a single git repository for the entire Airflow folder, but now I have a separate git per project (makes it more organized and easier to grant permissions on Gitlab since projects are so unrelated). This means that each project folder also as a .git and .gitignore, etc as well
I tend to save the raw data and then 'rest' a modified copy of the data which is exactly what gets copied into the database. I have to heavily modify some of the raw data due to different formats from different clients (Excel, web scraping, HTML email scraping, flat files, queries from SalesForce or other database sources...)
Example tree:
├───dags
│ ├───common
│ │ ├───hooks
│ │ │ pysftp_hook.py
│ │ │
│ │ ├───operators
│ │ │ docker_sftp.py
│ │ │ postgres_templated_operator.py
│ │ │
│ │ └───scripts
│ │ delete.py
│ │
│ ├───project_1
│ │ │ dag_1.py
│ │ │ dag_2.py
│ │ │
│ │ └───sql
│ │ dim.sql
│ │ fact.sql
│ │ select.sql
│ │ update.sql
│ │ view.sql
│ │
│ └───project_2
│ │ dag_1.py
│ │ dag_2.py
│ │
│ └───sql
│ dim.sql
│ fact.sql
│ select.sql
│ update.sql
│ view.sql
│
└───data
├───project_1
│ ├───modified
│ │ file_20180101.csv
│ │ file_20180102.csv
│ │
│ └───raw
│ file_20180101.csv
│ file_20180102.csv
│
└───project_2
├───modified
│ file_20180101.csv
│ file_20180102.csv
│
└───raw
file_20180101.csv
file_20180102.csv
Update October 2021. I have a single repository for all projects now. All of my transformation scripts are in the plugins folder (which also contains hooks and operators - basically any code which I import into my DAGs). DAG code I try to keep pretty bare so it basically just dictates the schedules and where data is loaded to and from.
├───dags
│ │
│ ├───project_1
│ │ dag_1.py
│ │ dag_2.py
│ │
│ └───project_2
│ dag_1.py
│ dag_2.py
│
├───plugins
│ ├───hooks
│ │ pysftp_hook.py
| | servicenow_hook.py
│ │
│ ├───sensors
│ │ ftp_sensor.py
| | sql_sensor.py
| |
│ ├───operators
│ │ servicenow_to_azure_blob_operator.py
│ │ postgres_templated_operator.py
│ |
│ ├───scripts
│ ├───project_1
| | transform_cases.py
| | common.py
│ ├───project_2
| | transform_surveys.py
| | common.py
│ ├───common
| helper.py
| dataset_writer.py
| .airflowignore
| Dockerfile
| docker-stack-airflow.yml
I would love to benchmark folder structure with other people as well. Maybe it will depend on what you are using Airflow to but I will share my case. I am doing data pipelines to build a data warehouse so in high level I basically have two steps:
Dump a lot of data into a data-lake (directly accessible only to a few people)
Load data from data lake into a analytic database where the data will be modeled and exposed to dashboard applications (many sql queries to model the data)
Today I organize the files into three main folders that try to reflect the logic above:
├── dags
│ ├── dag_1.py
│ └── dag_2.py
├── data-lake
│ ├── data-source-1
│ └── data-source-2
└── dw
├── cubes
│ ├── cube_1.sql
│ └── cube_2.sql
├── dims
│ ├── dim_1.sql
│ └── dim_2.sql
└── facts
├── fact_1.sql
└── fact_2.sql
This is more or less my basic folder structure.
I am using Google Cloud Composer. I have to manage multiple projects with some additional SQL scripts and I want to sync everything via gsutil rsync Hence I use the following structure:
├───dags
│ │
│ ├───project_1
│ │
│ ├───dag_bag.py
│ │
│ ├───.airflowignore
│ │
│ ├───dag_1
│ │ dag.py
│ │ script.sql
│
├───plugins
│ │
│ ├───hooks
│ │ hook_1.py
│ │
│ ├───sensors
│ │ sensor_1.py
│ │
│ ├───operators
│ │ operator_1.py
And the file dag_bag.py containes these lines
from airflow.models import DagBag
dag_bag = DagBag(dag_folder="/home/airflow/gcs/dags/project_1", include_examples=False)
I am trying to deploy a simple Qt based chat program, that uses a WebWidget for the chat itself, QListWidgets and some labels. As well as QWebSocket for the network connection. But I do need to add 120 MB files to deploy it.
This are my QT and CONFIG variables in the pro file:
CONFIG += qt release
QT += gui websockets webkitwidgets widgets
This is the list of files I had to add:
│ D3Dcompiler_47.dll
│ icudt54.dll
│ icuin54.dll
│ icuuc54.dll
│ libEGL.dll
│ libgcc_s_dw2-1.dll
│ libGLESV2.dll
│ libstdc++-6.dll
│ libwinpthread-1.dll
│ opengl32sw.dll
│ Qt5Core.dll
│ Qt5Gui.dll
│ Qt5Multimedia.dll
│ Qt5MultimediaWidgets.dll
│ Qt5Network.dll
│ Qt5OpenGL.dll
│ Qt5Positioning.dll
│ Qt5PrintSupport.dll
│ Qt5Qml.dll
│ Qt5Quick.dll
│ Qt5Sensors.dll
│ Qt5Sql.dll
│ Qt5Svg.dll
│ Qt5WebChannel.dll
│ Qt5WebKit.dll
│ Qt5WebKitWidgets.dll
│ Qt5WebSockets.dll
│ Qt5Widgets.dll
│
├───audio
│ qtaudio_windows.dll
│
├───bearer
│ qgenericbearer.dll
│ qnativewifibearer.dll
│
├───iconengines
│ qsvgicon.dll
│
├───imageformats
│ qdds.dll
│ qgif.dll
│ qicns.dll
│ qico.dll
│ qjp2.dll
│ qjpeg.dll
│ qmng.dll
│ qsvg.dll
│ qtga.dll
│ qtiff.dll
│ qwbmp.dll
│ qwebp.dll
│
├───mediaservice
│ dsengine.dll
│ qtmedia_audioengine.dll
│
├───platforms
│ qwindows.dll
│
├───playlistformats
│ qtmultimedia_m3u.dll
│
├───position
│ qtposition_positionpoll.dll
│
├───printsupport
│ windowsprintersupport.dll
│
├───sensorgestures
│ qtsensorgestures_plugin.dll
│ qtsensorgestures_shakeplugin.dll
│
├───sensors
│ qtsensors_generic.dll
│
├───sqldrivers
│ qsqlite.dll
│ qsqlmysql.dll
│ qsqlodbc.dll
│ qsqlpsql.dll
│
└───translations
qt_ca.qm
qt_cs.qm
qt_de.qm
qt_en.qm
qt_fi.qm
qt_fr.qm
qt_he.qm
qt_hu.qm
qt_it.qm
qt_ja.qm
qt_ko.qm
qt_lv.qm
qt_ru.qm
qt_sk.qm
qt_uk.qm
QtPositioning, Sql dlls, Qml and QtQuick? Last time I deployed a Qt program was with Qt4; I remember I had less dependencies.. Is there something wrong?
You might want to do your own Qt build and cut it down as much as possible. It will still be a mess, but a smaller one. Remove optional modules you don't need, resort to using system libraries instead of those bundled with Qt wherever possible, don't use ICU - that alone will cut almost 30MB of dependencies.
The best option is to use a static build and link statically, but there are plenty of limitations at play, you either need a commercial license or to open your code, and still, deployment for QML projects is and has been broken for years. Sadly, it seems like making the lives of all of those using Qt for free as miserable as possible has become quite a priority, in order to force developers into spending on the expensive commercial license, which is the sole remedy to the situation, or at least it will be hopefully by the time Qt 5.7 is released.
BTW, if those DLLs got pulled in by the deployment tool - I advice against trusting it. I have tried it literally yesterday, and it turned out to be completely broken - failed to pull in half of the needed DLLs, half of those it pulled in weren't actually needed, and in terms of qml files, it did even worse.
If not by the deployment tool, those extra dlls are probably indirect dependencies - for example the web sockets define a QML API, so they might pull QML in as a dependency, which itself pulls a cascade of other modules and libraries. You should investigate if you can build those modules without their QML side.