I have a repository where I'm using git flow. I initialized the git flow, but when I clone it I need to run git flow init again. Is there an automated way of doing this!?
At initialization I have to setup:
develop, production branches
features, releases and bugfix prefix
version prefix.
I know this configuration by heart but another developers may not know and typing it by hand may be error prone.
Related
We are currently in the process of developing custom operators and sensors for our Airflow (>2.0.1) on Cloud Composer. We use the official Docker image for testing/developing
As of Airflow 2.0, the recommended way is not to put them in the plugins directory of Airflow but to build them as separate Python package. This approach however seems quite complicated when developing DAGs and testing them on the Docker Airflow.
To use Airflows recommended approach we would use two separate repos for our DAGs and the operators/sensors, we would then mount the custom operators/sensors package to Docker to quickly test it there and edit it on the local machine. For further use on Composer we would need to publish our package to our private pypi repo and install it on Cloud Composer.
The old approach however, to put everything in the local plugins folder, is quite straight forward and doesnt deal with these problems.
Based on your experience what is your recommended way of developing and testing custom operators/sensors ?
You can put the "common" code (custom operators and such) in the dags folder and exclude it from being processed by scheduler via .airflowignore file. This allows for rather quick iterations when developing stuff.
You can still keep the DAG and "common code" in separate repositories to make things easier. you can easily use a "submodule" pattern for that (add "common" repo as submodule of the DAG repo - this way you will be able to check them out together, you can even keep different DAG directories (for different teams) with different version of the common packages this way (just submodule-link it to different versions of the packages).
I think the "package" pattern if more of a production deployment thing rather than development. Once you developed the common code locally, it would be great if you package it together in common package and version accordingly (same as any other python package). Then you can release it after testing, version it etc. etc..
In the "development" mode you can checkout the code with "recursive" submodule update and add the "common" subdirectory to PYTHONPATH. In production - even if you use git-sync, you could deploy your custom operators via your ops team using custom image (by installing appropriate, released version of your package) where your DAGS would be git-synced separately WITHOUT the submodule checkout. The submodule would only be used for development.
Also it would be worth in this case to run a CI/CD with the Dags you push to your DAG repo to see if they continue working with the "released" custom code in the "stable" branch, while running the same CI/CD with the common code synced via submodule in "development" branch (this way you can check the latest development DAG code with the linked common code).
This is what I'd do. It would allow for quick iteration while development while also turning the common code into "freezable" artifacts that could provide stable environment in production, while still allowing your DAGs to be developed and evolve quickly, while also CI/CD could help in keeping the "stable" things really stable.
We're making use of a remote repository and are storing artifacts locally. However, we are running into a problem because of the fact the remote repository regularly rebuilds all artifacts that it hosts. In our current state, we update metadata (e.x. repodata/repomd.xml), but artifacts are not updated.
We have to continually clear our local remote-repository-cache out in order to allow it to download the rebuilt artifacts.
Is there any way we can configure artifactory to allow it to recache new artifacts as well as the new artifact metadata?
In our current state, the error we regularly run into is
https://artifactory/artifactory/remote-repo/some/path/package.rpm:
[Errno -1] Package does not match intended download.
Suggestion: run yum --enablerepo=artifactory-newrelic_infra-agent clean metadata
Unfortunately, there is no good answer to that. Artifacts under a version should be immutable; it's dependency management 101.
I'd put as much effort as possible to convince the team producing the artifacts to stop overriding versions. It's true that it might be sometimes cumbersome to change versions of dependencies in metadata, but there are ways around it (like resolving the latest patch during development, as supported in the semver spec), and in any way, that's not a good excuse.
If that's not possible, I'd look into enabling direct repository-to-client streaming (i.e. disabling artifact caching) to prevent the problem of stale artifacts.
Another solution might be cleaning up the cache using a user plugin or a script using JFrog CLI once you learn about newer artifacts being published in the remote repository.
I tried svn2git to test a migration from svn to git, like this:
svn2git https://my-svn.net/project/ --username $USERNAME
I execute this command where git is installed. I perform this in /home/myuser/svn2git/. After some waiting the execution of the commands end:
...
A src/xx
A src/xx
A src/xx
A src/xx
I do not see the repository getting downloaded to the location where i am running this command from. After the command runs fine, If i commit and push to GIT i do not see anything getting committed. I see .git folder getting created and on the display with verbose i can see all my history.
Can anyone tell me what is incorrect here?
For a one-time migration git-svn is not the right tool for conversions of repositories or parts of a repository. It is a great tool if you want to use Git as frontend for an existing SVN server, but for one-time conversions you should not use git-svn, but svn2git which is much more suited for this use-case.
There are plenty tools called svn2git, the probably best one is the KDE one from https://github.com/svn-all-fast-export/svn2git. I strongly recommend using that svn2git tool. It is the best I know available out there and it is very flexible in what you can do with its rules files.
The svn2git tool you used (the nirvdrum one) is based on git-svn and just tries to work around some of the quirks and drawbacks. In thus that svn2git tool suffers from most of the same drawbacks.
You will be easily able to configure svn2gits rule file to produce the result you want from your current SVN layout, including any complex histories that might exist and including producing several Git repos out of one SVN repo or combining different SVN repos into one Git repo cleanly in one run if you like and also excluding any paths or branches or tags you don't to have migrated, though I'm always crying a little if someone discards code history which is precious.
If you are not 100% about the history of your repository, svneverever from http://blog.hartwork.org/?p=763 is a great tool to investigate the history of an SVN repository when migrating it to Git.
Even though git-svn or the nirvdrum svn2git is easier to start with, here are some further reasons why using the KDE svn2git instead of git-svn is superior, besides its flexibility:
the history is rebuilt much better and cleaner by svn2git (if the correct one is used), this is especially the case for more complex histories with branches and merges and so on
the tags are real tags and not branches in Git
you can generate annotated tags instead of lightweight tags
with git-svn the tags contain an extra empty commit which also makes them not part of the branches, so a normal fetch will not get them until you give --tags to the command as by default only tags pointing to fetched branches are fetched also. With the proper svn2git tags are where they belong
if you changed layout in SVN you can easily configure this with svn2git, with git-svn you will loose history eventually
with svn2git you can also split one SVN repository into multiple Git repositories easily
or combine multiple SVN repositories in the same SVN root into one Git repository easily
the conversion is a gazillion times faster with the correct svn2git than with git-svn
You see, there are many reasons why git-svn is worse and the KDE svn2git is superior. :-)
I have a couple of large Symfony projects, and have noticed that after updating everything to Symfony 4 (Flex), when our deployment automation runs its normal process of:
composer install --no-dev
We end up with (for example) this:
Symfony operations: 2 recipes (72fad9713126cf1479bb25a53d64d744)
- Unconfiguring symfony/maker-bundle (>=1.0): From github.com/symfony/recipes:master
- Unconfiguring phpunit/phpunit (>=4.7): From github.com/symfony/recipes:master
Then, as expected, this results in changes to symfony.lock and config/bundles.php, plus whatever else, depending on what was included in require-dev in composer.json.
None of this is breaking, exactly, but it is annoying to have a production deploy that no longer has a clean git status output, and can lead to confusion as to what is actually deployed.
There are various workarounds for this, for example I could just put everything in require rather than require-dev since there is no real harm in deploying that stuff, or I could omit the --no-dev part of the Composer command.
But really, what is the right practice here? It seems odd that there is no way to tell Flex to make no changes to configuration if you are just deploying a locked piece of software. Is this a feature request, or have I missed some bit of configuration here?
This was briefly the case on old Syfmony Flex 1 versions.
It was reported here, and fixed here.
It this happens to you, it means you have a very old installation on your hands, and you should do your best to at least update Symfony Flex to a more current version (Symfony Flex 1 does not even work anymore, so switching to Flex 2 if possible would be the only way, or simply remove Flex in production).
If you deploy to prod from your master branch, you can setup a deploy branch instead. And in that branch you can prevent certain files from being merged in. See this post for more detail. This creates a situation in which you have a master branch, a version branch (example: 3.21.2) and you're having devs checkout master, work on it, then merge their changes into the version branch. From there you pick and choose what gets deployed to prod. (There will be a small balancing act here. You'll want to merge all dev changes into master until it matches your version branch and make sure master matches the version after you've deployed. This adds a bit of work and you have to keep an eye on it. etc.)
Another option is to separate your git repository from your deployment directory. In this example, a git directory is created in /var/repo/site.git and the deploy directory is /var/www/domain.com and a post-receive git hook is used to automatically update the www directory after a push is received to the repo/site directory. You obviously run composer, npm, gulp, whathaveyou in the www directory and the git directory is left as is.
Without getting into commercial options such as continuous deployment apps, you can write a deployment script. There are a lot of ways to write a shell script that takes one directory and copies it over, runs composer, runs npm, etc. all in one command -- separating your git from your deploy directories. Here's a simple one that makes use of the current datetime to name a directory then symlinking it to the deployment directory.
I'm not a Git novice but also not a guru either and I have a question. We want to create remote repos that appear as folders within our network. The folders will actually contain a large legacy ASP app running in a production manner.
Then we want to be able to make local changes and be able to push commits to these networked repos and thus dynamically update the production application.
We already have the repo on Github and our developers fork that and work locally (we use SmartGit for most day to day stuff).
However (because the app is huge and legacy) we have (prior to using Git) always had a process for copying changed files to the target systems (production, QA etc).
But it dawned on me that we may be able to treat the operational system "as" a repo that's checked out to master. Then (when suitably tested) we want to simply use SmartGit to do a "push to" the operational system and have the changes delivered that way.
I'm at the edge of my Git knowledge though and unsure if this is easy to do or risky.
We don't want to install Git on the operational machine (its restricted to running Windows 2003 - yes I know...) so want to simply treat the remote system just like it was a local folder - with Git installed on our local machines.
Any tips or suggestions?
My tip: don't bother.
You can only push to bare repositories. These are such that they only contain the files normally residing in .git, with no working directory at all. So you cannot "run" those on the server. You would need to push to a bare repos on the server, and then clone/checkout that bare repos into a non-bare local repos on the server itself (which can be done in a post-receive hook inside git). But as you said, you cannot even install git on the server. So git push does nothing for you.
Second option would be to mount the servers filesystem on whatever staging/deployment machine you have, presumably one which you can install git on. Then you can git push into a bare repos on that deployment machine, run git hooks, and copy newly pushed stuff into your non-git server filesystem.
Third option would be to package everything up locally, make a tarball (or, I guess, zip-ball...) and just unpack that on the server.
So. Automated, continuous deployment => great idea. Using git => great idea. Directly using git push, not so much, mainly due to your constraints.