I'm trying to test and migrate only a few revisions from a specific file from a 300K revisions svn repo.
My very basic rules file looks like
create repository test-onefile
end repository
match /basedir/src/libraries/lib1/tests/(test.py)
repository test-onefile
prefix \1
min revision 82400
max revision 82541
branch master
end match
# Ignore everything else
match /
end match
There are a couple changes on that file between 82400 and 82541 and I would the migration to process just those and end in a few seconds.
However, the migration takes longer than two hours and seems to be processing every commit ?
It keeps printing
"Exporting revision xxxxx done"
And now and then
"Exporting xxxxx yyyyyyyy was copied from zzzzzzz, where non of them are related to the matched file above.
What I'm trying to do is to run quick tests exporting a few revisions to quickly check the outcome instead of waiting a few hours.
Related
Running the .NET Core Pack task, how do I get the outputted NuGet package version to auto-increment itself?
So, for example, if my current version is 1.0.0, then the next time I call the Pack task, I would like to see 1.0.1.
I'm using environment build variables with Build.BuildNumber and getting outputs at the moment of e.g. 20180913-.2.0, etc. I would like to establish to a more traditional versioning system.
From the docs, the variable Rev:.r is the daily build revision count. The accepted "solution" would lead to one day finishing having a version of 1.0.12, then the next day it will be 1.0.1.
If you want a simple incremental and unique semver, use 1.0.$(BuildID).
$(BuildID) is an internal immutable counter for your builds, and thus far cleaner than $(BuildNumber).
BuildID will always be incrementing - no reset.
Thus after a minor bump, you'd end up having say 1.2.123 becoming 1.3.124.
If you want to perform this task well, this can be done using npm version or similar, such as pubspec_version for Dart or Flutter builds.
- script: npm version $RELEASE_TYPE
where $RELEASE_TYPE is a variable you can set based on build (ie: CI, PR etc), having a value of major, minor, patch, prerelease etc.
- script: npm version $RELEASE_TYPE
condition: startsWith(variables['build.sourceBranch'], 'refs/head/release/')
env:
releaseType: minor
Update: Bump Repo Version and Use In Build (using npm)
To have the repo version update, I ended up including npm version as a DevDependency, with it's precommit hook to bump the project version on any commit.
This technique can be applied to other project types, placing them in a subfolder - although can lead to complications with server OS requirements.
To use this version in your build, add this bash script task, which gets and exports the version as a task variable:
v=`node -p "const p = require('./package.json'); p.version;"`
echo "##vso[task.setvariable variable=packageVersion]$v"
.Net Core Task only version
Unfortunately, no repo-bump.
Workaround 1:
jobs:
- job: versionJob #reads version number from the source file
steps:
- powershell: |
$fv = Get-Content versionFile
Write-Host ("##vso[task.setvariable variable=versionFromFile;isOutput=true]$fv")
displayName: 'version from file'
name: setVersionStep
- job: buildJob # consumes version number, calculates incremental number and set version using assemblyinfo.cs
dependsOn: versionJob
variables:
versionFromFile: $[ dependencies.versionJob.outputs['setVersionStep.versionFromFile'] ] # please note that spaces required between $[ and dependencies
buildIncrementalNumber: $[ counter(dependencies.versionJob.outputs['setVersionStep.versionFromFile'],1) ] #can't use $versionFromFile here
steps:
- powershell: |
Write-Host ($env:versionFromFile)
Write-Host ($env:versionFromFile + '.' + $env:buildIncrementalNumber)
displayName: 'version from file output'
Workaround 2:
This post describes a couple of others, using version-prefix and automatically applying the BuildNumber as a version-suffix.
I may have figured it out. For anyone tearing their hair out, try this:
Pack Task:
Automatic Package Versioning: Use an environment variable
Environment variable: Build.BuildNumber
Then, up in the top menu where you have Tasks, Variables, Triggers, Options, click Options and set:
Build number format: 1.0$(Rev:.r)
Save and queue. This will produce e.g. 1.0.1.
(Please Correct me if I am wrong, or if this does not work long-term.)
If you're just looking to bump the major, minor or revision version number, using counter operator in a variable is a simple and elegant approach. It will automatically add one to the current value.
Here's what I use:
variables:
major: '1'
minor: '0'
revision: $[counter(variables['minor'], 1)] #this will get reset when minor gets bumped. The number after Counter is the seed number (in my case, I started at 1).
app_version: '$(major).$(minor).$(revision)'
If you would like to see a real-world 4-job pipeline that uses this, I have one here https://github.com/LanceMcCarthy/DevReachCompanion/blob/master/azure-pipelines.yml
For me it's enough to set Build number format on Options tab to
$(date:yyyy).$(date:MMdd)$(rev:.r)
and add next build argument:
/p:Version=1.$(Build.BuildNumber) /p:AssemblyVersion=1.$(Build.BuildNumber)
In this case we manage major version manually, but minor version and build number will be set automatically. Easy to understand what version you have deployed.
I am using the ado pipeline and a yaml build. What I've done is utilized the pipeline variables, a counter function, and an inline powershell function to create the version number. It auto-increments and has made the entire build process nice.
Another SO Post about something similar
I am attempting to make a migration from D7 to D8 using Migrate Manifest. I have all the migrate modules installed (i.e. the 3 core migrate modules, Migrate Upgrade, Migrate Plus, Migrate Tools and Migrate Manifest). I followed the directions at this link: https://www.drupal.org/node/2350651 to prepare my manifest file. This is the relevant part of my resulting yaml file (truncated for brevity):
- block_content_body_field
- block_content_entity_display
- block_content_entity_form_display
- block_content_type
- d7_block
- d7_custom_block
When I ran the migrate-manifest via Drush, as specified by the directions in the link above, I received an error declaring a parse error pointing to the first line in the yaml list. This is the exact error: Unable to parse at line 1 (near " - block_content_body_field").
I tried several debugging steps:
I checked a yaml linter to see whether my file was properly formatted. My file passed the test.
I moved the block migrations below the ones that are prefixed with a 'd7' to match the example in the link.
I attempted a piecemeal migration of a file from my list of migrations, it worked.
I tried to alter the spacing on the list items.
All these steps failed to resolve the issue I am seeing.
I am unsure what other debugging steps I should take. All the above failed to resolve my issue.
Any ideas would be appreciated.
I am writing a build process for a WordPress installation using Ansible. It doesn't have a application-level build system at the moment, and I've chosen Ansible so that it can cleanly integrate with server build scripts, so I can bring up a working server at the touch of a button.
Most of my WordPress plugins are being installed with the unarchive feature, pointing to versioned plugin builds on the official wordpress.org installation server. I've encountered a problem with just one of these, which is that it is always being marked as "changed" even though the files are exactly the same.
Having examined the state of ls -Rl before and after, I noticed that this plugin (WordPress HTTPS) is the only one to use internal sub-directories, and upon each decompression, the modification time of folders is getting bumped.
It may be useful to know that this is a project build script, with a connection of local. I guess therefore that means that SSH is not being used.
Here is a snippet of my playbook:
- name: Install the W3 Total Cache plugin
unarchive: >
src=https://downloads.wordpress.org/plugin/w3-total-cache.0.9.4.1.zip
dest=wp-content/plugins
copy=no
- name: Install the WP DB Manager plugin
unarchive: >
src=https://downloads.wordpress.org/plugin/wp-dbmanager.2.78.1.zip
dest=wp-content/plugins
copy=no
# #todo Since this has internal sub-folders, need to work out
# how to preserve timestamps of the original folders rather than
# re-writing them, which forces Ansible to record a change of
# server state.
- name: Install the WordPress HTTPS plugin
unarchive: >
src=https://downloads.wordpress.org/plugin/wordpress-https.3.3.6.zip
dest=wp-content/plugins
copy=no
One hacky way of fixing this is to use ls -R before and after, using options to include file sizes but not timestamps, and then md5sum that output. I could then mark it as changed if there is a change in checksum. It'd work but it's not very elegant (and I'd want to do that for all plugins, for consistency).
Another approach is to abandon the task if a plugin file already exists, but that would cause problems when I bump the plugin version number to the latest copy.
Thus, ideally, I am looking for a switch to present to unarchive to say that I want the folder modification times from the zip file, not from playbook runtime. Is it possible?
Update: a commenter asked if the file contents could have changed in any way. To determine whether they have, I wrote this script, which creates a checksum for (1) all file contents and (2) all file/directory timestamps:
#!/bin/bash
# Save pwd and then change dir to root location
STARTDIR=`pwd`
cd `dirname $0`/../..
# Clear collation file
echo > /tmp/wp-checksum
# List all files recursively
find wp-content/plugins/wordpress-https/ -type f | while read file
do
#echo $file
cat $file >> /tmp/wp-checksum
done
# Get checksum of file contents
sha1sum /tmp/wp-checksum
# Get checksum of file sizes
ls -Rl wp-content/plugins/wordpress-https/ | sha1sum
# Go back to original dir
cd $STARTDIR
I ran this as part of my playbook (running it in isolation using tags) and received this:
PLAY [Set this playbook to run locally] ****************************************
TASK [setup] *******************************************************************
ok: [localhost]
TASK [jonblog : Run checksum command] ******************************************
changed: [localhost]
TASK [jonblog : debug] *********************************************************
ok: [localhost] => {
"checksum_before.stdout_lines": [
"374fadc4df1578f78fd60b1be6758477c2c533fa /tmp/wp-checksum",
"10d66f7bdbbdd3af531d1b11a3db3059a5868838 -"
]
}
TASK [jonblog : Install the WordPress HTTPS plugin] ***************
changed: [localhost]
TASK [jonblog : Run checksum command] ******************************************
changed: [localhost]
TASK [jonblog : debug] *********************************************************
ok: [localhost] => {
"checksum_after.stdout_lines": [
"374fadc4df1578f78fd60b1be6758477c2c533fa /tmp/wp-checksum",
"719c9da94b525e723b1abe188ee9f5bbaf121f3f -"
]
}
PLAY RECAP *********************************************************************
localhost : ok=6 changed=3 unreachable=0 failed=0
The debug lines reflect the checksum hash of the contents of the files (this is identical) and then the checksum hash of ls -Rl of the file structure (this has changed). This is in keeping with my prior manual finding that directory checksums are changing.
So, what can I do next to track down why folder modification times are incorrectly flagging this operation as changed?
Rather than overwriting all files each time and find a way to keep the same modification datetime, you may want to use the creates option of the unarchive module.
As you maybe already know, this tells Ansible that a specific file/folder will be created as a result of the task. Thus, next time the task will not be run again if that file/folder already exists.
See http://docs.ansible.com/ansible/unarchive_module.html#options
My solution is to modify the checksum script and to make that a permanent feature of the Ansible process. It feels a bit hacky to do my own checksumming, when Ansible should do it for me, but it works.
New answers that explain that I am doing something wrong, or that a new version of Ansible fixes the problem, would be most welcome.
If I get a moment, I will raise this as a possible bug with the Ansible team. However I do sometimes wonder about the effort/reward ratio when raising bugs on a busy tracker - I already have one item outstanding, it has been waiting a while, and I've chosen to work around that too.
Update (18 months later)
This Ansible build system never made it into live. It felt like I was always working around something. Recently, when I decided I needed to move my blog to another server, I finally Dockerised it. This took several weeks (since there is a surprising amount of things to think about in a real WordPress installation) but in general I found the process much nicer than using orchestration tools.
New to salt,and i add first server(wx-1),it works ,but when i add a differnt server, test.ping is ok,but when execute salt 'qing' state.highstate, it fails,the error info is:
No Top file or external nodes data matches found
Here is my top.sls:
base:
'wx-1':
- bin.nginx
- git
- web
- mongo
- redis
'qing':
- bin.nginx
qing is a new server and it's config is different to wx-1,don't know if this is ok,thanks for your help:)
If you make changes to your sls files. Make sure that you restart the master in order for it to update. This solved my problem when receiving the same error...
You didn't give much information. But here are a few things to check:
test if salt qing state.sls bin.nginx works, if not continue reading
make sure file_roots:base in master config points to /srv/salt
use salt-master/minion --version to check salt versions, make sure they are the same. Because different versions might diff
Give further info if you tried all the above.
We're running nexus on some old hardware which is limited in disk space and would like to remove artifacts older than a certain threshold.
Is there any way to do this other than a combination of find and curl?
There is a scheduled task that can automatically remove old snapshot releases:
http://www.sonatype.com/people/2009/09/nexus-scheduled-tasks/
http://www.sonatype.com/books/nexus-book/reference/confignx-sect-managing-tasks.html
Unfortunately, this does not work for hosted release repositories.
As mentioned on a Sonatype blog post linked from a comment in the blog in gavenkoa's answer, since Nexus 2.5 there is a built in "Remove Releases From Repository" scheduled task which can be configured to delete old releases keeping a defined number.
This is sufficient to meet our needs.
Delete all files to which no one access more then 100 days and not modified more then 200 days:
find . -type f -atime +100 -mtime 200 -delete
To cleanup empty directories:
find . -type d -empty -delete
Or alternatively look to https://github.com/akquinet/nexus_cleaner/blob/master/nexus_clean.sh and corresponding blog entry http://blog.akquinet.de/2013/12/09/how-to-clean-your-nexus-release-repositories/ (delete all except last 10 releases).
auto purge older than 30 days(u can change it) not download docker images from nexus 3
https://gist.github.com/anjia0532/4a7fee95fd28d17f67412f48695bb6de
# nexus3's username and pwd
username = 'admin'
password = 'admin123'
# nexus host
nexusHost = 'http://localhost:8081'
# purge repo
repoName = 'docker'
# older than days
days = 30
#change and run it
For Nexus2, you can use my Spring Boot application https://github.com/vernetto/nexusclean , you can define rules based on date and on a minimum number of Artifacts to retain, and it generates "rm -rf" commands (using the REST API is damn slow).
For Nexus3, I would definitely use a Groovy script as a "Execute Admin Task". One is posted here groovy script to delete artifacts on nexus 3 (not nexus 2)