I am trying to run the GATK best practices pipeline in google cloud and get below error
here's the gcloud command:
gcloud alpha genomics pipelines run \
--pipeline-file wdl_pipeline.yaml \
--regions us-central1 \
--inputs-from-file WDL=${GATK_GOOGLE_DIR}/PairedEndSingleSampleWf.gatk4.0.wdl,\
WORKFLOW_INPUTS=${GATK_GOOGLE_DIR}/PairedEndSingleSampleWf.hg38.inputs.json,\
WORKFLOW_OPTIONS=${GATK_GOOGLE_DIR}/PairedEndSingleSampleWf.gatk4.0.options.json \
--env-vars WORKSPACE=${GATK_OUTPUT_DIR}/work,\
OUTPUTS=${GATK_OUTPUT_DIR}/output \
--logging ${GATK_OUTPUT_DIR}/logging/
ERROR: (gcloud.alpha.genomics.pipelines.run) Unable to read file [wdl_pipeline.yaml]: [Errno 2] No such file or directory: 'wdl_pipeline.yaml'
[UPDATE]
The WDL Runner repository location had changed. The tutorial at https://cloud.google.com/genomics/docs/tutorials/gatk has been updated.
[ORIGINAL]
If you're following the tutorial at https://cloud.google.com/genomics/docs/tutorials/gatk make sure to do step 4 where you change directories. It is telling you that the wdl file is not in the directory where you are.
Related
I'm trying to use nix for building dotnet (sdk 5) projects.
The configuration is something like
stdenv.mkDerivation {
# builder = "${bash}/bin/bash";
# args = [ ./builder.sh ];
name = "mypackage";
src = ./.;
HOME = "/tmp/test-home";
DOTNET_CLI_TELEMETRY_OPTOUT = 1;
nativeBuildInputs = [ nodejs ];
buildInputs = [ dotnet openssl cacert ];
configurePhase = ''
export HOME=$PWD/home
runHook preConfigure
dotnet nuget list source
dotnet nuget locals all --list
dotnet restore
runHook postConfigure
'';
buildPhase = ''
export DOTNET_CLI_TELEMETRY_OPTOUT=1
export DOTNET_NOLOGO=1
${dotnet}/bin/dotnet publish ProjectDirectory --self-contained -r linux-x64 -c Release
'';
installPhase = ''
mkdir -p $out/
'';
};
The HOME attribute in argument for stdenv.mkDerivation is for nix-shell only. Since nix-shell will inherit normal $HOME, I need to set $HOME to another empty directory to mimic behavior of build phase.
When using nix-shell and run dotnet restore manually, it works fine.
When using nix-build, build failed with following error when running dotnet restore
Registered Sources:
nuget.org [Enabled]
https://api.nuget.org/v3/index.json
http-cache: /build/PROJECTNAME/home/.local/share/NuGet/v3-cache
global-packages: /build/PROJECTNAME/home/.nuget/packages/
temp: /build/NuGetScratch
plugins-cache: /build/PROJECTNAME/home/.local/share/NuGet/plugins-cache
Determining projects to restore...
/nix/store/fvfyn01fjmawvyn7vlhhrgkzyy6321wl-dotnet-sdk-5.0.202/sdk/5.0.202/NuGet.targets(131,5): error : Unable to load the service index for source https://api.nuget.org/v3/index.json. [/build/PROJECTNAME/PET-CT-machine-service.sln]
/nix/store/fvfyn01fjmawvyn7vlhhrgkzyy6321wl-dotnet-sdk-5.0.202/sdk/5.0.202/NuGet.targets(131,5): error : Name or service not known (api.nuget.org:443) [/build/PROJECTNAME/PET-CT-machine-service.sln]
/nix/store/fvfyn01fjmawvyn7vlhhrgkzyy6321wl-dotnet-sdk-5.0.202/sdk/5.0.202/NuGet.targets(131,5): error : Name or service not known [/build/tomopioneer/PET-CT-machine-service.sln]
What's the difference between nix-build and nix-shell --pure when HOME is manually set?
By reading code from github NixOS/nixpkgs repository, it seems several dotnet packages were using nuget manually and using dotnet restore with local source. Is that necessary?
I've make more tests about api.nuget.org.
running curl https://api.nuget.org/v3/index.json -v, I got:
works fine when directly run under terminal of nixos
works fine in nix-shell --pure
error in nix-build, curl: (6) Could not resolve host: api.nuget.org
nix-shell has network access which is not allowed in the sandbox that nix-build uses without a fixed output derivation.
By reading code from github NixOS/nixpkgs repository, it seems several dotnet packages were using nuget manually and using dotnet restore with local source. Is that necessary?
With sandbox enabled yes.
I am trying to use DataflowJavaOperator on our testing composer environment, but I am running into a 403 forbidden error. My intention is to kick off a Dataflow Java job on a different project using the test composer environment.
t2 = DataFlowJavaOperator(
task_id = "run-java-dataflow-job",
jar="gs://path/to/dataflow-jar.jar",
dataflow_default_options=config_params["dataflow_default_options"],
gcp_conn_id=config_params["gcloud_config"]["conn_id"],
dag=dag
)
My default options look like
'dataflow_default_options': {
'project': 'other-project',
'input': 'other-project:dataset.table',
'output': 'other-project:dataset.table'
...
}
I have tried creating a temporary composer test environment in the same project as the Dataflow, and this allows me to use DataflowJavaOperator as expected. Only when the composer environment resides in a different project as the Dataflow, does DataflowJavaOperator not work as expected.
My current workaround is to use BashOperator, use "env" to set the GOOGLE_APPLICATION_CREDENTIALS as gcp_conn_id path, store the jar file in our test composer bucket, and just run this bash command:
java -jar /path/to/dataflow-jar.jar \
[... all Dataflow job options]
Is it possible to use DataflowJavaOperator to kick off Dataflow jobs on another project?
You need a different GCP connection created for Composer to interact with your 2nd GCP project and you need to pass that connection id to gcp_conn_id in DataFlowJavaOperator
I'm trying to set up Symfony 2.5 via Composer.
First I am calling:
php5.3.8-cli /kunden/81425/composer.phar create-project symfony/framework-standard-edition hhcadm/ "2.5.*"
This works until at some point it is internally not calling php5.3.8-cli anymore but somehow uses php. The problem is that php is version 4.4.9 and this causes following effect:
Nothing to install or update
Generating autoload files
Updating the "app/config/parameters.yml" file
X-Powered-By: PHP/4.4.9
Content-type: text/html
<br />
<b>Parse error</b>: syntax error, unexpected T_STRING, expecting T_CONSTANT_ENCAPSED_STRING or '(' in <b>/kunden/81425/hhcadm/vendor/sensio/distribution-bundle/Sensio/Bundle/DistributionBundle/Resources/bin/build_bootstrap.php</b> on line <b>13</b><br />
Script Sensio\Bundle\DistributionBundle\Composer\ScriptHandler::buildBootstrap handling the post-install-cmd event terminated with an exception
[RuntimeException]
An error occurred when generating the bootstrap file.
The error makes sense as 4.4.9 is too old. How can I tell Composer/Symfony installer which php installation to use during the install routine?
Same happens for me when I call:
php5.3.8-cli /kunden/81425/composer.phar install
I am pretty shure there must be a config / param for that but I can't find it for hours now.
Thank you very much.
'php' must be the PHP5 executable.
The only solution i see is to modify the PATH, launch the composer command (and eventually restore the old path)
For example :
OLDPATH=$PATH
PATH=/usr/local/php5/bin:$PATH
php /kunden/81425/composer.phar install
PATH=$OLDPATH
To help you, you can also create a small script 'composer.sh' :
#!/bin/sh
OLDPATH=$PATH
PATH=/usr/local/php5/bin:$PATH
php /kunden/81425/composer.phar $*
PATH=$OLDPATH
And you call 'composer.sh install'
Edit : if all php binaries are in the same, you can create symbolic links
For example :
mkdir /usr/local/bin/php5
ln -s /usr/local/bin/php53-cli /usr/local/bin/php5/php
And so PATH=$PATH:/usr/local/bin/php5:$PATH
Edit: If you are not allowed to create /usr/local/bin/php5 then just use any other folder like:
#!/bin/sh
OLDPATH=$PATH
PATH=/kunden/81425/php5:$PATH
php /kunden/81425/composer.phar $*
PATH=$OLDPATH
I am trying to create an OpenShift application using the --from-code option to grab the application code from GitHub. I've created two different OpenShift QuickStarts -- with one, the --from-code option works, and with the other, it doesn't work.
So clearly I'm doing something wrong in the QuickStart that isn't working. But I can't see what I'm doing wrong. I either get error 504 or an error occurred, neither of which tells me what the problem is, and there doesn't seem to be a verbose flag to get more details on the error.
Tests-Mac:~ testuser$ rhc app create sonr diy-0.1 http://cartreflect-claytondev.rhcloud.com/reflect?github=smarterclayton/openshift-redis-cart --from-code https://github.com/citrusbyte/SONR.git
The cartridge 'http://cartreflect-claytondev.rhcloud.com/reflect?github=smarterclayton/openshift-redis-cart' will be downloaded and installed
Application Options
-------------------
Domain: schof
Cartridges: diy-0.1, http://cartreflect-claytondev.rhcloud.com/reflect?github=smarterclayton/openshift-redis-cart
Source Code: https://github.com/citrusbyte/SONR.git
Gear Size: default
Scaling: no
Creating application 'sonr' ... Server returned an unexpected error code: 504
Tests-Mac:~ testuser$ rhc app create sonr diy-0.1 http://cartreflect-claytondev.rhcloud.com/reflect?github=smarterclayton/openshift-redis-cart --from-code https://github.com/citrusbyte/SONR.git
The cartridge 'http://cartreflect-claytondev.rhcloud.com/reflect?github=smarterclayton/openshift-redis-cart' will be downloaded and installed
Application Options
-------------------
Domain: schof
Cartridges: diy-0.1, http://cartreflect-claytondev.rhcloud.com/reflect?github=smarterclayton/openshift-redis-cart
Source Code: https://github.com/citrusbyte/SONR.git
Gear Size: default
Scaling: no
Creating application 'sonr' ...
An error occurred while communicating with the server. This problem may only be temporary. Check that you have correctly specified your
OpenShift server 'https://openshift.redhat.com/broker/rest/domain/schof/applications'.
Tests-Mac:~ testuser$
That's creating an application with --from-code using this repo: https://github.com/citrusbyte/SONR . If I use this repo it works flawlessly: https://github.com/citrusbyte/openshift-sinatra-redis
The code itself seems to be good, as I can create an empty new application, merge the SONR code in, and it works flawlessly.
What am I doing wrong?
UPDATE: I've worked around this issue by creating the app in two stages instead of doing it in one stage:
rhc app create APPNAME diy-0.1 http://cartreflect-claytondev.rhcloud.com/reflect?github=smarterclayton/openshift-redis-cart
cd APPNAME
git remote add github -f https://github.com/citrusbyte/SONR.git
git merge github/master -s recursive -X theirs
git push origin master
I'd still love to know why doing it in one step was failing, though.
#developercorey had the right idea.
I tried with a ridiculous timeout of 99999, and then got a different timeout error that I don't think I can change:
$ rhc app create APPNAME diy-0.1 http://cartreflect-claytondev.rhcloud.com/reflect?github=smarterclayton/openshift-redis-cart --from-code https://github.com/citrusbyte/SONR.git --timeout 99999
...
Creating application 'APPNAME' ...
The initial build for the application failed: Shell command '/sbin/runuser -s /bin/sh 5328a9385973ca70150002af -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c5,c974' /bin/sh -c \"gear postreceive --init >> /tmp/initial-build.log 2>&1\""' exceeded timeout of 229
The fix I mentioned in my earlier update is working perfectly, and that's what I recommend anyone with a similar problem try -- I'm creating the app as empty without the --from-code option, and then merging in the code I wanted to use in a separate step:
rhc app create APPNAME diy-0.1 http://cartreflect-claytondev.rhcloud.com/reflect?github=smarterclayton/openshift-redis-cart
cd APPNAME
git remote add github -f https://github.com/citrusbyte/SONR.git
git merge github/master -s recursive -X theirs
git push origin master
It could be that the application takes to long to clone/setup, and the creation is timing out. Something you can try is to create the application without the --from-code, then clone it locally, and merge in your code from github, then do a git push. This operation has a much longer timeout period, and will also let you see what, if any, errors that you get since the application won't disappear if it doesn't succeed, unlike an app create.
For the purposes of CI, I need to be able to generate an XCARCHIVE and an IPA file in our nightly build. The IPA is for our testers, to be signed with our ad-hoc keys, and the XCARCHIVE is to send to the client so that they can import it into Xcode and submit it to the app store when they're happy with it.
Generating the IPA is simple enough with a bit of googling, however how to generate the .XCARCHIVE file is what eludes me. The closest I've found is:
xcodebuild -scheme myscheme archive
However, this stores the .xcarchive in some hard-to-find folder, eg:
/Users/me/Library/Developer/Xcode/Archives/2011-12-14/MyApp 14-12-11 11.42 AM.xcarchive
Is there some way to control where the archive is put, what its name is, and how to avoid having to re-compile it? I guess the best possible outcome would be to generate the xcarchive from the DSYM and APP that are generated when you do an 'xcodebuild build' - is this possible?
Xcode 5 now supports an -archivePath option:
xcodebuild -scheme myscheme archive -archivePath /path/to/AppName.xcarchive
You can also now export a signed IPA from the archive you just built:
xcodebuild -exportArchive -exportFormat IPA -exportProvisioningProfile my_profile_name -archivePath /path/to/AppName.xcarchive -exportPath /path/to/AppName.ipa
Starting with Xcode 4 Preview 5 there are three environment variables that are accessible in the scheme archive's post-actions.
ARCHIVE_PATH: The path to the archive.
ARCHIVE_PRODUCTS_PATH: The installation location for the archived product.
ARCHIVE_DSYMS_PATH: The path to the product’s dSYM files.
You could move/copy the archive in here. I wanted to have a little more control over the process in a CI script, so I saved a temporary file that could easily be sourced in my CI script that contained these values.
BUILD_DIR=$PROJECT_DIR/build
echo "ARCHIVE_PATH=\"$ARCHIVE_PATH\"" > $BUILD_DIR/archive_paths.sh
echo "ARCHIVE_PRODUCTS_PATH=\"$ARCHIVE_PRODUCTS_PATH\"" >> $BUILD_DIR/archive_paths.sh
echo "ARCHIVE_DSYMS_PATH=\"$ARCHIVE_DSYMS_PATH\"" >> $BUILD_DIR/archive_paths.sh
echo "INFOPLIST_PATH=\"$INFOPLIST_PATH\"" >> $BUILD_DIR/archive_paths.sh
Then in my CI script I can run the following:
xcodebuild -alltargets -scheme [Scheme Name] -configuration [Config Name] clean archive
source build/archive_paths.sh
ARCHIVE_NAME=AppName-$APP_VERSION-$APP_BUILD.xcarchive
cp -r "$ARCHIVE_PATH" "$BUILD_DIR/$ARCHIVE_NAME"
I have just solved this one - just add the argument -archivePath to your xcode build command line, given the initial question that would mean:
xcodebuild -scheme myscheme archive
becomes ...
xcodebuild -scheme myscheme archive -archivePath Build/Archive
(Note: paths are relative, I output my build to $PWD/Build)
This will then place your .app folder in:
Build/Archive.xarchive/Products/Application
If your build target already has your signing certificate and provisioning profile in it you can then create your IPA file without re-signing using the following command:
xcrun -v -sdk iphoneos PackageApplication -v `pwd`'/Build/Archive.xarchive/Products/Application/my.app' -o `pwd`'/myapp.ipa'
(Note: xcrun doesn't like relative paths hence the pwd)
The -v args dump lots of useful information - this command can fail to sign properly and still exit with code 0, sigh!
If you are finding that you can't run the built .ipa it's probably a signing issue that you can do a double check on using:
codesign --verify -vvvv myapp.app
If it's signed correctly and un-tampered with the output will have this in:
myapp.app: valid on disk
myapp.app: satisfies its Designated Requirement
If not you will see something similar to this:
Codesign check fails : /blahpath/myapp.app: a sealed resource is missing or invalid
file modified: /blahpath/ls-ios-develop.app/Assets.car
... which generally means you are trying to use an intermediate output directory rather than the proper archive.
My current solution is to rename the user's existing archives folder, run the build, and do a 'find' to copy the archives where i want, then delete the archives folder and rename the old folder back as it was, with code like this in my ruby build script:
# Move the existing archives out of the way
system('mv ~/Library/Developer/Xcode/Archives ~/Library/Developer/Xcode/OldArchivesTemp')
# Build the .app, the .DSYM, and the .xcarchive
system("xcodebuild -scheme \"#{scheme}\" clean build archive CONFIGURATION_BUILD_DIR=\"#{build_destination_folder}\"")
# Find the xcarchive wherever it was placed and copy it where i want it
system("find ~/Library/Developer/Xcode/Archives -name *.xcarchive -exec cp -r {} \"#{build_destination_folder}\" \";\"")
# Delete the new archives folder with this new xcarchive
system('rm -rf ~/Library/Developer/Xcode/Archives')
# Put the old archives back
system('mv ~/Library/Developer/Xcode/OldArchivesTemp ~/Library/Developer/Xcode/Archives')
Its a bit hacky but i don't see a better solution currently. At least it preserves the user's 'archives' folder and all their pre-existing archives.
--Important note!--
I since found out that the line of code where i find the archive and cp it to the folder i want doesn't copy the symlinks inside the archive correctly, thus breaking the code signing in the app. You'll want to replace that with a 'mv' or something that maintains symlinks. Cheers!
Here's a bit of bash that I've come up with for our Jenkins CI system. These commands should be run in a script immediately after the xcodebuild archive command finishes.
BUILD_DIR="${WORKSPACE}/build"
XCODE_SCHEME="myscheme"
# Common path and partial filename
ARCHIVE_BASEPATH="${HOME}/Library/Developer/Xcode/Archives/$(date +%Y-%m-%d)/${XCODE_SCHEME}"
# Find the latest .xcarchive for the given scheme
NEW_ARCHIVE=$(ls -td "${ARCHIVE_BASEPATH}"* | head -n 1)
# Zip it up so non-Apple systems won't treat it as a dir
pushd "${NEW_ARCHIVE%/*}"
zip -r "${BUILD_DIR}/${NEW_ARCHIVE##*/}.zip" "${NEW_ARCHIVE##*/}"
popd
# Optional, disk cleanup
rm -rf "${NEW_ARCHIVE}"
The BUILD_DIR is used to collect artifacts so that it's easy to archive them from Jenkins with a glob such as build/*.ipa,build/*.zip
Similar to the others, but perhaps a little simpler since I try to record the .xcarchive file's location. (I also don't move the archives folder, so this will work better if you're doing multiple builds at the same time.)
My caller build script generates a new tempfile and sets its path to an environment variable named XCARCHIVE_PATH_TMPFILE. This environment variable is available in my scheme's Archive post-action shell script, which then that writes the .xcarchive's path to that file. The build script that can then read that file after it calls xcodebuild archive.
post-action shell script
echo $ARCHIVE_PATH > "$XCARCHIVE_PATH_TMPFILE"
On Xcode 4.6 it is possible to specify a post-build action for the scheme to be compiled into an xcarchive:
echo "ARCHIVE_PATH=\"$ARCHIVE_PATH\"" > $PROJECT_DIR/archive_paths.sh
A build script can be used to check if $ARCHIVE_PATH is defined after running xcodebuild and if this is the case, the output xcarchive can be moved into a designated folder.
This method is not very maintainable if the targets in the project are a large number, as for each one it is necessary to tag the corresponding scheme as 'shared' and add the post-build action.
To address this problem, I have created a build script that generates the archive path programmatically by extracting the last build that matches the target name on the current day. This method works reliably as long as there aren't multiple builds with the same target name running on the machine (this may be a problem in production environments where multiple concurrent builds are run).
#!/bin/bash
#
# Script to archive an existing xcode project to a target location.
# The script checks for a post-build action that defines the $ARCHIVE_PATH as follows:
# echo "ARCHIVE_PATH=\"$ARCHIVE_PATH\"" > $PROJECT_DIR/archive_paths.sh
# If such post-build action does not exist or sourcing it doesn't define the $ARCHIVE_PATH
# variable, the script tries to generate it programmatically by finding the latest build
# in the expected archiving folder
#
post_build_script=archive_paths.sh
build_errors_file=build_errors.log
OUTPUT=output/
XCODEBUILD_CMD='/Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild'
TARGET_SDK=iphoneos
function archive()
{
echo "Archiving target '$1'"
# Delete $post_build_script if it already exists as it should be generated by a
# post-build action
rm -f $post_build_script
# Use custom provisioning profile and code sign identity if specified, otherwise
# default to project settings
# Note: xcodebuild always returns 0 even if the build failed. We look for failure in
# the stderr output instead
if [[ ! -z "$2" ]] && [[ ! -z "$3" ]]; then
${XCODEBUILD_CMD} clean archive -scheme $1 -sdk "${TARGET_SDK}" \
"CODE_SIGN_IDENTITY=$3" "PROVISIONING_PROFILE=$2" 2>$build_errors_file
else
${XCODEBUILD_CMD} clean archive -scheme $1 -sdk "${TARGET_SDK}"
2>$build_errors_file
fi
errors=`grep -wc "The following build commands failed" $build_errors_file`
if [ "$errors" != "0" ]
then
echo "BUILD FAILED. Error Log:"
cat $build_errors_file
rm $build_errors_file
exit 1
fi
rm $build_errors_file
# Check if archive_paths.sh exists
if [ -f "$post_build_script" ]; then
source "$post_build_script"
if [ -z "$ARCHIVE_PATH" ]; then
echo "'$post_build_script' exists but ARCHIVE_PATH was not set.
Enabling auto-detection"
fi
fi
if [ -z "$ARCHIVE_PATH" ]; then
# This is the format of the xcarchive path:
# /Users/$USER/Library/Developer/Xcode/Archives/`date +%Y-%m-%d`/$1\
# `date +%d-%m-%Y\ %H.%M`.xcarchive
# In order to avoid mismatches with the hour/minute of creation of the archive and
# the current time, we list all archives with the correct target that have been
# built in the current day (this may fail if the build wraps around midnight) and
# fetch the correct file with a combination of ls and grep.
# This script can break only if there are multiple targets with exactly the same
# name running at the same time.
EXTRACTED_LINE=$(ls -lrt /Users/$USER/Library/Developer/Xcode/Archives/`date
+%Y-%m-%d`/ | grep $1\ `date +%d-%m-%Y` | tail -n 1)
if [ "$EXTRACTED_LINE" == "" ]; then
echo "Error: couldn't fetch archive path"
exit 1
fi
# ls -lrt prints lines with the following format
# drwxr-xr-x 5 mario 1306712193 170 25 Jul 17:17 ArchiveTest 25-07-2013
# 17.17.xcarchive
# We can split this line with the " " separator and take the latest bit:
# 17.17.xcarchive
FILE_NAME_SUFFIX=$(echo $EXTRACTED_LINE | awk '{split($0,a," "); print a[11]}')
if [ "$FILE_NAME_SUFFIX" == "" ]; then
echo "Error: couldn't fetch archive path"
exit 1
fi
# Finally, we can put everything together to generate the path to the xcarchive
ARCHIVE_PATH="/Users/$USER/Library/Developer/Xcode/Archives/`date
+%Y-%m-%d`/$1 `date +%d-%m-%Y` $FILE_NAME_SUFFIX/"
fi
# Create output folder if it doesn't already exist
mkdir -p "$OUTPUT"
# Move archived xcarchive build to designated output folder
mv -v "$ARCHIVE_PATH" "$OUTPUT"
}
# Check number of command line args
if [ $# -lt 1 ]; then
echo "Syntax: `basename $0` <target name> [/path/to/provisioning-profile]
[<code sign identity]"
exit 1
fi
if [ ! -z "$2" ]; then
PROVISIONING_PROFILE="$2"
fi
if [ ! -z "$3" ]; then
SIGN_PROVISIONING_PROFILE="$3"
else
if [ ! -z "$PROVISIONING_PROFILE" ]; then
SIGN_PROVISIONING_PROFILE=$(cat "$PROVISIONING_PROFILE" | egrep -a -o
'[A-Fa-f0-9]{8}-[A-Fa-f0-9]{4}-[A-Fa-f0-9]{4}-[A-Fa-f0-9]{4}-[A-Fa-f0-9]{12}')
fi
fi
archive "$1" "$PROVISIONING_PROFILE" "$SIGN_PROVISIONING_PROFILE"
Full source code with an example Xcode project can be found here:
https://github.com/bizz84/Xcode-xcarchive-command