I am getting started with azure-pipelines.yml
I wanted to have 2 jobs within the same stage. One to build a solution and the other to run unit tests.
The problem is that the second job executed a script step and it does not find a folder Release that the previous one should have created:
trigger:
- master
pool:
vmImage: 'ubuntu-18.04'
stages:
- stage: CI
jobs:
- job: Build
steps:
- task: NuGetAuthenticate#0
- script: dotnet restore --no-cache --force
- script: dotnet build --configuration Release --no-restore
- job: UnitTests
dependsOn: Build
steps:
- script: dotnet vstest test/*UnitTests/bin/Release/**/*UnitTests.dll
However if I add all the steps within the same job it works:
trigger:
- master
pool:
vmImage: 'ubuntu-18.04'
stages:
- stage: CI
jobs:
- job: Build
steps:
- task: NuGetAuthenticate#0
- script: dotnet restore --no-cache --force
- script: dotnet build --configuration Release --no-restore
- script: dotnet vstest test/*UnitTests/bin/Release/**/*UnitTests.dll
I cannot find an answer on why a dependent job cannot find on the file system the folders that a previous one has generated. Any explanation or link to something that clarifies that would be much appreciated.
I have used gitlab in the past and I don't recall a similar behavior although I don't know whether it had the concept of job as a different thing to steps.
The key element that you are missing is that jobs run on independent agents (separate computers) and do not have any kind of shared filesystem.
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml
Any files created in one job that you want to make available on a dependent job must be explicitly staged (in job 'A') and then explicitly downloaded (in job 'B').
See publish:
https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/utility/publish-build-artifacts?view=azure-devops
And download:
https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/utility/download-build-artifacts?view=azure-devops
Related
Problem Background:
I have a class library project that contains the database migrations (MyProject.MigrationProject.csproj). And in startup.cs of the entry project (Web API), I have explicitly included migration assembly like following.
services.AddDbContext<ApplicationDbContext>(options =>
options.UseSqlServer(
configuration.GetConnectionString("DefaultConnection"),
x => x.MigrationsAssembly("MyProject.MigrationProject")));
Then I run the dotnet ef migration command locally using powershell. The command I'm using is:
dotnet ef migrations script --no-build -o D:\migrations\script.sql --idempotent --project D:\...\src\MyProject.MigrationProject\MyProject.MigrationProject.csproj --startup-project D:\...\src\MyProject.WebApi\MyProject.WebApi.csproj
The above command executes successfully on my machine and creates the desired script.sql file on output location. The same command is then used in the build pipeline (using command line task) in Azure Devops but for some reason it fails there. The command on Devops looks like this:
dotnet ef migrations script --no-build -o $(Build.ArtifactStagingDirectory)\migrations\script.sql --idempotent --project $(Build.SourcesDirectory)\src\MyProject.MigrationProject\MyProject.MigrationProject.csproj --startup-project $(Build.SourcesDirectory)\src\MyProject.WebApi\MyProject.WebApi.csproj
The Error I get from Devops:
Script contents:
dotnet ef migrations script --no-build -o D:\a\1\a\migrations\script.sql --idempotent --project D:\a\1\s\src\MyProject.MigrationProject\MyProject.MigrationProject.csproj --startup-project D:\a\1\s\src\MyProject.WebApi\MyProject.WebApi.csproj
##[debug]AGENT_VERSION: '2.193.1'
##[debug]AGENT_TEMPDIRECTORY: 'D:\a\_temp'
##[debug]Asserting container path exists: 'D:\a\_temp'
##[debug]Asserting leaf path exists: 'C:\Windows\system32\cmd.exe'
========================== Starting Command Output ===========================
##[debug]Entering Invoke-VstsTool.
##[debug] Arguments: '/D /E:ON /V:OFF /S /C "CALL "D:\a\_temp\37fc4a71-a144-4332-9a84-04e6138a2538.cmd""'
##[debug] FileName: 'C:\Windows\system32\cmd.exe'
##[debug] WorkingDirectory: 'D:\a\1\s'
"C:\Windows\system32\cmd.exe" /D /E:ON /V:OFF /S /C "CALL "D:\a\_temp\37fc4a71-a144-4332-9a84-04e6138a2538.cmd""
An error occurred while accessing the Microsoft.Extensions.Hosting services. Continuing without the applicgation service provider. Error: A certificate with the thumbprint 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' could not be found.
Unable to create an object of type 'ApplicationDbContext'. For the different patterns supported at design time, see https://go.microsoft.com/fwlink/?linkid=851728
##[debug]Exit code: 1
##[debug]Leaving Invoke-VstsTool.
##[error]Cmd.exe exited with code '1'.
##[debug]Processed: ##vso[task.logissue type=error]Cmd.exe exited with code '1'.
##[debug]Processed: ##vso[task.complete result=Failed]Error detected
##[debug]Leaving D:\a\_tasks\CmdLine_d9bafed4-0b18-4f58-968d-86655b4d2ce9\2.182.0\cmdline.ps1.
Finishing: CmdLine
At times, by tweaking the YAML file, I was able to get rid of the first error but the 2nd one never disappeared on devops. The issue is pretty much because of having separate project for Migrations but I think that's how it should be...
My Build pipline's YAML:
trigger:
- develop
pool:
vmImage: 'windows-latest'
variables:
solution: '**/*.sln'
buildPlatform: 'Any CPU'
buildConfiguration: 'Release'
steps:
- task: DotNetCoreCLI#2
displayName: Restore
inputs:
command: restore
projects: '**/MyProject.WebApi.csproj'
- task: DotNetCoreCLI#2
displayName: Build
inputs:
projects: '**/MyProject.WebApi.csproj'
arguments: '--no-restore'
- task: DotNetCoreCLI#2
displayName: Test
inputs:
command: test
projects: '**/*[Tt]ests/*.csproj'
arguments: '--no-restore --no-build'
- task: DotNetCoreCLI#2
displayName: 'Publish WebApi'
inputs:
command: publish
publishWebProjects: false
projects: '**/MyProject.WebApi.csproj'
arguments: '--configuration $(BuildConfiguration) --output $(Build.ArtifactStagingDirectory) --runtime -r $(runtime)'
- task: CopyFiles#2
inputs:
Contents: '**'
TargetFolder: '$(Build.ArtifactStagingDirectory)'
- task: UseDotNet#2
inputs:
packageType: 'sdk'
version: '5.x'
- task: DotNetCoreCLI#2
displayName: Install dotnet-ef
inputs:
command: 'custom'
custom: 'tool'
arguments: 'install --global dotnet-ef --version 5.0.10 --ignore-failed-sources'
- task: CmdLine#2
inputs:
script: dotnet ef migrations script --no-build -o $(Build.ArtifactStagingDirectory)\migrations\script.sql --idempotent --project $(Build.SourcesDirectory)\src\MyProject.MigrationProject\MyProject.MigrationProject.csproj --startup-project $(Build.SourcesDirectory)\src\MyProject.WebApi\MyProject.WebApi.csproj
- task: PublishBuildArtifacts#1
displayName: 'Publish Artifact: drop'
My Suspicion:
It could be an issue with the directory where the command is executed from (on ADO powershell). I suspect this because, on my local machine, before calling the method x.MigrationsAssembly("MyProject.MigrationProject"), the following command failed when I executed it from a directory other than the entry project's directory but when I navigated the powershell to entry project and executed the same command, it went successful. The command at that time was:
dotnet ef migrations script -o D:\migrations\script.sql --idempotent --project D:\...\src\MyProject.MigrationProject\MyProject.MigrationProject.csproj
I'm already using the same YAML in another project but that contains everything in single Web API project and so, I do not get any issue there.
Question:
What am I doing wrong here? What can I do to fix this issue? Any help would be appreciated.
Project Details
DotNet 5.0
EntityFramewokCore 5.0.10
Visual Studio 2019
If I'm missing anything, please ask.
Updates:
My suspicion about working directory for executing the dotnet ef command appears to be wrong as I tried that by supplying workingDirectory parameter to the command line tasks. It works on local machine though.
Thanks #jane-ma-msft
The Error message shows that it is a Certificate error. Please follow the work around to fix the issue.
Try to generate a new certificate or cancel certificate validation.
Check your .sln file. If it has PackageCertificateKeyFile & PackageCertificateThumbprint please try to remove the properties and restart a pipeline.
Or Check that it is configured correctly and that you have uploaded the correct certificate file to the appropriate path.
Make sure that the agent windows-latest has all the .NET SDK versions you need and all the software your project needs to reference. If not, use the task or command line to download them. Click this link to view the software installed on the windows-latest agent.
If you are using the Microsoft-hosted Windows agents in your pipeline, please try to use the self-hosted Agent in your pipeline. Click this document for detailed steps.
Refer here Link 1 & Link 2
I'm facing trouble with my GitLab CI. The expected workflow for my Test stage is :
fetching changes
checking out
downloading artifacts
run dotnet test
uploading artifacts
The current workflow is :
fetching changes
checking out (removing all bin/obj folder)
uploading artifacts -> this line failed due to inexistance of my test-results.xml file
My solution looks like this:
-- yml file
-- MyApp
-- MyApp.Tests
The version of my gitlab-runner is 12.4.1 and I'm using the shell executor on Windows environment.
Please find below my yml file:
image: mcr.microsoft.com/dotnet/core/sdk:3.1
variables:
EXE_RELEASE_FOLDER: 'Oncolin.UI.Desktop\bin\Release'
# the folder inside the deploy folder
DEPLOY_FOLDER_APP: 'Oncolin.UI.Desktop\Builds'
CONFIGURATION: 'release'
#NUGET_PATH: 'C:\NuGet\nuget.exe'
#MSBUILD_PATH: 'C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Bin\MSBuild.exe'
#MSTEST_PATH: 'C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\MSTest.exe'
NUNIT_PATH: '.\packages\NUnit.ConsoleRunner.3.9.0\tools\nunit3-console.exe'
SOLUTION: 'Oncolin.sln'
stages:
- build
- test
build:
stage: build
tags:
- windows
script:
- '& nuget restore' # restore Nuget dependencies
- '& dotnet restore'
- '& SonarScanner.MSBuild.exe begin /k:Oncolin /d:sonar.host.url=https://docker:10500 '
- '& dotnet build -c release'
- '& "$env:MSBUILD_PATH" "$env:SOLUTION" /t:Build /p:Configuration=Release' # build the project
- '& SonarScanner.MSBuild.exe end'
artifacts:
expire_in: 1 week # save gitlab server space, we copy the files we need to deploy folder later on
paths:
- '$env:EXE_RELEASE_FOLDER\*' # saving exe to copy to deploy folder
- 'RNC.RULES.QUALITY.TESTS\*' # saving entire Test project so NUnit can run tests
- 'RNC.MODEL.TNM.TESTS\*' # saving entire Test project so NUnit can run tests
- 'Oncolin.Model.Tests\*' # saving entire Test project so NUnit can run tests
test:
stage: test
tags:
- windows
allow_failure: false
script:
- '& dotnet test --test-adapter-path:. --logger:"junit;LogFilePath=$CI_PROJECT_DIR\artifacts\{assembly}-test-result.xml;MethodFormat=Class;FailureBodyFormat=Verbose"'
artifacts:
expire_in: 1 week # save gitlab server space, we copy the files we need to deploy folder later on
when: always
paths:
- $CI_PROJECT_DIR\artifacts\*test-result.xml
reports:
junit:
- $CI_PROJECT_DIR\artifacts\*test-result.xml
dependencies:
- build
Can someone explain to me why the downloading and dotnet test steps are skipped ? I hope I gave enough informations.
Thanks in advance.
The issue came from the sonar scanner. Even if I used the .Net Core sonar scanner, I had the same issue during my test stage.
I don't know why and I have already lost too many time to investigate on that.
I resolved it by creating a new stage specificly to run the sonar and ran it after the test stage.
I am new to CI and tried to add the following yaml to our project.
image: microsoft/dotnet:latest
stages:
- test
test:
stage: test
before_script:
- 'echo Stage - Test started'
- 'dotnet tool install dotnet-reportgenerator-globaltool --tool-path tools'
script:
- 'cd "./My Project"'
- 'dotnet test --settings:./Settings/CodeCoverage.runsettings --collect:"XPlat Code Coverage"'
- './tools/reportgenerator "-reports:./Tests/*/TestResults/*/coverage.cobertura.xml" "-targetdir:Reports_Coverage" -reportTypes:TextSummary'
- 'cat ./Reports_Coverage/Summary.txt'
after_script:
- 'echo Stage - Test finished'
On the job I get the following output:
Running with gitlab-runner 13.7.0 (943fc252) on ***
Resolving secrets
Preparing the "kubernetes" executor
Using Kubernetes namespace: gitlab-pipeline
Using Kubernetes executor with image microsoft/dotnet:latest ...
Preparing environment
Running on*** via ***...
Getting source from Git repository
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/***/.git/
Created fresh repository.
Checking out 07e98444 as ***...
Skipping Git submodules setup
Executing "step_script" stage of the job script
$ echo Stage - Test started
Stage - Test started
$ dotnet tool install dotnet-reportgenerator-globaltool --tool-path tools
You can invoke the tool using the following command: reportgenerator
Tool 'dotnet-reportgenerator-globaltool' (version '4.8.4') was successfully installed.
$ cd "./My Project"
$ dotnet test --settings:./Settings/CodeCoverage.runsettings --collect:"XPlat Code Coverage"
Running after_script
Running after script...
$ echo Stage - Test finished
Stage - Test finished
Cleaning up file based variables
ERROR: Job failed: command terminated with exit code 1
I tinkered around a little bit and realized, that this minified yaml also fails:
image: microsoft/dotnet:latest
stages:
- test
test:
stage: test
before_script:
- 'cd "./My Project"'
- 'dotnet restore'
script:
- 'dotnet test --no-restore'
This generates an equal output, except it stops at dotnet restore.
I honestly don't know what to do, since this is the most minified version of a test I found.
Did I mess up something within the project, or is the problem within the GitLab Runner itself?
Project Version: .net Core App 3.1
Not a real solution, but I found the issue behind this. You can set the verbosity level of the dotnet test command (documentation). While doing so, I found out that you obviously cannot run a WPF project within a Linux based GitLab Runner.
Have a yaml file build script and all works fine except for the unit test which fails after various warnings eventually raising MSB3027 and MSB3021 errors.
This is in a pipeline in Atlassian.
Seen issues mentioning long PATH names but how to resolve here? Have similar pipelines setup and working before.
image: mcr.microsoft.com/dotnet/core/sdk:3.1
pipelines:
branches:
develop:
- step:
name: Build and Test
caches:
- dotnetcore
script:
- export ENV_NAME=develop
- export PROJECT_NAME=Myapi
- export ZIP_FILE=Myapi-dev.zip
- export TEST_PROJECT=Myapi.Tests
- dotnet build $PROJECT_NAME
- dotnet test $TEST_PROJECT --no-build --configuration Release
# this will publish our app output into a subfolder so that we can package it up
- dotnet publish $PROJECT_NAME --output $ENV_NAME/publish --configuration release
/usr/share/dotnet/sdk/3.1.402/Microsoft.Common.CurrentVersion.targets(4364,5):
warning MSB3026: Could not copy "/root/.nuget/packages/microsoft.testplatform.objectmodel/16.7.1/lib/netstandard2.0/pt-BR/Microsoft.TestPlatform.CoreUtilities.resources.dll"
to "bin/Release/netcoreapp3.1/pt-BR/Microsoft.TestPlatform.CoreUtilities.resources.dll". Beginning retry 1 in 1000ms.
Could not find a part of the path '/opt/atlassian/pipelines/agent/build/myapi/bin/Release/netcoreapp3.1/pt-BR/Microsoft.TestPlatform.CoreUtilities.resources.dll'.
In the end this was a third party component and the resource file the Test project needed was not in the expected location for the build pipeline. I worked around the issue by copying the files it needed into the target location.
# Workaround for resource file case sensitivity issue
- ls -lR .
- cp -R ./Myapi/bin ./Myapi.Tests/bin
- cp -R ./Myapi/bin/Debug/netcoreapp3.1/pt-br ./Myapi.Tests/bin/Debug/netcoreapp3.1/pt-BR
- dotnet test $TEST_PROJECT
I have a simple solution that have two projects in it and I want it go through the azure build pipeline. One project is pure class files that will get build a DLL but that code is not the GIT repo in the same folder. And the other project is ASP.NET project and I have configured that in the pipeline. When I am trying to build the project it is giving me an error as below:
MSBUILD : error MSB1011: Specify which project or solution file to use
because this folder contains more than one project or solution file.
[error]Cmd.exe exited with code '1'.
Yaml File
# ASP.NET Core
# Build and test ASP.NET Core projects targeting .NET Core.
# Add steps that run tests, create a NuGet package, deploy, and more:
# https://learn.microsoft.com/azure/devops/pipelines/languages/dotnet-core
trigger:
- master
pool:
name: 'buildserver'
variables:
buildConfiguration: 'Release'
steps:
- script: dotnet build --configuration $(buildConfiguration)
displayName: 'dotnet build $(buildConfiguration)'
The error is asking you to identify the .sln or .csproj (or other proj type) file b/c dotnet doesn't know what you want it to do.
I tend to use variables for this kind of thing because I'm often using the solution name in other tasks.
example:
trigger:
- master
pool:
name: 'buildserver'
variables:
buildConfiguration: 'Release'
slnName: 'mySol'
solution: 'some/dir/$(slnName).sln'
steps:
- script: dotnet build $(solution) --configuration $(buildConfiguration)
displayName: 'dotnet build $(buildConfiguration)'
.csproj example:
With a repository (and solution) structure as follows:
(this is the part of your question that remains unclear)
myRepo
|--.git
|--src
|--proj1
| |--proj1.csproj
|
|--proj2
| |--proj2.csproj
|
|--mySol.sln
You would simply call out the .csproj file you want the pipeline to build.
trigger:
- master
pool:
name: 'buildserver'
variables:
buildConfiguration: 'Release'
projName: 'proj1'
project: 'src/$(projName)/$(projName).csproj'
steps:
- script: dotnet build $(project) --configuration $(buildConfiguration)
displayName: 'dotnet build $(buildConfiguration)'
With the above examples you shouldn't need to specify the .sln or .csproj as each potential build target lives in its own directory, and the dotnet cli searches from $pwd if you don't give it a value. Therefore, if your pipeline is working in the root of the repo (default behavior), dotnet should find the .sln file first and build it.
However
If your directory structure looks like the following, then you would need to specify:
myRepo
|--.git
|--src
|--proj1.csproj
|--class1.cs
|--class2.cs
|--proj2
| |--proj2.csproj
| |--class1.cs
|
|--mySol.sln
In the above dotnet doesn't know whether you want to build mySol.sln or proj1.csproj, so indicating the file to build should solve your problem, but I might suggest that you restructure your repository.
If proj2 doesn't make its home in myRepo
And is a dependency of proj1 then you will need to do some other acrobatics (ie: manual git repo clone) in the pipeline to get that project and it's files where they need to be. If this is the case, I would strongly suggest you treat proj2 as a completely independent product and deliver it to those projects (proj1) that depend upon it via NuGet or other package delivery method.