How to select certain test to run in VSTS?
I have few unit tests and I run them via build in VSTS - actually I have to run all of them from assembly. What I'm looking for is a feature similar to Visual Studio's Test Explorer - I want to select certain tests and run them from VSTS (not all of them).
Any ideas, please?
In the "Visual Studio Test" task you can put value in Test filter criteria field:
This field is additional criteria to filter tests from Test assemblies (for example: Priority=1|Name=MyTestMethod).
You could then use the criteria to specify the specific class you want to run tests from using:
FullyQualifiedName~NameSpace.Test
Related
I have a ASP.NET Web Forms project which is hosted on a Azure DevOps Repository. The Source Control is TFS.
I Created a continuous integration Pipeline in Azure Devops for the Repository.
Here is a Image of the Pipeline:
All the Task from the Pipeline are running correctly beside the VSTest-TestAssemblies Task.
My NUnit Test is failing in my local Visual Studio, as it should:
But when running my CI, it tells my that no test failed:
Here you can see my settings and the path to my tests of the VSTest-TestAssemblies Task:
This is what the Folder Structure of my Project looks like:
NR.TestAutomation
- NR.TestAutomation
- NR.TestAutomation.Tests
- bin
- obj
- DemoTests.cs
- NR.TestAutomation.Tests.csproj
- packages
- NR.TestsAutomation.sln
The Output of the CI Run tells me, that the folder exist and it even finds the test:
Source filter: **/*Tests/*.cs
SystemVssConnection exists true
vstest.console.exe "D:\a\9\s\NR.TestAutomation.Tests\DemoTests.cs"
Why are my failing tests not failing inside the Azure DevOps Pipeline?
You should specify the (output of the) TestProject to run, not the individual files that contain a test. The test runner cannot execute a .cs file.
So in short; the failing test doesn't run successful in an Azure DevOps pipeline... it doesn't run in the pipeline at all.
Test files
Run tests from the specified files. Ordered tests and webtests can be run by specifying the .orderedtest and .webtest files respectively. To run .webtest, Visual Studio 2017 Update 4 or higher is needed. The file paths are relative to the search folder. Supports multiple lines of minimatch patterns.
Source Visual Studio Test task - Arguments
Possibly interesting as well: more info on File matching patterns reference
EDIT:
what would be the proper Source Path to my tests?
Of course that depends on the name of your test project. Looking at your screenshots, you could use **/*.Tests.dll. This recursively matches all the assemblies where the name ends with .Tests.dll.
Since your project's name is NR.TestAutomation.Tests, the assembly should be named NR.TestAutomation.Tests.dll so it will match. It's also ready for future additions of test projects, as long as their name ends with .Tests.
So, I generate the code coverage report by running dotnet test --no-build -l trx --collect "Code coverage". Unfortunately, it also computes the code coverage for the test assemblies. How do I tell it not to collect for test assemblies?
The answer is here - https://learn.microsoft.com/en-us/visualstudio/test/customizing-code-coverage-analysis?view=vs-2019 and here - https://learn.microsoft.com/en-us/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file?view=vs-2019
For some reason Microsoft thinks that scanning the test assemblies is a good default. Too bad for us. At the end I went up with the custom runsettings file that excludes the test assemblies manually and includes only those having our company name.
I am pretty new to CI/CD and mm currently struggling with getting VSTS to find my xUnit tests. My solution is a pure experiment, to try and isolate the problem and to learn. Before getting into my setup and what's been done so far, the result is this line in the VSTS build log:
Warning: No test is available in C:\agent\_work\1\s\Quotifier.Models.xUnit.Test\bin\Release\netcoreapp2.0\Quotifier.Models.xUnit.Test.dll. Make sure that installed test discoverers & executors, platform & framework version settings are appropriate and try again.
From having googled the message I interpret it as VSTS not being able to find an xUnit test adapter. Is that a correct assumption? Also, I see the terms "test adapter", "test runner" and "test explorer" alot. Are these terms the same thing? Having checked out this article it seems they are indeed the same thing. Can anyone clarify?
Now, for my setup ...
The xUnit test project is a .NET Core 2 class library referencing NUget packages xunit and xunit.runner.visualstudio (2.2.0). The test project is also referencing a shared class library called "Quotifier.Models" which contains the tested code.
To make it easier to diagnose I am running a local (on prem) build agent on my PC. This enables me to investigate the file structure.
The build step of my build definition was initially a "Visual Studio Build" that built the whole solution. I can see that the resulting binaries ends up in
C:\agent\_work\1\s\Quotifier.Models.xUnit.Test\bin\Release\netcoreapp2.0.
Among the binaries are also xunit.runner.visualstudio.dotnetcore.testadapter.dll. Is this the xUnit test adapter VSTS can't find?
More googling tells me that the typical place for the test adapter to be found is in the $(Build.SourcesDirectory)\packages folder and, so, that's what the test step's "Path to custom test adapters" should be set to. That gave me this log entry:
Warning: The path 'C:\agent\_work\1\s\packages' specified in the 'TestAdapterPath' does not contain any test adapters, provide a valid path and try again..
I also checked that local folder and, indeed, the xunit NUget packages does not end up in there. Can anyone guess as to why?
Assuming the xunit.runner.visualstudio.dotnetcore.testadapter.dll is indeed the test adapter VSTS is looking for, I thought I should try and help it a bit. So I created a seperate MSBuild step and specified the output path via the "MSBuild arguments":
/p:OutputPath="$(build.binariesdirectory)/$(BuildConfiguration)/test-assemblies".
I then set the test step's "Path to custom test adapters" property to point at that same folder. With that I'm back to the original warning in the log:
Warning: No test is available in C:\agent\_work\1\b\release\test-assemblies\Quotifier.Models.xUnit.Test.dll. Make sure that installed ... bla ... bla
Now, to summarize ...
running xUnit test projects out of the box doesn't work with VSTS
The xUnit NUget packages does not end up in the /**/packages folder
I am assuming the xunit.runner.visualstudio.dotnetcore.testadapter.dll is the test adapter needed. Is this assumtion correct?
I'm out of ideas for now so any hints, links or suggestions will be greatly appreciated.
Use "dotnet test" to run Your NetCore-UnitTests.
The task is called ".NET Core" and has a command dropdown where you select build/restore/test etc... You also need to add "NET Core Tool Installer" if you are using a hosted agent
And your xunit-packages are not supposed to end up in "packages"-directory. The netcore2 projects handle nuget-packages differently and will reference them from the local nuget-cache
I have a suite with more than 500 test scenarios written in Web driver. I am using TestNG framework to execute the test cases.
Could you please suggest me how to execute specific test cases from the suite like 31, 45, 68 etc?
Thank you
If you have prefaced all of your tests with #Test, then in Eclipse with TestNG plugin, you can right click on the method name, and run it as a TestNG Test. You can also click on a class file, and only run that class. However, if you want to run test X and test Y that are in different tests, you can press Run Configurations... and add which methods you want run.
If you don't have the TestNG plugin, you can write an XML file that has the classes, methods, groups, or suites you want to run. It is described here
I have created a new build definition for TFS 2010. After building my C# solution I would like it to execute a couple of unit tests. These unit tests require an XML input file, so I have a [DeploymentItem] attribute to the test methods which provides the relative path the XML files. If I run the unit tests from within Visual Studio they pass ok.
When the unit tests get run following a build (via my build definition), they fail with: "Microsoft.BizTalk.TestTools.BizTalkTestAssertFailException: Input file does not exist..."
It would be great if I could get to a trace of what the build agent was trying to do, to help with troubleshooting.
Does anyone know how to get such a trace output? I guess I could increase the verbosity of the trace output from the main solution under test but I don't think that would give me any indication of where the build agent was looking for the test input XML or why?
Thanks
Rob
I found it! Needed to click the "View Log" link from the screen that's displayed following the build. I had been looking at the default view of "View Summary"