I'm writing Robot Framework tests for a Plone project.
Here is the code.
I want tests to be executed on Sauce Labs, but with two different OS's:
iOS
Android
I'm starting by replacing the current default tests (Firefox / WinXP) with iOS, using the following parameters in my .travis.yml's matrix:
- TARGET=test ROBOT_DESIRED_CAPABILITIES="platform:OS X 10.8,browserName:iPad,version:6,tunnel-identifier:$TRAVIS_JOB_ID"
Tests are now failing with the following issues:
Could not proxy http://localhost:3001/wd/hub/session, exception: java.net.ConnectException: Connection refused
I have no idea why. I have read all the stuff in example.product and I didn't find any differences.
Here are my builds at Travis CI, which show the errors.
And you can find my Sauce Labs results here.
Currently, you have to run your test ZServer (by settings ZSERVER_PORT environment variable) at some port supported by Sauce Labs: https://saucelabs.com/docs/connect#localhost
I have a working example at: https://github.com/datakurre/example.product
I filed a request for SauceLabs to add support for port 55001 used by Plone, and the port will be included in the next SauceConnect-release.
Update: There was an another issue. p.a.robotframework defines Selenium version for Sauce Labs, currently 2.31.0 (UPDATE: next release will not set the default value). I asked this from Sauce Labs and it seems that selenium version should not be set for mobile tests at all.
I have now a working p.a.robotframework example at: https://github.com/datakurre/example.product/tree/p.a.robotframework
To reset Sauce Labs Selenium version set by p.a.robotframework, you could set an environment variable ROBOT_SELENIUM_VERSION=" " for mobile Travis builds as in https://github.com/datakurre/example.product/commit/4deafc843b89a66da8cf6a85955a2b936ada79b4
Technically ROBOT_SELENIUM_VERSION="" should be enough for p.a.robotframework to use the default Selenium version of Sauce Labs, but when I tested that in .travis.yml, Travis didn't set that environment variable at all and it didn't override p.a.robotframework's default. Anyway, the next p.a.robotframework-release (>= 0.6.2) will set no selenium-version for Sauce Labs by default.
Background:
Sauce Labs documentation related to Selenium version: https://saucelabs.com/docs/additional-config#selenium-version
p.a.robotframework code for defining SELENIUM_VERSION-variable for Robot Framework and forcing selenium-version Selenium desired capabilities string for Sauce Labs: https://github.com/plone/plone.app.robotframework/blob/master/src/plone/app/robotframework/saucelabs.robot#L25
Finally robotsuite, which is used used to wrap Robot Framework tests for zope.testrunner, passes all ROBOT_-prefixed environment variables for Robot Framework test runner without the prefix, effectively overriding the default set in p.a.robotframework.
Related
I've created a test console app using .net core 3.1, added it to github, and setup a GitHub action using the template provided by Sonarcloud.io.
The build analysis part looks like this:
.\.sonar\scanner\dotnet-sonarscanner begin /k:"<projectkey>" /o:"<org>" /d:sonar.login="${{ secrets.SONAR_TOKEN }}" /d:sonar.host.url="https://sonarcloud.io" /d:sonar.scanner.force-deprecated-java-version=true
dotnet build --configuration Release --no-restore
.\.sonar\scanner\dotnet-sonarscanner end /d:sonar.login="${{ secrets.SONAR_TOKEN }}"
This builds fine and reports no errors. Yet when I go to sonarcloud.io although I can see it's received the analysis, it's not showing any issues, despite my purposely adding some:
The master branch displays this:
Yet, the code is visible in the code tab:
For PRs I see:
But there are issues (I can see them locally when using SonarLint pointing at sonarcloud.io).
What am I missing?
It turns out this wasn't showing any issues because it was considering all my files as test code. I had created this whole repo as a Sonarcloud test, and my project was called SonarCloudTest.csproj and Sonar has some regex rules around categorising code.
I am pretty new to CI/CD and mm currently struggling with getting VSTS to find my xUnit tests. My solution is a pure experiment, to try and isolate the problem and to learn. Before getting into my setup and what's been done so far, the result is this line in the VSTS build log:
Warning: No test is available in C:\agent\_work\1\s\Quotifier.Models.xUnit.Test\bin\Release\netcoreapp2.0\Quotifier.Models.xUnit.Test.dll. Make sure that installed test discoverers & executors, platform & framework version settings are appropriate and try again.
From having googled the message I interpret it as VSTS not being able to find an xUnit test adapter. Is that a correct assumption? Also, I see the terms "test adapter", "test runner" and "test explorer" alot. Are these terms the same thing? Having checked out this article it seems they are indeed the same thing. Can anyone clarify?
Now, for my setup ...
The xUnit test project is a .NET Core 2 class library referencing NUget packages xunit and xunit.runner.visualstudio (2.2.0). The test project is also referencing a shared class library called "Quotifier.Models" which contains the tested code.
To make it easier to diagnose I am running a local (on prem) build agent on my PC. This enables me to investigate the file structure.
The build step of my build definition was initially a "Visual Studio Build" that built the whole solution. I can see that the resulting binaries ends up in
C:\agent\_work\1\s\Quotifier.Models.xUnit.Test\bin\Release\netcoreapp2.0.
Among the binaries are also xunit.runner.visualstudio.dotnetcore.testadapter.dll. Is this the xUnit test adapter VSTS can't find?
More googling tells me that the typical place for the test adapter to be found is in the $(Build.SourcesDirectory)\packages folder and, so, that's what the test step's "Path to custom test adapters" should be set to. That gave me this log entry:
Warning: The path 'C:\agent\_work\1\s\packages' specified in the 'TestAdapterPath' does not contain any test adapters, provide a valid path and try again..
I also checked that local folder and, indeed, the xunit NUget packages does not end up in there. Can anyone guess as to why?
Assuming the xunit.runner.visualstudio.dotnetcore.testadapter.dll is indeed the test adapter VSTS is looking for, I thought I should try and help it a bit. So I created a seperate MSBuild step and specified the output path via the "MSBuild arguments":
/p:OutputPath="$(build.binariesdirectory)/$(BuildConfiguration)/test-assemblies".
I then set the test step's "Path to custom test adapters" property to point at that same folder. With that I'm back to the original warning in the log:
Warning: No test is available in C:\agent\_work\1\b\release\test-assemblies\Quotifier.Models.xUnit.Test.dll. Make sure that installed ... bla ... bla
Now, to summarize ...
running xUnit test projects out of the box doesn't work with VSTS
The xUnit NUget packages does not end up in the /**/packages folder
I am assuming the xunit.runner.visualstudio.dotnetcore.testadapter.dll is the test adapter needed. Is this assumtion correct?
I'm out of ideas for now so any hints, links or suggestions will be greatly appreciated.
Use "dotnet test" to run Your NetCore-UnitTests.
The task is called ".NET Core" and has a command dropdown where you select build/restore/test etc... You also need to add "NET Core Tool Installer" if you are using a hosted agent
And your xunit-packages are not supposed to end up in "packages"-directory. The netcore2 projects handle nuget-packages differently and will reference them from the local nuget-cache
New to Unit testing so please, forgive my naiveness. Browsing through the code provided by the IE Developer Channel. I adopted the code into my MVC project by adding a Unit Test project. On debugging the test I write using the WebDriver API. The first step creating a session using the Command
const string newSessionCommand = #"
{
""command"": {
""name"": ""newSession"",
""parameters"": {
""desiredCapabilities"": {},
""requiredCapabilities"": {}
},
""sessionId"": null
}
}";
response = ExecuteCommand(newSessionCommand, null);
The command fails with the error "WebDriver will only run against the DeveloperPreview build. Please run this EXE with the following argument", compared the sample, added all that is needed like the x64 architecture, registering the dll, NOTE that is works in the BING search sample sln, but NOT as a test case, also added to the debug the command line arguments:
/appvve:9BD02EED-6C11-4FF0-8A3E-0B4733EE86A1_6A0357B5-AB99-4856-8A59-CF2C38579E78
You are getting the "WebDriver will only run against the DeveloperPreview build." because either you not are not running your built application with the "/appvve:9BD02EED-6C11-4FF0-8A3E-0B4733EE86A1_6A0357B5-AB99-4856-8A59-CF2C38579E78" argument or you do not have the Developer Channel build installed.
You need to do this because the IE Developer Channel runs as an AppV application. As such, any application that makes calls into the Developer Channel version using the WebDriver API needs to run in the same AppV environment (which is why you must use the /appvve argument).
Make sure you have the IEDeveloperChannel build install and try running your Unit test executable with the /Appvve argument
Example:
MyUnitTest.exe /appvve:9BD02EED-6C11-4FF0-8A3E-0B4733EE86A1_6A0357B5-AB99-4856-8A59-CF2C38579E78
You can get the Developer Channel here: http://devchannel.modern.ie/
I'm looking for a .NET coverage tool, and had been trying out PartCover, with mixed success.
I see that OpenCover is intended to replace PartCover, but I've so far been unable to link it with TypeMock Isolator so my mocked-out tests pass while gathering coverage info.
I tried replicating my setup for Partcover, but there's no defined profilename that works with the "link" argument for Isolator. Thinking that OpenCover was based on Partcover, I tried to tell Isolator to link with Partcover, and it didn't complain (I still had Partcover installed), but the linking didn't work - Isolator thought it wasn't present.
Am I missing a step? Is there a workaround? Or must I wait for an Isolator version that is friends with OpenCover?
Note: I work at Typemock
I poked around with the configuration a little bit and managed to get OpenCover to run nicely with Isolator. Here's what you can do to make them work together, until we add official support:
Register OpenCover profiler by running runsvr32 OpenCover.Profiler.dll (you will need an Administrator's access for this).
Locate the file typemockconfig.xml, it should be under your installation directory, typically C:\Program Files (x86)\Typemock\Isolator\6.0.
Edit the file, and add the following entry towards the end of the file, above </ProfilerList>:
<Profiler Name="OpenCover" Clsid="{1542C21D-80C3-45E6-A56C-A9C1E4BEB7B8}" DirectLaunch="false">
<EnvironmentList />
</Profiler>
Save the file, you will now have a new entry in the Typemock Configuration utility, called OpenCover. Press the Link button to link them. You will now be able to run your tests using OpenCover.Console.exe and Isolator. For example, here's how to run your tests with MSTest:
OpenCover.Console.exe
-target:"C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe"
-targetargs:"/testcontainer:"d:\code\myproject\mytests.dll"
-output:opencovertests.xml
There is still a minor issue running this with TMockRunner -link (that is, with late linking). I will need to look at it further at work.
Hope that helps.
I haven't been able to find any up to date info on using the OCUnit tests with Xcode 4, even though it is a redesigned feature.
Apples own documentation in the Organizer is from 2005 (OS X 10.3).
I thought I just go ahead and try set it up myself.
1: Made a new test target. (TestUnit)
2: Made sure all Base SDK settings in project and targets is set to iOS 4.3.
3: Added a test Class
4: Made sure the framework search path pointed to the framework folder
(4.1?) Manually added the SenTesting framework to my testTarget
When I set my scheme to the "TestUnit iOS 4.3 Simulator" and try and run it an error pop up says "The selected run destination is not valid for this action.".
I have no idea if the above is the correct way to add OCUnit testing to Xcode 4, so if someone has a resource or could point out the steps I would be grateful.
Thank you in advance.
In Xcode 4 'longpress' on the run button until a drop down comes out and select test.
The following steps worked for me to add unit testing to an iPhone project running in the simulator:
Add a Cocoa Touch Unit Testing target to the project. Adding the target added a unit testing class to the project.
Modify the application's scheme to run the unit tests. This involved adding the unit testing target to the Test step of the scheme. I used the application's scheme to unit test instead of the unit test scheme Xcode adds when you add the unit testing target.
Choose Product > Test.
You do not need to add the SenTesting framework to your project. Also, make sure the Test Host build setting is blank if you're unit testing an iPhone app running in the simulator. The simulator does not support application-hosted unit tests.
In addition, to avoid the "The selected run destination is not valid for this action" message when you are in the UnitTest target, open its Biuld scheme and select Run