Playwright tests work locally but fail when run in "npx playwright test" in Azure DevOps Pipelines with "error page.goto: net::ERR_CONNECTION_RESET" - automated-tests

Question

Context:
Playwright Version: Version 1.25.0
Operating System: windows-latest
Node.js version: v16.13.1
Browser: Firefox and Chrome
Extra: Run via Azure DevOps Pipelines in the cloud. The application login starts with the app domain, then goes to Azure login domain, then returns to original domain in the login, if it matters.
What is the goal? The goal is to include our Playwright test suite running in Azure DevOps Pipeline with a trigger.
The problem: The example test suite works locally with "npx playwright test" running all tests in separate files. But in Azure Pipelines, the example test suite works only with 1 or 2 tests, eg. "npx playwright test example.spec.ts". -> If we use "npx playwright test" and run all tests in azure-pipelines.yml, 1-2 test will pass and all others will fail with usually the login failing "page.goto: net::ERR_CONNECTION_RESET" (see errors below in Azure Pipelines log) or "page.goto: NS_ERROR_NET_RESET" with Firefox. I also sometimes "get page.goto: NS_ERROR_UNKNOWN_HOST"
What I have tried already: I've tried switching from ubuntu to windows in the .yml. I've tried running with only 1 worker and with fullyParallel: false. I also limited the browsers to Chrome only or Firefox only. I have tried to find a solution online, but I haven't encoutered this specific problem.
azure-pipelines.yml
trigger:
- tests/playwright
pool:
vmImage: 'windows-latest'
steps:
- script: echo Running azure-pipelines.yml...
displayName: 'Run a one-line script'
- task: NodeTool#0
inputs:
versionSpec: '16.x'
displayName: 'nodetool 16.x'
- task: Npm#1
inputs:
command: 'ci'
- task: CmdLine#2
inputs:
script: 'npx playwright install --with-deps'
- task: CmdLine#2
inputs:
script: 'set CI=true && echo %CI% && npx playwright test'
Azure Pipelines Job log with the errors
Retry #1 ---------------------------------------------------------------------------------------
page.goto: net::ERR_CONNECTION_RESET at https://mywebsite.com
=========================== logs ===========================
navigating to "https://mywebsite.com", waiting until "load"
============================================================
6 | console.log(`beforeEach initiated, running ${testInfo.title}`);
7 | const lg = new login(page);
> 8 | await lg.loginToAppWithAllLicenses();
| ^
9 | });
10 |
at loginToAppWithAllLicenses (D:\a\1\s\pages\login.ts:13:25)
7) [chromium] › example.spec.ts:17:5 › example test suite › Check that News and Messages is present
Test timeout of 60000ms exceeded while running "beforeEach" hook.
3 | import { login } from '../pages/login';
4 |
> 5 | test.beforeEach(async ({ page }, testInfo) => {
| ^
6 | console.log(`beforeEach initiated, running ${testInfo.title}`);
7 | const lg = new login(page);
8 | await lg.loginToAppWithAllLicenses();
page.click: Target closed
An example test that can fail
import { test, expect, Page } from '#playwright/test';
import { Utils } from '../pages/utils';
import { login } from '../pages/login';
test.beforeEach(async ({ page }, testInfo) => {
console.log(`beforeEach initiated, running ${testInfo.title}`);
const lg = new login(page);
await lg.loginToAppWithAllLicenses();
});
test.describe(example test suite', () => {
test("Check that News and Messages is present", async ({ page }) => {
await page.goto('https://mywebsite.com');
// Check that News and Messages are visible to assert that page has loaded
await expect(page.locator('ls-home-news >> text=News'))
.toHaveText('News');
await expect(page.locator('ls-home-messages >> text=Messages'))
.toHaveText('Messages');
});
});
The login that is performed in beforeEach
import { chromium, Page } from '#playwright/test';
export class login {
private page: Page;
constructor(page: Page) {
this.page = page;
}
async loginToAppWithAllLicenses() {
await this.page.goto('https://mywebsite.com');
// Click div[role="button"]:has-text("Email")
await Promise.all([
this.page.waitForNavigation(),
this.page.locator('div[role="button"]:has-text("Email")').click(),
]);
// Click [placeholder="Email Address"]
await this.page.click('[placeholder="Email Address"]');
await this.page.locator('[placeholder="Email Address"]').fill('email here..');
// Click [placeholder="Password"]
await this.page.click('[placeholder="Password"]');
await this.page.locator('[placeholder="Password"]').fill('password here..');
// Click button:has-text("Sign in")
await this.page.click('button:has-text("Sign in")');
// Select company
await this.page.click('.b-number-cell');
await this.page.waitForLoadState('networkidle');
}
}
playwright.config.ts
import type { PlaywrightTestConfig } from '#playwright/test';
import { devices } from '#playwright/test';
const config: PlaywrightTestConfig = {
testDir: './tests',
timeout: 60 * 1000,
expect: {
timeout: 5000
},
fullyParallel: true,
forbidOnly: !!process.env.CI,
retries: 1,
workers: process.env.CI ? 1 : undefined,
reporter: 'html',
use: {
actionTimeout: 0,
screenshot: 'only-on-failure',
trace: 'off',
},
projects: [
{
name: 'chromium',
use: {
...devices['Desktop Chrome'],
},
},
],
outputDir: 'test-results/',
};
export default config;

Answer 1

I finally managed to solve this by switching the test servers to Azure. It seems that at some point, the firewall of the server where the application was running, simply didn't want to communicate anymore.
So the problem wasn't really a Playwright-problem, but a network issue caused by the server's firewall settings.
After we moved the test application to Azure, we haven't faced such issues anymore, and the code didn't change.