What a Google Search Can't Tell You About These JS Testing Frameworks

August 9, 2022 · 22 min read

This blog post gives detailed insights into the most prevalent JavaScript E2E (end-to-end) testing frameworks. We break down our article into two main sections:

Our experience - we tried six different frameworks by writing the same test case on the same web application and provided detailed pros/cons of working with those technologies. All the code examples are available here.
Deep dive - to help you decide the best framework for your project, we pulled together many different data sources - e.g., the state of JS survey, social networks, GitHub, etc. We evaluate each framework across four categories: feature set, adoption & popularity, DX (developer experience), and maintenance.

The frameworks

Let's start by briefly exploring the background of the chosen frameworks. We're listing our picks in alphabetical order.

Cypress - created by Brian Mann, was opened to the world in 2017. Cypress became more than a framework. Besides the core library with APIs to write tests, Cypress expanded into a platform that offers a complete overview of the test's performance, presenting an all-in-one monitoring dashboard.
Nightwatch.js - initially released under a company called pineview.io, now Nighwatch.js belongs to BrowserStack, with Andrei Rusu, creator of Nightwatch.js, still contributing actively to the project. With a relatively small API, Nighwatch.js focus on essentials (e.g., integrated runner, assert API).
Playwright - backed by Microsoft, with initial developments led by Pavel Feldman and Andrey Lushnikov, Playwright wants to come across as a resilient, robust, and unified tool to run E2E tests against any browser (Opera and Safari included). Playwright goes beyond the JavaScript ecosystem offering support for additional programming languages.
Protractor 🎖(honorable mention) - we're not going to look into Protractor due to its narrow focus on Angular applications. Regardless, we want to mention this pioneer open-source technology that played its role in the history of JavaScript.
Puppeteer - backed by Google, Puppeteer had its initial development primarily led by Andrey Lushnikov, who now works on Playwright. The first thing to note is that Puppeteer is a library and not a framework; hence many of the features you would expect to find in a testing framework are not present, such as assert API.
TestCafe - backed by DevExpress, this framework has good cross-browser support and a simple yet rich testing API. Like Cypress, TestCafe expands into a platform, TestCafe Studio.
Webdriver.io - with Christian Bromann as lead maintainer, this framework had a major rewrite in 2021 (v7 release), which reassured its presence amongst other frameworks in this list. The main selling point of Webdriver.io is interoperability. Webdriver.io allows developers to test native mobile applications (iOS and Android), stretching beyond the scope of web application testing.

🧪 Our experience

This section documents our journey while setting up and writing one test with every framework. A hands-on experience with each technology will help us interpret the community trends, which we'll cover later in this article.

Framework	Time to setup and write first test	LOC [1]	DX[2]
Cypress	30min	12	Excellent
Nightwatch.js	30min	19	Very Good
Playwright	18min	17	Excellent
Puppeteer	30min	34	Poor
TestCafe	25min	21	Good
Webdriver.io	50min	18	Poor

1 - Lines of Code | 2 - our opinion

Test use case

We've built a simple react application that contains a heading element, a button, and a paragraph. When clicking the button, we trigger an ajax request and display a quote on the screen.

"sample application used for testing"

It's time to see some code! We'll go through the frameworks in alphabetical order. For each framework, we'll list items we've liked and disliked.

Cypress

Cypress has detailed documentation about its APIs and contains guides/articles with recommendations and good testing practices.

/tests/cypress.spec.js
const selectors = require('./common/selectors');
const mock = require('./common/mock');

it('Cypress', () => {
  cy.intercept('/random-quote', mock).as('getRandomQuotes');
  cy.visit('http://localhost:3000/');
  cy.get(selectors.title).should('have.text', 'Quotez');
  cy.get(selectors.contentEmpty).should('be.visible');
  cy.get(selectors.contentEmpty).should('have.text', '...');
  cy.get(selectors.trigger).click();
  cy.get(selectors.content).should('have.text', `${mock.quote} by ${mock.author}`);
});

source

👎 Disliked

We faced overwhelming documentation during onboarding; we felt pulled in many different directions. Onboarding should be a smooth user journey where one learns to use the basics of the framework.
Cypress is highly opinionated about scaffolding. On the other hand, everything is configurable, so you need to dig through the docs to figure out how to configure the project according to your preference.
Cypress has a list of trade-offs that might be a deal-breaker for many projects. It's noteworthy that the Cypress team is transparent about this.

👍 Liked

"cypress dashboard running test"

The automated onboarding generates quite some files, but all in all, it's fantastic.
Cypress refreshes the browser as you update your test files, offering a seamless developer experience.
Cypress has a beautiful API design that is easy to understand (navigation, selectors, mocking). It took seconds to intercept a request and provide mock data.
```
cy.intercept('/random-quote', mock).as('getRandomQuotes');
```
Intellisense works remarkably well even on JavaScript-only projects, where you get autocompletion for should statements such as 'have.text' or 'be.visible'.
The time-traveling debugging is a few miles ahead of the competitors, making debugging much more convenient.
Our test has only 12 LOC, making it the shortest test we've written during our experiments.

Nightwatch.js

We implemented our test smoothly with the help of a quick start guide and automated onboarding. Nightwatch.js offers the possibility of integrating existing projects, supporting both JavaScript and TypeScript.

/tests/nightwatch.spec.js
const selectors = require('./common/selectors');
const mock = require('./common/mock');

describe('Nightwatch.js', function () {
  it('Nightwatch.js', async function () {
    await browser
      .mockNetworkResponse('http://localhost:3000/random-quote', {
        status: 200,
        body: JSON.stringify(mock),
      })
      .navigateTo('http://localhost:3000')
      .assert.textEquals(selectors.title, 'Quotez')
      .assert.elementPresent(selectors.contentEmpty)
      .assert.textEquals(selectors.contentEmpty, '...')
      .click(selectors.trigger)
      .assert.elementPresent(selectors.content)
      .assert.textMatches(selectors.content, `${mock.quote} by ${mock.author}`);
  });
});

source

👎 Disliked

Does not support mock network requests across all supported browsers.
We found outdated documentation. The code generated during onboarding is not in sync with the documentation. The docs mention npx nightwatch tests/ecosia.js, which with the new structure should be npx Nightwatch tests/specs/basic/ecosia.js.
Having the tests break because you've deleted empty auto-generated folders is a frustrating experience.

👍 Liked

No config tweaking is necessary to run the test.
The fluent API is super consistent; you can do everything on the go: mocks, assertions, and interactions.
Another pleasant surprise was the debug API .debug({ timeout: 20000 }), which is neat to troubleshoot your tests - many other frameworks adopt a similar API.
The error log is nicely highlighted and formatted. Many tools don't offer visually friendly logs.
After successfully running the test, we're positively impressed by the good terminal text reports. Logs are well formatted and have nifty step-by-step assertion statements, all by default.

"nightwatch.js error logs"

"nightwatch.js test passing logs"

Playwright

With an automated onboarding, offering support for JavaScript and Typescript, it also comes with options to generate CI integrations (e.g., GitHub Actions). During onboarding, Playwright installs everything for us and provides a list of commands for you to start immediately.

/tests/playwright.spec.js
const { test, expect } = require('@playwright/test');
const selectors = require('./common/selectors');
const mock = require('./common/mock');

test('Playwright', async ({ page }) => {
  await page.route('http://localhost:3000/random-quote', (route) =>
    route.fulfill({
      status: 200,
      body: JSON.stringify(mock),
    }),
  );
  await page.goto('http://localhost:3000/');
  await expect(page.locator(selectors.title)).toHaveText('Quotez');
  await expect(page.locator(selectors.contentEmpty)).toBeVisible();
  await expect(page.locator(selectors.contentEmpty)).toHaveText('...');
  await page.locator(selectors.trigger).click();
  await expect(page.locator(selectors.content)).toBeVisible();
  await expect(page.locator(selectors.content)).toHaveText(`${mock.quote} by ${mock.author}`);
});

source

👎 Disliked

There's nothing major to dislike about Playwright. Like other test libraries and frameworks that leverage async/await, tests might feel bloated in the number of await statements. Still, we believe this is a minor detail and subjective to the developer's preference.

👍 Liked

Outstanding onboarding, fast and practical, giving us the commands to run a test immediately. The example generated as boilerplate it's small but contains just the right amount of information for anyone to get started.
Friendly test reporting, with a local server delivering an HTML report that outlines test failures and passes. For fails, it points to the line of code where you made a mistake and gives a nice call log in a human-readable format.
The debugging mode has a red dot (pointer) to indicate the cursor position and show you where the click interactions happened; this is extremely handy; it can help uncover the most challenging bugs.
The stepping debugger in a separate window allows developers to step through a test script line by line! The debugger is impressive, and we haven't seen anything like it.
The config file is super intuitive; it comes with inline comments for each config property.
The assert API feels like working with Jest; hats off to the Playwright team on an intelligent API design, not reinventing the wheel, reducing the friction between different technologies.
Mocking requests only require a single statement.

Puppeteer

Puppeteer does warn us that it's a general-purpose library, not a testing library; this is felt right from the start. Alone, Puppeteer is not enough; this is the first time we face this entry barrier, given all the other frameworks have an integrated test runner. We've picked Jest to bypass the setup process as quickly as possible.

/tests/puppeteer.spec.js
const puppeteer = require('puppeteer');
const selectors = require('./common/selectors');
const mock = require('./common/mock');

it('Puppeteer (powered by Jest)', async () => {
  const browser = await puppeteer.launch({ headless: true }); // headless: false to see the browser
  const page = await browser.newPage();
  await page.setRequestInterception(true);
  page.on('request', (request) => {
    if (request.url().endsWith('/random-quote')) {
      request.respond({
        status: 200,
        contentType: 'application/json',
        body: JSON.stringify(mock),
      });
    } else {
      request.continue();
    }
  });
  await page.goto('http://localhost:3000', { waitUntil: 'networkidle0' });
  const titleEl = await page.$(selectors.title);
  const title = await titleEl.evaluate((e) => e.textContent);
  expect(title).toEqual('Quotez');
  const emptyContentEl = await page.$(selectors.contentEmpty);
  const emptyContentText = await emptyContentEl.evaluate((e) => e.textContent);
  expect(emptyContentText).toEqual('...');
  const triggerEl = await page.$(selectors.trigger);
  await triggerEl.click();
  await page.waitForNetworkIdle();
  const contentEl = await page.$(selectors.content);
  const contentText = await contentEl.evaluate((e) => e.textContent);
  expect(contentText).toEqual(`${mock.quote} by ${mock.author}`);
  await browser.close();
});

source

👎 Disliked

In this case, we need to set up everything from scratch, including additional tooling (Jest).
Error logs are not readable. Pay attention not to make errors while you write your tests (especially missing the await keyword somewhere 🔥). Unreadable error stack traces can be demotivating for anyone trying to troubleshoot a failing test.
Tests are too verbose, which is reflected by the number of LOC for our simple test compared with other alternatives.
Having run our simple test a couple of times in the non-headless mode, we've ended up with lingering Chromium instances that require forceful shutdown. Lingering chromium instances are most likely related to our Jest integration and how we lack proper handling for teardown. We think this is an unnecessary extra problem, which we can easily circumvent by picking another framework.

👍 Liked

API is very minimalistic. With .evaluate you can do everything "just" knowing JavaScript.
The ability to explicitly wait for the page to idle is neat; it gives you more control over what's happening in the browser compared to the auto-retry timeout approach many other frameworks adopt. Nope, this control won't save you from flaky tests.
```
await page.goto('http://localhost:3000', { waitUntil: 'networkidle0' });
```
Mocking requests is super easy. The event-driven interception API is simple to use.

TestCafe

Instead of onboarding, TestCafe jumps from the installation to writing the first test because there's no setup required!

/tests/testcafe.spec.js
const { Selector, RequestMock } = require('testcafe');
const selectors = require('./common/selectors');
const mock = require('./common/mock');

const tcMock = RequestMock().onRequestTo('http://localhost:3000/random-quote').respond(JSON.stringify(mock), 202);

fixture`TestCafe`.page('http://localhost:3000').requestHooks(tcMock);

test('TestCafe', async (t) => {
  await t
    .expect(Selector(selectors.title).innerText)
    .eql('Quotez')
    .expect(Selector(selectors.contentEmpty).exists)
    .ok()
    .expect(Selector(selectors.contentEmpty).innerText)
    .eql('...')
    .click(Selector(selectors.trigger))
    .expect(Selector(selectors.content).exists)
    .ok()
    .expect(Selector(selectors.content).innerText)
    .eql(`${mock.quote} by ${mock.author}`);
});

source

👎 Disliked

"sample security prompt for TestCafe"

There are some security concerns with the amount of OS permission-related popups we've faced when running the test for the first time. We haven't experienced this with any other framework. It seems unnecessary to have video recording enabled by default (for example).
Inconvenient API design for RequestMock. Any other alternative does a better job here.
Overall, TestCafe's API design does not match its competitors, but overall a good experience, excluding setting up network request mocking.
Although our example is minimalistic, the dashboard is much slower to start than the Cypress dashboard or the Playwright debugger.

👍 Liked

There's no setup required and no boilerplate, which feels fantastic. Zero configuration is why we wrote our first test in 25 minutes, one of our fastest.
Just like with Nightwatch.js, we're impressed with the text reports on the terminal.
Error stack traces make recommendations about correct API usage.

Webdriver.io

The only framework whose test case we couldn't fully complete is Webdriver.io. Although it has a quick start guide and familiar APIs, there are inconsistencies in the documentation that made the experience a bit bumpy. We spent about 30 minutes debugging the following error:

Proxy error: Could not proxy request /random-quote from localhost:3000 to http://localhost:3000.

See https://nodejs.org/api/errors.html#errors\_common\_system\_errors for more information (ECONNRESET).

As of today, we still don't know how to mock network requests with Webdriver.io, while we managed to do it effortlessly with all other alternatives. Again, this is just a detail, and the problem is undoubtedly on our end, but our level of effort here was way higher than other frameworks, which should tell us something.

/tests/webdriverio.spec.js
const selectors = require('./common/selectors');
const mock = require('./common/mock');

describe('Webdriver.IO', () => {
  it('Webdriver.IO', async () => {
    // Proxy error: Could not proxy request /random-quote from localhost:3000 to http://localhost:3000
    // const iomock = await browser.mock('http://localhost:3000/random-quote')
    // iomock.respond(mock)
    // Proxy error: end
    await browser.url('http://localhost:3000/');
    await expect($(selectors.title)).toHaveText('Quotez');
    expect(await $(selectors.contentEmpty).isDisplayedInViewport()).toBe(true);
    await expect($(selectors.contentEmpty)).toHaveText('...');
    const btn = await $(selectors.trigger);
    await btn.click();
    // ...
  });
});

source

👎 Disliked

Upon running the init command, Wedriver.io changed the formatting of our package.json; this feels a bit intrusive.
The generated boilerplate was out of place even though we provided our tests directory path during the onboarding terminal prompt; this is an evident inconsistency.
Running the first test, we are overwhelmed by the amount of logging that appears in the terminal. A small test generated a lot of noise in the logs. You can tune logging in the file wdio.conf.js through the logLevel property, but a more balanced default logging would bring a better experience.
Writing the test took about one hour (without request mocking).
Intellisense is not working for the most critical API, the selector $ interface.
Error stack traces are lengthy and hard to extract useful information.

👍 Liked

A surprising aspect of Webdriver.io is that it allows us to plug a custom reporter with many different options.
The selector API is neat; using the $ character as a selector feels familiar and fast to type.
Plugging in Mocha was straightforward without requiring additional setup besides filling in the command line prompt during onboarding.

🎯 Deep dive

We break down this analysis into four major categories.

Features - a comprehensive list of features that place our frameworks vis-à-vis.
Adoption & Popularity - how many people out there are using it? Are developers aware of this technology?
DX (Developer Experience) - how easy is it to write, run and debug tests? What do users have to say about these technologies? Are they satisfied after using them for a while?
Maintenance - are these technologies under active development? Who is contributing to them? Are issues quickly addressed?

Features

Feature	Cypress	Nightwatch	Playwright	Puppeteer	TestCafe	Webd.io
Integrated runner	✅	✅	✅	❌[2]	✅	✅
Assert/Expect APIs	✅	✅	✅	❌	✅	✅
Browser support	Chrome Edge Firefox	Chrome Edge Firefox Safari	Chrome Edge Firefox Safari Opera	Chrome Firefox [3]	Chrome Edge Firefox IE 11+ Safari Opera	All [5]
Support multiple tabs/windows	❌ [1]	✅	✅	✅	✅ [4]	✅
Auto-waiting Automatic DOM query retries	✅	✅	✅	❌	✅	✅
Intercept and mock requests	✅	✅	✅	✅	✅	✅ [6]
Browser extensions support	❌[9]	✅	✅	✅	✅	✅
Integrated dashboard	✅	❌	❌	❌	✅	❌
Integrated stepping debugger	✅	❌	✅	❌	❌	❌
Screenshots	✅	✅	✅	✅	✅	✅
Video recording	✅	❌	✅	❌	❌	❌ [7]
Code coverage	✅	❌	✅	✅	❌	✅
Automated onboarding	✅	✅	✅	❌	❌	✅

1 - see trade-offs | 2 - being a library puppeteer has naturally a narrower feature set | 3 - it has been experimental for a long time | 4 - not supported by all browsers | 5 - no specifics available | 6 - but we couldn't make it work | 7 - endorces 3rd party package | 8 - not supported by all browsers | 9 - Cypress allows you to install extensions, but you won't be able access the URL scheme chrome-extension://...

Cypress and Playwright stand out with an almost perfect score in our feature table. The differences in the table might look minor, but there's a lot more to it. Cypress runs on the browser, which means the test scripts you write are embedded with the target application; hence it is very likely that Cypress will never support features such as multi-window orchestration. Another limitation (fixed recently) is that Cypress can only visit same-origin domains. That's not the case with Playwright; it runs on Node.js and interacts with the browser through APIs such as Chrome Devtools Protocol, bypassing Cypress limitations. How much value can Cypress deliver while running embedded in the browser? And how much of that will be unmatched by Playwright? Only time will tell.

We were surprised to learn that TestCafe and Webdriver.io are the browser support category's champ, but that won't be a differentiator for much longer now that Microsoft has finally sunsetted Internet Explorer. Options such as Nightwatch.js and Playwright are now equally competitive in browser support.

Adoption & Popularity

Metric	Cypress	Nightwatch	Playwright	Puppeteer	TestCafe	Webd.io
GitHub stars	39k	11k	41k	79k	9k	8k
NPM downloads (per week)	4M	169k	742k	3M	251k	366k
Interest [2]	72%	n/a [3]	70%	58%	n/a [3]	38%
Usage [2]	42%	n/a [3]	10%	37%	n/a [3]	10%
Awareness [2]	83%	n/a [3]	34%	78%	n/a [3]	27%
Stack Overflow (questions tagged)	8k	1k	1k	7k	2k	2k
Twitter followers	20k	3k	5k	- [4]	2k	4k
Youtube subscribers [5]	11k	-	2k	- [4]	-	-

1 - absolute values are rounded | 2 - source is 2021.stateofjs.com | 3 - technologies with less than 10% awareness not considered in the 2021.stateofjs.com | 4 - dedicated social only. e.g., Puppeteer points to Chrome Developers, so we don't track it

Cypress has a good reputation, which means it will be around for a good while, whether you love it or hate it. Cypress has a broad social presence and a large community. We think Puppeteer scores well due to its broader scope as a library. An interesting observation on Webdriver.io is that despite the low levels of usage reported in the state of js survey, it has quite some traffic on npm; we believe this is due to native mobile developers using Webdriver.io to write tests. Most likely, mobile developers do not overlap with the respondents' group for the state of js survey.

DX (Developer Experience)

Metric	Cypress	Nightwatch	Playwright	Puppeteer	TestCafe	Webd.io
Satisfaction	92%	n/a [1]	93%	82%	n/a [1]	46%
Would use again	37%	n/a [1]	16%	30%	n/a [1]	9%
Would not use again	4%	n/a [1]	1%	7%	n/a [1]	11%

1 - technologies with less than 10% awareness not considered in the 2021.stateofjs.com

We're happy to see that these numbers align with our experience. We believe onboarding and documentation are pilar to creating a good DX and reducing the learning curve. Another vital aspect that the two technologies (Cypress and Playwright), with the best DX scores, can't afford to miss is debugability. Functionalities such as stepping debuggers and friendly stack traces reduce friction, making developers less frustrated.

Maintenance

Parameter	Cypress	Nightwatch	Playwright	Puppeteer	TestCafe	Webd.io
Backers	-	BrowserStack	Microsoft	Google	DevExpress	-
Recurrent contributors [2]	2	1	4	2 [3]	3	1
Open issues	2k	155	602	327	390	158
Issue resolution time median [1]	15 days	9 days	3 days	21 days	9 days	12 hours
Open pull requests	44	20	18	14	5	6
% Open issues & pull requests (the lower the better) [1]	38%	12%	21%	18%	16%	4%
Commits past month (across all branches)	343	28	363	85	23	242
Merged pull requests past month	105	11	355	42	42	83
Total contributors	378	107	266	424	113	427

1 - source is isitmaintained.com | 2 - number of developers that signed ~10% or more of the total of commits to the framework's codebase | 3 - one of the individuals that most contributed now works on Playwright

Playwright has superior numbers here. The number of monthly commits and the average time for issue resolution indicates that the Playwright team is developing faster than others. On top of that is backed by Microsoft. We can not ignore that companies such as Cypress are more prone to impact during unstable economic periods.

Another critical aspect is whether there are knowledge silos in the frameworks' core teams. A project primarily maintained by a single developer is at a higher risk of being discontinued. Life is unpredictable, and there are many reasons why that person would stop contributing to that project. Consider that Nightwatch.js and Webdriver.io run as a "one-person operation", while others have larger core teams behind the scenes.

🔮 Predictions

This section is great because it will allow us to come back in a few years and understand how wrong we were.

Conspiracy theory: With VSCode and the acquisition of GitHub, Microsoft has demonstrated to be investing strongly in the open source ecosystem for the past few years. We think that Microsoft could buy out Cypress.io to monopolize the JavaScript browser testing space; this could be another space for Microsoft to thrive (although, realistically, JavaScript testing frameworks feel too niche for a company like Microsoft).
Applications are increasingly complex; we think the restrictions imposed by Cypress will pave the road for Playwright to catch up in a year or two. We foresee the usage scores to go up for Playwright.
The Browserstack acquisition of Nightwatch.js will positively impact the project. Now backed by a larger tech organization with a powerful presence in the software testing industry, Nightwatch.js has the resources to thrive.
The discontinuation of Internet Explorer means that TestCafe and Webdriver.io will lose the edge of broader browser compatibility; things even out in this category, leaving fewer excuses for developers not to adopt Cypress, Playwright, or even Nightwatch.js.

⚖️ Conclusion: Our pick

The excellent DX, quick start and documentation make Playwright the best option at this time.

We hope this article becomes a good decision-making tool for you to pick the best JavaScript E2E testing framework.

If you liked this article, consider sharing (tweeting) it to your followers.

Did you like this article?

Thanks for your feedback!

The frameworks​

🧪 Our experience​

Test use case​

Cypress​

👎 Disliked​

👍 Liked​

Nightwatch.js​

👎 Disliked​

👍 Liked​

Playwright​

👎 Disliked​

👍 Liked​

Puppeteer​

👎 Disliked​

👍 Liked​

TestCafe​

👎 Disliked​

👍 Liked​

Webdriver.io​

👎 Disliked​

👍 Liked​

🎯 Deep dive​

Features​

Adoption & Popularity​

DX (Developer Experience)​

Maintenance​

🔮 Predictions​

⚖️ Conclusion: Our pick​

Did you like this article?

The frameworks

🧪 Our experience

Test use case

Cypress

👎 Disliked

👍 Liked

Nightwatch.js

👎 Disliked

👍 Liked

Playwright

👎 Disliked

👍 Liked

Puppeteer

👎 Disliked

👍 Liked

TestCafe

👎 Disliked

👍 Liked

Webdriver.io

👎 Disliked

👍 Liked

🎯 Deep dive

Features

Adoption & Popularity

DX (Developer Experience)

Maintenance

🔮 Predictions

⚖️ Conclusion: Our pick