The Curious Case of Playwright Tests: Why npm run Sometimes Can’t Find Your Tests

Have you ever found yourself staring at a terminal, utterly perplexed, as your Playwright tests execute flawlessly when run directly, yet mysteriously vanish with a “No tests found” error when invoked via an npm script? If this scenario rings a bell, rest assured, you’re not alone in this peculiar corner of JavaScript development. This common “gotcha” often boils down to a subtle, yet crucial, difference in how various shells and npm itself interpret command-line arguments, especially those involving special characters.

Join me as we dive into a real-world debugging journey to unravel this enigmatic behavior.

The Challenge I Faced: My Playwright Tests Seemingly Disappeared

My recent experience involved a perfectly functional Playwright test suite. Consider a file like ui/UIBasicstest.spec.js, where specific tests were clearly marked with a @smoke tag for easy filtering:

// ui/UIBasicstest.spec.js

import { test, expect } from '@playwright/test';

test('Browser Context Playwright Test', { tag: '@smoke' }, async ({ page }) => {
  // ... test code ...
});

test('Page Context Playwright Test', { tag: '@smoke' }, async ({ page }) => {
  // ... test code ...
});

// ... other tests ...

My goal was straightforward: list these smoke tests. When I executed the command directly in my terminal (using Git Bash/MINGW64 on Windows), the results were exactly as expected:

$ npx playwright test --grep '@smoke' --list

Expected and Actual Output (Working Flawlessly!):

Listing tests:
  ui\UIBasicstest.spec.js:8:1 › Browser Context Playwright Test
  ui\UIBasicstest.spec.js:15:1 › Page Context Playwright Test
Total: 2 tests in 1 file

“Great!” I thought. To streamline the workflow, I naturally moved this command into an npm script within my package.json:

// package.json
{
  "name": "playwright_tests",
  "version": "1.0.0",
  "scripts": {
    // ... other scripts ...
    "test:smoke": "npx playwright test --grep '@smoke' --list"
  },
  // ... rest of package.json
}

Confident, I then ran the npm script:

$ npm run test:smoke

Unexpected Output (The Baffling Failure!):

> playwright_tests@1.0.0 test:smoke
> npx playwright test --grep '@smoke' --list


Error: No tests found
Listing tests:
Total: 0 tests in 0 files

Frustration mounted. The command was identical, the working directory was correct, yet one execution worked flawlessly while the other failed spectacularly. What was going on?

The Breakthrough: Decoding Shell Interpretation and Quoting

The heart of this mystery lies in the intricate dance between your operating system’s shell (like Git Bash/MINGW64, cmd.exe, or PowerShell) and how npm processes commands defined in your package.json scripts. Specifically, it’s a tale of string parsing and the subtle power of special characters.

When you encapsulate @smoke in single quotes (') within your package.json script, different shells can interpret the @ symbol or the single quotes themselves in unexpected ways. While some shells might treat characters within single quotes literally when run directly, the npm process adds another layer of interpretation before handing the command over to the underlying shell. This double-layer parsing can mangle the argument string, preventing Playwright from receiving --grep '@smoke' precisely as intended. The @ character, often used for special purposes in various shell contexts, further complicates this.

In essence, the combination of a special character like @ and single quotes within an npm script can lead to the argument being malformed or misinterpreted by Playwright by the time it executes.

The Simple Fix: Embracing Double Quotes!

After some head-scratching and experimentation, the solution proved surprisingly straightforward: replace single quotes (') with properly escaped double quotes (\") for the --grep argument within your package.json script.

Here’s the updated package.json entry:

// package.json (Updated)
{
  "name": "playwright_tests",
  "version": "1.0.0",
  "scripts": {
    // ... other scripts ...
    "test:smoke": "npx playwright test --grep \"@smoke\" --list"
  },
  // ... rest of package.json
}

A quick but crucial note: The inner double quotes must be escaped with a backslash (\") because the entire script string in package.json is already enclosed in double quotes. This is standard JSON string escaping.

Now, when you run the npm script again, watch the magic unfold:

$ npm run test:smoke

Expected and Actual Output (Sweet Success!):

> playwright_tests@1.0.0 test:smoke
> npx playwright test --grep "@smoke" --list

Listing tests:
  ui/UIBasicstest.spec.js:8:1 › Browser Context Playwright Test
  ui/UIBasicstest.spec.js:15:1 › Page Context Playwright Test
Total: 2 tests in 1 file

Why This Works: A Deeper Dive into Quoting Mechanics

Using escaped double quotes (\"...\) ensures that the entire string "@smoke" is passed as a single, literal argument to the playwright command. When a string is enclosed in double quotes within a shell context (and after npm’s initial parsing), it’s typically treated as a single unit. While double quotes do allow for variable expansion (which isn’t happening here), when properly escaped within the JSON, they effectively “protect” the inner content from any premature or incorrect interpretation by npm and the intermediate shell process that executes the command.

This robust quoting mechanism guarantees that Playwright receives the --grep argument precisely as grep "@smoke", allowing it to correctly identify and filter your smoke tests.

Key Takeaways & Best Practices

This seemingly small quoting nuance highlights a fundamental lesson in command-line scripting, especially when dealing with npm scripts across different operating systems and shells:

• Be Mindful of Shell Interpretation: Commands behave differently depending on the shell environment (e.g., Git Bash, PowerShell, cmd.exe, Zsh, Bash). What works directly in one might require adjustments when run via npm.
• Prioritize Robust Quoting: When passing arguments with special characters (like @, $, *, etc.) or spaces, always default to using double quotes ("). If the command itself is within a double-quoted string (like in package.json), remember to escape inner double quotes with a backslash (\").
• Test Your npm Scripts Early: Don’t assume a script will work just because the direct command does. Always test your npm scripts in the target environment(s).
• Consult npm and Shell Documentation: When in doubt, a quick check of how your specific shell handles quoting or how npm executes scripts can save hours of debugging.

This experience, though frustrating at the time, was a valuable lesson in the subtle complexities of shell scripting and npm. Understanding these nuances empowers you to write more robust and reliable automation scripts.

Happy testing, and may your Playwright tests always be found! 🚀