[Bug]: Generated files, which are part of .gitignore, are not picked up for testing

[BrianMar]

Version

1.38.1

Steps to reproduce

Create a repo with test files, each having one test, named:
generate-foo.test.ts
generated-foo.test.ts
generatedfoo.test.ts

set your testMatch to "gener*.test.ts"

(In my repo I had the following files:)
gen-0.playwright-test.ts
gener-00.playwright-test.ts
genera-00.playwright-test.ts
generat-00.playwright-test.ts
generate-00.playwright-test.ts
generated-00.playwright-test.ts
generated-command-script-00.playwright-test.ts
generated-command-script-01.playwright-test.ts
generated-command-script-02.playwright-test.ts
generated-command-script-03.playwright-test.ts
generated-command-script-04.playwright-test.ts
generated-command-script-05.playwright-test.ts
generated-foobar.playwright-test.ts
generatedABuchOfStuff.playwright-test.ts

Expected behavior

The tests in each file should be found.

Actual behavior

Any files that start with "generated-" are not found and show up as having no tests.

Additional context

This only seems to happen at the beginning of the file. I can put "generated" in a longer name, like "command-script-generated-00.playwright-test.ts".

Environment

System:
    OS: Windows 10 10.0.22631
    Memory: 27.52 GB / 63.95 GB
  Binaries:
    Node: 20.11.1 - C:\Program Files\nodejs\node.EXE
    Yarn: 1.22.21 - C:\.tools\.npm-global\yarn.CMD
    npm: 10.2.4 - C:\Program Files\nodejs\npm.CMD
  Languages:
    Bash: 5.2.15 - C:\Program Files\Git\usr\bin\bash.EXE

[yury-s]

Can you share full reproduction? I cannot reproduce it with the following project: https://github.com/yury-s/bug-30443.git

[alchemist7991]

If you are using the regex as gener*.test.ts, this will match names like generated-00.playwright.test.ts instead of generated-00.playwright-test.ts.
So either you change all the test name from *-test.ts to *.test.ts, or you can update the regex from gener*.test.ts to gener*test.ts (this will match both *-test.ts and *.test.ts).

Please let me know if I missed anything else.

[yury-s]

If you are using the regex as gener*.test.ts, this will match names like generated-00.playwright.test.ts instead of generated-00.playwright-test.ts.

This exact regex will not much any of those names, so it was most likely a glob.

[BrianMar]

I think I've finally figured out what the actual issue is.

TLDR:
*My generated files are in my .gitignore, because we don't want them checked in.
*'cachedCollectFiles' has, for its second parameter, a "respectGitIgnore" value.

• project.respectGitIgnore is not part of the configuration, but is set by

    this.respectGitIgnore = !projectConfig.testDir && !config.testDir;

Since I'm not explicitly setting a test directory in my config, Playwright is respecting the .gitignore, and therefore these files are being skipped. So whatever I name my generated files, I would then update my .gitignore to match, and Playwright would continue to not find the files.

I'll propose a change in a new comment below, but here's the details on how I figured this out. It was typing this out that led me to the answer. Feel free to jump to the next comment unless you want a lot of detail

============================

I finally got a chance to debug into this farther, and it looks like the issue is in "cachedCollectFiles".

I have added logging to "collectFilesForProject" in projectUtils.js, so it now looks like this:

async function collectFilesForProject(project, fsCache = new Map()) {
  const extensions = new Set(['.js', '.ts', '.mjs', '.mts', '.cjs', '.cts', '.jsx', '.tsx', '.mjsx', '.mtsx', '.cjsx', '.ctsx']);
  const testFileExtension = file => extensions.has(_path.default.extname(file));
  const allFiles = await cachedCollectFiles(project.project.testDir, project.respectGitIgnore, fsCache);
  const testMatch = (0, _util2.createFileMatcher)(project.project.testMatch);
  const testIgnore = (0, _util2.createFileMatcher)(project.project.testIgnore);

  _fs.default.writeFileSync("c:/src/collectFilesForProject-project.json", `${JSON.stringify(project.project)}`);
  _fs.default.writeFileSync("c:/src/collectFilesForProject-allFiles.json", `${JSON.stringify(allFiles)}`);

  const testFiles = allFiles.filter(file => {
    if (!testFileExtension(file)) return false;
    const isTest = !testIgnore(file) && testMatch(file);

    _fs.default.writeFileSync(`c:/src/filecheck-${_path.default.basename(file)}.json`, `${file}: isTest:${isTest} testIgnore:${testIgnore(file)} testMatch:${testMatch(file)}`)

    if (!isTest) return false;
    return true;
  });

  _fs.default.writeFileSync("c:/src/collectFilesForProject-testFiles.json", `${JSON.stringify(testFiles)}`);

  return testFiles;
}

Here is a truncated list of the files in my automation direclory from the "allFiles" output. eliding large chunks we don't care about:

[
...,
MY-PROJECT\\jest.setup.js",
MY-PROJECT\\just-task.js",
MY-PROJECT\\lib\\automation\\AutomationHelper.js",
MY-PROJECT\\lib\\automation\\AutomationHelper.js.map",
MY-PROJECT\\lib\\automation\\automationTestFileUtils.js",
MY-PROJECT\\lib\\automation\\cmdscrip08.playwright-test.js",
MY-PROJECT\\lib\\automation\\cmdscrip08.playwright-test.js.map",
MY-PROJECT\\lib\\automation\\cmdscripp00.playwright-test.js",
MY-PROJECT\\lib\\automation\\cmdscripp00.playwright-test.js.map",
MY-PROJECT\\lib\\automation\\cmdscript00.playwright-test.js",
MY-PROJECT\\lib\\automation\\cmdscript00.playwright-test.js.map",
MY-PROJECT\\lib\\automation\\cmdscript01.playwright-test.js",
MY-PROJECT\\lib\\automation\\cmdscript01.playwright-test.js.map",
MY-PROJECT\\lib\\automation\\cmdscript02.playwright-test.js",
MY-PROJECT\\lib\\automation\\cmdscript02.playwright-test.js.map",
MY-PROJECT\\lib\\automation\\cmdscript03.playwright-test.js",
MY-PROJECT\\lib\\automation\\cmdscript03.playwright-test.js.map",
MY-PROJECT\\lib\\automation\\cmdscript04.playwright-test.js",
MY-PROJECT\\lib\\automation\\cmdscript04.playwright-test.js.map",
MY-PROJECT\\lib\\automation\\cmdscript05.playwright-test.js",
MY-PROJECT\\lib\\automation\\cmdscript05.playwright-test.js.map",
MY-PROJECT\\lib\\automation\\cmdscript11.playwright-test.js",
MY-PROJECT\\lib\\automation\\cmdscript11.playwright-test.js.map",
MY-PROJECT\\lib\\automation\\cmdscripts07.playwright-test.js",
MY-PROJECT\\lib\\automation\\cmdscripts07.playwright-test.js.map",
...,
MY-PROJECT\\lib\\automation\\generated-test-partitions.json",
MY-PROJECT\\lib\\automation\\ListAuthors.cmd",
MY-PROJECT\\lib\\automation\\playwright-environment.js",
MY-PROJECT\\lib\\automation\\playwright-environment.js.map",
MY-PROJECT\\lib\\automation\\test00.playwright-test.js",
MY-PROJECT\\lib\\automation\\test00.playwright-test.js.map",
MY-PROJECT\\lib\\automation\\TestAuthors.md",
MY-PROJECT\\lib\\automation\\TestRunner.js",
MY-PROJECT\\lib\\automation\\TestRunner.js.map",
MY-PROJECT\\lib\\CommandScript\\CommandScript.js",
...,
MY-PROJECT\\src\\.jest-cache\\jest-transform-cache-127ae6e527aec712620987fa30832274-79ef2876fae7ca75eedb2aa53dc48338\\f9\\SelectionHighlightpuppeteertest_f9e79d19c9d6becbc696f1dfdac8fbe1.map",
MY-PROJECT\\src\\.jest-cache\\perf-cache-127ae6e527aec712620987fa30832274-da39a3ee5e6b4b0d3255bfef95601890",
MY-PROJECT\\src\\automation\\AutomationHelper.ts",
MY-PROJECT\\src\\automation\\automationTestFileUtils.js",
MY-PROJECT\\src\\automation\\cmdscrip08.playwright-test.ts",
MY-PROJECT\\src\\automation\\cmdscripp00.playwright-test.ts",
MY-PROJECT\\src\\automation\\cmdscripts07.playwright-test.ts",
MY-PROJECT\\src\\automation\\CommandScript\\0-ad hoc all.cmdScript",
...
MY-PROJECT\\src\\automation\\CommandScript\\Widget-BlockDeleteCommand.cmdScript",
MY-PROJECT\\src\\automation\\ListAuthors.cmd",
MY-PROJECT\\src\\automation\\playwright-environment.ts",
MY-PROJECT\\src\\automation\\test00.playwright-test.ts",
MY-PROJECT\\src\\automation\\TestAuthors.md",
MY-PROJECT\\src\\automation\\TestRunner.ts",
MY-PROJECT\\src\\CommandScript\\CommandScript.md",
];

The files I expect to test are the ones in "src/automation/cmdscript0?.playwright-test.ts". As you can see, the initial file list doesn't pick those up.

I don't have a testDir defined. So according to the logic here:

class FullProjectInternal {
 ...
  constructor(configDir, config, fullConfig, projectConfig, configCLIOverrides, throwawayArtifactsPath) {
    this.project = void 0;
...
    this.expect = takeFirst(projectConfig.expect, config.expect, {});
    this.respectGitIgnore = !projectConfig.testDir && !config.testDir;
    _fs.default.writeFileSync("c:/src/FullProjectInternal.json", `projectConfig.testDir:${JSON.stringify(projectConfig.testDir)} config.testDir:${config.testDir} respectGitIgnore:${this.respectGitIgnore}`);
  }
}

And with my project not having a testDir defined, the output is:

projectConfig.testDir:undefined config.testDir:undefined respectGitIgnore:true

So my test files are getting ignored because Playwright is defaulting to respecting .gitignore if no testDir is set in the config.

[pavelfeldman]

So my test files are getting ignored because Playwright is defaulting to respecting .gitignore if no testDir is set in the config.

That sounds right and that was a deliberate decision in order for Playwright not traverse node_modules, out and alike when no test dir is specified. Deliberate, but undocumented as I can see. Sounds like you can solve your problem via pointing Playwright to the generated tests after all?

[BrianMar]

I've created a proposal to expose the flag. However, you make a good point with "node_modules", which we probably never want to search.

I'm going to test with "testDir" now, I just wanted to get the brain dump complete while it was fresh in my mind.

#30553

[pavelfeldman]

Folding this into #30553