Incorrect Jest coverage

[stephane-arista]

When using @swc/jest to transpile ts to commonjs and running jest --coverage certain branches are shown as not covered (console logging in these branches show that tests do run the code path). Using babel to transpile and run the tests shows the correct coverage.

[pspeter3]

I added this to my Jest configuration in package.json which seemed to help.

"jest": {
    "collectCoverage": true,
    "transform": {
      "^.+\\.(t|j)sx?$": [
        "@swc/jest",
        {
          "sourceMaps": true
        }
      ]
    }
  }

[krutoo]

@pspeter3 Thank you for your code snippet, it helps, but now there is another problem:

It looks like not covered code highlights are not displayed correctly.

There is an example of coverage report from project with:

• jest@27.3.1
• @swc/core@1.2.118
• @swc/jest@0.2.11

the yellow highlight here says that the comma is supposedly not covered

[sebald]

@krutoo ran into the same issues and guessed it is caused by the conversion to older JS syntax. We could fix these false-positives by targeting a newer ES version.

  transform: {
    '^.+\\.(t|j)sx?$': [
      '@swc/jest',
      {
        jsc: {
          target: 'es2021',
        },
        sourceMaps: true,
      },
    ],
  },

[krutoo]

@sebald Thank you, looks like it realy works with target: "es2021"

[JoshuaKGoldberg]

Another reproduction here: typescript-eslint/tslint-to-eslint-config#1367

I ended up only needing target: "es2021". Thanks!

[kdy1]

I think swc reusing span for various places may cause this.

Can you try https://sokra.github.io/source-map-visualization/#custom by manually invoking swc?

[bobaaaaa]

@kdy1 does this help? Maybe the Object.keys(_node).forEach produces the mismatch? Jest complains that this line is not covered: export * as node from './BodyNodesBuilder';

Edit: tested with

• @swc/jest 0.2.20
• jest 27.5.1

[kdy1]

@bobaaaaa Can you share some code? I think the test file would be enough, as it does not have good sourcemap.

[kdy1]

Oh... Maybe assumption about monotonic increment of source map position can be the cause.
I'll add it to my tasklist

[bobaaaaa]

@kdy1 You can find it here: https://gist.github.com/bobaaaaa/3649b3a7e6312793a257bf67c500128a
Let me know if something is missing.

(thx for investigating ❤️)

[kdy1]

You need sourceMaps: true or sourceMaps: "inline".
I verified that it's working

[bobaaaaa]

@kdy1 Hm, I tested both sourceMaps in .swcrc and as an option in the jest.config.json. Both did not work for me. Even with target: "es2021".

[kdy1]

Hmm... Source maps are valid, maybe emitted tokens without sourcemap entry can be the cause I guess?
Some sourcemap libraries have bugs related to it.

[bobaaaaa]

@kdy1 I updated the gist with the generated .js + .js.map files: https://gist.github.com/bobaaaaa/3649b3a7e6312793a257bf67c500128a

[Krzywy14]

Useing coverageProvider: 'v8' will probably solve the problem.
Solution funded at stackoverflow https://stackoverflow.com/a/74851858/8770040

[danr-za]

@Krzywy14 that completely drops the coverage percentage - did you have any success with it?

[stefee]

My specific problem seems to have been resolved by swc-project/swc-node#673, for what it's worth.

[Krzywy14]

@Krzywy14 that completely drops the coverage percentage - did you have any success with it?

danr-za
IMO coverage percentage is better than when i used babel.
I check that on few files. With babel, even when all function/lines has been coverage, still not showing 100% coverage. When change to "v8" looks like coverage have really 100%.

"coverageProvider": "v8" used with:

"@swc-node/jest": "^1.5.5",
"@swc/core": "^1.3.23",

And I forgot. In transform remove ts from regex

"transform": {
  "\\.(tsx|jsx|js)$": [
    "@swc-node/jest",
      {
        "dynamicImport": true
      }
    ]
 },

[danr-za]

well, exactly the opposite on my end here - I had 100% with babel, once I switched to swc I had around 99%, but with coverageProvider: v8 I even have around 86% 🤷

[kdy1]

Did someone try target: es2022. without decorators, and with externalHelpers: true?

swc has an enormous amount of transforms, and to fix this, I have to narrow down the pass which is producing the wrong source map.

[danr-za]

@kdy1 we use decorators a lot in our project, so its a must in our case

[dmregister]

Since the project does not use decorators, setting decorators false and externalHelpers true fixed the coverage issues

[RobinTail]

I tried:

• externalHelpers: true
• decorators: false
• target: "es2021"
• sourceMaps: true
• in different combinations.

Unfortunately, it does not help to restore the coverage and even also makes it worse.

[RobinTail]

Also found that unused exports are marked as uncovered statements:

Removing export makes them covered.

[jougene]

Same with ts-node and mocha (((

[rogisolorzano]

For my NestJS project, I also noticed that test coverage dropped quite a bit when switching from ts-jest to @swc/jest.

Digging into it, it seems to be related to how the Typescript metadata for constructors are transpiled.

Take this constructor:

constructor(
  @Inject(AppService)
  private readonly appService: AppService,
) {}

The output of swc for the design:paramtypes metadata for the constructor is:

_ts_metadata("design:paramtypes", [
    typeof _appservice.AppService === "undefined" ? Object : _appservice.AppService
])

Notice the typeof ternary. This gets flagged as an "uncovered branch" by istanbul, the
coverage calculator used by jest.

Compare this to how ts-jest outputs this:

__metadata("design:paramtypes", [app_service_1.AppService])

No ternary, so no impact to code coverage.

I created a workaround that tells istanbul to ignore these lines that incorrectly impact code coverage. It inserts an /* istanbul ignore next */ comment in the necessary places, resulting in transpiled code like this:

_ts_metadata("design:paramtypes", [
    /* istanbul ignore next */typeof _appservice.AppService === "undefined" ? Object : _appservice.AppService
])

I setup this repo with instructions to reproduce: https://github.com/rogisolorzano/nest-swc-coverage

The workaround I'm using is in create-swc-transformer.js

Curious if there are any thoughts on this and potential solutions that don't involve having to insert /* istanbul ignore next */ into the transpiled output by swc?

[ctsstc]

I saw mention of using @swc-node/jest vs @swc/jest what are the differences?

I'm using NestJS as well and this seems to be related to decorators, so removing the legacyDecorator and/or decoratorMetadata is not an approach I can test.

This seems to be very related to an issue ts-jest was/is having here, maybe there's some insight to be taken from there? If they did resolve it I think it was via a similar method as rogisolorzano provided which is adding istanbul ignore comments.

I think I've tried all configuration permutations listed here so far without any luck, as well as installing @swc-node/jest & @swc/helpers and trying it as the jest transformer. One thing to note to keep you sane is that you may need to run npx jest --clearCache between configuration changes it seems before running your next jest coverage run (I noticed removing the transform decorators in my swcrc config broke tests, and still showed they were broken after reverting everything ?!).

Potentially Related Issues on Other Projects

• why is a constructor parameter considered a branch and not covered? istanbuljs/istanbuljs#70
• fix for decorator coverage kulshekhar/ts-jest#488

Ultimately it sounds like source mapping can help some scenarios, changing the target in other scenarios, but specific decorators don't seem to care about either of those in some scenarios like Angular and NestJS commonly it seems.

[klutzer]

I've tested here with vitest+swc in a Nestjs project and the issue still happens. I can't apply a transformer as @rogisolorzano suggested (I guess transformers are just for Jest).

There's also an issue in istanbul related to the same problem: istanbuljs/istanbuljs#70

[ctsstc]

I took some time to checkout the related "fix for decorator coverage" that I linked above. It does seem they took the same approach as @rogisolorzano 👍

I'm not sure if this project has a way to implement a "pre-processor" but maybe that's something @rogisolorzano could open a PR for is this is how folks seem to be getting around this. Otherwise it sounds like there needs to be a major rework on the system to implement so that it doesn't require a ternary (or is this an underlying transpilation implementation issue of these frameworks with how the decorator is written?).

https://github.com/kulshekhar/ts-jest/pull/488/files#diff-0548fd42c45d69a916cd9689c388d2c60b74746b4b7626a008167273557b4e57

PS: Thanks @rogisolorzano for the minimal reproduction and digging deep into this.

@klutzer are you not using Jest? I believe that's the scope of this issue, but I believe the coverage issue is not directly related to jest but the coverage tool [istanbul].

[unicornware]

i'm using vitest instead of jest, but i'm dealing with the same coverage issues outlined above in my nestjs project. i ended up creating a custom plugin that adds /* c8 ignore next */ comments to constructor parameters in my source code. adding comments to the transformed code didn't work for me (this may or may not have something to do with using v8 as my coverage provider, but i'm not sure)

my vitest.config.ts:

/**
 * @file Vitest Configuration
 * @module config/vitest
 * @see https://vitest.dev/config/
 */

import { DECORATOR_REGEX } from '@flex-development/decorator-regex'
import pathe from '@flex-development/pathe'
import { cast, split, type Nullable } from '@flex-development/tutils'
import swc from '@swc/core'
import ci from 'is-ci'
import tsconfigpaths from 'vite-tsconfig-paths'
import GithubActionsReporter from 'vitest-github-actions-reporter'
import {
  defineConfig,
  type UserConfig,
  type UserConfigExport
} from 'vitest/config'
import { BaseSequencer, type WorkspaceSpec } from 'vitest/node'
import tsconfig from './tsconfig.json' assert { type: 'json' }

/**
 * Vitest configuration export.
 *
 * @const {UserConfigExport} config
 */
const config: UserConfigExport = defineConfig((): UserConfig => {
  /**
   * [`lint-staged`][1] check.
   *
   * [1]: https://github.com/okonet/lint-staged
   *
   * @const {boolean} LINT_STAGED
   */
  const LINT_STAGED: boolean = !!Number.parseInt(process.env.LINT_STAGED ?? '0')

  return {
    define: {},
    plugins: [
      {
        enforce: 'pre',
        name: 'decorators',

        /**
         * Transforms source `code` containing decorators.
         *
         * @see https://github.com/swc-project/swc/issues/3854
         *
         * @param {string} code - Source code
         * @param {string} id - Module id of source code
         * @return {Promise<Nullable<{ code: string }>>} Transform result
         */
        async transform(
          code: string,
          id: string
        ): Promise<Nullable<{ code: string }>> {
          // do nothing if source code does not contain decorators
          DECORATOR_REGEX.lastIndex = 0
          if (!DECORATOR_REGEX.test(code)) return null

          /**
           * Regular expression used to match constructor parameters.
           *
           * @see https://regex101.com/r/kTq0JK
           *
           * @const {RegExp} CONSTRUCTOR_PARAMS_REGEX
           */
          const CONSTRUCTOR_PARAMS_REGEX: RegExp =
            /(?<=constructor\(\s*)([^\n)].+?)(?=\n? *?\) ?{)/gs

          // add "/* c8 ignore next */" before constructor parameters
          for (const [match] of code.matchAll(CONSTRUCTOR_PARAMS_REGEX)) {
            code = code.replace(match, (params: string): string => {
              return split(params, '\n').reduce((acc, param) => {
                return acc.replace(
                  param,
                  param.replace(/(\S)/, '/* c8 ignore next */ $1')
                )
              }, params)
            })
          }

          return {
            code: (
              await swc.transform(code, {
                configFile: false,
                filename: id,
                inlineSourcesContent: true,
                jsc: {
                  keepClassNames: true,
                  parser: {
                    decorators: true,
                    dynamicImport: true,
                    syntax: 'typescript',
                    tsx: pathe.extname(id) === '.tsx'
                  },
                  target: cast<swc.JscTarget>(tsconfig.compilerOptions.target),
                  transform: {
                    decoratorMetadata:
                      tsconfig.compilerOptions.emitDecoratorMetadata,
                    legacyDecorator: true,
                    useDefineForClassFields:
                      tsconfig.compilerOptions.useDefineForClassFields
                  }
                },
                sourceMaps: 'inline',
                swcrc: false
              })
            ).code
          }
        }
      },
      tsconfigpaths({ projects: [pathe.resolve('tsconfig.json')] })
    ],
    test: {
      allowOnly: !ci,
      benchmark: {},
      chaiConfig: {
        includeStack: true,
        showDiff: true,
        truncateThreshold: 0
      },
      clearMocks: true,
      coverage: {
        all: !LINT_STAGED,
        clean: true,
        cleanOnRerun: true,
        exclude: [
          '**/__mocks__/**',
          '**/__tests__/**',
          '**/index.ts',
          'src/interfaces/',
          'src/metadata/',
          'src/types/'
        ],
        extension: ['.ts'],
        include: ['src'],
        provider: 'v8',
        reporter: [...(ci ? [] : (['html'] as const)), 'lcovonly', 'text'],
        reportsDirectory: './coverage',
        skipFull: false
      },
      environment: 'node',
      environmentOptions: {},
      globalSetup: [],
      globals: true,
      hookTimeout: 10 * 1000,
      include: [
        `**/__tests__/*.${LINT_STAGED ? '{spec,spec-d}' : 'spec'}.{ts,tsx}`
      ],
      isolate: true,
      mockReset: true,
      outputFile: { json: './__tests__/report.json' },
      passWithNoTests: true,
      reporters: [
        'json',
        'verbose',
        ci ? new GithubActionsReporter() : './__tests__/reporters/notifier.ts'
      ],
      /**
       * Stores snapshots next to `file`'s directory.
       *
       * @param {string} file - Path to test file
       * @param {string} extension - Snapshot extension
       * @return {string} Custom snapshot path
       */
      resolveSnapshotPath(file: string, extension: string): string {
        return pathe.resolve(
          pathe.resolve(pathe.dirname(pathe.dirname(file)), '__snapshots__'),
          pathe.basename(file).replace(/\.spec.tsx?/, '') + extension
        )
      },
      restoreMocks: true,
      root: process.cwd(),
      sequence: {
        sequencer: class Sequencer extends BaseSequencer {
          /**
           * Determines test file execution order.
           *
           * @public
           * @override
           * @async
           *
           * @param {WorkspaceSpec[]} specs - Workspace spec objects
           * @return {Promise<WorkspaceSpec[]>} `files` sorted
           */
          public override async sort(
            specs: WorkspaceSpec[]
          ): Promise<WorkspaceSpec[]> {
            return (await super.sort(specs)).sort(([, file1], [, file2]) => {
              return file1.localeCompare(file2)
            })
          }
        }
      },
      setupFiles: ['./__tests__/setup/index.ts'],
      silent: false,
      singleThread: true,
      slowTestThreshold: 3000,
      snapshotFormat: {
        callToJSON: true,
        min: false,
        printBasicPrototype: false,
        printFunctionName: true
      },
      testTimeout: 10 * 1000,
      typecheck: {
        allowJs: false,
        checker: 'tsc',
        ignoreSourceErrors: false,
        include: ['**/__tests__/*.spec-d.ts'],
        tsconfig: pathe.resolve('tsconfig.typecheck.json')
      },
      unstubEnvs: true,
      unstubGlobals: true
    }
  }
})

export default config

[kdy1]

I think #7900 may improve the situation.

[DavideDaniel]

Our issue was that components that were using emotion were being tested but not captured in the coverage reporting. Using sourceMaps: 'inline' was all it took to get coverage back up to where it was.