V8 coverage - limit to certain url pattern?

[gullerya]

It is possible to collect a JS coverage running node with NODE_V8_COVERAGE env variable set.
The coverage dumped includes much more than the 'user-land' code, mostly node's own internal content.

Is it possible (how, if any?) to specify a url pattern that will be taken as a filter to reduce the reported data?

[kaizhu256]

yes, jslint-org/jslint's ci generates coverage-reports using only nodejs with no dependencies.
it filters out internal content with the following real-world-code:

    cwd = process.cwd().replace((
        /\\/g
    ), "/") + "/";
    await Promise.all(<v8-coverage-object>.map(async function ({
        functions,
        url
    }) {
...
        pathname = url.replace((
            process.platform === "win32"
            ? "file:///"
            : "file://"
        ), "").replace((
            /\\\\/g
        ), "/");
        // filter out internal-content who's pathname does not begin with <cwd>,
        // or pathnames containing "node_modules"
        if (
            pathname.indexOf(cwd) !== 0 ||
            pathname.indexOf(cwd + "[") === 0 ||
            pathname.indexOf("/node_modules/") >= 0
        ) {
            return;
        }
...

you can tool-lessly create your own html-coverage-report with the following simple shell script:

#!/bin/sh
curl -L -O https://raw.githubusercontent.com/jslint-org/jslint/v2021.5.23/ci.sh
printf 'console.log(true ? "hello world" : "bye world");' > foo.js
sh ci.sh shRunWithCoverage node foo.js
find .

# stdout/stderr:
#   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
#                                  Dload  Upload   Total   Spent    Left  Speed
# 100 30614  100 30614    0     0  57981      0 --:--:-- --:--:-- --:--:-- 57871
# hello world
# 
# coverage-report
# +----------------------------------+----------+
# | files covered                    |    lines |
# +----------------------------------+----------+
# | ./                               |      0 % |
# | ________________________________ |    0 / 1 |
# +----------------------------------+----------+
# | ./foo.js                         |      0 % |
# | ________________________________ |    0 / 1 |
# +----------------------------------+----------+
#
# .
# ./.build
# ./.build/coverage
# ./.build/coverage/coverage-badge.svg
# ./.build/coverage/coverage-report.txt
# ./.build/coverage/coverage-v8.json
# ./.build/coverage/foo.js.html
# ./.build/coverage/index.html
# ./ci.sh
# ./foo.js

• file - ./.build/coverage/index.html
  

• file - ./.build/coverage/foo.js.html
  

• file - ./.build/coverage/coverage-badge.svg
  

[gullerya]

First of all, thanks!

I'll probably rephrase the question a bit, since, well, I do know that I can read the node's report and filter it out by the URLs of the entries, so that's no brainer.
I'm actually owning some low level test runner, so that I'm looking for a more 'close-to-native' possibility, and that's only from performance perspective.

So the question is more about: can we make NodeJS to narrow down the coverage dump in the first place, even before that data touches the hard disk.

Another possibility - is it possible to get handle to that coverage object in the memory, say, from parent process (something like this Profiler API of Chromium Dev Tools)?

[kaizhu256]

+@bcoe

[kaizhu256]

it seems nodejs has an internal profiler that's normally not accessible. the following build script (in git-bash-for-windows) will allow you to access the profiler. unfortunately, i see nothing that allows you tell v8 to selectively disable profiling for certain files.

a bigger question is are you experiencing any real-world performance hits when turning on NODE_V8_COVERAGE?
unlike istanbul, i suspect much of the coverage-instrumentation is generated for free by v8's builtin profiler.

#!/bin/sh

# clone nodejs source code
git clone https://github.com/nodejs/node --single-branch --branch=v14.17.0
cd node

# patch lib/internal/bootstrap/loaders.js to grant access to internal-modules
sed -i \
    -e "s/internalBinding = /internalBinding = globalThis.internalBinding = /" \
    lib/internal/bootstrap/loaders.js

# build nodejs in windows
./vcbuild.bat

# inspect internal profiler module
./Release/node.exe -e '
console.log("inspect profiler module:");
console.log(internalBinding("profiler"));
'

# stdout:
# inspect profiler module:
# BaseObject {
#   setCoverageDirectory: [Function: setCoverageDirectory],
#   setSourceMapCacheGetter: [Function: setSourceMapCacheGetter],
#   takeCoverage: [Function: takeCoverage],
#   stopCoverage: [Function: stopCoverage]
# }

[kaizhu256]

sorry i misread the question. if user wants to get coverage-dump before hitting harddisk, maybe the profiler could have an additional function getCoverage()? something like following:

diff --git a/src/inspector_profiler.cc b/src/inspector_profiler.cc
index e40b4b99cc..03eebeeb40 100644
--- a/src/inspector_profiler.cc
+++ b/src/inspector_profiler.cc
@@ -478,6 +478,12 @@ static void SetSourceMapCacheGetter(const FunctionCallbackInfo<Value>& args) {
   env->set_source_map_cache_getter(args[0].As<Function>());
 }

+static isolate* GetCoverage() {
+  return env_->isolate();
+}
+
 static void TakeCoverage(const FunctionCallbackInfo<Value>& args) {
   Environment* env = Environment::GetCurrent(args);
   V8CoverageConnection* connection = env->coverage_connection();
@@ -516,6 +522,7 @@ static void Initialize(Local<Object> target,
   Environment* env = Environment::GetCurrent(context);
   env->SetMethod(target, "setCoverageDirectory", SetCoverageDirectory);
   env->SetMethod(target, "setSourceMapCacheGetter", SetSourceMapCacheGetter);
+  env->SetMethod(target, "getCoverage", GetCoverage);
   env->SetMethod(target, "takeCoverage", TakeCoverage);
   env->SetMethod(target, "stopCoverage", StopCoverage);
 }
@@ -523,6 +530,7 @@ static void Initialize(Local<Object> target,
 void RegisterExternalReferences(ExternalReferenceRegistry* registry) {
   registry->Register(SetCoverageDirectory);
   registry->Register(SetSourceMapCacheGetter);
+  registry->Register(GetCoverage);
   registry->Register(TakeCoverage);
   registry->Register(StopCoverage);
 }

[gullerya]

:) Yes, this is closer to my point, thanks!
And actually... there is there the TakeCoverage method, which right now does exactly that - takes the coverage data from V8, but it's void in return. Why not just return the result of that TakeCoverage?

And I'm not yet clear of how to invoke there inspector methods from the JS land, but assuming they are exposed somehow?

[gullerya]

I addition to the drill into the details with @kaizhu256 , I'd just like to pause moment on the following thought:

• coverage possibility in Node currently returns mostly internal stuff (running in with empty script already brought to me about 150K of coverage data)
• from the consumer perspective, it is almost likely NOT relevant data, opted IN by default
• I'd suggest either:
  • allow to opt out the internal staff by another env switch
  • even better - to opt it out by default and allow to opt in
  • even better - both: opt out internals by default + allow to get the data in memory instead of disk

All this might seem like pre-mature optimizations, but i'm dwelling now on some 'coverage-per-test' cross platform scenario and having, say 400-600 tests adding up overhead of: dump of full coverage by Node => read/parse it by test runner => filter out vast majority of that - all those just looks like a case to do some optimization here...

[dyst5422]

I hate to necro something that's years old, but wanted to point out a parallel element of code coverage that may or may not be relevant at the low level of v8 coverage, but I think is highly desirable to make the native test runner fully featured enough to support base use cases where coverage is desired.

The code coverage report does not currently allow for selecting files to include in the report. This means any files that AREN'T exercised at all don't show up in the report, and the totals are only relevant to those files that are executed. Additionally, it includes coverage for the test files themselves, which is likely not behavior expected by most.

I believe that at the node level (not V8), it should be able to either take a directory path to include and filtering options to ignore files in the report OR take a glob pattern of files to include in the report (basically both of the above combined).

I think this could be handled in post-processing by node in just reporting the files NOT included in the V8 coverage as zero coverage and ignoring those files according to the above filter.