cgen,pref: add `-coverage` support + `vcover` tool #21154

felipensp · 2024-03-31T20:07:34Z

Building with coverage support

v -cov ./dir-output file.v
or
v -cov ./dir-output run file.v

Every run of ./file will generate a cover.<build hash>.<time hash>.json file.

VCOVDIR env variable can be used to change the output dir.
VCOVDIR=/tmp/cover ./file

Reporting to stdout

$ v cover -file vcover.01HYNV1D0CSFFXKKJT5QB7T624.2040586979 
  17.44 |   15 |   86 | /home/felipe/github/v/vlib/strings/builder.c.v
   9.21 |   14 |  152 | /home/felipe/github/v/vlib/math/bits/bits.v
  65.66 |  130 |  198 | /home/felipe/github/v/vlib/strconv/f64_str.c.v
  32.10 |   52 |  162 | /home/felipe/github/v/vlib/strconv/utilities.c.v
  28.57 |   28 |   98 | /home/felipe/github/v/vlib/strconv/utilities.v
  44.44 |    4 |    9 | /home/felipe/github/v/vlib/builtin/array.v
  26.42 |   28 |  106 | /home/felipe/github/v/vlib/builtin/array_d_gcboehm_opt.v
  28.57 |    6 |   21 | /home/felipe/github/v/vlib/builtin/backtraces_nix.c.v
  34.45 |   72 |  209 | /home/felipe/github/v/vlib/builtin/builtin.c.v
   3.57 |    1 |   28 | /home/felipe/github/v/vlib/builtin/builtin.v
  14.29 |    1 |    7 | /home/felipe/github/v/vlib/builtin/builtin_d_gcboehm.c.v
  33.33 |    2 |    6 | /home/felipe/github/v/vlib/builtin/builtin_nix.c.v
  46.67 |    7 |   15 | /home/felipe/github/v/vlib/builtin/cfns_wrapper.c.v
   7.25 |    5 |   69 | /home/felipe/github/v/vlib/builtin/float.c.v
  35.44 |   73 |  206 | /home/felipe/github/v/vlib/builtin/int.v
  20.00 |    3 |   15 | /home/felipe/github/v/vlib/builtin/result.v
   2.33 |   22 |  946 | /home/felipe/github/v/vlib/builtin/string.v
  16.53 |   60 |  363 | /home/felipe/github/v/vlib/builtin/string_interpolation.v
  15.94 |   11 |   69 | /home/felipe/github/v/vlib/builtin/utf8.v
  11.43 |    8 |   70 | /home/felipe/github/v/vlib/json/json_primitives.c.v
 100.00 |    2 |    2 | bug.v
Total coverage: 21 files, 19.18% coverage

Merging and reporting to stdout

v cov -dir ./coverage-dir

Test coverage

v -cov /tmp/cover test vlib/json/
v cover -dir /tmp/cover

Filtering results

v cover -dir /tmp/cover -filter vlib/json,vlib/toml

spytheman

Excellent work.

spytheman · 2024-05-14T10:13:11Z

@felipensp, I've tested running this:

if true {
	println('branch 1')
	if true {
		println('branch 1.1')
		if true {
			println('branch 1.1.1')
		} else {
			println('branch 1.1.2')
		}
	} else {
		println('branch 1.2')
		if true {
			println('branch 1.2.1')
		} else {
			println('branch 1.2.2')
		}
	}
} else {
	println('branch 2')
	if true {
		println('branch 2.1')
		if true {
			println('branch 2.1.1')
		} else {
			println('branch 2.1.2')
		}
	} else {
		println('branch 2.2')
		if true {
			println('branch 2.2.1')
		} else {
			println('branch 2.2.2')
		}
	}
}

with: v -coverage - run branches.v, and I got:

#0 13:07:08 ^ v_coverage /v/oo>v -coverage - run branches.v
branch 1
branch 1.1
branch 1.1.1
V coverage
--------------------------------------------------
[   0.00%]    0 /   98 | /space/v/oo/vlib/strings/builder.c.v
[   0.00%]    0 /   43 | /space/v/oo/vlib/strings/similarity.v
[   0.00%]    0 /   18 | /space/v/oo/vlib/strings/strings.c.v
[   0.00%]    0 /   59 | /space/v/oo/vlib/strings/strings.v
[   0.00%]    0 /  152 | /space/v/oo/vlib/math/bits/bits.v
[   0.00%]    0 /    0 | /space/v/oo/vlib/math/bits/bits_tables.v
[   0.00%]    0 /    8 | /space/v/oo/vlib/math/bits/unsafe_bits.v
[   0.00%]    0 /  191 | /space/v/oo/vlib/strconv/atof.c.v
[   0.00%]    0 /   60 | /space/v/oo/vlib/strconv/atofq.c.v
[   0.00%]    0 /  104 | /space/v/oo/vlib/strconv/atoi.v
[   0.00%]    0 /  176 | /space/v/oo/vlib/strconv/f32_str.c.v
[   0.00%]    0 /  198 | /space/v/oo/vlib/strconv/f64_str.c.v
[   0.00%]    0 /    0 | /space/v/oo/vlib/strconv/f64_str.v
[   0.00%]    0 /   13 | /space/v/oo/vlib/strconv/format.v
[   0.00%]    0 /  269 | /space/v/oo/vlib/strconv/format_mem.c.v
[   0.00%]    0 /    4 | /space/v/oo/vlib/strconv/ftoa.c.v
[   0.00%]    0 /   38 | /space/v/oo/vlib/strconv/number_to_base.c.v
[   0.00%]    0 /    0 | /space/v/oo/vlib/strconv/tables.v
[   0.00%]    0 /  166 | /space/v/oo/vlib/strconv/utilities.c.v
[   0.00%]    0 /   98 | /space/v/oo/vlib/strconv/utilities.v
[   0.00%]    0 /  433 | /space/v/oo/vlib/strconv/vprintf.c.v
[   0.00%]    0 /    9 | /space/v/oo/vlib/builtin/array.v
[   3.36%]    4 /  119 | /space/v/oo/vlib/builtin/array_d_gcboehm_opt.v
[   0.00%]    0 /    4 | /space/v/oo/vlib/builtin/backtraces.c.v
[   0.00%]    0 /   21 | /space/v/oo/vlib/builtin/backtraces_nix.c.v
[  18.48%]   39 /  211 | /space/v/oo/vlib/builtin/builtin.c.v
[   0.00%]    0 /   28 | /space/v/oo/vlib/builtin/builtin.v
[  14.29%]    1 /    7 | /space/v/oo/vlib/builtin/builtin_d_gcboehm.c.v
[  33.33%]    2 /    6 | /space/v/oo/vlib/builtin/builtin_nix.c.v
[  20.00%]    3 /   15 | /space/v/oo/vlib/builtin/cfns_wrapper.c.v
[   0.00%]    0 /    2 | /space/v/oo/vlib/builtin/chan.v
[   0.00%]    0 /   11 | /space/v/oo/vlib/builtin/character_inout.c.v
[   0.00%]    0 /   69 | /space/v/oo/vlib/builtin/float.c.v
[   0.00%]    0 /  207 | /space/v/oo/vlib/builtin/int.v
[   0.00%]    0 /   34 | /space/v/oo/vlib/builtin/map.c.v
[   0.00%]    0 /  306 | /space/v/oo/vlib/builtin/map.v
[   0.00%]    0 /   38 | /space/v/oo/vlib/builtin/map_d_gcboehm_opt.v
[   0.00%]    0 /    3 | /space/v/oo/vlib/builtin/option.c.v
[   0.00%]    0 /    8 | /space/v/oo/vlib/builtin/option.v
[   0.00%]    0 /   15 | /space/v/oo/vlib/builtin/result.v
[   0.00%]    0 /   24 | /space/v/oo/vlib/builtin/rune.v
[   0.00%]    0 /  206 | /space/v/oo/vlib/builtin/sorted_map.v
[   0.00%]    0 /  951 | /space/v/oo/vlib/builtin/string.v
[   0.00%]    0 /    9 | /space/v/oo/vlib/builtin/string_charptr_byteptr_helpers.v
[   0.00%]    0 /  364 | /space/v/oo/vlib/builtin/string_interpolation.v
[   0.00%]    0 /   18 | /space/v/oo/vlib/builtin/utf8.c.v
[   0.00%]    0 /   69 | /space/v/oo/vlib/builtin/utf8.v
[  54.55%]    6 /   11 | branches.v
--------------------------------------------------
Total coverage: 48 files, 1.12% coverage
#0 13:07:41 ^ v_coverage /v/oo>

Why does it print all the .v files for all modules (used in the program), given that for most programs, most of their functionality, will not be used?

I think that skipping all the ones, for which there is 0 coverage will be more useful and more compact.

felipensp · 2024-05-21T21:40:58Z

I am planning to do:

Add env var for dir for output files
Generate the counter file on each run
Generate the separated metadata file
cmd tool (vcover) to merging/reporting

vlib/v/gen/c/coverage.v

JalonSolov · 2024-05-24T14:47:27Z

The Windows gcc failure appears to be completely unrelated... and very, very weird:

  FAIL  [ 252/1864] C:  3547.7 ms, R:    39.108 ms vlib/net/unix/use_net_and_net_unix_together_test.v
         retry: 1
      comp_cmd: "D:\a\v\v\v.exe" -skip-running   -o "C:\Users\runneradmin\AppData\Local\Temp\v_0\tsession_1368_01HYNCRVAR27QE8T5X47RA2JE4\251_2\use_net_and_net_unix_together_test.exe" "D:\a\v\v\vlib\net\unix\use_net_and_net_unix_together_test.v"
       run_cmd: "C:\Users\runneradmin\AppData\Local\Temp\v_0\tsession_1368_01HYNCRVAR27QE8T5X47RA2JE4\251_2\use_net_and_net_unix_together_test.exe"
failure code: 1; foutput.len: 437; failure output:
---------------------------------------------------------------------------------------------------- retry: 0 ; max_retry: 0 ; r.exit_code: 1 ; trimmed_output.len: 269
V panic: net: resolve_addrs Unix socket address is too long; could not resolve path C:\Users\runneradmin\AppData\Local\Temp\v_0\tsession_1368_01HYNCRVAR27QE8T5X47RA2JE4\net_unix_test\domain_socket

v hash: c46c1e3

print_backtrace_skipping_top_frames is not implemented

spytheman · 2024-05-25T08:28:53Z

The Windows gcc failure appears to be completely unrelated... and very, very weird:

It is my fault. It is fixed on master now.

felipensp added 8 commits March 31, 2024 17:06

fix

5cad749

fix

c255914

improvement

f9ed550

fix

a1c524c

improvement

68bd90d

improve

f05811a

Merge remote-tracking branch 'origin/master' into v_coverage

322d2f4

fix merge

5a8ecb8

felipensp marked this pull request as ready for review May 13, 2024 23:47

spytheman changed the title ~~v: code coverage~~ cgen,pref: add -coverage support May 14, 2024

spytheman approved these changes May 14, 2024

View reviewed changes

felipensp marked this pull request as draft May 14, 2024 17:00

felipensp added 3 commits May 23, 2024 10:56

Merge remote-tracking branch 'origin/master' into v_coverage

8344d02

wip

0adcd12

wip

b5d9fa6

JalonSolov reviewed May 24, 2024

View reviewed changes

vlib/v/gen/c/coverage.v Outdated Show resolved Hide resolved

felipensp added 3 commits May 24, 2024 10:15

fix

cf6bb6f

Merge remote-tracking branch 'origin/master' into v_coverage

0f4f4a7

wip

c223eb9

wip

791d7ba

felipensp changed the title ~~cgen,pref: add -coverage support~~ cgen,pref: add -coverage support + vcover tool May 24, 2024

wip

a561426

felipensp added 4 commits May 25, 2024 10:40

improve

01ed4d7

fix

1096fb5

added -filter

c80b978

ignore test funcs

73c3878

improve filtering

0d6665f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cgen,pref: add `-coverage` support + `vcover` tool #21154

cgen,pref: add `-coverage` support + `vcover` tool #21154

felipensp commented Mar 31, 2024 •

edited

spytheman left a comment

spytheman commented May 14, 2024 •

edited

felipensp commented May 21, 2024

JalonSolov commented May 24, 2024

spytheman commented May 25, 2024

cgen,pref: add -coverage support + vcover tool #21154

Are you sure you want to change the base?

cgen,pref: add -coverage support + vcover tool #21154

Conversation

felipensp commented Mar 31, 2024 • edited

Building with coverage support

Reporting to stdout

Merging and reporting to stdout

Test coverage

Filtering results

spytheman left a comment

Choose a reason for hiding this comment

spytheman commented May 14, 2024 • edited

felipensp commented May 21, 2024

JalonSolov commented May 24, 2024

spytheman commented May 25, 2024

cgen,pref: add `-coverage` support + `vcover` tool #21154

cgen,pref: add `-coverage` support + `vcover` tool #21154

felipensp commented Mar 31, 2024 •

edited

spytheman commented May 14, 2024 •

edited