test options do not work for compiled binaries

[hansbogert]

Godog.BindCommandLineFlags in init does not work for binaries

When using test flags like -coverprofile they are not honored when trying a godog application which has been created by go test -c ... -o ...

happy case

Using the a test as suggested by the README:

package foo

import (
	"os"
	"testing"

	"github.com/cucumber/godog"
	"github.com/cucumber/godog/colors"
	"github.com/spf13/pflag" // godog v0.11.0 and later
)

var opts = godog.Options{
	Output: colors.Colored(os.Stdout),
	Format: "progress", // can define default values
}

func init() {
	godog.BindCommandLineFlags("godog.", &opts) // godog v0.11.0 and later
}

func TestMain(m *testing.M) {
	pflag.Parse()
	opts.Paths = pflag.Args()

	status := godog.TestSuite{
		Name:    "godogs",
		Options: &opts,
	}.Run()

	// Optional: Run `testing` package's logic besides godog.
	if st := m.Run(); st > status {
		status = st
	}

	os.Exit(status)
}

Running this directly is working:

$ go test  -coverprofile foo.txt  godog_test.go
ok  	command-line-arguments	0.060s	coverage: [no statements] [no tests to run]

Running the compiled variant:

$ go test  -c -coverprofile foo.txt  godog_test.go
$ ./foo.test -coverprofile foo.txt  godog_test.go 
./foo.test -coverprofile foo.txt  godog_test.go 
invalid argument "overprofile" for "-c, --godog.concurrency" flag: strconv.ParseInt: parsing "overprofile": invalid syntax
Usage of ./foo.test:
  -c, --godog.concurrency int   run the test suite with concurrency (default 1)
  -d, --godog.definitions       print all available step definitions
  -f, --godog.format string     will write a report according to the selected formatter
                                
                                usage:
                                  -f <formatter>

Notice that the help file does not show any --test.coverprofile as suggestion.

✨ Do you have a proposal for making it better?

It seems not having godog.BindCommandLineFlags("godog.", &opts) in a init() function and placing it just before pflag.Parse() is enough. And seems to work for test binaries as well as with directly ran go test applications

Continuing with the code given above

17,20d16
< func init() {
< 	godog.BindCommandLineFlags("godog.", &opts) // godog v0.11.0 and later
< }
< 
21a18
> 	godog.BindCommandLineFlags("godog.", &opts) // godog v0.11.0 and later

Rerunning:

./foo.test -coverprofile foo.txt  godog_test.go         
invalid argument "overprofile" for "-c, --godog.concurrency" flag: strconv.ParseInt: parsing "overprofile": invalid syntax
Usage of ./foo.test:
  -c, --godog.concurrency int                       run the test suite with concurrency (default 1)
  ......
      --test.bench regexp                           run only benchmarks matching regexp
      --test.benchmem                               print memory allocations for benchmarks
      --test.benchtime d                            run each benchmark for duration d (default 1s)
      --test.blockprofile file                      write a goroutine blocking profile to file
      --test.blockprofilerate rate                  set blocking profile rate (see runtime.SetBlockProfileRate) (default 1)
      --test.count n                                run tests and benchmarks n times (default 1)
      --test.coverprofile file                      write a coverage profile to file
      --test.cpu list                               comma-separated list of cpu counts to run each test with
      --test.cpuprofile file                        write a cpu profile to file
      --test.failfast                               do not start new tests after the first test failure
      --test.fuzz regexp                            run the fuzz test matching regexp
....

Still not working, though notice the added --test-.... options now.

Simply using --test.coverprofile now works:

rm foo.txt; ./foo.test --test.coverprofile foo.txt > /dev/null ; ls -alh foo.txt
testing: warning: no tests to run
-rw-rw-r-- 1 hvdb hvdb 10 Jan 12 21:43 foo.txt

📚 Any additional context?

I'm unsure if there was a good reason to have the above line in the init() in the first place (?)

---

This text was originally generated from a template, then edited by hand. You can modify the template here.

[vearutop]

Please check #498 and #495.

It seems, while godog.BindCommandLineFlags has some ergonomic benefits, it is not great from perspective of stdlib/go test compatibility. Maybe it would even make sense to deprecate and phase it out in favor of godog.BindFlags.