Not all tests are run, and some of the skipped are broken

[marco-m]

• Task version: 3.4.2 (but this bug is independent)
• Operating System: macOS 11.3
• Go version: go 1.16.3

Hello @andreynering :-)

With reference to the Taskfile of Task itself, the test task definition is

  test:
    desc: Runs test suite
    deps: [install]
    cmds:
      - go test {{catLines .GO_PACKAGES}}

GO_PACKAGES is set as

vars:
  GO_PACKAGES:
    sh: go list ./...

$ go list ./...
github.com/go-task/task/v3
github.com/go-task/task/v3/args
github.com/go-task/task/v3/cmd/task
github.com/go-task/task/v3/internal/compiler
github.com/go-task/task/v3/internal/compiler/v2
github.com/go-task/task/v3/internal/compiler/v3
github.com/go-task/task/v3/internal/execext
github.com/go-task/task/v3/internal/logger
github.com/go-task/task/v3/internal/output
github.com/go-task/task/v3/internal/status
github.com/go-task/task/v3/internal/summary
github.com/go-task/task/v3/internal/templater
github.com/go-task/task/v3/taskfile
github.com/go-task/task/v3/taskfile/read

But a plain go test shows the following errors. At least some of them are due to the fact that recently Task started using ANSI escape sequences for color output.

$ go test
--- FAIL: TestStatus (0.00s)
    task_test.go:277: Wrong output message: task: Task "gen-foo" is up to date
        
--- FAIL: TestPrecondition (0.00s)
    task_test.go:302: Wrong output message: task: 1 != 0 obviously!
        
    task_test.go:310: Wrong output message: task: 1 != 0 obviously!
        
    task_test.go:317: Wrong output message: task: 1 != 0 obviously!
        
--- FAIL: TestGenerates (0.02s)
    task_test.go:373: Wrong output message: task: Task "sub/src.txt" is up to date
        task: Task "rel.txt" is up to date
        
    task_test.go:373: Wrong output message: task: Task "sub/src.txt" is up to date
        task: Task "abs.txt" is up to date
        
    task_test.go:373: Wrong output message: task: Task "sub/src.txt" is up to date
        task: Task "my text file.txt" is up to date
        
--- FAIL: TestStatusChecksum (0.01s)
    task_test.go:410: 
                Error Trace:    task_test.go:410
                Error:          Not equal: 
                                expected: "task: Task \"build\" is up to date\n"
                                actual  : "\x1b[0mtask: Task \"build\" is up to date\n\x1b[0m"
                            
                                Diff:
                                --- Expected
                                +++ Actual
                                @@ -1,2 +1,2 @@
                                -task: Task "build" is up to date
                                -
                                +task: Task "build" is up to date
                                +
                Test:           TestStatusChecksum
--- FAIL: TestDry (0.00s)
    task_test.go:629: 
                Error Trace:    task_test.go:629
                Error:          Not equal: 
                                expected: "task: [build] touch file.txt"
                                actual  : "\x1b[0mtask: [build] touch file.txt\n\x1b[0m"
                            
                                Diff:
                                --- Expected
                                +++ Actual
                                @@ -1 +1,2 @@
                                -task: [build] touch file.txt
                                +task: [build] touch file.txt
                                +
                Test:           TestDry
--- FAIL: TestSummary (0.00s)
    task_test.go:754: 
                Error Trace:    task_test.go:754
                Error:          Not equal: 
                                expected: "task: task-with-summary\n\nsummary of task-with-summary - line 1\nline 2\nline 3\n\ndependencies:\n - dependend-task-1\n - dependend-task-2\n\ncommands:\n - echo 'task-with-summary was executed'\n - echo 'another command'\n - exit 0\n\n\ntask: other-task-with-summary\n\nsummary of other-task-with-summary\n\ncommands:\n - echo 'other-task-with-summary was executed'\n"
                                actual  : "\x1b[0mtask: task-with-summary\n\x1b[0m\x1b[0m\n\x1b[0m\x1b[0msummary of task-with-summary - line 1\n\x1b[0m\x1b[0mline 2\n\x1b[0m\x1b[0mline 3\n\x1b[0m\x1b[0m\n\x1b[0m\x1b[0mdependencies:\n\x1b[0m\x1b[0m - dependend-task-1\n\x1b[0m\x1b[0m - dependend-task-2\n\x1b[0m\x1b[0m\n\x1b[0m\x1b[0mcommands:\n\x1b[0m\x1b[0m - echo 'task-with-summary was executed'\n\x1b[0m\x1b[0m - echo 'another command'\n\x1b[0m\x1b[0m - exit 0\n\x1b[0m\x1b[0m\n\x1b[0m\x1b[0m\n\x1b[0m\x1b[0mtask: other-task-with-summary\n\x1b[0m\x1b[0m\n\x1b[0m\x1b[0msummary of other-task-with-summary\n\x1b[0m\x1b[0m\n\x1b[0m\x1b[0mcommands:\n\x1b[0m\x1b[0m - echo 'other-task-with-summary was executed'\n\x1b[0m"
                            
                                Diff:
                                --- Expected
                                +++ Actual
                                @@ -1,23 +1,23 @@
                                -task: task-with-summary
                                -
                                -summary of task-with-summary - line 1
                                -line 2
                                -line 3
                                -
                                -dependencies:
                                - - dependend-task-1
                                - - dependend-task-2
                                -
                                -commands:
                                - - echo 'task-with-summary was executed'
                                - - echo 'another command'
                                - - exit 0
                                -
                                -
                                -task: other-task-with-summary
                                -
                                -summary of other-task-with-summary
                                -
                                -commands:
                                - - echo 'other-task-with-summary was executed'
                                -
                                +task: task-with-summary
                                +
                                +summary of task-with-summary - line 1
                                +line 2
                                +line 3
                                +
                                +dependencies:
                                + - dependend-task-1
                                + - dependend-task-2
                                +
                                +commands:
                                + - echo 'task-with-summary was executed'
                                + - echo 'another command'
                                + - exit 0
                                +
                                +
                                +task: other-task-with-summary
                                +
                                +summary of other-task-with-summary
                                +
                                +commands:
                                + - echo 'other-task-with-summary was executed'
                                +
                Test:           TestSummary
FAIL
exit status 1
FAIL    github.com/go-task/task/v3      0.325s

EDIT

Mhhh, I am now confused, it seems that the tests are actually run, but they fail only when invoking go test directly?

[andreynering]

This was very strange indeed: go test . and go test ./... were passing, but go test failed. I really don't know why.

Anyway, I forced NO_COLOR=1 on tests to fix this, which recently got support by fatih/color#137 (came in a good time 🙂).

[marco-m]

Ah, good catch!