implements new gleak goroutine leak detection experimental package (#538

)

- provides goroutine discovery as goroutine.Goroutine
- provides HaveLeaked matcher for reasoning on slices of goroutines for leak detection
- provides set of IgnoringXXX matchers for identifying good (non-leaky) goroutines
- provides unit tests for gleak and gleak/goroutine packages
- updates documentation and adds Gleakiee, the leaky gopher, mascot

docs/css/layout.css

@@ -59,6 +59,13 @@ img[alt="Ginkgo"] {
 	margin-right: auto;
 }
 
+img[alt="Leakiee"] {
+	max-width: 80%;
+	max-height: 150px;
+	display: block;
+	float: right;
+}
+
 /* code styling */
 
 .markdown-body div.highlight {

docs/index.md

@@ -2984,4 +2984,292 @@ Describe("server performance", func() {
 })
 ```
 
+## `gleak`: Finding Leaked Goroutines
+
+![Leakiee](./images/leakiee.png)
+
+The `gleak` package provides support for goroutine leak detection.
+
+> **Please note:** gleak is an experimental new Gomega package.
+
+### Basics
+
+Calling `Goroutines` returns information about all goroutines of a program at
+this moment. `Goroutines` typically gets invoked in the form of
+`Eventually(Goroutines).ShouldNot(...)`. Please note the missing `()` after
+`Goroutines`, as it must be called by `Eventually` and **not before it** with
+its results passed to `Eventually` only once. This does not preclude calling
+`Goroutines()`, such as for taking goroutines snapshots.
+
+Leaked goroutines are then detected by using `gleak`'s `HaveLeaked` matcher on
+the goroutines information. `HaveLeaked` checks the actual list of goroutines
+against a built-in list of well-known runtime and testing framework goroutines,
+as well as against any optionally additional goroutines specifications passed to
+`HaveLeaked`. Good, and thus "non-leaky", Goroutines can be identified in
+multiple ways: such as by the name of a topmost function on a goroutine stack or
+a snapshot of goroutine information taken before a test. Non-leaky goroutines
+can also be identified using basically any Gomega matcher, with `HaveField` or
+`WithTransform` being highly useful in test-specific situations.
+
+The `HaveLeaked` matcher _succeeds_ if it finds any goroutines that are neither
+in the integrated list of well-known goroutines nor in the optionally specified
+`HaveLeaked` arguments. In consequence, any _success_ of `HaveLeaked` actually
+is meant to be a _failure_, because of leaked goroutines. `HaveLeaked` is thus
+mostly used in combination with `ShouldNot` and `NotTo`/`ToNot`.
+
+### Testing for Goroutine Leaks
+
+In its most simple form, just run a goroutine discovery with a leak check right
+_after_ each test in `AfterEach`:
+
+> **Important:** always use `Goroutines` and not `Goroutines()` in the call to
+> `Eventually`. This ensures that the goroutine discovery is correctly done
+> repeatedly as needed and not just a single time before calling `Eventually`.
+
+```go
+AfterEach(func() {
+    Eventually(Goroutines).ShouldNot(HaveLeaked())
+})
+```
+
+Using `Eventually` instead of `Ω`/`Expect` has the benefit of retrying the leak
+check periodically until there is no leak or until a timeout occurs. This
+ensures that goroutines that are (still) in the process of winding down can
+correctly terminate without triggering false positives. Please refer to the
+[`Eventually`](#eventually) section for details on how to change the timeout
+interval (which defaults to 1s) and the polling interval (which defaults to
+10ms).
+
+This form of goroutine leak test can cause false positives in situations where a
+test suite or dependency module uses additional goroutines. This simple form
+only looks at all goroutines _after_ a test has run and filters out all
+_well-known_ "non-leaky" goroutines, such as goroutines from Go's runtime and
+the testing frameworks (such as Go's own testing package and Gomega).
+
+### Using Goroutine Snapshots in Leak Testing
+
+Often, it might be sufficient to cover for additional "non-leaky" goroutines by
+taking a "snapshot" of goroutines _before_ a test and then _afterwards_ use ths
+snapshot to filter out the supposedly "non-leaky" goroutines.
+
+```go
+var ignoreGood []goroutine.Goroutine
+
+BeforeEach(func() {
+    ignoreGood = Goroutines()
+})
+
+AfterEach(func() {
+    Eventually(Goroutines).ShouldNot(HaveLeaked(ignoreGood))
+})
+```
+
+### `HaveLeaked` Matcher
+
+```go
+Eventually(ACTUAL).ShouldNot(HaveLeaked(NONLEAKY1, NONLEAKY2, NONLEAKY3, ...))
+```
+
+causes a test to fail if `ACTUAL` after filtering out the well-known "good" (and
+non-leaky) goroutines of the Go runtime and test frameworks, as well as
+filtering out the additional non-leaky goroutines passed to the matcher, still
+results in one or more goroutines. The ordering of the goroutines does not
+matter.
+
+`HaveLeaked` always takes the built-in list of well-known good goroutines into
+consideration and this list can neither be overriden nor disabled. Additional
+known non-leaky goroutines `NONLEAKY1`, ...  can be passed to `HaveLeaks` either
+in form of `GomegaMatcher`s or in shorthand notation:
+
+- `"foo.bar"` is shorthand for `IgnoringTopFunction("foo.bar")` and filters out
+  any (non-leaky) goroutine with its topmost function on the backtrace stack
+  having the exact name `foo.bar`.
+
+- `"foo.bar..."` is shorthand for `IgnoringTopFunction("foo.bar...")` and
+  filters out any (non-leaky) goroutine with its topmost function on the
+  backtrace stack beginning with the prefix `foo.bar.`; please notice the
+  trailing `.` dot.
+
+- `"foo.bar [chan receive]"` is shorthand for `IgnoringTopFunction("foo.bar
+  [chan receive]")` and filters out any (non-leaky) goroutine where its topmost
+  function on the backtrace stack has the exact name `foo.bar` _and_ the
+  goroutine is in a state beginning with `chan receive`.
+
+- `[]goroutine.Goroutine` is shorthand for
+  `IgnoringGoroutines(<SLICEOFGOROUTINES>)`: it filters out the specified
+  goroutines, considering them to be non-leaky. The goroutines are identified by
+  their [goroutine IDs](#goroutine-ids).
+
+- `IgnoringInBacktrace("foo.bar.baz")` filters out any (non-leaky) goroutine
+  with `foo.bar.baz` _anywhere_ in its backtrace.
+
+- additionally, any other `GomegaMatcher` can be passed to `HaveLeaked()`, as
+  long as this matcher can work on a passed-in actual value of type
+  `goroutine.Goroutine`.
+
+### Goroutine Matchers
+
+The `gleak` packages comes with a set of predefined Goroutine matchers, to be
+used with `HaveLeaked`. If these matchers succeed (that is, they match on a
+certain `Goroutine`), then `HaveLeaked` considers the matched goroutine to be
+non-leaky.
+
+#### IgnoringTopFunction(topfname string)
+
+```go
+Eventually(ACTUAL).ShouldNot(HaveLeaked(IgnoringTopFunction(TOPFNAME)))
+```
+
+In its most basic form, `IgnoringTopFunction` succeeds if `ACTUAL` contains a
+goroutine where the name of the topmost function on its call stack (backtrace)
+is `TOPFNAME`, causing `HaveLeaked` to filter out the matched goroutine as
+non-leaky. Different forms of `TOPFNAME` describe different goroutine matching
+criteria:
+
+- `"foo.bar"` matches only if a goroutine's topmost function has this exact name
+  (ignoring any function parameters).
+- `"foo.bar..."` matches if a goroutine's topmost function name starts with the
+  prefix `"foo.bar."`; it doesn't match `"foo.bar"` though.
+- `"foo.bar [state]"` matches if a goroutine's topmost function has this exact
+  name and the goroutine's state begins with the specified state string.
+
+`ACTUAL` must be an array or slice of `goroutine.Goroutine`s.
+
+#### IgnoringGoroutines(goroutines []goroutine.Goroutine)
+
+```go
+Eventually(ACTUAL).ShouldNot(HaveLeaked(IgnoringGoroutines(GOROUTINES)))
+```
+
+`IgnoringGoroutines` succeeds if `ACTUAL` contains one or more goroutines which
+are elements of `GOROUTINES`, causing `HaveLeaked` to filter out the matched
+goroutine(s) as non-leaky. `IgnoringGoroutines` compares goroutines by their
+`ID`s (see [Goroutine IDs](#gorotuine-ids) for background information).
+
+`ACTUAL` must be an array or slice of `goroutine.Goroutine`s.
+
+#### IgnoringInBacktrace(fname string)
+
+```go
+Eventually(Goroutines).ShouldNot(HaveLeaked(IgnoringInBacktrace(FNAME)))
+```
+
+`IgnoringInBacktrace` succeeds if `ACTUAL` contains a groutine where `FNAME` is
+contained anywhere within its call stack (backtrace), causing `HaveLeaked` to
+filter out the matched goroutine as non-leaky. Please note that
+`IgnoringInBacktrace` uses a (somewhat lazy) `strings.Contains` to check for any
+occurence of `FNAME` in backtraces.
+
+`ACTUAL` must be an array or slice of `goroutine.Goroutine`s.
+
+#### IgnoringCreator(creatorname string)
+
+```go
+Eventually(Goroutines).ShouldNot(HaveLeaked(IgnoringCreator(CREATORNAME)))
+```
+
+In its most basic form, `IgnoringCreator` succeeds if `ACTUAL` contains a
+groutine where the name of the function creating the goroutine matches
+`CREATORNAME`, causing `HaveLeaked` to filter out the matched goroutine(s) as
+non-leaky. `IgnoringCreator` uses `==` for comparing the creator function name.
+
+Different forms of `CREATORNAME` describe different goroutine matching
+criteria:
+
+- `"foo.bar"` matches only if a goroutine's creator function has this exact name
+  (ignoring any function parameters).
+- `"foo.bar..."` matches if a goroutine's creator function name starts with the
+  prefix `"foo.bar."`; it doesn't match `"foo.bar"` though.
+
+### Adjusting Leaky Goroutine Reporting
+
+When `HaveLeaked` finds leaked goroutines, Gomega prints out a description of
+(only) the _leaked_ goroutines. This is different from panic output that
+contains backtraces of all goroutines.
+
+However, `noleak`'s goroutine dump deliberately is not subject to Gomega's usual
+object rendition controls, such as `format.MaxLength` (see also [Adjusting
+Output](#adjusting-output)).
+
+`noleak` will print leaked goroutine backtraces in a more compact form, with
+function calls and locations combined into single lines. Additionally, `noleak`
+defaults to reporting only the package plus file name and line number, but not
+the full file path. For instance:
+
+    main.foo.func1() at foo/bar.go:123
+
+Setting `noleak.ReportFilenameWithPath` to `true` will instead report full
+source code file paths:
+
+    main.foo.func1() at /home/coolprojects/ohmyleak/mymodule/foo/bar.go:123
+
+### Well-Known Non-Leaky Goroutines
+
+The well-known good (and therefore "non-leaky") goroutines are identified by the
+names of the topmost functions on their stacks (backtraces):
+
+- signal handling:
+  - `os/signal.signal_recv` and `os/signal.loop` (covering
+  varying state),
+  - as well as `runtime.ensureSigM`.
+- Go's built-in [`testing`](https://pkg.go.dev/testing) framework:
+  - `testing.RunTests [chan receive]`,
+  - `testing.(*T).Run [chan receive]`,
+  - and `testing.(*T).Parallel [chan receive]`.
+- the [Ginko testing framework](https://onsi.github.io/ginkgo/):
+  - `github.com/onsi/ginkgo/v2/internal.(*Suite).runNode` (including anonymous
+  inner functions),
+  - the anonymous inner functions of
+  `github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts`,
+  - and finally
+  `github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).registerForInterrupts`
+  (for v1 support).
+
+Additionally, any goroutines with `runtime.ReadTrace` in their backtrace stack
+are also considered to be non-leaky.
+
+### Goroutine IDs
+
+In order to detect goroutine identities, we use what is generally termed
+"goroutine IDs". These IDs appear in runtime stack dumps ("backtrace"). But …
+are these goroutine IDs even unambiguous? What are their "guarantees", if there
+are any at all?
+
+First, Go's runtime code uses the identifier (and thus term) [`goid` for
+Goroutine
+IDs](https://github.com/golang/go/search?q=goidgen&unscoped_q=goidgen). Good to
+know in case you need to find your way around Go's runtime code.
+
+Now, based on [Go's `goid` runtime allocation
+code](https://github.com/golang/go/blob/release-branch.go1.18/src/runtime/proc.go#L4130),
+goroutine IDs never get reused – unless you manage to make the 64bit "master
+counter" of the Go runtime scheduler to wrap around. However, not all goroutine
+IDs up to the largest one currently seen might ever be used, because as an
+optimization goroutine IDs are always assigned to Go's "P" processors for
+assignment to newly created "G" goroutines in batches of 16. In consequence,
+there may be gaps and later goroutines might have lower goroutine IDs if they
+get created from a different P.
+
+Finally, there's [Scott Mansfield's blog post on Goroutine
+IDs](https://blog.sgmansfield.com/2015/12/goroutine-ids/). To sum up Scott's
+point of view: don't use goroutine IDs. He spells out good reasons for why you
+should not use them. Yet, logging, debugging and testing looks like a sane and
+solid exemption from his rule, not least `runtime.Stack` includes the `goids`
+for
+some reason.
+
+### Credits
+
+The _Leakiee the gopher_ mascot clearly has been inspired by the Go gopher art
+work of [Renee French](http://reneefrench.blogspot.com/).
+
+The `gleak` package was heavily inspired by Uber's fine
+[goleak](https://github.com/uber-go/goleak) goroutine leak detector package.
+While `goleak` can be used with Gomega and Ginkgo in a very specific form, it
+unfortunately was never designed to be (optionally) used with a matcher library
+to unlock the full potential of reasoning about leaky goroutines. In fact, the
+crucial element of discovering goroutines is kept internal in `goleak`. In
+consequence, Gomega's `gleak` package uses its own goroutine discovery and is
+explicitly designed to perfectly blend in with Gomega (and Ginkgo).
+
 {% endraw  %}