docstore/memdocstore: fatal error: concurrent map iteration and map write

[eqinox76]

Describe the bug

Our unit tests fail in ~1/10 cases with:

fatal error: concurrent map iteration and map write

goroutine 66 [running]:
runtime.throw(0x123488e, 0x26)
        /snap/go/current/src/runtime/panic.go:1117 +0x72 fp=0xc000195248 sp=0xc000195218 pc=0x43a5d2
runtime.mapiternext(0xc000195318)
        /snap/go/current/src/runtime/map.go:858 +0x54c fp=0xc0001952c8 sp=0xc000195248 pc=0x412cac
gocloud.dev/docstore/memdocstore.decoder.DecodeMap(0x10ec2e0, 0xc0003d8870, 0xc0006bd400)
        /home/user/go/pkg/mod/gocloud.dev@v0.22.0/docstore/memdocstore/codec.go:191 +0xa5 fp=0xc000195388 sp=0xc0001952c8 pc=0x95a3c5
gocloud.dev/docstore/memdocstore.(*decoder).DecodeMap(0xc0006c0ea0, 0xc0006bd400)
        <autogenerated>:1 +0x4f fp=0xc0001953b0 sp=0xc000195388 pc=0x960b6f
gocloud.dev/docstore/driver.decodeStruct(0x11d6e40, 0xc0006c2750, 0x199, 0x147ba38, 0xc0006c0ea0, 0xc000195520, 0xc0006c0ea0)
        /home/user/go/pkg/mod/gocloud.dev@v0.22.0/docstore/driver/codec.go:678 +0x17d fp=0xc000195420 sp=0xc0001953b0 pc=0x913a9d
gocloud.dev/docstore/driver.Document.Decode(0x1079160, 0xc0006c2750, 0x0, 0x11d6e40, 0xc0006c2750, 0x199, 0xc0005abc00, 0xc, 0x10, 0x147ba38, ...)
        /home/user/go/pkg/mod/gocloud.dev@v0.22.0/docstore/driver/document.go:184 +0x145 fp=0xc000195478 sp=0xc000195420 pc=0x917285
gocloud.dev/docstore/memdocstore.decodeDoc(0xc0003d8870, 0x1079160, 0xc0006c2750, 0x0, 0x11d6e40, 0xc0006c2750, 0x199, 0xc0005abc00, 0xc, 0x10, ...)
        /home/user/go/pkg/mod/gocloud.dev@v0.22.0/docstore/memdocstore/codec.go:119 +0x1ca fp=0xc000195540 sp=0xc000195478 pc=0x959e0a
gocloud.dev/docstore/memdocstore.(*docIterator).Next(0xc0006397c0, 0x1470510, 0xc0000b4010, 0x1079160, 0xc0006c2750, 0x0, 0x11d6e40, 0xc0006c2750, 0x199, 0xc0005abc00, ...)
        /home/user/go/pkg/mod/gocloud.dev@v0.22.0/docstore/memdocstore/query.go:153 +0xf0 fp=0xc0001955c8 sp=0xc000195540 pc=0x95ee10
gocloud.dev/docstore.(*DocumentIterator).Next(0xc0006a7530, 0x1470510, 0xc0000b4010, 0x1079160, 0xc0006c2750, 0x0, 0x146e6a0)
        /home/user/go/pkg/mod/gocloud.dev@v0.22.0/docstore/query.go:234 +0x287 fp=0xc0001956d8 sp=0xc0001955c8 pc=0x9363e7
jobs.allJobs(0x1470510, 0xc0000b4010, 0xc0006a7530, 0x1, 0x0, 0x0, 0x0, 0xc0006a7530)
        /home/user/go/pkg/mod/internallib/jobs/job_storage.go:147 +0x97 fp=0xc000195758 sp=0xc0001956d8 pc=0xc43f17

To Reproduce

I was not able to create a minimal example. The calling code is pretty simple:

func allJobs(ctx context.Context, iter *docstore.DocumentIterator) (result []*Job, err error) {
	for {
		var job Job
		err := iter.Next(ctx, &job)
		if err != nil {
			if err == io.EOF {
				return result, nil
			}
			return nil, err
		}
		result = append(result, &job)
	}
}

I suspect that the culprit is the usage of nested structs.

type Job struct {
	Specific interface{}
}
// JobInfo is used as value for Job.Specific
// but will be unmarshalled as map[string]interface{}
type JobInfo struct {
	StartDate   time.Time
	EndDate     time.Time
	Auth        string
}

Expected behavior

No error when reading all document from the memdocstore

Version

go version go1.16.2 linux/amd64
gocloud.dev@v0.22.0

Additional context

Thanks again for your work!