[bug] Write error not returned if reader already signals EOF #494

hmoragrega · 2022-02-17T19:21:32Z

During the last chunk of a sequential write, when the reader already returned io.EOF, if the writer fails, the error is lost.

This happens due to err being already io.EOF so the condition to overwrite the error with the write error is false

if err == nil { // write error lost if err == io.EOF
    err = err2
}

I've provided a new test, not sure if it could be tested as a test case on another one though.

puellanivis

For Testing is dark, and full of terrors.

There’s nothing wrong here, just small things.

puellanivis · 2022-02-18T18:45:25Z

client_integration_test.go

+type writerFunc func(b []byte) (int, error)
+
+func (f writerFunc) Write(b []byte) (int, error) {
+	return f(b)
+}


This is not an efficient way to build mocks in Go. We’re already writing a receiver method here for a type, so we don’t need to wrap a generic function like this.

type errWriter struct{ … } with one receiver method is the same amount of code, and fewer function calls.

puellanivis · 2022-02-18T18:45:47Z

client_integration_test.go

+	defer cmd.Wait()
+	defer sftp.Close()
+
+	sftp.disableConcurrentReads = true


You’ve copied code from another test that is irrelevant to the testing here. Please remove this condition as it is irrelevant noise.

puellanivis · 2022-02-18T18:46:19Z

client_integration_test.go

+	content := []byte("hello world")
+	f.Write(content)


content is not used after these two lines. Skip the temporary variable, and just pass it in directly as a function argument.

puellanivis · 2022-02-18T18:47:54Z

client_integration_test.go

+	defer sftpFile.Close()
+
+	want := errors.New("error writing")
+	n, got := io.Copy(writerFunc(func(b []byte) (int, error) {


We do not know for sure here that we’re calling into sftpFile.writeToSequential. Instead, we should just directly actually call the function itself rather than depend on io.Copy to call the expected function.

puellanivis · 2022-02-18T18:49:36Z

client_integration_test.go

+
+	require.Error(t, got)
+	assert.ErrorIs(t, want, got)
+	assert.Equal(t, int64(10), n)


This magic number is greater than it needs to be, and specified in two places.

We want to ensure that it returns whatever short write value we return, but it should be so clearly under the number of bytes expected to be copied that we can be sure at a glance that it has done a short write, and still returned the short write values. For a value of 10 here, I’m not sure if that’s a full write with error, or supposed to be a short write.

For this, an ideal value is going to be 2. No one is going to think a write of "hello world" is supposed to return 2 bytes written.

puellanivis · 2022-02-18T18:51:32Z

client_integration_test.go

+	require.NoError(t, err)
+	defer sftpFile.Close()
+
+	want := errors.New("error writing")


Test errors should be as clearly test errors as possible. Prefer errors.New("test error"). Who knows if somewhere in the code, there might be returned an "error writing" error. But nothing should ever return a "test error" error except in test code.

puellanivis · 2022-02-18T18:53:27Z

client_integration_test.go

+	defer sftp.Close()
+
+	sftp.disableConcurrentReads = true
+	d, err := ioutil.TempDir("", "sftptest-writesequential")


We want this directory name to be as precisely narrow as possible. If we make a TestClientWriteSequential and repeat this TempDir construction, now we have tests that step on each other’s toes.

Add -writeerr to the directory name.

puellanivis · 2022-02-18T18:59:06Z

client_integration_test.go

+	}), sftpFile)
+
+	require.Error(t, got)
+	assert.ErrorIs(t, want, got)


The appropriate scope of assertion here is that it is not io.EOF. We don’t need to test that it’s the exact same error that we returned, (which is why you’re capturing it with want := errors.New(…) because otherwise the error is an opaque pointer and errors.Is(errors.New("foo"), errors.New("foo")) == false)

We just need to know it’s not an io.EOF.

puellanivis · 2022-02-18T19:03:56Z

client.go

+			m, wErr := w.Write(b[:n])
 			written += int64(m)

-			if err == nil {
-				err = err2
+			if wErr != nil {
+				if err == nil || err == io.EOF {
+					err = wErr
+				}


If we want to prioritize returning an error from Write, then we don’t need any of this complicated logic anymore. We can just return err here directly.

m, err := w.Write(b[:n]) written += int64(m) if err != nil { return written, err }

I was in doubt about this one myself, I do like the early return more too, I'll change it.

puellanivis · 2022-02-18T19:06:22Z

client_integration_test.go

+
+	require.Error(t, got)
+	assert.ErrorIs(t, want, got)
+	assert.Equal(t, int64(10), n)


Testing just a single write here for written probably isn’t fully reliable. Consider, if the function were implemented as:

m, err := w.Write(b[:n]) if err != nil { return m, err } written += int64(w) `̀ ` We might need to use `timeBobWriter` instead, with a very small chunk size?

hmoragrega · 2022-02-21T09:53:46Z

I'll work on the test side, thanks!

puellanivis

More testing gotchas. Don’t worry, this sort of thing is expected when testing is given the same attention to detail as the code itself. (Intentionally looking for bugs in the tests, etc.)

puellanivis · 2022-02-21T14:46:11Z

client.go

@@ -1176,11 +1176,11 @@ func (f *File) writeToSequential(w io.Writer) (written int64, err error) {
 		if n > 0 {
 			f.offset += int64(n)

-			m, err2 := w.Write(b[:n])
+			m, wErr := w.Write(b[:n])


there’s no need to rename err error. Since we now have an early return, there is no need to access the two err variables at the same time, so shadowing is no longer a problem.

puellanivis · 2022-02-21T14:48:14Z

client_integration_test.go

+func (w *lastChunkErrSequentialWriter) Write(b []byte) (int, error) {
+	chunkSize := len(b)
+	w.written += chunkSize
+	if w.written == w.expected {


This test is too strict. If we shoot past w.expected then we will never enter this loop.

puellanivis · 2022-02-21T14:49:03Z

client_integration_test.go

+}
+
+func (w *lastChunkErrSequentialWriter) Write(b []byte) (int, error) {
+	chunkSize := len(b)


This alias ends up adding more characters of code than it saves. Sometimes, repetition isn’t a bad thing.

puellanivis · 2022-02-21T14:50:53Z

client_integration_test.go

+		content    = []byte("hello world")
+		shortWrite = 2
+	)
+	w := lastChunkErrSequentialWriter{


w is only ever access by pointer. So just directly declare it with a pointer: &lastChunkErrWriter (sequential is imprecisely inaccurate in this name; it need not only be used in this test.)

Unfortunately, this test still does not ensure that the same bug mentioned before could still slip through. We’re still only ever making one call to Write() as the first Write still errors out immediately. We have to set chunkSize. We need to set MaxPacketChecked(2) so that we’re writing in say 2-byte chunks. Then we want the first to succeed, then the second return early (1), so we end up with written == 3.

In the face of all of this, I think special casing all of the functionality of the writer is appropriate. First run returns len(b), nil the second returns 1, errors.New("test error"). Then we also test to ensure that the function was called only twice.

puellanivis · 2022-02-21T14:54:18Z

client_integration_test.go

+	fname := f.Name()
+	n, err := f.Write(content)
+	require.NoError(t, err)
+	require.Equal(t, n, len(content))


os.File.Write: … Write returns a non-nil error when n != len(b).

If we already passed the NoError above, then we can rely upon the contract, that n can be known to be len(content).

puellanivis · 2022-02-21T14:55:04Z

client_integration_test.go

+	f, err := ioutil.TempFile(d, "write-sequential-writeerr-test")
+	require.NoError(t, err)
+	fname := f.Name()
+	n, err := f.Write(content)


Speaking of which, let’s use WriteString so we can have a const for content. But also since as noted below, we don’t need to test for n != len(content) anyways, we don’t need to use content more than once either, so just use the string directly.

puellanivis · 2022-02-21T14:58:48Z

client_integration_test.go

+	require.NoError(t, err)
+	defer sftpFile.Close()
+
+	gotWritten, gotErr := sftpFile.writeToSequential(&w)


We know logically that we could not have reached this code unless err == nil. So, there’s no point in renaming err here, as the previous value is already irrelevant, so the name can be reused.

Likewise, there will now be no other gotXY, so, we can shorten that variable name to just that: got.

puellanivis · 2022-02-21T15:38:22Z

client_integration_test.go

+	require.NotErrorIs(t, io.EOF, gotErr)
+	require.Equal(t, int64(shortWrite), gotWritten)


The API is confusing on this matter:
func ErrorIs(t TestingT, err error, target error, msgAndArgs ...interface{}),
but:
func Equal(t TestingT, expected interface{}, actual interface{}, msgAndArgs ...interface{})

This is one of the central reasons why I tell people to avoid using require and assert. Getting the arguments in the wrong order is way too easy to do, and without checking the documentation, who knows which one is which. (Especially, since it’s require.ErrorIs(t, actual, expect) but require.Equals(t, expect, actual) )

This is especially true, when one is coming from a language that incentivizes writing equality tests as constValue == testValue so that mistakenly only using one = will not produce an unexpectedly valid assignment, but an invalid unassignable left-hand value. In Go, you cannot put an assignment in a condition, and so there is no way to accidentally use a single = assignment like this. This idiom is probably the very reason why require.Equal orders its parameters in that order… an order that is discouraged in Go.

Instead, something like this (without the comments which are here for study, rather than for good code practice):

if got != int64(shortWrite) { // In the `require.Equals`, the two arguments are cast into `interface{}`, // which means you lose the same compile-time type safety you would normally have. // It’s way better to use the code the exact same as your caller would use the code, // with verbose `if err != nil { }` et al. included. // Note the order here, `if got != expected { … }`, and the message orders `got` then `expected`. t.Errorf("sftpFile.Write() = %d, but expected %d", got, shortWrite) }

Also, we don’t want to require either of these tests anyways. We only want to assert it in this case. (Since both written and err have valid values, testing both of them every test makes sense. require is only appropriate when the test is a requirement for the remaining tests to have any meaningful result and/or not panic.

drakkan · 2022-03-03T11:23:47Z

@hmoragrega thanks for noticing this bug. I'm trying to fix the test cases as requested by @puellanivis and your commits will be merged as part of #499 once that test cases are ok

puellanivis · 2022-03-03T11:25:26Z

Thanks for spotting the issue and getting us started on a fix @hmoragrega

hmoragrega · 2022-03-03T11:27:30Z

No worries! I couldn't found time to apply the latest changes these week, glad you can take it from here

Return writer error if not nil in writeSequential

2d34a53

puellanivis reviewed Feb 18, 2022

View reviewed changes

Improved test with CR feedback

1c605bf

puellanivis reviewed Feb 21, 2022

View reviewed changes

drakkan mentioned this pull request Mar 3, 2022

writeToSequential: improve tests for write errors #499

Merged

drakkan closed this Mar 3, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[bug] Write error not returned if reader already signals EOF #494

[bug] Write error not returned if reader already signals EOF #494

hmoragrega commented Feb 17, 2022 •

edited

puellanivis left a comment

puellanivis Feb 18, 2022

puellanivis Feb 18, 2022

puellanivis Feb 18, 2022

puellanivis Feb 18, 2022

puellanivis Feb 18, 2022

puellanivis Feb 18, 2022

puellanivis Feb 18, 2022

puellanivis Feb 18, 2022

puellanivis Feb 18, 2022

hmoragrega Feb 21, 2022 •

edited

puellanivis Feb 18, 2022

hmoragrega commented Feb 21, 2022

puellanivis left a comment

puellanivis Feb 21, 2022

puellanivis Feb 21, 2022

puellanivis Feb 21, 2022

puellanivis Feb 21, 2022

puellanivis Feb 21, 2022

puellanivis Feb 21, 2022

puellanivis Feb 21, 2022

puellanivis Feb 21, 2022

drakkan commented Mar 3, 2022

puellanivis commented Mar 3, 2022

hmoragrega commented Mar 3, 2022

		require.NotErrorIs(t, io.EOF, gotErr)
		require.Equal(t, int64(shortWrite), gotWritten)

[bug] Write error not returned if reader already signals EOF #494

[bug] Write error not returned if reader already signals EOF #494

Conversation

hmoragrega commented Feb 17, 2022 • edited

puellanivis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hmoragrega Feb 21, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hmoragrega commented Feb 21, 2022

puellanivis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drakkan commented Mar 3, 2022

puellanivis commented Mar 3, 2022

hmoragrega commented Mar 3, 2022

hmoragrega commented Feb 17, 2022 •

edited

hmoragrega Feb 21, 2022 •

edited