New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
virtualfile.read can lead to infinite loops #108
Comments
virtualDir.readdir is also wrong. If n is <= 0, os.Readdir returns all the remaining files and never returns an io.EOF. rice's virtualDir.readdir always returns all files (ignoring the offset, and even resetting the offset), and returns an io.EOF. |
Can you bit more specific how you triggered these conditions? I cannot reproduce any of them. :( |
The reader (in mattes/migrate) wasn't implemented in the most robust way, and if it receives some bytes, it tries to read again, it doesn't check for the io.EOF error. The correct behavior from rice would be to return zero bytes and an io.EOF again on the subsequent read. Instead it resets the offset to zero and starts returning the file from the start again, leading to an infinite loop. You can reproduce it by writing a for loop that reads until io.EOF, then doing another read. That last read should return 0 bytes and io.EOF. Instead it returns the start of the file again. I've fixed these bugs in my fork: nkovacs@341d7a8, nkovacs@96d5b2a |
I would be happy to merge those fixes upstream! |
@nkovacs Do you want create a PR for those? Or shall I just pull them in manually? |
This |
Heh, after digging in deep to debug an infinite loop in arduino-builder, I found the problem is in the rice read function, find this bug report and see that @cmaglie has already done (probably) exactly the same... But with a fix, just tested it and it works perfectly :-D |
I faced the exact issue. Because go-migrate is so complicated, I initially thought it was some edge case on their side. But after spending many hours reading the related logic, I managed to reduce the reproducible code to contain only I tried just using |
I'm using rice with https://github.com/mattes/migrate. It has a pretty complicated combination of buffered readers, pipes and bytes.Buffer, and I am getting an infinite loop, consuming all memory.
The problem is with the implementation of virtualfile.read. If the buffer is larger than the remaining contents of the file, you return the remaining bytes and an
io.EOF
. It looks like some reader wasn't prepared for that, and expects more content if it gets some bytes. Because you resetvf.offset
to zero, the next read will again read some bytes, and this ended up being an infinite loop that consumes all available memory: https://github.com/GeertJohan/go.rice/blob/master/virtual.go#L86While investigating this, I found a related problem. This will output the file contents twice:
test.txt is:
The text was updated successfully, but these errors were encountered: