[Error] unexpected EOF on client connection with an open transaction

[rllola]

I am trying to COPY IN a huge amount of data but I keep getting this error. This is happening once I call finish() after a write_all.

I can reproduce this systematically. I have no idea if the invalid message length is being caused by the client and then it crash.

Error message

2023-01-16 07:43:57.115 UTC [71] LOG:  invalid message length
2023-01-16 07:43:57.115 UTC [71] CONTEXT:  COPY transactions, line 1
2023-01-16 07:43:57.115 UTC [71] STATEMENT:  COPY dogecoin_mainnet.transactions FROM stdin (DELIMITER ',')
2023-01-16 07:43:57.116 UTC [71] ERROR:  unexpected EOF on client connection with an open transaction
2023-01-16 07:43:57.116 UTC [71] CONTEXT:  COPY transactions, line 1
2023-01-16 07:43:57.116 UTC [71] STATEMENT:  COPY dogecoin_mainnet.transactions FROM stdin (DELIMITER ',')
2023-01-16 07:43:57.117 UTC [71] FATAL:  terminating connection because protocol synchronization was lost

[sfackler]

Can you provide a self contained example that reproduces this?

[rllola]

I am not sure...

I see two options, 1) I give you access to the private repo (the project is going to be public so I don't care) or 2) I can try to investigate this week end and let you know on this thread what I have found.

Also please note that I am running on a 4GB of RAM and 2Cores server (the cheapest kimsufi server) but I don't think it is link.

[rllola]

I believe I might have found the issue.

I have been tracking the CopyData message length I am sending to postgres.

write CopyData message
[/usr/local/cargo/git/checkouts/rust-postgres-62932403471beb1c/f513635/postgres-protocol/src/message/frontend.rs:160] self.len = 1365071846

The value is 1.27132 GB which according to postgres is bigger than max authorize.

So we are hitting this message here:
https://github.com/postgres/postgres/blob/master/src/backend/libpq/pqcomm.c#L1220-L1225

And it returns EOF.

The maxLen value is define here:
https://github.com/postgres/postgres/blob/master/src/include/libpq/libpq.h#L31
And MaxAllocSize is here and it is 1GB
https://github.com/postgres/postgres/blob/master/src/common/stringinfo.c#L28

Solution
We could add this check inside the lib to be sure we are not going over this limit and return a more informative error message.

[sfackler]

I don't think I want to hardcode an internal constant into this library. It could be reasonable to mention this constraint in the copy in documentation.

[rllola]

I understand that you don't want to hardcode value. I am however in favor of having it because it will make it fail early and help return a proper error message regarding the length of the buffer. Having it mentioned in the documentation won't be enough because of the confusing error message about unexpected EOF in my opinion.

[sfackler]

The entire purpose of copy queries are to allow very large data imports to be handled in a streaming fashion to avoid buffering the entire thing in memory. As a result, I don't expect many people to be running into this limitation (which is probably why the Postgres backend doesn't try to send a nice error message).

[rllola]

One year later I have faced this issue again. After some internet research, I have found this old issue I opened. I am again in favor of a hardcoded limit as it is in the postgres code to avoid the confusing error message.

However adding it to the documentation it is a start and should be enough to close this issue.