Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

StreamEntry (XREAD) does not support binary because it uses Map<String, String> #3566

Open
raylau1234 opened this issue Oct 3, 2023 · 4 comments

Comments

@raylau1234
Copy link

raylau1234 commented Oct 3, 2023

StreamEntry uses a Map<String, String> for fields.

As a result, binary data which contains illegal utf-8 sequences get mangled.

Reproduce by using XADD with a key or value which contains an illegal utf-8 sequence like 0xc3 0x28.

Retrieve via XREAD.

Expected result -- API to get byte array instead of a String, which is lossy with illegal utf-8 sequences.

@sazzad16
Copy link
Collaborator

sazzad16 commented Oct 3, 2023

First of all, what is XGET?

Secondly, there are xrange(byte[] key, ...) methods. Don't these help?

@raylau1234
Copy link
Author

Sorry, I meant XREAD.

You are correct on the alternate XRANGE signature. But I am really using XREAD. Let me update the title and to clarify.... XREAD only is problematic.

@raylau1234 raylau1234 changed the title StreamEntry (XRANGE / XGET) does not support binary because it uses Map<String, String> StreamEntry (XREAD) does not support binary because it uses Map<String, String> Oct 3, 2023
@raylau1234
Copy link
Author

I see there are alternate signatures for xread as well. Let me see of those help. I found the StreamEntry ones first. Give me a few minutes to explore and will udpate.

@raylau1234
Copy link
Author

OK, List<byte[]> xread(XReadParams, Entry<byte[], byte[]>... streams) is a possible workaround, but is cumbersome bec I need to deserialize the byte into the map of key / message id / values. i.e. I need to replicate everything that List<Map, Entry<String, List<StreamEntry>>> xread(XReadParams, Map<String, StreamEntryID>) is doing in terms of decoding with STREAM_READ_RESPONSE, STREAM_ENTRY_LIST, etc. (but skipping the SafeEncoder.encode() and keeping the binary there).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants