Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Errors and meta fields when streaming #754

Open
facundolinlaud opened this issue Dec 22, 2019 · 3 comments · May be fixed by #1015
Open

Errors and meta fields when streaming #754

facundolinlaud opened this issue Dec 22, 2019 · 3 comments · May be fixed by #1015

Comments

@facundolinlaud
Copy link

facundolinlaud commented Dec 22, 2019

Hello, how are you?

I've been using PapaParse in Nodejs to parse big remote files with streams. The library is great, thank you very much for your dedication.

One thing I'm trying to achieve is to be able to retrieve the errors field of an output in PapaParse's stream. When you process the data in one go (that is, when not using a stream), the output you get is of the following structure:

{
   data: [...],
   errors: [...],
   meta: {...}
}

However, when piping PapaParse's read stream to a writable stream, I don't receive an object with errors and meta fields, but a parsed row only. I would like to know if there is any way of receiving this whole object for each parsed row as I'm very interested in the TooFewFields and TooManyFields errors.

This is important for me because I would like to maintain the same error checking interface when parsing with and without PapaParse's stream, even if it means having an impact in the parsing performance.

Happy holidays!

@facundolinlaud
Copy link
Author

I just tried passing a callback to Papaparse's chunk field that does in fact receives an object of the form:

{
   data: [],
   errors: [],
   meta: {...}
}

However, the data and errors fields always arrive as empty arrays, even though the stream ends up parsing the csv file as expected. The same behavior can be seen at the Papaparse's demo, where – if you try to stream parse something – you will receive an empty array of data and errors no matter the input.

Is this an expected behavior?

Thank you for your time!

@pokoli
Copy link
Collaborator

pokoli commented Dec 23, 2019

Hi @facundolinlaud,

IIRC this is some feature that is not currently suported when using NodeStreams due to the async way of NodeStreams.

Any suggestion/improvement will be very welcome.

@80avin
Copy link

80avin commented Jun 24, 2023

@pokoli Can you describe how async way of NodeStreams affects it ?
and why it isn't as easy as changing data to results in this line ?

if (!stream.push(data) && !this._handle.paused()) {

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants