You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Please provide an implementation of a Transformer that can parse large CSV files in a browser environment.
Motivation
I tried to implement such a CsvParseTransformer.
However, when dealing with large files that contain long data with line breaks in a single column and put pressure on the browser’s memory,
It was necessary to use internal APIs not exported by csv-parse, as shown below.
This raised concerns about future functionality.
And, I was unable to apply the AbortController to the CsvParseTransformer and could not implement the purging of its internal buffer.
also,I was unable to implement the csv-parse compliant procedure for interrupting the CsvTransformAPI as AbortController .
import { transform as CsvTransformAPI } from ‘…/…/node_modules/csv-parse/lib/api/index.js’
Alternative
Therefore, I would like you to provide a Transformer that is guaranteed to work continuously.
I think it would a good idea to pass a max_buffer_size configuration parameter to the parser constructor so that we can better control how much memory the parser is allowed to use.
I'm having a similar problem parsing csv files in a worker on edge where there are hard memory limits.
I'd like to add a +1 to this as we're encountering a similar issue. We're trying to parse large input files via the ESM streaming API without excessive memory pressure in the browser and hitting issues.
Summary
Please provide an implementation of a Transformer that can parse large CSV files in a browser environment.
Motivation
I tried to implement such a CsvParseTransformer.
However, when dealing with large files that contain long data with line breaks in a single column and put pressure on the browser’s memory,
It was necessary to use internal APIs not exported by csv-parse, as shown below.
This raised concerns about future functionality.
And, I was unable to apply the AbortController to the CsvParseTransformer and could not implement the purging of its internal buffer.
also,I was unable to implement the csv-parse compliant procedure for interrupting the CsvTransformAPI as AbortController .
Alternative
Therefore, I would like you to provide a Transformer that is guaranteed to work continuously.
** CsvParseTransformer Draft**
** exsample usage of CsvParseTransformer
It is convenient to be able to transfer data(from <input type=file>) to a restful API using pipelining.
The text was updated successfully, but these errors were encountered: