Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorrect parse with trim: true and utf16le encoding #349

Closed
farisn opened this issue Jun 28, 2022 · 1 comment
Closed

Incorrect parse with trim: true and utf16le encoding #349

farisn opened this issue Jun 28, 2022 · 1 comment

Comments

@farisn
Copy link

farisn commented Jun 28, 2022

Describe the bug

When using the trim: true option with utf16le encoding, the content is modified.

csv@5.5.3 contain csv-parse@4.16.3
Also reproduced for csv@6.1.3, csv-parse@5.2.0.

To Reproduce
In csv-parse@5.2.0:

const assert = require('assert')

const {parse: csvParse} = require('csv-parse/sync')

const data = 'ф, ы'
const encoding = 'utf16le'
const delimiter = ','
const record_delimiter = '\n'

const records = csvParse(Buffer.from(data, encoding), {
  //delimiter: Buffer.from(delimiter, encoding),
  //record_delimiter: Buffer.from(record_delimiter, encoding)
  encoding,
  trim: true
})

const correctRecords = data
  .split(record_delimiter)
  .map(
    record => record
      .split(delimiter)
      .map(str => str.trim())
  )
assert.deepStrictEqual(records, correctRecords)

Additional context

I need to parse utf16le encoded csv file with custom delimiter and record_delimiter. Please release bugfix also for csv-parse@4.16.3.

@wdavidw
Copy link
Member

wdavidw commented Jun 29, 2022

  • csv-parse: 5.2.0 => 5.2.1
  • csv: 6.1.3 => 6.1.4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants