New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: use bl for a faster delimiter parser #1693
Conversation
Codecov Report
@@ Coverage Diff @@
## master #1693 +/- ##
==========================================
+ Coverage 64% 64.04% +0.04%
==========================================
Files 22 22
Lines 839 840 +1
==========================================
+ Hits 537 538 +1
Misses 302 302
|
Any metrics for speed improvements? Is it possible to add some information in the description of this PR as to why |
It copies buffers less, I think it might be slower for large delimiters but for the common case of a single byte it's doing much less work. I'm really not sure how to test it. Probably need 2-3 benchmarks for different kinds of data. |
Cool, just want to add some context to the PR for people curious about the addition of this dependency. 😄 |
I'm actually kind of worried it will be slower for some workloads. Do you have any favorite tools for testing performance? |
This was my approach, the new bl one was not faster (even though my hunch is it uses less memory), so I'm working on making const Benchmark = require('benchmark')
const DelimiterParser = require('./')
const smallBuffers = new Array(1000).map(() => Buffer.concat([Buffer.alloc(14, 1), Buffer.from('\n')]))
const largeBuffers = new Array(1000).map(() => Buffer.concat([Buffer.alloc(1024, 1), Buffer.from('\n')]))
const mixedBuffers = new Array(1000).map((val, i) => Buffer.concat([Buffer.alloc((i * i * (Math.random() + 1)) % 1024, 1), Buffer.from('\n')]))
const suite = new Benchmark.Suite()
suite.add('smallBuffers', () => {
let count = 0
const parser = new DelimiterParser({ delimiter: '\n' })
parser.on('data', () => count++)
smallBuffers.map(data => parser.write(data))
return count
})
suite.add('largeBuffers', () => {
let count = 0
const parser = new DelimiterParser({ delimiter: '\n' })
parser.on('data', () => count++)
largeBuffers.map(data => parser.write(data))
return count
})
suite.add('mixedBuffers', () => {
let count = 0
const parser = new DelimiterParser({ delimiter: '\n' })
parser.on('data', () => count++)
mixedBuffers.map(data => parser.write(data))
return count
})
suite
.on('cycle', event => {
console.log(String(event.target))
})
.on('complete', () => {
console.log('done')
})
.run({ async: true }) |
Even with the bl perf improvements unless we have huge data events this isn't any faster. |
|
No description provided.