Skip to content

πŸ”ͺ chunk/split a string by length without cutting/truncating words.

Notifications You must be signed in to change notification settings

algolia/chunk-text

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 

History

43 Commits
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

Chunk Text

chunk/split a string by length without cutting/truncating words.

const out = chunk('hello world how are you?', 7);
/* ['hello', 'world', 'how are', 'you?'] */

Installation

$ npm install chunk-text
# yarn add chunk-text

Usage

All number values are parsed according to Number.parseInt.

const chunk = require('chunk-text');

chunk(text, chunkSize);

Chunks the text string into an array of strings that each have a maximum length of chunkSize.

const out = chunk('hello world how are you?', 7);
/* ['hello', 'world', 'how are', 'you?'] */

If no space is detected before chunkSize is reached, then it will truncate the word to always ensure the resulting text chunks have at maximum a length of chunkSize.

const out = chunk('hello world', 4);
/* ['hell', 'o', 'worl', 'd'] */

chunk(text, chunkSize, chunkOptions);

Chunks the text string into an array of strings that each have a maximum length of chunkSize, as determined by chunkOptions.charLengthMask.

The default behavior if chunkOptions.charLengthMask is excluded is equal to chunkOptions.charLengthMask=-1.

For single-byte characters, chunkOptions.charLengthMask never changes the results.

For multi-byte characters, chunkOptions.charLengthMask allows awareness of multi-byte glyphs according to the following table:

chunkOptions.charLengthMask result
-1 - same as default, same as chunkOptions.charLengthMask=1
- each character counts as 1 towards length
0 - each character counts as the number of bytes it contains
>0 - each character counts as the number of bytes it contains, up to a limit of chunkOptions.charLengthMask=N
- a 7-byte ZWJ emoji such as runningPerson+ZWJ+femaleSymbol (πŸƒπŸ½β€β™€οΈ) counts as 2, when chunkOptions.charLengthMask=2

You can also substitute from the default chunkOptions.charLengthType property of length to TextEncoder.

This enables you to pass any object to chunkOptions.textEncoder which matches the signature, chunkOptions.textEncoder.encode(text).length

If your environment natively contains the TextEncoder prototype and chunkOptions.textEncoder isn't provided,

the module attempts new TextEncoder() in order to use this chunkOptions.charLengthType.

If

  • chunkOptions.charLengthType is set to TextEncoder.
  • chunkOptions.textEncoder isn't provided.
  • TextEncoder prototype isn't provided by the environment.

Then

  • ReferenceError will occur.

End If

// one woman runner emoji with a colour is seven bytes, or five characters
// RUNNER(2) + COLOUR(2) + ZJW + GENDER + VS15
// (actually encodes to 17)
const runner = 'πŸƒπŸ½β€β™€οΈ';

const outDefault = chunk(runner+runner+runner, 4);
/* [ 'πŸƒπŸ½β€β™€οΈπŸƒπŸ½β€β™€οΈπŸƒπŸ½β€β™€οΈ' ] */

const outZero = chunk(runner+runner+runner, 4, { charLengthMask: 0 });
/* [ 'πŸƒπŸ½β€β™€οΈ', 'πŸƒπŸ½β€β™€οΈ', 'πŸƒπŸ½β€β™€οΈ' ] */

const outTwo = chunk(runner+runner+runner, 4, { charLengthMask: 2 });
/* [ 'πŸƒπŸ½β€β™€οΈπŸƒπŸ½β€β™€οΈ', 'πŸƒπŸ½β€β™€οΈ' ] */

// FLAG + RAINBOW
// 2 each as length, 4 each as TextEncoder
// 4 as length, 8 as TextEncoder
// Node v14.5.0 does not provide TextEncoder natively.
const flags = 'πŸ³οΈβ€πŸŒˆπŸ³οΈβ€πŸŒˆ';

// \/ will fail if your environment doesn't already have TextEncoder prototype \/
chunk(flags, 8, { charLengthMask: 0, charLengthType: 'TextEncoder' });
// [ 'πŸ³οΈβ€πŸŒˆ', 'πŸ³οΈβ€πŸŒˆ' ]
// /\ will fail if your environment doesn't already have TextEncoder prototype /\

chunk(flags, 4, {
  charLengthMask: 0,
  charLengthType: 'TextEncoder',
  textEncoder: new TextEncoder(),
})
// [ 'πŸ³οΈβ€πŸŒˆ', 'πŸ³οΈβ€πŸŒˆ' ]

chunk(flags, 999, {
  charLengthMask: 0,
  charLengthType: 'TextEncoder',
  textEncoder: {
    encode: () => ({ length: 999 }),
  },
})
// [ 'πŸ³οΈβ€πŸŒˆ', 'πŸ³οΈβ€πŸŒˆ' ]

Usage in Algolia context

This library was created by Algolia to ease the optimizing of record payload sizes resulting in faster search responses from the API.

In general, there is always a unique large "content attribute" per record, and this packages will allow to chunk that content into small chunks of text.

The text chunks can then be distributed over multiple records.

Here is an example of how to split an existing record into several ones:

var chunk = require('chunk-text');
var record = {
  post_id: 100,
  content: 'A large chunk of text here'
};

var chunks = chunk(record.content, 600); // Limit the chunk size to a length of 600.
var records = [];
chunks.forEach(function(content) {
  records.push(Object.assign({}, record, {content: content}));
});