Feature: .slashes() tokenize transform

[retorquere]

I'm tokenizing using compromise/one. Can I have 'IEEE/WIC/ACM' be recognized as 3 slash-separated words rather than one?

[spencermountain]

hey Emiliano, good idea.
I've changed the tokenizer to allow for 3 slashes per word.

nlp(`IEEE/WIC/ACM`).match('wic').found //true

note that the words are not actually split. we have an awkward, but safe interpretation of slashes, so they don't get bunged-up by other transformations.

cheers

[spencermountain]

released in 14.13.0, thanks

[retorquere]

on 14.13 I still see

const nlp = require('compromise/one')
const doc = nlp('IEEE/WIC/ACM')

for (const sentence of doc.json({offset:true})) {
  for (const term of sentence.terms) console.log(term)
}

showing one token with the combined words. How do I extract the separate words in 14.13?

[spencermountain]

Hey, sorry for delay - yes, this is not possible now, but is a good idea.
The thinking was that slashed words should be one term, for most purposes, but should be able to be accessed individually with matches and things. You can see the slashed words are tokenized in the .json() response in an 'alias' property.

It would be cool (and possible) to add a .slashes().split() method. I can try to add it in an upcoming release
Cheers

[retorquere]

I find them there, but they've been lowercased. I use the tokenizer for a sentence-casing algorithm so I need case intact.

[retorquere]

Would the split method recreate location info? And this slashes.split would be something I would run on individual terms?