Skip to content

vikesh-raj/go-sentencepiece-encoder

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Sentence Piece Encoder

This is pure go implementation of the sentencepiece encoder. Create an encoder for the given sentencepiece model and then use use the Tokenize function to split the input text into tokens.

Example:

import "github.com/vikesh-raj/go-sentencepiece-encoder/sentencepiece"

text := "This is a sample text"
spm, _ := sentencepiece.NewSentencepieceFromFile("spm.model", false)
tokens := spm.Tokenize(text)

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Packages

No packages published