| external help file | Module Name | online version | schema |
|---|---|---|---|
PSOpenAI-help.xml |
PSOpenAI |
2.0.0 |
BPE tokeniser for use with OpenAI's models.
ConvertTo-Token
[-Text] <String>
[[-Encoding] <String>]
[<CommonParameters>]
ConvertTo-Token
[-Text] <String>
[-Model] <String>
[<CommonParameters>]
Encode text to tokens for use with OpenAI's models. (tokenize)
The output values are compatible with OpenAI tiktoken.
$Text = Hello, world!
ConvertTo-Token -Text $Text -Model 'gpt-4'
# Output: (9906, 11, 1917, 0)'🍈🍒🍑' | ConvertTo-Token -Encoding 'o200k_base'
# Output: (102415, 230, 102415, 240, 102415, 239)Specifies texts to be encoded.
Type: String
Parameter Sets: (All)
Required: True
Position: 0
Accept pipeline input: True (ByValue)Specifies the encoding name. Currently cl100k_base and o200k_base are supported.
It cannot be specified with the model name.
Type: String
Parameter Sets: encoding
Accepted values: cl100k_base, o200k_base
Required: False
Position: 1
Default value: cl100k_baseSpecifies the model name. such like gpt-4 or text-embedding-3-small.
It cannot be specified with the encoding name.
Type: String
Parameter Sets: model
Required: True
Position: 1
Default value: NoneThis cmdlet supports the common parameters: -Debug, -ErrorAction, -ErrorVariable, -InformationAction, -InformationVariable, -OutVariable, -OutBuffer, -PipelineVariable, -Verbose, -WarningAction, and -WarningVariable. For more information, see about_CommonParameters.