How to log the input tokens count and output tokens count for each request to openai api call using open ai as llm component , in a single operation ?

[akar5h]

I am currently using ner v3 pipeline to create custom ner tags , I have used Chain of Thoughts with few examples for few shot learning, I want to know the count of i/o tokens that each operation does while making api call to OpenAI ? For some cost analysis and optimisations .

[rmitsch]

Hi @akar5h, spacy-llm doesn't offer this natively, as counting the tokens properly depends on the mechanics of the underlying model. There are other libraries for that, e. g. https://github.com/AgentOps-AI/tokencost. We might consider integrating a token counting library in the future so that we can log this information.

[rmitsch]

I. e. we recommend that you log your raw prompt output (with save_io = True) and use another library like tokencost to calculate the token count for your chosen LLM provider/model.