tiktoken
Here are 44 public repositories matching this topic...
Specify a github or local repo, github pull request, arXiv or Sci-Hub paper, Youtube transcript or documentation URL on the web and scrape into a text file and clipboard for easier LLM ingestion
-
Updated
Jun 6, 2024 - Python
forked from @anse-app/chatgpt-demo. Index site at https://free-chat.asia
-
Updated
May 16, 2024 - TypeScript
Analysis of OpenAI's ChatGPT
-
Updated
Jan 28, 2024 - Jupyter Notebook
The ChatGPT Long Term Memory package is a powerful tool designed to empower your projects with the ability to handle a large number of simultaneous users and external sources.
-
Updated
Jul 31, 2023 - Python
This project implements token calculation for OpenAI's gpt-4 and gpt-3.5-turbo model, specifically using `cl100k_base` encoding.
-
Updated
Jun 3, 2024 - C#
Predict the exact openai token usage of functions
-
Updated
Sep 20, 2023 - Python
📐 GPT token estimation and context size utilities without a full tokenizer
-
Updated
May 14, 2024 - TypeScript
A custom GPT based on [Zero To Hero](https://karpathy.ai/zero-to-hero.html) utilizing tiktoken with the intent to augment AI Transformer-model education and reverse engineer GPT models from scratch.
-
Updated
Jun 19, 2023 - Python
Count tokens in a text file.
-
Updated
Sep 26, 2023 - Python
Improve this page
Add a description, image, and links to the tiktoken topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the tiktoken topic, visit your repo's landing page and select "manage topics."