Tiktoken api

The tiktoken API refers to the programmatic methods exposed by OpenAI"s open-source BPE tokenization library (`tiktoken`) written in Rust with Python bindings. Developers call functions like `enc.encode("text")` to rapidly convert natural text strings into mathematical token ID arrays.