yethee / tiktoken
PHP version of tiktoken
Installs: 1 154 936
Dependents: 9
Suggesters: 0
Security: 0
Stars: 122
Watchers: 3
Forks: 27
Open Issues: 5
Requires
- php: ^8.1
- symfony/service-contracts: ^2.5 || ^3.0
Requires (Dev)
- doctrine/coding-standard: ^13
- mikey179/vfsstream: ^1.6.11
- phpbench/phpbench: ^1.2
- phpunit/phpunit: ^10.5.45
- psalm/plugin-phpunit: ^0.19.2
- vimeo/psalm: 6.8.9
Suggests
- ext-ffi: To allow use LibEncoder
README
This is a port of the tiktoken.
Installation
$ composer require yethee/tiktoken
Usage
use Yethee\Tiktoken\EncoderProvider; $provider = new EncoderProvider(); $encoder = $provider->getForModel('gpt-3.5-turbo-0301'); $tokens = $encoder->encode('Hello world!'); print_r($tokens); // OUT: [9906, 1917, 0] $encoder = $provider->get('p50k_base'); $tokens = $encoder->encode('Hello world!'); print_r($tokens); // OUT: [15496, 995, 0]
Cache
The encoder uses an external vocabularies, so caching is used by default to avoid performance issues.
By default, the directory for temporary files is used.
You can override the directory for cache via environment variable TIKTOKEN_CACHE_DIR
or use EncoderProvider::setVocabCache()
:
use Yethee\Tiktoken\EncoderProvider; $encProvider = new EncoderProvider(); $encProvider->setVocabCache('/path/to/cache'); // Using the provider
Lib mode
Experimental
You can use tiktoken-rs library via FFI binding. This can improve performance when need to encode medium or large texts. However, the overhead of data marshalling can lead to poor performance for small texts.
use Yethee\Tiktoken\Encoder\LibEncoder; use Yethee\Tiktoken\EncoderProvider; // LibEncoder::init('/path/to/lib'); $encProvider = new EncoderProvider(true); // Force using the lib encoder
You need to provide path to the lib before using the provider. There are several ways to do this:
- Use
Yethee\Tiktoken\Encoder\LibEncoder::init()
method. - Use
Yethee\Tiktoken\Encoder\LibEncoder::preload()
method, inside opcache preload script. - Use environment variable
TIKTOKEN_LIB_PATH
orLD_LIBRARY_PATH
Build lib
Requirements
- Rust >= 1.85
git clone git@github.com:yethee/tiktoken-php.git
cd tiktoken-php
cargo build --release
Copy binary from target/release
:
libtiktoken_php.so
for linuxlibtiktoken_php.dylib
for MacOStiktoken_php.dll
for Windows
NOTE: You can see .docker/Dockefile
for an example.
Benchmark
You can see benchmark result in #27 or run it locally:
composer bench
TODO
- Add implementation for
Yethee\Tiktoken\Encoder\LibEncoder::encodeInChunks()
method
Limitations
- Encoding for GPT-2 is not supported.
- Special tokens (like
<|endofprompt|>
) are not supported.