yurunsoft / tiktoken
PHP version of tiktoken
v0.1.4
2023-09-19 06:32 UTC
Requires
- php: ^8.1
- symfony/service-contracts: ^2.5 || ^3.0
Requires (Dev)
- doctrine/coding-standard: ^11.1
- phpunit/phpunit: ^10.0
- psalm/plugin-phpunit: ^0.18.3
- vimeo/psalm: 5.9.0
This package is auto-updated.
Last update: 2024-10-19 08:48:21 UTC
README
This is a port of the tiktoken.
Installation
$ composer require yethee/tiktoken
Usage
use Yethee\Tiktoken\EncoderProvider; $provider = new EncoderProvider(); $encoder = $provider->getForModel('gpt-3.5-turbo-0301'); $tokens = $encoder->encode('Hello world!'); print_r($tokens); // OUT: [9906, 1917, 0] $encoder = $provider->get('p50k_base'); $tokens = $encoder->encode('Hello world!'); print_r($tokens); // OUT: [15496, 995, 0]
Limitations
- Encoding for GPT-2 is not supported.
- Special tokens (like
<|endofprompt|>
) are not supported.