yurunsoft/tiktoken

PHP version of tiktoken

v0.1.4 2023-09-19 06:32 UTC

This package is auto-updated.

Last update: 2024-10-19 08:48:21 UTC


README

Packagist Version Build status License

This is a port of the tiktoken.

Installation

$ composer require yethee/tiktoken

Usage

use Yethee\Tiktoken\EncoderProvider;

$provider = new EncoderProvider();

$encoder = $provider->getForModel('gpt-3.5-turbo-0301');
$tokens = $encoder->encode('Hello world!');
print_r($tokens);
// OUT: [9906, 1917, 0]

$encoder = $provider->get('p50k_base');
$tokens = $encoder->encode('Hello world!');
print_r($tokens);
// OUT: [15496, 995, 0]

Limitations

  • Encoding for GPT-2 is not supported.
  • Special tokens (like <|endofprompt|>) are not supported.

License

MIT