mehrab-wj/tiktoken-php

a clone of python tiktoken but for PHP! fast BPE tokeniser for use with OpenAI's models.

Installs: 44 934

Dependents: 3

Suggesters: 0

Security: 0

Stars: 17

Watchers: 0

Forks: 4

Open Issues: 0

pkg:composer/mehrab-wj/tiktoken-php

v1.0.0 2023-04-19 18:36 UTC

This package is auto-updated.

Last update: 2025-09-11 22:41:48 UTC


README

PHP Text Tokenizer for GPT models

About

A PHP toolkit to tokenize text like GPT family of models process it.

Forked from semji/gpt3-tokenizer-php to bug fixes and improvement.

Requirements

Usage

First install the package using composer:

composer require mehrab-wj/tiktoken-php
use TikToken\Encoder;
$prompt = "Ai is cool";
$encoder = new Encoder();

$tokens = $encoder->encode($prompt); // [32, 72, 318, 3608]

// Get tokens count:
echo count($tokens); // 4