README

Folds text to a comparable form for search and deduplication: lowercase, diacritics removed, punctuation collapsed — while keeping the text in its own script.

$normalizer->normalize('Crème Brûlée');   // "creme brulee"
$normalizer->normalize('مَدْرَسَة');          // "مدرسه"  — still Arabic

Is this what you need?

Most PHP libraries in this space produce an ASCII slug for URLs. This one produces a comparison key in the original script. Pick accordingly:

You want	Use
A URL slug (`crème` → `creme`)	`cocur/slugify` or `symfony/string`
Transliteration into Latin (`مدرسة` → `madrasa`)	`ext-intl` `Transliterator`
A comparison key that stays Arabic (`مَدْرَسَة` → `مدرسه`)	this package

The distinction matters for search. Transliterating Arabic to Latin collapses unrelated roots onto the same consonant skeleton and produces a key you cannot display, highlight, or feed back into an Arabic index.

Installation

composer require cleatsquad/php-text-normalizer

PHP 8.2+. Uses ext-intl when present, and falls back to symfony/polyfill-intl-normalizer otherwise.

Usage

use CleatSquad\TextNormalizer\TextNormalizer;

$normalizer = new TextNormalizer();

$normalizer->normalize('Quelle est la MÉTÉO à Rabat ?');
// "quelle est la meteo a rabat"

$normalizer->tokenize('token-2024');
// ['token', '2024']

Analyzing text with metadata

$result = $normalizer->analyze('  Météo   à   RABAT !!! ');

$result->normalized;    // "meteo a rabat"
$result->original;      // "  Météo   à   RABAT !!! "
$result->wasModified(); // true
$result->length();      // 13
$result->profileName;   // "arabic_search_latin"

Script profiles

use CleatSquad\TextNormalizer\NormalizerProfile;

new TextNormalizer(NormalizerProfile::latin());                    // Latin only
new TextNormalizer(NormalizerProfile::arabic());                   // Arabic search mode (ة -> ه)
new TextNormalizer(NormalizerProfile::arabic(searchEquivalences: false)); // Arabic strict mode (preserves ة)
new TextNormalizer(NormalizerProfile::cyrillic());                 // Cyrillic (ё -> е, і/ї -> i)
new TextNormalizer(NormalizerProfile::greek());                    // Greek (ς -> σ, tonos stripped)
new TextNormalizer(NormalizerProfile::all());                      // default (all scripts)
new TextNormalizer(new NormalizerProfile());                       // punctuation only, folds nothing

Profiles are scoped: a Latin profile leaves Arabic harakat exactly where they are. Compose your own, or extend a shipped one:

$profile = NormalizerProfile::latin()->merge(
    new NormalizerProfile(characterMap: ['ĳ' => 'ij'])
);

What it folds

Diacritics — by Unicode canonical decomposition, not a table. Košice, Ṣāliḥ, Đà Nẵng, Ĝangalo all fold correctly, in every script, because NFD reaches every decomposable letter. A hand-written table only ever covers the ones someone remembered.

Letters that carry no mark — by table, since decomposition cannot reach them: æ œ ß ø ł đ ð þ ħ ı ŋ ŧ ƶ.

Arabic orthographic equivalences — by table, because Unicode considers them distinct letters and no normalization form unifies them:

Fold	Why
`أ إ آ ٱ` → `ا`	Alif variants
`ة` → `ه`	Ta Marbuta, as search indexes conventionally do
`ى ی ې ۍ` → `ي`	Alef Maksura, and Persian/Urdu/Pashto Yeh
`ک ګ` → `ك`	Keheh (Persian/Urdu Kaf)
`ہ ھ` → `ه`	Heh Goal, Heh Doachashmee
`٠-٩` and `۰-۹` → `0-9`	Arabic-Indic and Extended Arabic-Indic digits
tatweel `ـ` removed	decorative elongation, never lexical
ZWNJ/ZWJ removed	invisible, and Persian puts them inside words

Without these, علي typed on an Arabic keyboard and علی typed on a Persian one are two different strings, and your index answers nothing.

Design notes

Idempotent. Normalizing an already-normalized string returns it unchanged.

Output stays in NFC whenever valid UTF-8 Unicode normalization succeeds, making it safe to store and compare byte-wise. Malformed UTF-8 inputs are returned untouched rather than converted to an empty string.

Combining marks a profile does not claim are preserved, attached to their letter rather than treated as word boundaries.

No clock, no I/O, no configuration files. One object, two methods.

Testing

composer install
composer test      # PHPUnit
composer analyse   # PHPStan, max level

License

MIT. See LICENSE.

cleatsquad / php-text-normalizer

Maintainers

Package info

Statistics

Security