illuma-law / laravel-semantic-deduper

v0.1.5 2026-10-23 14:04 UTC

Requires

• php: ^8.3
• illuminate/contracts: ^11.0|^12.0|^13.0
• illuminate/database: ^11.0|^12.0|^13.0
• illuminate/support: ^11.0|^12.0|^13.0
• spatie/laravel-package-tools: ^1.16

Requires (Dev)

• larastan/larastan: ^3.0
• laravel/pint: ^1.14
• mrpunyapal/peststan: ^0.2.12
• nunomaduro/collision: ^8.8
• orchestra/testbench: ^10.1
• pestphp/pest: ^4.0
• pestphp/pest-plugin-arch: ^4.0
• pestphp/pest-plugin-laravel: ^4.0
• phpstan/extension-installer: ^1.4
• phpstan/phpstan-deprecation-rules: ^2.0
• phpstan/phpstan-phpunit: ^2.0
• spatie/laravel-ray: ^1.35

MIT

• illuma-law

Published on Packagist.org on 2026-10-23 14:04 UTC

README

Group and deduplicate Laravel search or RAG results using cosine similarity of their embedding vectors. Also includes fuzzy string deduplication and text chunking utilities.

Requires PHP 8.3+ and Laravel 11, 12, or 13.

Install

composer require illuma-law/laravel-semantic-deduper
php artisan vendor:publish --tag=semantic-deduper-config

Cluster results

Pass items with an embedding vector and a group key. The clusterer removes near-duplicates, limits each group, and returns the retained items as DTOs.

use IllumaLaw\SemanticDeduper\SemanticClusterer;

$results = [
    ['id' => 1, 'source' => 'web', 'embedding' => [0.1, 0.2, 0.3]],
    ['id' => 2, 'source' => 'web', 'embedding' => [0.11, 0.19, 0.31]],
    ['id' => 3, 'source' => 'docs', 'embedding' => [0.8, 0.3, -0.2]],
];

$grouped = SemanticClusterer::make()
    ->groupBy('source')
    ->threshold(0.95)
    ->maxPerGroup(3)
    ->cluster($results);

$items = $grouped->allItems();
$ids = $grouped->collectIdentifiers();

Defaults for the threshold and result limits are in config/semantic-deduper.php. Use embeddingKey() or idKey() if your input uses different field names.

Utilities

Laravel collections include the dedupeFuzzy macro after package discovery:

$unique = collect(['Laravel', 'Laraval', 'Symfony'])
    ->dedupeFuzzy(fn (string $name) => $name, threshold: 85);

Split text into overlapping character-based chunks:

use IllumaLaw\SemanticDeduper\Utils\TextChunker;

$chunks = TextChunker::chunk($text, chunkSize: 1200, overlap: 150);

Test

composer test
composer analyse
vendor/bin/pint --test

MIT licensed. See LICENSE.