xp-framework/tokenize

Text tokenizing

v9.1.0 2024-03-24 13:28 UTC

Requires

• php: >=7.0.0
• xp-framework/core: ^12.0 | ^11.0 | ^10.0 | ^9.0 | ^8.0 | ^7.0

Requires (Dev)

• xp-framework/test: ^2.0 | ^1.0

BSD-3-Clause

• module
• xp

README

Tokenizing text

use text\{StringTokenizer, StreamTokenizer};
use io\File;

// Supports strings and streams
$tokens= new StringTokenizer('He asked: Can you parse this?', ' .?!,;:', true);
$tokens= new StreamTokenizer((new File('parse-me.txt'))->in(), ' .?!,;:', true);

// Can iterate using foreach...
foreach ($tokens as $token) {
  Console::writeLine($token);
}

// ...or with an iterator API
while ($tokens->hasMoreTokens()) {
  Console::writeLine($tokens->nextToken());
}

// Returns: ["He", " ", "asked", ":", " ", "Can", " ", "you", " ", "parse", " ", "this", "?"]