Search by

hamdallah90 / parafork

hamdallah90

Run one PHP function over a list of inputs in parallel, in worker processes forked from your script.

Package info

github.com/hamdallah90/parafork

Type:php-ext

Ext name:ext-parafork

pkg:composer/hamdallah90/parafork

Statistics

Installs: 0

Dependents: 0

Suggesters: 0

Stars: 0

Open Issues: 0

v1.0.0 2026-11-14 21:15 UTC

This package is auto-updated.

Last update: 2026-11-14 21:29:11 UTC


README

Parafork is a PHP extension that runs one function over a list of inputs in parallel, in worker processes forked from your script. A worker starts as a copy of the calling process, so it already has every function, class and constant the script has defined; nothing is loaded or bootstrapped again. Arguments are copied once into shared memory for the worker, and each result is checked and copied back into the calling process as ordinary PHP values. Values include arrays of scalars and, as data, objects of your own classes, enum cases and dates (see Objects). Closures, resources and connections never cross; a worker gets them by being a copy of the process.

Requirements

  • PHP: 8.3, 8.4, 8.5 or 8.6, non-thread-safe (NTS) builds only. A thread-safe (ZTS) build is refused when you run ./configure. 8.6 is tested on its release candidate until 8.6.0 is out. 8.7 is not supported.
  • Operating system: Linux x86-64 with glibc is the platform the extension is built and tested on. Linux aarch64 (glibc, Docker on Apple Silicon) and Linux x86-64 on musl (Alpine) run the whole PHPT suite on PHP 8.3 to 8.5 (musl also on 8.6) with no failure; neither has a benchmark. macOS (arm64 and x86-64) is for development only: every macOS PHP process runs extra system threads, so set parafork.allow_threads=1 there; there is no CPU placement and no parent-death signal, so a busy worker can outlive a killed parent until its task ends (see Settings). Windows, the BSDs and 32-bit builds are not supported (Windows has no fork()).
  • SAPI: the command-line SAPI (cli) and php-fpm (fpm-fcgi). Under any other SAPI (cli-server, apache2handler, cgi-fcgi, embed, phpdbg) Parafork\map() throws ParaforkException with reason unsupported_sapi. See php-fpm.
  • Loading: from php.ini (or -d extension=...) at startup. Loading Parafork with dl() ends the script with "Fatal error: Unable to start parafork module", because the extension must stay loaded for the life of the process. In php-fpm, loading it per pool (php_value[extension]) or through FastCGI is unsupported: it loads, then every call throws guard_not_installed.
  • opcache: nothing to change. opcache, JIT included, can stay on; workers run the code the calling process has already compiled. Two configurations are limited: with opcache.preload set and JIT on, or with opcache.enable_file_override=1, a request that runs with opcache switched off (for example because an opcache_reset() was pending when it started) cannot fan out and gets opcache_restart_pending. The same happens when opcache_get_status is in disable_functions or refused by opcache.restrict_api in those two configurations.

Install and build

With PIE

PIE builds and installs the extension from the package:

pie install hamdallah90/parafork

The package is for non-thread-safe PHP builds (support-zts is false) on Linux and macOS (macOS is for development only and needs parafork.allow_threads=1, see Requirements). Check that the extension is loaded as shown below.

From source

You need the PHP development files (phpize, php-config) and a C compiler.

phpize
./configure --enable-parafork
make
make test            # the test suite is written for Linux
sudo make install

Then add the extension to php.ini:

extension=parafork

Check that it is loaded:

php -r 'var_dump(extension_loaded("parafork"));'   # bool(true)
php --ri parafork                                   # "parafork support => enabled" and the settings

./configure --enable-parafork-testing builds a test version with fault-injection points; do not use it in production.

Usage

Parafork\map($fn, $argLists, $timeoutMs = null) calls $fn once for each element of $argLists, in workers, and returns the results under the same keys and in the same order. Each element of $argLists is the argument list of one call, as for call_user_func_array().

The examples below are complete scripts. On macOS, run them with php -d parafork.allow_threads=1.

A named function

<?php
function square(int $n): int
{
    usleep(100000); // stands for real work
    return $n * $n;
}

// four calls: square(1), square(2), square(3), square(4)
print_r(Parafork\map('square', [[1], [2], [3], [4]]));

Output:

Array
(
    [0] => 1
    [1] => 4
    [2] => 9
    [3] => 16
)

The four calls run at the same time, so the script takes about 0.1 s rather than 0.4 s when four CPUs are free.

Static methods, keys and named arguments

A public static method can be given in three forms. Keys of $argLists are kept in the result, and string keys inside an argument list are named arguments.

<?php
final class Invoice
{
    public static function total(array $lines, int $taxPercent = 0): int
    {
        $sum = array_sum(array_column($lines, 'amount'));
        return intdiv($sum * (100 + $taxPercent), 100);
    }
}

$jobs = [
    'north' => [[['amount' => 100], ['amount' => 50]]],
    'south' => ['lines' => [['amount' => 200]], 'taxPercent' => 10],
];

print_r(Parafork\map('Invoice::total', $jobs));
print_r(Parafork\map([Invoice::class, 'total'], $jobs));
print_r(Parafork\map(Invoice::total(...), $jobs));

Output:

Array
(
    [north] => 150
    [south] => 220
)
Array
(
    [north] => 150
    [south] => 220
)
Array
(
    [north] => 150
    [south] => 220
)

strlen(...) and other first-class callables of a function or a public static method are accepted too. Closures are not (see Limits).

Arrays as task arguments

An argument can be any mix of null, bool, int, float, string, arrays of them and objects (next section), nested up to 512 levels. To pass an array as the single argument of a call, wrap it in its argument list. A common pattern is to split a large input into one chunk per worker:

<?php
function score_chunk(array $rows): array
{
    $out = [];
    foreach ($rows as $row) {
        $out[$row['id']] = $row['qty'] * $row['price'];
    }
    return $out;
}

$rows = [];
for ($i = 1; $i <= 1000; $i++) {
    $rows[] = ['id' => "order-$i", 'qty' => $i % 7, 'price' => 3];
}

$chunks = array_chunk($rows, 250);                              // 4 chunks
$parts = Parafork\map('score_chunk', array_map(fn ($c) => [$c], $chunks));
$scores = array_merge(...$parts);

echo count($scores), "\n";
echo $scores['order-10'], "\n";
echo array_sum($scores), "\n";

Output:

1000
9
9009

The closure passed to array_map() runs in the calling process, which is fine; only the function given to Parafork\map() must be named.

Each task gets its own copy of its arguments. Changes a task makes to them, to globals or to static variables stay in the worker; only the return value comes back.

Objects as arguments and results

Objects cross as data: the receiver builds its own objects from the sender's property values. No constructor, destructor, __get or __set runs while an object is built, and there is no autoloading for a result.

<?php
enum Currency: string { case Eur = 'EUR'; case Usd = 'USD'; }

final class Line
{
    public function __construct(
        public readonly string $sku,
        public readonly int $qty,
        public readonly int $cents,
    ) {}
}

final class Order
{
    /** @var Line[] */
    public array $lines = [];
    public Currency $currency = Currency::Eur;

    #[Parafork\Skip]
    public ?object $logger = null;     // a service: left out, arrives as null

    public function __construct(public readonly string $id) {}
}

function price(Order $order): Order
{
    $total = 0;
    foreach ($order->lines as $line) {
        $total += $line->qty * $line->cents;
    }
    $priced = new Order($order->id . ':priced');
    $priced->lines = [new Line('total', 1, $total)];
    $priced->currency = $order->currency;
    return $priced;
}

$order = new Order('A-1');
$order->lines = [new Line('apple', 3, 50), new Line('pear', 2, 120)];
$order->logger = new stdClass;

$out = Parafork\map('price', ['a' => [$order]]);
echo $out['a']->id, ' ', $out['a']->lines[0]->cents, ' ', $out['a']->currency->name, "\n";
var_dump($out['a']->currency === Currency::Eur);
var_dump($out['a'] === $order);

Output:

A-1:priced 390 Eur
bool(true)
bool(false)

What crosses:

  • Objects of user classes (and stdClass): every declared property, public, protected and private, private properties of parent classes included, typed or untyped, readonly or not, initialized or not, and dynamic properties where the class allows them. The same object reached twice in one task's arguments, or in one result, arrives as one object, and cycles arrive as cycles. Two tasks that receive the same object get one copy each, and a result is never the caller's original object.
  • Enum cases, pure or backed, by name: the receiver gets its own case, so === against Currency::Eur holds.
  • Dates: DateTime, DateTimeImmutable, DateTimeZone, DateInterval, and classes that extend them (Carbon, for example), in both directions.
  • Classes with serialization methods (__serialize and __unserialize, or __sleep and __wakeup) cross the way serialize() and unserialize() move them. As a result, such an object is accepted only when its class, a parent class or an interface is listed in parafork.result_methods, because its __unserialize or __wakeup then runs in the calling process on data a worker chose.
  • Instance methods as tasks: [$object, 'method'] and invokable objects. The object is copied with the arguments and the worker calls the method on its copy. A public static method given this way runs as a static method and the object is not sent.
  • #[Parafork\Skip] on a property leaves it out: a typed property arrives uninitialized, an untyped one null. Use it for a logger, a repository or any service a data object holds.

What does not cross (type_not_transferable for an argument, result_not_transferable for a result, with the path to the value in the message and getPath()):

  • closures, generators, fibers, weak references and weak maps, resources, anonymous classes, classes with a destructor, abstract classes, interfaces;
  • internal classes that keep state outside their properties and are not in the list below: PDO, mysqli, the DOM and XML classes, the SPL iterators other than ArrayIterator and RecursiveArrayIterator, ZipArchive, and so on;
  • objects that implement Psr\Container\ContainerInterface (service containers, refused at once so that a framework container does not fill max_payload);
  • of the internal classes only these cross: stdClass, the date classes and DatePeriod, the ArrayObject family (ArrayObject, ArrayIterator, RecursiveArrayIterator), SplObjectStorage, SplDoublyLinkedList, SplQueue, SplStack, SplFixedArray, GMP, Random\Randomizer with the Mt19937, PcgOneseq128XslRr64 and Xoshiro256StarStar engines, every Exception and Error class, and any other internal class that has no custom object creation and the standard handlers. Each of them (the date classes and stdClass excepted) is checked against the exact shape of its own state before the receiver's internal __unserialize runs, and as a result it needs a listing in parafork.result_methods. Every other internal class is refused with "the class is not supported yet".

The receiving process needs the class: a worker has every class the calling process had when it was forked or can autoload it; the calling process needs the classes of a result. A class that differs between the two (a different property list or types) is refused. With parafork.result_classes set, a result may hold only objects of the listed classes, their subclasses and implementers.

Services are not copied. A worker is a copy of the calling process, so it already has the container, singletons and statics the process had when the worker was forked; a task takes its services from there and the arguments carry data. An inherited database connection is not usable in a worker (its file descriptor is replaced by /dev/null), so open one per worker with a start hook:

<?php
function open_db(): void { $GLOBALS['conn'] = 'opened in ' . getmypid(); }
function query(int $n): string { return $GLOBALS['conn'] . " #$n"; }

Parafork\onWorkerStart('open_db');           // runs once in each new worker, before its first task
$results = Parafork\map('query', [[1], [2]]);

An inherited SQLite connection is pinned in each new worker, so dropping it in a task never closes the parent's database files.

Overlapping work: submit() and Future

Parafork\submit($fn, $args = [], $timeoutMs = null) starts one task and returns a Parafork\Future at once. The task starts on a free worker, or waits in a queue when every worker is busy; tasks start in the order they were submitted. $args is one argument list, as one element of $argLists is for map(), and the same rules for callables and arguments apply. The arguments are checked before submit() returns, and a later change to your variables does not reach the task.

<?php
function slow_square(int $n): int
{
    usleep(200000);
    return $n * $n;
}

$a = Parafork\submit('slow_square', [4]);
$b = Parafork\submit('slow_square', [5]);
// the parent can do its own work here while the workers compute

echo $a->await() + $b->await(), "\n";
var_dump($a->isDone());

$c = Parafork\submit('sleep', [30]);
var_dump($c->cancel());
try {
    $c->await();
} catch (Parafork\TaskException $e) {
    echo $e->getReason(), "\n";
}

Output:

41
bool(true)
bool(true)
cancelled
  • await() waits for the result and returns it, or throws the TaskException of a failed task (its getTaskKey() is null). Later calls return the same value or throw an equal exception. await($ms) with a positive number of milliseconds gives up after that long with reason await_timeout; the task keeps running.
  • isDone() never blocks. cancel() removes a queued task or kills the worker that runs it, and returns false when the task had already finished; await() then throws cancelled.
  • A Future that is destroyed before its task finished cancels the task, and a script that ends with tasks running cancels them.
  • There is no background thread: queued tasks start, and finished results are read, only inside Parafork calls (submit(), await(), isDone(), cancel(), map(), status()). A task that has finished waits in its worker until one of them runs. $timeoutMs bounds a task from the submit() call, queue time included.
  • parafork.max_pending limits the unfinished futures of the process; one more submit() throws too_many_pending.

Errors

Failures of a call are Parafork\ParaforkException (or a subclass), a RuntimeException, and getReason() returns a short machine-readable reason. Bad arguments raise PHP's TypeError or ValueError. Two subclasses carry more detail:

  • Parafork\TransferException: an argument cannot be sent to a worker. getPath() gives the keys that lead to the bad value.
  • Parafork\TaskException: a task failed. getTaskKey() gives its key in $argLists, and for a task that threw, getRemoteClass(), getRemoteMessage(), getRemoteCode(), getRemoteFile(), getRemoteLine() and getRemoteTrace() describe the original exception.
<?php
function parse_quantity(string $s): int
{
    if (!ctype_digit($s)) {
        throw new InvalidArgumentException("not a quantity: '$s'", 42);
    }
    return (int) $s;
}

// a task throws
try {
    Parafork\map('parse_quantity', ['a' => ['12'], 'b' => ['x1'], 'c' => ['7']]);
} catch (Parafork\TaskException $e) {
    echo $e->getReason(), "\n";
    echo $e->getTaskKey(), "\n";
    echo $e->getRemoteClass(), ' (', $e->getRemoteCode(), '): ', $e->getRemoteMessage(), "\n";
}

// a task runs past the deadline of the call (200 ms here)
try {
    Parafork\map('sleep', [[5]], 200);
} catch (Parafork\TaskException $e) {
    echo $e->getReason(), ': ', $e->getMessage(), "\n";
}

// an argument holds a closure
try {
    Parafork\map('count', ['first' => [['items' => [fn () => 1]]]]);
} catch (Parafork\TransferException $e) {
    echo $e->getReason(), ' at ', json_encode($e->getPath()), "\n";
}

// a closure cannot run in a worker
try {
    Parafork\map(fn ($x) => $x * 2, [[1]]);
} catch (Parafork\ParaforkException $e) {
    echo $e->getReason(), "\n";
}

Output:

task_exception
b
InvalidArgumentException (42): not a quantity: 'x1'
timeout: task 0 did not finish within the call's deadline of 200 ms
type_not_transferable at ["first",0,"items",0]
closure_not_shippable

When several tasks fail, map() throws the failure of the task that comes first in $argLists, as a plain foreach would; tasks after it are cancelled.

Reasons, by class:

Class Reason Meaning
ParaforkException not_callable $fn is not a function or a public static method: unknown name, a private, protected or non-static method given by name, a method of an object that is not public or is magic (other than __invoke) or is served by __call, a name served only by __callStatic, $object->method(...).
closure_not_shippable $fn is a closure.
reentrant_call A Parafork function was called while another Parafork call was running (for example from a destructor).
unsupported_sapi The SAPI is neither cli nor fpm-fcgi.
opcache_restart_pending The request runs with opcache off while opcache's shared memory is in use, in a configuration where that is refused (see Requirements).
guard_not_installed Parafork was not loaded from php.ini at startup (for example per php-fpm pool), so its forbidden-function guard is not installed; nothing is forked.
unguarded_function A module loaded with dl() or per pool registered a function Parafork forbids in workers; nothing is forked.
sigchld_ignored The process ignores SIGCHLD (pcntl_signal(SIGCHLD, SIG_IGN)), so no worker can be forked.
threads_present The process runs other threads and parafork.allow_threads is off.
spawn_failed No worker could be started (the system call that failed is named).
too_many_pending submit() was called with parafork.max_pending futures unfinished.
await_timeout Future::await($ms) gave up; the task keeps running.
TransferException type_not_transferable An argument holds a closure, a resource, or an object or enum that cannot cross (see Objects); the message names the cause.
too_deep An argument nests deeper than 512 levels or holds a reference cycle.
payload_too_large The arguments of one task are larger than parafork.max_payload.
TaskException task_exception The task threw; see the getRemote*() methods.
task_fatal The task hit a fatal error, such as memory_limit or the time limit; the message is the fatal error's.
task_exit The task called exit().
timeout The task did not finish before the call's deadline.
cancelled Future::cancel() was called, shutdown() ended the task, or the script ended.
worker_crashed The worker was killed by a signal; getSignal() gives it.
worker_lost The worker ended without a result, or other code reaped it. Exit status 137 is usually PHP's memory hard limit.
nested_call The task called Parafork\map(), Parafork\submit(), Parafork\shutdown(), Parafork\onWorkerStart() or a Future method; raised in the worker and forwarded.
forbidden_in_worker The task called a function that acts on the calling process (such as header() or pcntl_fork()); raised in the worker and forwarded. The forbidden list is header() and the other response functions, the session functions, register_shutdown_function(), fastcgi_finish_request(), header_register_callback(), getallheaders(), apache_request_headers(), fpm_get_status(), dl() and the pcntl_fork and pcntl_exec family.
not_callable_in_worker Rare: the name resolved in the calling process but not in a worker; the task is retried once on a fresh worker first.
result_not_transferable The result holds a closure, a resource, an object or enum that cannot cross, or nests deeper than 512 levels or holds a reference cycle; the message names the cause and the path.
result_too_large The result is larger than parafork.max_payload.
result_corrupt The worker's result failed validation.
internal_error A defensive check failed: the task could not be sent to its worker, a class that crossed is missing in the worker, or a value changed while it was being copied.

Settings

Setting Default Range Can be changed Meaning
parafork.workers 0 0 to 64 anywhere (ini_set() too) Most workers per process. 0 means one per CPU the process may run on (the affinity mask at the first call, capped by a cgroup v2 CPU quota), at most 64. Workers are forked only as tasks need them. A change applies once no worker is alive.
parafork.task_timeout 60000 0 to 86400000 (ms) anywhere Default deadline of a call, in milliseconds, used when $timeoutMs is null; it is also capped by the time left under max_execution_time. 0 means no deadline from the extension.
parafork.max_pending 1024 1 to 65536 anywhere Most unfinished Futures per process (queued or running).
parafork.max_payload 1G 1M to 4G php.ini or -d only Largest argument set of one task, and largest result.
parafork.arena_recycle 64M 1M to 4G, at most max_payload php.ini or -d only A worker whose argument memory has passed this size is replaced after its current task.
parafork.allow_threads 0 0 or 1 php.ini or -d only 1 allows forking while the process runs other threads, at your own risk; without it a call that must fork throws threads_present. Needed on macOS, where every PHP process has two system threads. With grpc loaded, every php-fpm child has gRPC's threads from its first request on, so the default refuses every fork there, and GRPC_ENABLE_FORK_SUPPORT=1 does not change that. rdkafka's threads exist only while a producer exists, so code that destroys its producer before fanning out needs no switch. A worker blocked on a lock another thread held ends only at its task's deadline.
parafork.lifetime request request or process php.ini or -d only How long a worker lives. request: one parent request; in php-fpm every request that fans out forks its workers. process is experimental, see Lifetime.
parafork.placement claim claim, claim_parent, off php.ini or -d only Core placement: claim pins each task to a CPU that no other Parafork task in the process tree holds, and runs it unpinned when none is free; off pins nothing (the only mode on macOS); claim_parent also tries the parent's CPU for the first task of a call and is experimental. The empty string means off (an unquoted off in php.ini is read as false).
parafork.claim_scope tree tree or user php.ini or -d only Who shares the claims. tree: the process that loaded the extension and its descendants (in php-fpm, every child of the master, all pools). user: every process of the same user on the host, through /dev/shm/parafork.v1.<uid>; experimental.
parafork.result_methods empty class and interface names, comma separated php.ini or -d only Classes whose __unserialize or __wakeup may run in the calling process on a result. A class, a parent class or an interface in the list admits it, ignoring case. See Objects.
parafork.result_classes empty as above php.ini or -d only When not empty, a result may hold objects of the listed classes only, their subclasses and implementers (enum cases and dates included; DateTimeInterface admits DateTime, DateTimeImmutable and Carbon, DateTimeZone and DateInterval are listed by name). Empty: any class the calling process has loaded that passes the checks.
parafork.process_ignore_constants LARAVEL_START constant names, comma separated php.ini or -d only Under lifetime=process, constants whose values a kept worker may differ in.
<?php
ini_set('parafork.workers', '2');   // before the first call of the script

Parafork\map('strlen', [['a'], ['bb'], ['ccc']]);
$status = Parafork\status();
echo $status['workers_max'], "\n";
echo count($status['workers']), " workers alive\n";

Parafork\shutdown();
echo count(Parafork\status()['workers']), " workers alive\n";

Output:

2
2 workers alive
0 workers alive

Workers stay alive between calls of the same script and are stopped when the script ends, or earlier by Parafork\shutdown(). Changing an ini setting with ini_set() (any setting that differs from the one a worker was forked under) retires the workers; the next call forks new ones.

Lifetime

  • request (default). A worker serves one parent request. In the CLI a script is one request, so workers live until the script ends. In php-fpm every request that fans out forks its workers, and they end with the request.
  • process (experimental). Idle workers stay between the requests of one php-fpm child, which saves the fork. A kept worker carries what its parent held when it forked, statics and globals tasks wrote, and the arguments of earlier requests in its argument memory until it is replaced. Use it only where all requests share one trust level and tasks are pure functions of their arguments. A worker is retired when the code or the constants of the request differ from the ones it was forked with, when opcache restarted, when an ini setting changed, and when parafork.arena_recycle is passed. In php-fpm, tasks of a kept worker see $_SERVER and $_ENV rebuilt from the process environment only (no FastCGI parameters); under request they see their own request's. One log line per process says the setting is experimental. It needs opcache to be readable; where it is not, a request runs under request.
  • Cost of the default in php-fpm. The parent forks each worker, and fork time grows with the parent's heap: about 1.7 ms for a small page, 12.5 ms at a 200 MiB heap. See Performance. A request whose parallel work is a few milliseconds, or whose heap is large, may gain nothing from fanning out.

php-fpm

  • Load the extension from php.ini (extension=parafork), not per pool.
  • Processes. Workers belong to the php-fpm child that forked them, under every pm mode. When a child ends (pm.max_requests, idle timeout, reload, request_terminate_timeout, SIGKILL) its workers are killed with it. The ceiling of extra PHP processes is pm.max_children x parafork.workers; with workers=0 on a large host that is high (64 CPUs and 50 children allow 3,200). Set it per pool, for example php_admin_value[parafork.workers] = 4, together with parafork.task_timeout.
  • Task state. A task sees no request data: $_GET, $_POST, $_COOKIE, $_FILES, $_REQUEST and $_SESSION are empty, there are no request headers or body, output buffers started by the parent are not inherited, and output and headers of a task go nowhere. header(), setcookie(), the session functions, register_shutdown_function() and fastcgi_finish_request() throw forbidden_in_worker in a task. fastcgi_finish_request() in the parent is fine; submitted tasks go on and can be awaited after the response is sent.
  • Tasks run with the parent's code and settings at the moment of the fork. Services the parent booted before its first call are current in the workers; a service registered after the first call is stale in workers forked before it. Call Parafork\shutdown() after changing services.
  • Placement. With claim_scope=tree all pools of one master share the claim table, whatever their users. A pool that must not be influenced by the others sets php_admin_value[parafork.placement] = off.
  • Busy hosts. When every core is already busy with requests, fan-out adds no throughput: see the full-load cells in Performance. Use it to cut latency at low and medium concurrency.

Performance

Each result above "Reported, without a target" was measured in two separate sessions, and the two agree, except the placement=claim_parent empty round trip and the memory known limit, which were measured once. A target is the minimum result a cell has to reach. The cells under "Reported, without a target" are reported as measured, with no target set for them.

Host and method. One 30 vCPU virtual machine (1 thread per core), Intel Xeon Platinum 8358 @ 2.60 GHz, 222 GiB RAM, Linux 6.8 (kernel 6.8.0-1064-nvidia) x86-64. PHP 8.3.36, 8.4.27, 8.5.12 and 8.6.0RC7, all NTS, built with -O2, opcache on with the tracing JIT. Every cell ran in its own container on dedicated cores, with the other cores idle (checked before each run). parafork.* are the defaults, so lifetime=request and placement=claim, unless a row says otherwise. php-fpm cells use pm=static, 8 children, nginx and a load generator on their own cores, and a page that runs four CPU-bound tasks of about 20 ms; "sequential" is the same page running the four tasks in the child. Every result and every response body was checked against the sequential result. Numbers are medians; the PHP versions are listed where they differ.

Speed-up of CPU-bound work (command line)

Cell Result PHP
8 tasks of about 200 ms each on 8 cores, against running them in a loop 7.86x to 8.00x faster (8.3: 7.99x, 8.4: 7.95x, 8.5: 8.00x, 8.6: 7.86x) 8.3 to 8.6
Round trip of an empty task on a warm pool median 10 to 18 us, p99 13 to 22 us (8.3: 17.8 / 22.3 us, 8.4: 10.0 / 12.9 us, 8.5: 10.0 / 13.3 us, 8.6: 17.4 / 21.8 us) 8.3 to 8.6
4 tasks of about 5 ms on 4 cores, against running them in a loop 3.71x to 3.94x faster 8.3 to 8.6

php-fpm, lifetime=request

Sequential requests take 80.6 ms (four 20 ms tasks).

Cell Result PHP
One request at a time (c=1) median latency 22.6 to 22.8 ms 8.3 to 8.6
2 concurrent requests, 8 cores 3.52x to 3.58x the requests per second of sequential requests; p99 24 to 27 ms against 83 ms 8.3 to 8.6
8 concurrent requests, cores saturated 0.934x to 0.940x the requests per second of sequential requests; p99 107 to 108 ms against 83 ms 8.3 to 8.6
16 children, 16 concurrent requests, 8 task cores 0.933x the requests per second of sequential requests 8.3
First map() of a request (4 empty tasks, 4 forks), small page median 1.72 to 1.87 ms, p99 1.88 to 2.07 ms 8.3 to 8.6
cgroup v2 CPU quota of 4 CPUs on 8 cores 1.895x at 2 concurrent requests, 0.939x at 8 8.3
Two php-fpm masters of one user, claim_scope=user, 2 concurrent requests each each 3.43x and 3.44x of its sequential rate; combined on shared cores 1.823x of the combined sequential rate 8.3
placement=claim against placement=off, 8 concurrent requests 0.997x 8.3

With the cores saturated, fan-out adds no throughput: it keeps 93% to 94% of the requests per second of sequential requests and raises p99 latency. It cuts latency at low and medium concurrency.

placement=claim_parent stays experimental: its empty round trip is 9.97 us median and it does not beat claim at 2 or 8 concurrent requests (0.995x and 1.001x of claim).

Large arguments and results

Task: reads all five fields of 1,000,000 rows (arrays of an int, a string, an int, a float and a string) and returns a list of 1,000,000 ints; 2 cores, defaults. The serialize pool and ext-parallel use one worker. The serialize pool is a hand-written pcntl worker that exchanges serialize()d data over a socket pair; ext-parallel is its ZTS build on the same PHP version.

PHP Parafork\map() serialize pool ext-parallel same work inline map() against the serialize pool map() against ext-parallel
8.3.36 402.2 ms 1810.9 ms 1201.7 ms 183.5 ms 4.50x faster 2.99x faster
8.4.27 410.7 ms 1840.4 ms 1411.7 ms 186.9 ms 4.48x faster 3.44x faster
8.5.12 406.8 ms 1814.7 ms 1356.5 ms 170.1 ms 4.46x faster 3.33x faster
8.6.0RC7 408.5 ms 1892.8 ms 1190.5 ms 171.9 ms 4.63x faster 2.91x faster

A task that moves this much data and does little with it costs more than running it inline (the last column of timings). Faster copy-out of results is future work.

Objects

PHP 8.3.36, one warm worker, a task that returns its argument. W1: 100,000 objects of one class with six typed properties, as 25,000 trees with some shared and back references. W2: the same data as stdClass objects with dynamic properties. The comparison is a serialize pool that moves serialize()d data.

Cell Result
Round trip of W1 311.7 ms against 1417.2 ms: 0.218 times the serialize pool
Round trip of W2 420.1 ms against 1032.0 ms: 0.409 times
Building the W1 result in the parent 0.164 times unserialize() of the same data

Memory

Cell Result PHP
10,000 map() calls of 4 tasks with 1 MiB arguments each, lifetime=process, 4 workers worker RSS stays at about 9.7 MiB idle and 13.5 MiB at most (below the 75.7 MiB limit of idle size plus arena_recycle plus twice the largest argument set); 624 workers replaced by arena_recycle 8.3

Known limit: memory after a large payload. After a payload of the size of the 1,000,000-row task above and the end of its worker generation (lifetime=process), the parent's RssAnon plus RssShmem was 409,248 KiB above its idle value (8,232 KiB), and a new worker's was 409,252 KiB above its idle value (5,820 KiB). A control that built and freed the same rows without calling map() left the parent at 415,356 KiB, so 407,124 KiB of the growth is PHP's allocator keeping memory, and Parafork's own share is 2,124 KiB (about 2.1 MiB). The goal was to stay within 1 MiB of idle; it is not met. Finding the 2.1 MiB is future work.

Reported, without a target

These cells are reported as measured. No target is claimed for them.

Cost of the default lifetime in php-fpm (first map() of a request, 4 empty tasks), by parent heap size. PHP 8.3.36, one request at a time, lifetime=request. The 50, 200 and 500 MiB rows are 350 requests each; the first row is the first-dispatch cell of the php-fpm table above, taken over many more requests. Fork time grows with the parent's heap, so a request whose parallel work is a few milliseconds, or whose heap is large, may gain nothing from fanning out. lifetime=process avoids the fork after the first dispatch.

Parent heap (memory_get_usage()) Median p99 Fork CPU (median)
0.3 MiB (a small page) 1.72 ms 1.88 ms 1.44 ms
50 MiB 4.73 ms 5.59 ms 4.30 ms
200 MiB 12.50 ms 13.68 ms 12.06 ms
500 MiB 25.93 ms 29.54 ms 25.52 ms

A booted Laravel application (laravel/framework v13.40.0, production caches, no database, 8 children, PHP 8.3.36). One route ends with a map() of 4 empty tasks; another runs four 20 ms tasks.

lifetime=request lifetime=process
First map() of a request median 2.08 ms, p99 2.63 ms median 76 us, p99 120 us
Second map() of the same request median 54 us median 38 us
Four 20 ms tasks, 1 concurrent request 3.23x the requests per second of running them inline 3.62x
Four 20 ms tasks, 2 concurrent requests 3.11x 3.55x

Linux aarch64 (reported, no target): not measured for this release.

What the extension adds at run time

Generated from parafork.stub.php and checked against php --re parafork. Everything is in the Parafork namespace.

Functions

  • map(string|array|object $fn, array $argLists, ?int $timeoutMs = null): array: runs $fn once per element of $argLists in workers and returns the results under the same keys, in order. $fn is a function name, a public static method, or an object method (see Objects); closures throw closure_not_shippable.
  • submit(string|array|object $fn, array $args = [], ?int $timeoutMs = null): Future: starts one task, or queues it when every worker is busy.
  • status(): array: a snapshot of this process's workers, limits and counters (keys lifetime, lifetime_refused, threads, forks_with_threads, workers_max, workers, dying, counters, in_worker and placement; inside a task it returns that worker's own view with fewer keys).
  • shutdown(): void: stops this process's workers; the next call forks new ones.
  • onWorkerStart(string|array $callable): void: registers a start hook, a named function or a public static method ('f', 'C::m', ['C', 'm']), that each new worker runs once, after its setup and before its first task, inside that task's time limit. Use it for work a worker should do once, such as opening its own database connection instead of using the one it inherited. If it throws, the first task fails with that exception and the worker is retired. Registering the same name again changes nothing; a different one replaces the workers at the next call. It gets no arguments and its return value is ignored.

Classes

  • Future (final): the handle submit() returns. await(?int $timeoutMs = null): mixed, isDone(): bool, cancel(): bool.
  • Skip (final, an attribute for properties): #[Parafork\Skip] leaves a property out when its object is transferred.
  • ParaforkException extends \RuntimeException: base class of every call failure the extension reports.
    • getReason(): string (final): the reason code.
  • TransferException extends ParaforkException (final): an argument could not be sent to a worker.
    • getPath(): array: the keys from the argument list down to the value that could not be sent.
  • TaskException extends ParaforkException (final): a task failed.
    • getTaskKey(): int|string|null: the failed task's key in $argLists.
    • getRemoteClass(): ?string: class of the exception the task threw.
    • getRemoteMessage(): ?string: its message.
    • getRemoteCode(): int: its code (0 when the code is not an integer).
    • getRemoteFile(): ?string: the file it was thrown in.
    • getRemoteLine(): int: the line it was thrown at.
    • getRemoteTrace(): string: its trace as a string.
    • getSignal(): int: the signal that killed the worker, or 0.

Interfaces: none.

Constants: none.

ini settings

  • parafork.workers: most workers per process (0 = one per available CPU).
  • parafork.task_timeout: default deadline of a call, in milliseconds.
  • parafork.max_payload: largest argument set of one task, and largest result.
  • parafork.arena_recycle: argument memory after which a worker is replaced.
  • parafork.allow_threads: allow forking while the process runs other threads.
  • parafork.max_pending: most unfinished Futures per process.
  • parafork.lifetime: request or process (experimental).
  • parafork.placement, parafork.claim_scope: core placement and who shares it.
  • parafork.result_methods, parafork.result_classes: which result classes may run __unserialize or __wakeup, and which may be returned at all.
  • parafork.process_ignore_constants: constants a kept worker may differ in.

Limits

  • Closures are refused, of every kind (fn, function, bound or static), with closure_not_shippable. Pass the name of a function or a public static method, or a first-class callable such as strlen(...). $object->method(...) is refused too (not_callable); pass [$object, 'method'].
  • Some objects do not cross. Closures, resources, generators, anonymous classes, classes with a destructor, service containers and internal classes that keep state outside their properties are refused. What crosses is stdClass, user classes, enum cases, the date classes, and a fixed list of internal classes (the ArrayObject family, the SPL lists and SplObjectStorage, SplFixedArray, GMP, the Random engines, exceptions); every other internal class is refused as "not supported yet". See Objects.
  • A result with a serialization method needs a listing. An object whose class defines __unserialize or __wakeup comes back as a result only when its class, a parent or an interface is in parafork.result_methods.
  • Worker exception traces carry the parent's frames from the fork point, because a worker is a copy of the parent: an exception created in a task has a trace that starts with the task's frames and continues with the frames of the calling process at the moment of the fork. With zend.exception_ignore_args=0 (the CLI default) each frame holds its arguments, so such a trace often holds a value that cannot cross (a closure, a resource), and a result that is an exception object with Throwable listed in parafork.result_methods is then often refused with result_not_transferable, however simple the exception is. With zend.exception_ignore_args=1 the arguments are absent and the result crosses. Parafork does not cut the trace. TaskException::getRemoteTrace() shows the same parent frames as text. Set zend.exception_ignore_args=1 where tasks return exceptions.
  • Only the CLI and php-fpm. Other SAPIs get unsupported_sapi. Under php-fpm, Parafork must be loaded from php.ini, not per pool or through FastCGI.
  • NTS only. ZTS builds are refused when you run ./configure; Windows is not supported (no fork()). Linux x86-64 with glibc is the tested platform; macOS is for development (needs parafork.allow_threads=1, no CPU placement, no parent-death signal); aarch64 Linux and musl run the PHPT suite, with no benchmark for either; the BSDs and 32-bit builds are not supported.
  • Processes that run other threads. A call that must fork while the process runs other threads (a loaded grpc extension, an rdkafka producer, any macOS PHP) throws threads_present unless parafork.allow_threads=1.
  • Output of a task is not shown. A task's echo output is discarded and so are its headers and response functions. Its warnings, notices and deprecations are forwarded: at most 16 per task, each cut to 1024 bytes, and re-raised in the calling process after the call has collected every result, so the calling process's error handler sees them. Return what you need, or throw.
  • State between tasks. A worker runs several tasks, so statics and globals a task writes stay visible to the next task of the same worker, and which worker runs which task is not defined. Do not rely on either. A task's arguments are its own copy; changes to them, to globals or to statics never come back, only the return value does.
  • Settings and definitions. A worker keeps the code and ini values it was forked with, but the calling process retires its workers before a call when an ini setting changed or when a function, class or constant was defined after the fork, so the next call forks new workers (a fork per worker). Do not call dl() after workers exist: a dl() that replaces a user class by name is not noticed.
  • Cost of a fork. Fork time grows with the calling process's heap (see Performance); under the default lifetime each php-fpm request that fans out pays it. Tiny tasks gain nothing: each costs a few microseconds plus copying its arguments; run work of less than a few hundred microseconds per task inline.
  • Large repeated payloads. A task whose arguments are larger than parafork.arena_recycle (64M) retires its worker after that task, so the next call forks again and touches fresh pages. Raise arena_recycle above the payload if you send such payloads repeatedly.
  • Address space per worker. Each worker reserves arena_recycle + 2 x max_payload + 128 KiB of virtual memory (about 2.1 GiB at the defaults), only touched as used. A low RLIMIT_AS (ulimit -v) or vm.overcommit_memory=2 can stop workers from starting: the call then runs with fewer workers, or fails with spawn_failed. Lower parafork.max_payload there.
  • Code that reaps any child or ignores SIGCHLD. pcntl_wait() or pcntl_waitpid(-1, ...) while workers exist can take a worker's exit status: its task fails with worker_lost, and processes the worker started may keep running. A process that ignores SIGCHLD cannot fork workers (sigchld_ignored).
  • Processes started by tasks. proc_open(), exec() and system() children die with their worker's process group on a timeout or cancellation, unless they left it (setsid). When the calling process itself is killed, its workers are killed but the processes they started are not.
  • No background thread. submit()d tasks start and finish only inside Parafork calls; a finished task waits in its worker until the next call.
  • lifetime=process is experimental and unsuitable where requests differ in trust (see Lifetime). claim_scope=user and placement=claim_parent are experimental too; a same-user process can truncate the user table, which ends every process that uses it with SIGBUS.
  • opcache. With preloading and JIT, or with opcache.enable_file_override=1, a request that runs with opcache off cannot fan out (opcache_restart_pending; see Requirements). A worker that is still running 20 ms after its request ended is logged and reaped later.
  • Results with colliding hash keys. Copying back a result whose array keys collide in PHP's hash table takes quadratic time, as building the same array in plain PHP does.
  • Workers are not a sandbox. They run with the same user, files and permissions as the calling process. See docs/THREAT_MODEL.md.
  • Side effects run in no fixed order. Tasks run at the same time, so their writes to files or databases happen in any order.