README

Unit testing framework for XenForo

Compatibility

Install the newest version your XenForo and PHP allow. Older lines stay on Packagist.

version	XenForo	PHP	still gets fixes
v5.x	2.3	8.3+	yes - the current line
v4.x	2.3	8.3+	yes - fixes only
v3.x	2.3	8.1+	yes - for PHP 8.1 and 8.2
v2.x	2.2	8.1+	no
v1.x	2.1	5.5+	no

v4 and v5 support the same XenForo and PHP versions; v5 is the current line, and UPGRADING.md describes moving to it.

Put the version you picked in your addon's require-dev as ^5.32, ^3.0 and so on - the installation section below shows the whole file.

Upgrading

Upgrade notes are in UPGRADING.md, newest first. CHANGELOG.md lists everything that changed.

1. Introduction

Unit testing is a process by which individual components (units) of your software are tested. The objective is to isolate a section of code and validate its correctness - ensure that it behaves as expected for the given inputs.

There are multiple levels of testing typically used in software development, including Unit testing, Integration testing, System testing, Acceptance testing and so on. Unit testing is generally the lowest level of testing performed and the goal is to test a small section of code in isolation. Each test should run independently and should have no side effects on future tests. Ideally, unit tests should cover all potential code paths through the code being tested and verify that invalid inputs cause the expected failures.

Unit testing is typically executed using tools designed to run a suite of tests, reporting back to the developer on the success or failure of each test. The most popular and widely unit testing framework for PHP is PHPUnit. Learning how to use PHPUnit when developing PHP applications will make your code more robust, help you identify issues early on in the development lifecycle and generally make your code cleaner and more maintainable.

The most useful thing I've found with unit testing - other than the obvious debugging process - is when making major changes to your codebase, such as refactoring or upgrading to a new version of a framework or library being used. Having a comprehensive set of unit tests working before you begin, allows you to quickly identify code that no longer works the way it used to or should.

There are lots of tutorials on how to use PHPUnit and unit testing in general - I'm not going to cover that here. The purpose of this tutorial is to explain the theory and process behind unit testing XenForo addons.

Testing within a monolithic application framework such as XenForo can be problematic - especially given that we are not building stand-alone applications, but instead we are extending or adding functionality to an existing application.

One of the most important functions of unit testing is isolating your code from the surrounding framework and swapping out other code with test stubs, mock objects, fakes and test harnesses - which allow you to inject expected behaviours or responses from external code for testing purposes and avoiding side effects.

Fortunately, XenForo v2 has been architected in a way that makes it much easier for us to isolate parts of the framework and inject mock objects.

2. XenForo Application Container

One of the most significant architectural decisions made by the XenForo developers when designing XenForo v2 was to implement an application container pattern. The application container gives us a central location from which to gain access to all of the subsystems that allow XenForo to run - whether that is to access the database, send an email, create an alert, and so on.

What makes this so significant is that the container pattern also allows us to simply swap out those subsystems with systems of our own that can mimic or mock the behaviour we want to see for our testing purposes.

So for example, rather than actually sending an email - we can swap out the mail handling classes with mock objects that pretend to do the same thing without actually causing emails to be sent. It would be rather annoying to be sending lots of emails every time you ran your test suite!

What's more - we can set expectations on those mock classes to ensure that specific methods were called, perhaps with a certain set of parameters - as part of validating that our code executed as expected for our tests.

To make unit testing of XenForo addons easier, I have developed a unit testing framework which relies on the ability to arbitrarily swap out the subsystems from the application container with mock objects and application stubs - both for setting expectations and for avoiding side effects from our testing.

3. XenForo Application Architecture

To understand how we execute unit tests against our addons, we first need to understand how XenForo executes.

There are two possible execution triggers that we interact with as users or administrators of XenForo. The first is obviously the web interface. When we visit a XenForo site in our browser, the web server redirects our request via {forum_root}/index.php

If you look at this file, it's pretty simple (comments are mine):

<?php

// start by ensuring we are running the minimum required version of PHP
$phpVersion = phpversion();
if (version_compare($phpVersion, '5.6.0', '<'))
{
    die("PHP 5.6.0 or newer is required. $phpVersion does not meet this requirement. Please ask your host to upgrade PHP.");
}

// save the current directory from our index.php file - we'll use that later as our forum root
$dir = __DIR__;

// this is where we load in the main XenForo framework, but we aren't executing it yet
require($dir . '/src/XF.php');

// now let's boot the framework - this is what sets up some key variables, environment settings, runs our autoloader
// and registers our error handlers and shutdown functions
XF::start($dir);

// check whether we've got an API call
if (\XF::requestUrlMatchesApi())
{
    \XF::runApp('XF\Api\App');
}
else // ... or a regular web call
{
    \XF::runApp('XF\Pub\App');
}

You'll note that the API calls use the same entry point - making a HTTP call to {forum_root}/api gets redirected via the same index file.

Either way, \XF::runApp() is what actually starts execution of the XenForo framework. The first thing it does is to instantiate a new application container, store it in a static variable so we can access it globally, load the settings from config.php, start the autoloader for our addons, and then finally it starts handling the URL requested to work out whether we need to display a forum list, or a specific thread or take some other action.

The second entry point is via the CLI - the Command Line Interface. In this case, we instead execute {forum_root}/cmd.php, which looks remarkably similar to index.php (again, comments mine):

<?php

// start by ensuring we are running the minimum required version of PHP
$phpVersion = phpversion();
if (version_compare($phpVersion, '5.6.0', '<'))
{
    die("PHP 5.6.0 or newer is required. $phpVersion does not meet this requirement. Please ask your host to upgrade PHP.");
}

// save the current directory from our cmd.php file - we'll use that later as our forum root
$dir = __DIR__;

// this is where we load in the main XenForo framework, but we aren't executing it yet
require ($dir . '/src/XF.php');

// now let's boot the framework - this is what sets up some key variables, environment settings, runs our autoloader
// and registers our error handlers and shutdown functions
XF::start($dir);

// this is the important bit. Rather than "running" our application - we instead instantiate a CLI runner (based on
// Symfony's Console Component) and have that work our what command we're asking for and executing that for us.
$runner = new \XF\Cli\Runner();
$runner->run();

The $runner->run() command sets up the console input and output classes, and then sets up and starts our application framework, just like the web based entry point.

The difference is that instead of interpreting URLs to work out what we're asking it to do, it uses the command line parameters. Rather than generating a HTML (or JSON or XML etc) response back to a web browser or API client, it waits for output from the CLI command and writes that to the console.

In both cases though, we have our application container instantiated with all of the subsystems we might need, ready for our application to call them.

4. PHPUnit Architecture

Now, the important thing to understand with PHPUnit is that it is a console application. It has no web interface and does not interact with browsers or web servers. Indeed, it is much closer in function to XenForo's CLI interface.

The difference between the XenForo CLI and running PHPUnit is that our unit tests aren't giving XenForo a command to execute, we are feeding PHPUnit a set of test scripts, which it will execute one after the other and then output the results of those tests to the console.

So where does XenForo fit into this? Easy - we can have PHPUnit instantiate our XenForo application framework for us so that the application container is there waiting for us to call as our code is executed!

What's more, as part of our tests, we can selectively swap out or mock some of those subsystems in the application container so that we can give predictable responses to calls our code makes without any side effects for other tests.

There's quite a bit that goes on behind the scenes to make that happen - but I've created a Composer package you can include in your addons that does all the heavy lifting for you. All you need to do is write your tests and call various helper methods from my package to swap out subsystems as required.

5. Mock Objects

An important adjunct to PHPUnit is the ability to mock classes that our code being tested will interact with. This allows us to both ensure that our code makes the expected calls with the expected parameters, plus return predictable responses for our code to execute a specific code path.

In unit tests, mock objects simulate the behaviour of real objects. They can behave just like real objects, including being passed to function calls that declare type expectations and providing concrete implementations of interfaces, without needing to provide any implementation details.

Mockery is a simple PHP mock object framework for use in unit testing with PHPUnit. Mockery is designed as a drop in alternative to PHPUnit's own mock objects library. We utilise Mockery heavily in our unit testing framework.

6. XenForo Unit Test Framework

See installation instructions later on in this tutorial to learn how to integrate the unit test framework into your addon. For now I'll give you a brief overview of how it works once installed.

At this point I should acknowledge the work of Taylor Otwell and other contributors to the Laravel PHP Framework - the XenForo Unit Test Framework was heavily inspired by the test framework developed for Laravel and some of the reflection classes are taken directly from the Illuminate\Support component.

If you're new to PHPUnit - this is the point where you should go away and read some tutorials. At least read the documentation: Writing Tests for PHPUnit

The test classes we write for PHPUnit typically inherit from PHPUnit\Framework\TestCase. What my XenForo unit test framework does is provide some additional layers between the base TestCase class and your own classes.

Hampel\Testing\TestCase extends PHPUnit\Framework\TestCase and provides most of the functionality for the unit test framework. This is provided by the Composer package.

You then copy one file into your unit testing directory, {addon_root}/tests/TestCase.php:

<?php namespace Tests;

use Hampel\Testing\TestCase as BaseTestCase;

abstract class TestCase extends BaseTestCase
{
    /**
     * @var string $rootDir path to your XenForo root directory, relative to the addon path
     *
     * Set $rootDir to '../../../..' if you use a vendor in your addon id (ie <Vendor/AddonId>)
     * Otherwise, set this to '../../..' for no vendor
     *
     * No trailing slash!
     */
    protected $rootDir = '../../../..';

    /**
     * @var array $addonsToLoad an array of XenForo addon ids to load
     *
     * Specifying an array of addon ids will cause only those addons to be loaded - useful for isolating your addon for
     * testing purposes
     *
     * Leave empty to load all addons
     */
    protected $addonsToLoad = [];

    /**
     * Helper function to load mock data from a file (eg json)
     * To use, create a "mock" folder relative to the tests folder, eg:
     * 'src/addons/MyVendor/MyAddon/tests/mock'
     *
     * @param $file
     *
     * @return false|string
     */
    protected function getMockData($file)
    {
        return file_get_contents(__DIR__ . '/mock/' . $file);
    }
}

You might need to edit the $rootDir variable if you don't include a vendor folder in your addon installation path.

This class is what your unit tests will extend, and the two properties above are the whole of its configuration.

The framework boots XenForo for you. Hampel\Testing\TestCase::createApplication() follows the same pattern as XenForo's own entry points: it includes XF.php, starts XenForo, and sets up the application with the add-on ids from $addonsToLoad. It runs in setUp(), so every test class extending Tests\TestCase has the application available.

If you need a different boot, override createApplication() in your tests/TestCase.php and pass ['xf-addons' => $this->addonsToLoad] to XF::setupApp().

A test extending plain PHPUnit\Framework\TestCase cannot load your addon's classes by itself. They are loaded by XenForo's autoloader, which exists only once a test has booted the application. Such a test passes in a full run when an earlier test has booted XenForo, and fails when run alone with Class "Vendorly\Addonista\..." not found.

To find these tests, run each test class on its own:

for f in $(find tests/Unit tests/Feature -name '*Test.php' | sort); do
    ./vendor/bin/phpunit "$f" >/dev/null 2>&1 || echo "FAILS ALONE: $f"
done

--order-by=random does not reliably find them, because the autoloader stays registered once any test has booted.

The same loop answers the opposite case, which fails the other way round. $addonsToLoad cannot remove a class extension that something earlier in the process has already resolved - XenForo caches the resolution and aliases an XFCP proxy, and a proxy cannot be undeclared. So a test asserting that an addon's extension is absent, perhaps to prove a column stays nullable for a forum without it, passes on its own and exercises the extension in a full run. Guard such a test with requireClassNotExtended(), which skips rather than passing, and run it in the per-class loop above.

Two fixes:

extend Tests\TestCase instead;
or map your addon's namespace in your own composer.json and run composer dump-autoload, so Composer can load plain classes without XenForo:

"autoload-dev": {
    "psr-4": {
        "Tests\\": "tests/",
        "Vendorly\\Addonista\\": ""
    }
}

The second works only for plain classes. A class built on an XFCP_ proxy - one extending a XenForo class - still needs the booted application.

7. Swapping XenForo Subsystems

The core functionality of the XenForo Unit Testing Framework is provided by some fairly simple helper functions which make it easy for us to swap out subsystems from the application container.

It helps to understand that most of the subsystems in the application container are not fully instantiated classes - instead they use closures to configure and instantiate the object that will actually provide the subsystem the first time it is called by the application. This is much more efficient - since we don't execute or instantiate any of that code until we need to use it. It also makes it trivial to swap out - we can simply over-write those closures in the container with our own objects.

swap() simply replaces the code at a specific container key with an instance we supply.

So if we have written a test harness which replaces certain functionality, we can just swap in our class in place of the core one. We do exactly this in our fakesMail() helper - we replace the mailer.transport container key with our own class that logs mails the application sent and lets us run assertions against that log, but never actually sends mail.

    protected function fakesMail()
    {
        // enableMailQueue is a config.php value, not an option - so mail sent with queue() goes
        // straight to the transport rather than becoming a MailSend job we would never see
        $this->setConfig('enableMailQueue', false);

        $this->swap('mailer.transport', function (Container $c)
        {
            return new TestTransport();
        });

        // XF\Mail\Mailer takes both the transport and the queue flag as constructor arguments, and
        // `mailer` is separately cached - so an already-built mailer would keep the real ones
        $this->app()->container()->decache('mailer');

        return $this->getMailTransport();
    }

Swapping a container key does not reach anything already built from it. A resolved object keeps the values it was constructed with, so if you swap a key yourself, decache() whatever was built from it, as fakesMail() does with mailer above.

The TestTransport class extends Symfony Mailer's AbstractTransport, accepting the same calls a real transport would, but stores the messages in an array rather than sending them.

mock() takes that one step further and lets us swap the closure function with a mock object that we can declare assertions on for testing purposes.

We supply an abstract class to use as the basis for the mock object and then optionally supply expectations on that mock.

For example, if our code queries the XF\Http\Request class to retrieve the IP address of the visitor - we can't test this from PHPUnit because there is no HTTP request when executing a console command! However, we can simply mock our request - it's stored in the 'request' key in the application container and so we might do this:

$this->mock('request', XF\Http\Request::class, function ($mock) {
   $mock->expects()->getIp(true)->once()->andReturns('10.0.0.1');
});

So we instruct XenForo to use our mock object when querying the Request object, and we tell PHPUnit that we are expecting our code to call XF\Http\Request::getIp(true); once, at which point our mock object will return the IP address 10.0.0.1.

spy() is the same swap again, with the expectations turned around. A mock declares what must happen and fails if it does not; a spy declares nothing, records everything, and lets us assert after the code under test has run:

$request = $this->spy('request', XF\Http\Request::class);

// ... run the code under test ...

$request->shouldHaveReceived('getIp');

That suits code we want to observe rather than constrain. It costs us the return value, though - a spy answers null for every method it was not explicitly told about, so where the code under test uses what it gets back, mock() is still the right tool.

We have helpers to mock many of the key subsystems:

mockDatabase
mockRepository
mockFinder
mockEntity
mockFactory
mockService
mockRequest
mockFs

We also have fake systems which log interactions with the subsystem and then allow us to query that after the fact:

fakesErrors
fakesJobs - and runJobToCompletion, which runs a job until it completes
fakesLogger
fakesMail
fakesSimpleCache
fakesRegistry
fakesHttp - and fakesHttpByUrl, which picks the response by request URL rather than by the order the calls happen in
fakesEvents - records the code events your addon fires and stops them reaching any listener
recordsLog - records what your addon logs through a PSR-3 logger of its own, rather than through XenForo's moderator log

Finally, we have some special helper functions for specific purposes:

assertBbCode lets you test the expected output of some BbCode, useful for testing custom codes.
expectPhrase mocks the phrase rendering process and allows us to return arbitrary strings for phrases.
setOption & setOptions allow us to directly set the values we want for our options so we don't have to mock the options repository. It restores options after each test is executed - keeping to our goal of no side-effects.
setTestTime lets us set the application execution time (\XF::$time) to a known specific time (optionally using the Carbon library), so that we can test functions that rely on time intervals or comparisons.
swapFs lets us swap the filesystem from local to memory so that we can make non-persistent changes to the filesystem and avoid side effects. This one needs league/flysystem-memory: ^1.0 in your own require-dev, which Composer will not install without the advisory note in the installation section below
setConfig sets a value in config.php
assertAddOnClassExtensionsApply asserts that every class extension your addon registers actually applies. One that does not is silent - XenForo's own class runs and your code never does - and a suite with no test on the extended class has nothing to go red, so it passes either way
createTempDirectory gives you an empty directory to write into, absolute and inside the system temporary directory, and removes it when the test finishes - so a test needs no cleanup of its own. Reach for it rather than pairing File::getTempDir() with File::deleteDirectory() by hand, which resolves a relative or empty path against your XenForo root
actingAs, actingAsMember and actingAsGuest run the code under test as a given visitor, with setVisitorPermissions and setVisitorContentPermissions granting permissions in memory rather than reading them from the database
makeEntity and createEntity build an entity with the values you give it - makeEntity touches no database, createEntity saves
createNodeFixture and createNodeTreeFixture create nodes without rebuilding the forum's node tree. Saving an XF:Node the ordinary way writes every row in xf_node and queues a permissionRebuild job, so inside a test's transaction every node on the forum is locked until the test ends
createThreadFixture creates a thread in a node, with its first post - a thread without one has a null FirstPost, so it fails wherever anything reads through it
dispatch runs one of your routes the way XenForo does and hands back the reply its controller produced, with assertReplyIsView, assertReplyTemplate, assertReplyViewClass, assertReplyParam, replyParam, assertReplyIsRedirect, assertReplyIsError, assertReplyIsMessage, assertReplyIsApiResult, replyApiResult and replyErrors to assert against it. Parameters the route reads are passed as the third argument, and the controller's access checks in preDispatch() run. describeReply summarises a reply in one line - reach for it rather than var_export, which on a reply's errors reaches a phrase, the language and the database adapter's configuration, password included
callAction calls a controller action directly with a POST, for testing saves, toggles and deletes. It skips preDispatch(), so pair it with dispatch to test access checks
followRedirect dispatches the route a redirect reply points at, so a test of a save can go on to assert what the page the user lands on shows. It reads the route type out of the url, which is the part worth not hand-rolling - the control panel has its own script, the api sits behind a prefix on the public one, and the friendly-urls option changes the public and api forms but not the admin one
actingAsApiKey runs api dispatches as a given api key, and restores \XF::$apiKey afterwards
renderTemplate renders a template to HTML with no web server, renderMacro renders one macro out of a template, and renderReply renders the template a dispatched reply named, with assertSee, assertDontSee, assertSeeText, assertDontSeeText and assertSeeInOrder to assert on the output, and textOf to read that output back with its markup stripped and its entities decoded
assertTemplateModificationApplied asserts that a template modification has applied, using its apply count
assertAddOnTemplateModificationsApply asserts that every template modification your add-on registers still applies, which a suite with no assertion on the markup they insert cannot otherwise notice - XenForo records a modification whose find stops matching as ok with an apply count of zero
renderTemplateWithoutModification renders a template with one of its modifications left out, so a test can show the markup it asserts on is that modification's work, without disabling it for every other suite on the forum. To pair it with a dispatch, hand it the reply's own parameters - renderTemplateWithoutModification($key, $reply->getParams()) - which is what renderReply renders the template with. $xf.reply is still absent on this side, which matters only for a template that reads it
assertNoTemplateErrors asserts that nothing the test rendered raised an error
assertNoUnresolvedPhrases asserts that none of your add-on's phrases came out as its own key, which is what XenForo renders for a phrase it cannot find. It takes a prefix or a whole phrase key, and searches the text the page displays, the page parameters the render set, and the side navigation and sidebar
pageParam reads back what a rendered template set with <xf:title> and friends, which never appear in the template's own output
sideNavHtml and sidebarHtml read back what a render contributed through <xf:sidenav> and <xf:sidebar>, which are in neither the rendered output nor the page parameters - XenForo's account pages build their whole sidebar that way. Both accumulate across renders within one test
globalTemplateData hands back the xf parameter a render sees, which is how a test asserts on what a templater_global_data listener of yours contributed - the event fires from inside XenForo's own method, so nothing else proves the listener is registered the way XenForo will call it
renderRawReply and renderJsonReply render a reply whose view builds its own output rather than naming a template - a file download shaped in renderRaw(), or an endpoint shaping its response in renderJson(). The first returns the XF\Http\Response with the body set, the second the decoded JSON document
renderBbCode renders BB code to HTML, the BB code twin of renderTemplate
renderWidgetHtml renders one widget to HTML, named by its definition class or by the key of a widget the forum has configured. A widget that decides not to show itself renders an empty string and that is preserved, so an assertion on the absence means the widget declined - the other ways a widget render comes back empty are faults and are refused instead
setVisitorAdminPermissions grants admin permissions to a built visitor
UsesDatabaseTransactions is a trait you opt into per test class: it wraps each test in a transaction and rolls it back, so tests can save entities without leaving anything behind. assertDatabaseHas, assertDatabaseMissing and assertDatabaseCount then assert against rows that are really there

isolateAddon() no longer exists; use the $addonsToLoad property in your tests/TestCase.php.

8. Installing the Framework

The unit testing framework is structured as a Composer package hampel/xenforo-test-framework - you'll need to have Composer installed on your dev server before you can use it. We use the require-dev directive to load the testing framework only in our development environment. We will later show the commands required to strip our test code from our addon during the build process - we don't want or need to deploy our unit tests to our production servers.

You can view the source code for the package here: XenForo Test Framework

If you need more guidance on using Composer packages in your XenForo addons - refer to my tutorial: Using Composer Packages in XenForo 2.1+ Addons Tutorial

I'm going to assume that your XenForo forum root is at /srv/www/xenforo and that your addon (let's call it "Vendorly/Addonista") is installed in /srv/www/xenforo/src/addons/Vendorly/Addonista

If you already have a composer.json file, I'll presume that you know what you are doing and will simply instruct you to add the require-dev and autoload-dev directives below to your file. Otherwise, if your package does not already use Composer, you can simply create a composer.json file with the following in the root of your addon (/srv/www/xenforo/src/addons/Vendorly/Addonista/composer.json):

{
    "require-dev": {
        "hampel/xenforo-test-framework": "^5.32",
        "nesbot/carbon": "^3.0"
    },
    "autoload-dev": {
        "psr-4": {
            "Tests\\": "tests/"
        }
    }
}

Note that nesbot/carbon is optional, but really useful.

If you add league/flysystem-memory for swapFs(), Composer will refuse to resolve it. Every flysystem release up to 3.35.2 carries a low-severity advisory, and Composer blocks an affected package by default, so composer update stops with "not loaded, because they are affected by security advisories". XenForo supplies flysystem 1.x, which has no patched release, so the version you need is the version that is flagged. To install it anyway, tell Composer you accept that one advisory, and keep the floor above the range of the critical one that is patched:

{
    "require-dev": {
        "league/flysystem": "^1.1.4",
        "league/flysystem-memory": "^1.0"
    },
    "config": {
        "policy": {
            "advisories": {
                "ignore-id": {
                    "PKSA-w9tt-7782-78jx": "no patched flysystem 1.x exists; dev-only, for the test suite"
                }
            }
        }
    }
}

composer audit then lists it as ignored with that reason rather than hiding it. Do not ignore PKSA-pwh8-d4fr-nywn - that one is a remote-code-execution advisory fixed in flysystem 1.1.4, which the floor above already excludes.

If your addon pins config.platform.php below 8.3, raising it to install this has a consequence. Composer uses that pin to guard every dependency's PHP requirement, so once it says 8.3 nothing stops a runtime dependency resolving to a release needing more than your addon's own floor. Check the packages array in composer.lock after each update.

Your dev dependencies are visible to other addons' test suites on the same install. If your addon declares composer_autoload, XenForo registers its whole vendor/ - dev tree included - on the class loader, so your copy of PHPUnit is present in another addon's test run unless that suite sets $addonsToLoad. Setting it in your own tests/TestCase.php protects your suite from theirs; it does not protect theirs from yours.

A suite with no $addonsToLoad can die before the first test on an install where another addon vendors a different PHPUnit. Two copies of PHPUnit on one class loader mean a class from one and its parent from the other - PHP then refuses with something like "Readonly class PHPUnit\Framework\TestStatus\Known cannot extend non-readonly class", and the run exits 255. The versions do not have to differ by a major for this: two patch releases of the same minor are enough. Worse, XenForo's error handler logs that fatal as a real row in xf_error_log - a row that an addon which reports on the error log will then send onward. Set $addonsToLoad and the whole class of problem goes away, because no other addon's autoloader is registered.

Change directory to your addon root, then run composer update to install the framework. We install PHPUnit and Mockery automatically for you.

$ cd /srv/www/xenforo/src/addons/Vendorly/Addonista/
$ composer update

Next, look in the {addon_root}/vendor/hampel/xenforo-test-framework/ directory - copy the entire tests directory to the root of your addon.

$ cd /srv/www/xenforo/src/addons/Vendorly/Addonista/
$ cp -r vendor/hampel/xenforo-test-framework/tests .

Inside the tests directory, you'll find the following directories and files:

/tests/Feature this is where feature tests go, if you write any
/tests/Feature/ExampleTest.php a placeholder that keeps the directory in git - phpunit.xml will not run without it. If you delete it before adding a feature test, commit a .gitkeep instead
/tests/Unit this is where all of your unit tests should go
/tests/Unit/ExampleTest.php this is a simple example test - edit or copy it as the basis for your own test classes
/tests/TestCase.php this is our base test class (Tests\TestCase) that all unit test classes should inherit from if you want to boot the XenForo application framework for use in your tests. It is the only file here you own and edit

Third step is to copy the phpunit.xml file from {addon_root}/vendor/hampel/xenforo-test-framework/phpunit.xml into the root of your addon:

$ cd /srv/www/xenforo/src/addons/Vendorly/Addonista/
$ cp vendor/hampel/xenforo-test-framework/phpunit.xml .

This file contains the configuration and directives for PHPUnit - importantly, look at the testsuite configuration options - they tell PHPUnit where to find our unit tests.

<?xml version="1.0" encoding="UTF-8"?>
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:noNamespaceSchemaLocation="https://schema.phpunit.de/10.5/phpunit.xsd"
         backupGlobals="false"
         backupStaticProperties="false"
         bootstrap="vendor/autoload.php"
         cacheDirectory=".phpunit.cache"
         colors="true"
         processIsolation="false"
         stopOnFailure="false"
         failOnDeprecation="true"
         failOnRisky="true"
         failOnNotice="true"
         failOnWarning="true"
         failOnEmptyTestSuite="true">
  <testsuites>
    <testsuite name="Unit">
      <directory suffix="Test.php">./tests/Unit</directory>
    </testsuite>

    <testsuite name="Feature">
      <directory suffix="Test.php">./tests/Feature</directory>
    </testsuite>
  </testsuites>
</phpunit>

The failOn* flags fail the suite on deprecations, notices, warnings, risky tests, and a run that executes no tests - such as a --filter that matches nothing.

failOnPhpunitDeprecation is deliberately absent: PHPUnit 10.5 does not have it, so on the oldest supported version the configuration fails validation and failOnWarning then fails every run.

failOnEmptyTestSuite fires on an empty run, not an empty suite. A Feature suite that collects nothing - or misses a test file not named *Test.php - still exits 0 while the Unit suite passes. When you add your first feature test, run vendor/bin/phpunit --testsuite Feature and check the test count includes it.

Finally, update your build.json file to clean up unit test code when we build our addon releases. Assuming you only use Composer for unit testing:

{
    "exec": [
        "rm -fv _build/upload/src/addons/{addon_id}/composer.json",
        "rm -fv _build/upload/src/addons/{addon_id}/composer.lock",
        "rm -fv _build/upload/src/addons/{addon_id}/phpunit.xml",
        "rm -rfv _build/upload/src/addons/{addon_id}/tests",
        "rm -rfv _build/upload/src/addons/{addon_id}/.phpunit.cache",
        "rm -rfv _build/upload/src/addons/{addon_id}/vendor"
    ]
}

-f because a path that is not there is not a problem: without it the build prints an error for every file you have already removed.

.phpunit.cache has to be named. XenForo skips a dot-entry by its own name but still walks into it, so .phpunit.cache/test-results reaches the zip unless the directory is removed.

Removals go before any mv of your documentation into _build/, if your build does that - exec runs in order.

(you can literally use the string {addon_id} - you don't need to hard code your addon ID in the build.json file !!)

... these directives will delete all of the files relating to our unit testing, including the composer files and vendor directory.

If you also use Composer for other non-development parts of your addon, then you don't want to remove everything - just add the following to your build.json instead:

{
    "exec": [
        "rm -v _build/upload/src/addons/{addon_id}/phpunit.xml",
        "rm -v -r _build/upload/src/addons/{addon_id}/tests"
    ]
}

... and make sure you specify the --no-dev option when running composer install during your build process.

For example, an addon which uses Composer for both dev and non-dev purposes:

{
    "exec": [
        "composer install --working-dir=_build/upload/src/addons/{addon_id}/ --no-dev --optimize-autoloader",
        "rm -fv _build/upload/src/addons/{addon_id}/phpunit.xml",
        "rm -rfv _build/upload/src/addons/{addon_id}/tests",
        "rm -rfv _build/upload/src/addons/{addon_id}/.phpunit.cache"
    ]
}

Give every composer install a --working-dir pointing into _build/. XenForo runs each exec line from your addon's own root, so a bare composer install --no-dev strips the dev dependencies out of the working copy you develop in - this framework among them - every time you build. Reinstalling them is composer install, but nothing tells you it happened.

9. Configuring the Framework

There are two options you may need to configure, both in TestCase.php which will be in the tests directory.

Root Directory Path

The $rootDir variable specifies the path to your forum root, relative to the addon directory.

protected $rootDir = '../../../..';

Typically, addons will use the Vendor/AddonId structure, which makes the addon directory <forum-root>/src/addons/Vendor/AddonId - and thus we need to go back 4 levels to reach the forum root.

However, if you haven't used a Vendor in your addon id, you'll need to adjust this variable. If your addon directory is: <forum-root>/src/addons/AddonId, then you'll need to change it to: $rootDir = '../../..'

Addon Isolation

The other option is the addon isolation system - allowing us to instruct the XenForo framework to only load addons we need for testing, thus minimising any side effects or unintended interactions that may occur from other addons we have installed in our development forum.

Simply list the full addon ids of each addon you want to be loaded when the unit tests run, as an array.

For example:

protected $addonsToLoad = ['Vendor/AddonId', 'XFMG'];

Generally, you would simply list the addon id of the addon you are developing, thus keeping other addons out of the test run. Note that this also restricts Composer autoloading, which is a good way of checking that you've specified all requirements in composer.json correctly and aren't picking up packages from other addons.

Leaving the array empty will load all addons as normal.

Isolation filters three things: Composer autoloading, class extensions, and code event listeners - app_setup included. On v4 and earlier it did not filter listeners.

Content type handlers are not among them, because they come from the database - so every installed addon's stay registered, and an operation that walks every content type can fail inside XenForo's own code. Nor can an extension be added to a class another test has already resolved, and a test class whose list would add one is refused in setUp() rather than quietly exercising XenForo's own class. Both are answered by declaring the list once for the whole suite rather than per test class; DOCS.md has the detail, under "Unit Test Configuration".

A note on failOnRisky

XF::start() installs its own error and exception handlers, and TestCase removes them after each test so that failOnRisky can be used. As a result, a handler your own code leaves installed is removed rather than reported.

10. Running Unit Tests

Composer installed the PHPUnit executable at {addon_root}/vendor/bin/phpunit. To run our tests, we go to our addon root in the console and simply execute PHPUnit. The phpunit.xml file (which should also be in our addon root) tells PHPUnit where to find our tests.

$ cd /srv/www/xenforo/src/addons/Vendorly/Addonista/
$ ./vendor/bin/phpunit
PHPUnit 12.5.34 by Sebastian Bergmann and contributors.

..................                                                18 / 18 (100%)

Time: 544 ms, Memory: 32.00 MB

OK (18 tests, 70 assertions)

You could also set up a bash alias to make it easier to run. I have the following:

alias u="./vendor/bin/phpunit"

... so all I need to do is change directory to my addon root and then just run the u command to execute my unit tests.

11. Framework Documentation

DOCS.md — the API reference, one section per helper
TIPS.md — writing testable code, and what to reach for instead of mocking XenForo
UPGRADING.md — what to do when moving between versions
CHANGELOG.md — everything that changed, per release

12. Limitations

There are still things we can't effectively test, or which are problematic to test.

Controllers

dispatch() runs a route in process and returns the reply the controller produced, for public, admin and api routes. It covers the view returned, the parameters passed to the template, and the controller's access checks in preDispatch() - which a controller invoked directly never runs. renderReply() renders the template the reply named.

Out of reach: the whole page - navigation, header and footer come from XenForo's own app classes rather than from the template, though pageParam() reads the values the template set for that wrapper, such as the page title. Also anything needing the real front controller - index.php's bootstrap order, session cookies, web server rewrites - and JavaScript and visual appearance.

dispatch() sends a GET by default and any method you ask for. XenForo asserts a CSRF token in preDispatch() for anything that is not a GET, so a non-GET dispatch carries one, with a matching cookie - which puts a write path behind its own authorisation checks within reach. dispatchWithoutCsrfToken() sends none, for asserting that the check refuses. callAction() calls an action directly, skipping preDispatch() and so the permission checks too, for the cases where that is what you want to test.

Most public routes also need the visitor to hold general.view, which a built visitor does not, so a public dispatch returns a 403 until the test grants it. Eleven core controllers are exempt, account and login among them, so a route that works without the grant is not evidence that the next one will. See dispatch() in DOCS.md.

Name the controller your add-on extends, not your extension of it - 'XF:Tools', not MyVendor\MyAddon\XF\Admin\Controller\Tools. XenForo resolves the base to the most derived class, which is your extension, so resolving through the base is itself proof the extension applied. An extension named directly is refused, because the proxy it inherits from is declared while XenForo resolves the class being extended.

Database queries

Mocking the database adapter, entities and finders quickly becomes cumbersome for anything beyond simple queries. With the UsesDatabaseTransactions trait you can run the real query against the real database and assert on the result with assertDatabaseHas(); the transaction is rolled back when the test finishes.

Entity saving

save() on the base Entity class is final, so a mock cannot stop it reaching the database. Use UsesDatabaseTransactions: the save still runs, and is rolled back when the test finishes.

Functions which use `time()` rather than `\XF::$time`

We cannot override the return value of the PHP function time() with an arbitrary known value like we can with the \XF::$time variable. As such, we cannot know in advance what that value will return which can make some tests problematic.

Using Carbon::now() as a replacement for time() would solve this problem for us - since the Carbon library has the ability to set arbitrary return values for testing. However, we cannot control external libraries and the XenForo core

if functions they use rely on time() and we need to call them and can't mock the entire class, then we could have difficulty testing in some circumstances.

UI changes & template modifications

renderTemplate() renders a template to HTML, so a template modification applying, and a phrase resolving rather than showing a raw key, can both be asserted - see assertSee() in DOCS.md. A template modification must be installed in the forum the tests run against. On a development install, a template with an _output/ copy renders from that copy, so an edit to it is seen without xf-dev:import, and a change made only in the database is not - see renderTemplate() in DOCS.md.

The whole page, appearance and JavaScript behaviour still need checking by hand.

Code which checks what kind of application is running

This framework boots the base XF\App, not XF\Pub\App or XF\Admin\App, so code that checks $app instanceof \XF\Pub\App - typically a templater_global_data listener - does not run its body under a test.

Check $app->container('app.classType') === 'Pub' instead, as XenForo itself does; dispatch() sets that key. Or move the body into a method the test can call directly. A listener checking the class type can be tested like this:

$this->swap('app.classType', 'Pub');

$data = [];
\MyVendor\MyAddon\Listener::templaterGlobalData($this->app(), $data, $reply);

$this->assertArrayHasKey('myAddonData', $data);

Data in the database

Code which relies on particular data being present in the database is problematic, since that data can change.

UsesDatabaseTransactions removes anything a test writes. XenForo's own transactions nest inside it, so code under test can open and commit its own. Two things it cannot roll back: DDL commits implicitly in MySQL, so a test that alters the schema must clean up after itself - $db->emptyTable() counts, since it issues a TRUNCATE; and uncommitted rows are visible only to the connection that wrote them.

There is no way to seed a known set of data for a test to work against. Tests that rely on data already in your development forum are fragile.

Filesystem paths which aren't abstracted

swapFs() only helps when every access goes through $app->fs(). Code which writes to a real path - XF\Util\File::getTempDir(), File::getNamedTempFile() - and reads the result back through an abstracted path such as internal-data:// fails under a swapped filesystem, because the two no longer point at the same place. Test that code against the real filesystem, writing to a path you delete in tearDown().

$fs->has() does not reliably report directories, so call deleteDir() unconditionally, in a try/catch, when cleaning up.

Static classes

If we can't swap out a class with our own instance, because it relies on static variables or functions - then it will be much more difficult or impossible to test. This is a general limitation on unit testing rather than something specific to XenForo.

\XF::$time can be set with setTestTime(), and \XF::visitor() with actingAs(), which restores the previous visitor afterwards.

13. Writing testable code and other unit testing tips

Recognising code that is hard to test, and knowing how to restructure it, is most of what makes a suite worth having. Those tips have their own document: TIPS.md.

hampel / xenforo-test-framework

Package info

Security

Statistics

README

Compatibility

Upgrading

1. Introduction

2. XenForo Application Container

3. XenForo Application Architecture

4. PHPUnit Architecture

5. Mock Objects

6. XenForo Unit Test Framework

7. Swapping XenForo Subsystems

8. Installing the Framework

9. Configuring the Framework

10. Running Unit Tests

11. Framework Documentation

12. Limitations

Controllers

Database queries

Entity saving

Functions which use `time()` rather than `\XF::$time`

UI changes & template modifications

Code which checks what kind of application is running

Data in the database

Filesystem paths which aren't abstracted

Static classes

13. Writing testable code and other unit testing tips

hampel / xenforo-test-framework

Package info

Security

Statistics

README

Compatibility

Upgrading

1. Introduction

2. XenForo Application Container

3. XenForo Application Architecture

4. PHPUnit Architecture

5. Mock Objects

6. XenForo Unit Test Framework

7. Swapping XenForo Subsystems

8. Installing the Framework

9. Configuring the Framework

10. Running Unit Tests

11. Framework Documentation

12. Limitations

Controllers

Database queries

Entity saving

Functions which use time() rather than \XF::$time

UI changes & template modifications

Code which checks what kind of application is running

Data in the database

Filesystem paths which aren't abstracted

Static classes

13. Writing testable code and other unit testing tips

Functions which use `time()` rather than `\XF::$time`