netresearch / kite
Yet another build and deployment tool - inspired by TYPO3.Surf
Requires
- php: >=5.4.0
- symfony/console: ~2.7
- symfony/expression-language: ~2.7
- symfony/process: ~2.7
README
Kite: Make your projects fly
Kite is a build and automation tool inspired by TYPO3.Surf, written in PHP and utilizing PHP for configuration. It's...
- easy to use
- Kite ships with several preconfigured tasks, workflows and presets - just use a preset and be done.
- Jobs and workflows are available on the command line and there's also
--help
available for each of them.
- flexible
- The configuration can be completely done by arrays, fiddling your tasks together or by workflow classes that have all the tasks available as methods or both.
- Jobs, workflows and tasks can easily be reused at any point - new jobs can be composed of any of them.
- The variable system provides a JavaScript like variable inheritance: Sub tasks can access variables from parent but can set them on their own as well
- Jobs and workflows can expose variables as command line options and arguments.
- Advanced logic during execution possible by using Symfony Expression Language
- node based
- Unlimited number of remote targets possible
- Nodes can be set globally or only for specific (sub) tasks
- Remote tasks operate on all current nodes
- safe
- Everything can be
--dry-run
to preview what happens (yet the tasks to include need to be configured) - The complete debug output of previous tasks can be viewed with
kite log
- Everything can be
Contents
Appendix
Installation
You can install kite globally (recommended) or per project
Prerequesites
Locally
- PHP 5.4+ (for the composer installation, the application can have lower versions)
- Linux shell (windows might work but is untested)
- SSH/SCP installed (for working on nodes)
- git installed (if you use git tasks)
- composer installed (if you use composer tasks)
Remote
- PHP
- SSH access
Global installation
composer global require "netresearch/kite" ~/.composer/vendor/bin/kite -V
Per project installation
cd /var/www/project composer require "netresearch/kite" vendor/bin/kite -V
Configuration
Concepts
- Tasks
- Smallest, predefined steps
- See the task reference for tasks shipped with kite
- Workflows
- Special kind of task that allows to composer it's subtasks in a class
- Top level workflows can expose command line arguments and options
- See the workflow reference for workflows shipped with kite
- Jobs
- Outermost kind of task
- Available as commands on command line
- Set of tasks and/or workflows defined in arrays (in arbitrary depth)
- Configurable command line arguments and options
- Presets
- Configuration presets (including f.i. common jobs)
- Configuration file (typo3conf/Kite.php, app/etc/kite.php, kite.php)
- Defines the jobs; can load and override presets
Variables
The fact that all of the configured tasks are to be ran automated, introduces the need for a variable system that allows you to read from dynamic configurations or change it. Kite provides a basic syntax to access those variables from within strings (all options of tasks, nodes etc.):
Each string inside curly braces inside an option string are evaluated as Symfony Expression Language expressions - f.i.
<?php $this['messages'] = (object) ['default' => 'Hello world']; $this['jobs']['echo'] = [ 'description' => 'Output the default message from (\{config["messages"].default\})', 'task' => [ 'type' => 'output', 'message' => '{config["messages"].default}' ] ];
As you see above, by quoting the braces, you can avoid that the expression is evaluated. Please see the Symfony Expression Language Syntax for help on how to use the expressions.
Variable scopes
The variable scopes are very similar to those in JavaScript. This means that you can
access all variables from the parent scope within the current scope unless you have
a variable in the current scope that's name is the same. To disambiguate you can use
the special variables this
and parent
.
Task or workflow options are always bound to the scope of the task
This means, that they have to explicitly be set for the task or workflow and can not be read
from parent tasks (like jobs or workflows). However sub tasks of those tasks can
access those options without prefix when they don't have an option with the same
name or with parent
prefix otherwise.
Global variables
Additionally to the options from the current and parent tasks there are some global variables available:
job
- The current job object (instance of
\Netresearch\Kite\Job
)
- The current job object (instance of
kite
- Object with some information about the kite package (
path
and relativedir
)
- Object with some information about the kite package (
config
- The config array object (as in configuration file)
composer
- Composer service object providing keys
packages
androotPackage
- Composer service object providing keys
Special variables
this
- By using
this
you can force the variable to be not looked up in the parent scopes - but only within the current.
- By using
parent
- Points to the parent object
Available functions
Kite ships with the following expression language functions:
isset(variable)
andempty(variable)
- Behave just like their PHP equivalents. Only available for variable objects, such as
tasks
,nodes
,workflows
orjobs
and their objects (f.i. not for configuration arrays)
get(variableName, variableValue)
andset(variableName, variableValue)
- Get or set the variables (f.i.
set('job.owner', node.user)
- Get or set the variables (f.i.
answer(question)
- Let the (command line) user answer a question and return the result
answer(question)
- Let the (command line) user answer a confirmation question and return the result
select(question, options)
- Let the (command line) select from an array of options
- any PHP function
- Lets you call PHP functions as you are used to in PHP - e.g.
str_replace('\\\\', '/', config['somePath'])
(Note the four back slashes which are required to pass a single escaped backslash to Expression Language)
- Lets you call PHP functions as you are used to in PHP - e.g.
Kite configuration file
You need a file called "Kite.php" to set up config (where to deploy).
For TYPO3 projects it should be placed here: typo3conf/Kite.php
,
for Magento app/etc/kite.php
and for all other applications just kite.php
.
A basic example could be
<?php // Example for a project without a staging environment // This loads configuration with common jobs $this->loadPreset('common'); // This configuration is loaded on execution of deploy or rollout job $this['stages']['staging']['node'] = array( 'host' => 'set host here', 'deployPath' => 'set path on host here', 'webUrl' => 'set url here', 'php' => 'php56', ); // no staging is available unset($this['stages']['production']); ?>
Jobs
Jobs are to be configured in the key jobs
in the configuration. They can contain
a single task
, an array of tasks
or a workflow
(always only one of them).
<?php // Job, running a single task $this['jobs']['echo'] = [ 'description' => 'Output a message', 'arguments' => [ 'message' => [ 'type' => 'string', 'required' => true, 'label' => 'The message to output' ] ], 'task' => [ 'type' => 'output', 'message' => '{job.message}' ] ]; // Job, running a workflow $this['jobs']['diagnose'] = [ 'description' => 'Show status of packages', 'workflow' => 'Netresearch\Kite\Workflow\Composer\Diagnose' // can written as follows also: // 'workflow' => 'composer-diagnose' ];
Nodes
Whenever you set a key named node
or nodes
on a job, workflow or task
it's value will be mapped to an aggregate of node models. Those models have the
following default configuration:
<?php array( 'user' => '', 'pass' => '', 'port' => '', 'url' => '{(this.user ? this.user ~ "@" : "") ~ this.host}', // SCP/SSH URL 'sshOptions' => ' -A{this.port ? " -p " ~ this.port : ""}{this.pass ? " -o PubkeyAuthentication=no" : ""}', 'scpOptions' => '{this.port ? " -P " ~ this.port : ""}{this.pass ? " -o PubkeyAuthentication=no" : ""}', 'php' => 'php', // PHP executable 'webRoot' => '{this.deployPath}/current', // No default values, required to be set: // 'webUrl' => 'http://example.com', // 'host' => 'example.com', // 'deployPath' => '/var/www' );
Development packages
The default kite jobs checkout
and merge
work only on packages which are white listed as development packages. As deploy
and rollout
utilize those jobs they stick to this behaviour as well.
You can white list packages in three ways: by package name, by git remote url or by package path. When none of them is given all git packages will be used as development packages. You can change the white lists in your Kite configuration file:
<?php // The following whitelist types are available (evaluated by OR) // ... for the package names $this['composer']['whitelistNames'] = 'netresearch/.*'; // ... for the git remote urls $this['composer']['whitelistRemotes'] = 'git@github.com:netresearch/.*'; // ... for the package paths $this['composer']['whitelistPaths'] = 'vendor/netresearch/.*';
The default configuration is as follows:
<?php $this['composer']['whitelistNames'] = $this['composer']['whitelistPaths'] = null; $this['composer']['whitelistRemotes'] = '{preg_quote(preg_replace("#/[^/]+$#s", "", composer.rootPackage.remote), "#")}/.+';
This means that only packages which share the part before the last occurrence of / with the application remote will be considered as development packages (f.i. if your application git remote is https://github.com/netresearch/kite.git
then only packages with a remote like https://github.com/netresearch/*
will match).
Deployment configuration
Stages
As you saw in the example in Kite configuration file, there is a top level configuration
element named stages
. They are set by the common
preset and hold configuration
only used for each of it's keys (such as staging
and production
by default). They
are evaluated by workflows based on the stage-select
workflow, which takes the
stage(s) to use from either command line or a select question. After a stage was
selected ALL of it's values are set to the corresponding task (such as deploy
).
The stages have no special meaning and are not handled in a special way - they only
play together with the stage based tasks (deploy
and rollout
from the common
preset and ccr
from the typo3
preset) because those are configured so.
Deployment
The deployment
workflow deploys your application to exactly one stage (whereas the
rollout
just runs the deployment
workflow for each until the selected stage).
Thereby it does the following steps:
Run
kite composer diagnose
to assert that your application is at a defined state (nothing uncommited, unpushed, unpulled, lock file up to date etc.)- Run
composer checkout
with the parameters you provided for the stage: branch
- The branch to checkout. Incommon
preset they are configured as follows:<?php $this['stages'] = [ 'staging' => [ 'branch' => '{replace("/", "-", composer.rootPackage.name)}-staging', 'merge' => true, 'createBranch' => '{config["composer"]["whitelistNames"] || config["composer"]["whitelistRemotes"] || config["composer"]["whitelistPaths"]}' // add nodes or node in your config ], 'production' => [ 'branch' => 'master', // add nodes or node in your config ] ];
merge
- Whether to merge the currently checked out branch into the branch to checkoutcreateBranch
- Whether to create the branch if it doesn't exist. This is by default set to true for the staging stage, when Development packages are white listed (which are by default).rsync
- configuration for rsync task invoked (f.i. withexcludes
option)
- Run
Creates a new release from the current release on each
node
{deployPath}/releases
Rsync the current local state to the new release dir on each
node
Symlink shared directories and files (shared means shared between the releases) - the shared directories and files are expected to be at
{deployPath}/shared
. They can be configured as seen in the typo3 preset:<?php $this->merge( $this['jobs']['deploy']['task'], [ 'shared' => [ 'dirs' => ['fileadmin', 'uploads', 'typo3temp'] ] ] );
To illustrate the behaviour of the stage configuration here's an example setting the shared directories differently for each
stage
:<?php $this->merge( $this['stages'], [ 'staging' => [ 'shared' => [ 'dirs' => ['shared_dir_1', 'shared_dir_2'], 'files' => ['file1', 'file2'] ] ], 'production' => [ 'shared' => [ 'dirs' => '{config["stages"]["staging"]["shared"]["dirs"]}', 'file' => 'file' ] ] ] );
Switch the previous release pointer (
{deploypath}/previous
) to the current release.Switch the current release pointer (
{deploypath}/current
) to the new release.Invoke the
onReady
task if any. F.i.:<?php $this->merge( $this['jobs']['deploy']['task'], [ 'onReady' => [ 'type' => 'shell', 'command' => 'mail', 'optArg' => ['s' => 'Deployed to {stage}', 'user@example.com'] ], ] );
And to once again demonstrate that each of the
stages
can override any option on the deployment workflow:<?php $this->merge( $this['stages']['production'], [ 'onReady' => [ 'type' => 'shell', 'command' => 'mail', 'optArg' => ['s' => '[IMPORTANT] Deployed to {stage}', 'user@example.com'] ], ] );
When you invoke the deployment
or rollout
jobs with the rollback (--rollback
or -r
)
option, it
- switches the next release pointer (
{deploypath}/next
) to the current release - switches the current release pointer (
{deploypath}/current
) to the previous release - invokes the
onReady
task if any.
When you invoke the deployment
or rollout
jobs with the rollback (--activate
or -a
)
option, it invokes the last three steps of the deployment (switch symlinks, and invoke onReady
).
Usage
As stated above, all jobs are available as kite sub commands (kite job-name
). You can list the available commands by running
kite [list]
By running
kite help command #or kite command --help
you can show help for a specific job/command.
Common commands
kite [help [command]]
- Gives a list of all available commands (jobs) or shows help for the given one
kite --workflow=<workflow-name-or-class>
- Runs a workflow class without requiring it to be inside a job
kite --workflow=<workflow-name-or-class> --help
- Shows the docs (php class doc), arguments and options for a workflow
kite log [-l]
- Shows the last (default), specific (use with caret like ^2 shows the 2nd least
log or with a timestamp from
kite log -l
) or a list of the available log records
- Shows the last (default), specific (use with caret like ^2 shows the 2nd least
log or with a timestamp from
Common jobs
update [branch]
- Checks out
branch
when given - Runs git pull followed by composer update
- Checks out
kite checkout [--merge] branch
- Goes through all Development packages and checks out the branch there if it’s available
- After checking out the branch on a package it goes through all Development packages requiring it and updates the version constraint to that branch
- When
--merge
is passed, the currently checked out branch is merged into the branch to checkout - When
-c
is passed, the branch will be created in all Development packages when it doesn't exist there (you can use-p
option to limit the operation to certain packages)
kite merge [--squash] [--message=”Message”] branch
- Goes through all composer packages and merges the branch into the currently checked out
kite package-foreach [--git] command
- Runs a command for each composer package (optionally only
--git
packages)
- Runs a command for each composer package (optionally only
kite cc, kite ccr [stage]
- Clears caches locally (cc) or on all nodes of a specific stage (contained in TYPO3 preset only for now)
Development workflow
The default kite jobs are designed to ease the development of composer based applications which tends to be pretty complicated when it comes to working with development branches across several packages. What kite can do for you, is to manage the branches in the Development packages along with dependencies on them.
In the following you'll find the kite commands for the common workflow that you do your development within feature (eventually derived from topic branches) or bug fix branches, which you will merge into the master as soon as their changes were reviewed and tested.
Ensure application is on master and up to date
kite update master
Create the feature branch in the package(s) you want
This will create the FEATURE branch in package vendor/branch and all installed dependent packages and make them require it in
dev-FEATURE
(as your application directly or indirectly requires the package it will be checked out in FEATURE branch as well).kite checkout -c -p vendor/package FEATURE
Develop
cd vendor/package echo '<?php phpinfo(); ?>' > i.php git add i.php git commit -m 'Added php info' git push cd ../..
(Work on another existing branch)
kite checkout ANOTHER_FEATURE
Deploy FEATURE branch to staging
kite checkout FEATURE kite deploy staging
(Work on another existing branch)
kite checkout ANOTHER_FEATURE
FEATURE was reviewed, merge it into master and delete it
kite checkout master kite merge --delete FEATURE
Roll out master (now including FEATURE) over staging to production
kite rollout production
Deployment jobs
kite deploy [stage]
- Runs the deployment for all nodes on the given or selected stage
kite rollout [stage]
- Runs the deployment for all nodes for each stage until (including) the given stage
Use public key authentication
To prevent you to have to type your password several times during deployment you should set your public key on your server. Usually this is located here: "~/.ssh/authorized_keys".
Trouble shooting
Every task that's executed including it's output will be logged to a log file inside
your home directory. This includes f.i. each command ran on the local and remote shells,
their output, debug messages and a lot more. Basically it holds the output, you would
get by adding -vvv
to your kite command.
Just run kite log
when a job failed and you want to know exactly what went wrong.