,

Managing a Project's Virtualenvs with tox

A comprehensive beginner's introduction to tox.

Tox is a great tool for standardising and automating any development task that benefits from being run in an isolated virtualenv. But Tox suffers from an educational problem: Tox appears to be just for running tests, and that’s all that many projects use it for. It can also be unclear how best to use the tox.ini file format at first. This tutorial aims to clear up the confusion and teach you how to automate a project’s development tasks with Tox. It covers everything you need to know to understand the real hypothesis/h tox.ini file.

Why Tox is confusing at first

Tox can appear to be just a tool for running tests, rather than for development tasks in general:

  • Tox was originally for running your tests in an isolated virtual environment, but has grown into a more general project virtualenv / command management tool.

  • For many developers their first introduction to Tox will be when they come across it being used in some project to run the tests, while the project confusingly uses manually-created virtualenvs for other things that Tox could be used for, like building the documentation and running the development server.
  • Tox’s own documentation is unclear on the scope of Tox too, stating Tox’s vision as to automate and standardize testing in Python. But the same docs later describe Tox as a generic virtualenv management and test command line tool and include examples of using Tox to build documentation and to run development environments. Tox’s GitHub project’s description is Command line driven CI frontend and development task automation tool.
  • Internal terminology used by Tox also suggests that it’s just for testing: Tox refers to the virtualenvs that it creates as testenvs. We’ll be using both testenv and virtualenv in this guide too. Eventually we’ll get around to explaining exactly what a testenv is relative to a virtualenv.

In addition:

  • The tox.ini file format is just complicated enough that it requires some knowledge of how Tox works to understand how to write a good one.

  • The relationship between Tox and similar tools such as Docker, GNU Make, virtualenv, virtualenvwrapper, etc is unclear.

So what is Tox and what is it for?

Tox is a project virtualenv management and development task automation tool. You create a tox.ini file for your project, and in this file you define virtualenvs (Tox calls them testenvs) for all your project tasks that you want to standardize and automate with Tox: running the tests, running the development server, running the linters, building the documentation, publishing a release, etc. Here’s an example tox.ini file that’ll be explained in detail later:

[tox]
envlist   = tests
skipsdist = true

[testenv]
deps =
    tests: -r requirements.txt
    lint:  flake8
    docs:  sphinx-autobuild
commands =
    tests: pytest tests/
    lint:  flake8 src
    docs:  sphinx ...

You run Tox telling it which of your testenvs to run, e.g. tox -e docs to build the documentation or tox -e tests,lint to run the tests followed by the linter, and Tox creates a virtualenv, installs the dependencies that you’ve listed in tox.ini into it, and runs the commands that you’ve listed in tox.ini in it.

Tox’s workflow

More concretely, Tox is a tool that automates a certain workflow, as described in the System overview in the Tox docs:

  1. Read Tox config settings from a tox.ini file, command line options and environment variables.
  2. Optionally create a Python package (sdist) of your project. (We skip this step for Hypothesis projects.)
  3. Create a Python virtual environment using the version of Python selected in tox.ini.
  4. Install any Python dependencies listed in tox.ini (the deps setting) into the virtualenv. If an sdist of your project was created in step 2 install that too.
  5. Run the commands listed in tox.ini (the commands setting) in the virtualenv. By default fail if any of the commands exit with a non-zero exit code.
  6. Print out a report of the virtualenvs that were run and whether each succeeded or failed.

So Tox is really just a tool for creating virtualenvs and running commands in them, and any development task that can be automated by running commands in virtualenvs can be automated with Tox.

If you ask Tox to run more than one virtualenv at once it will loop repeating steps 3, 4 and 5 for each virtualenv, and finally report on them all in step 6.

If you run the same virtualenv again Tox will speed things up by reusing the previously created virtualenv. It’ll skip steps 3 and 4 and avoid unnecessarily recreating the virtualenv or reinstalling the dependencies unless you pass the tox --recreate option or unless Tox knows that the virtualenv needs to be recreated (for example because the deps in tox.ini have changed).

Benefits of using Tox

By defining all of your project’s tasks in a tox.ini file Tox simplifies things for developers. A developer only needs to install and run Tox, they don’t need to bother with creating and activating virtualenvs and installing dependencies themselves.

Tox also simplifies CI integration: CI scripts that just run Tox are much simpler than scripts that handle virtualenvs and dependencies themselves.

Tox also standardizes things, reducing differences between different development environments and between dev envs and CI and production. The tox.ini file defines exactly what dependencies to install into each virtualenv and Tox takes extra steps to isolate its virtualenvs from the outside system.

Tox can also reduce the chance of dependency conflicts by using a separate virtualenv for each task, rather than installing everything in one big virtualenv. The packages needed to build your documentation or run your linters don’t need to be installed in the virtualenv that runs your dev server.

Finally, Tox provides one place to define and document all of the available projects tasks – in the tox.ini file – with a simple and consistent command line interface for running those tasks.

I call Tox project virtualenv management because it uses a per-project configuration file, the tox.ini file, that you add to your project’s version control repository. This is different than other tools, such as virtualenvwrapper, that are more personal-use virtualenv managers. Your text editor is your personal text editing tool and text editor configuration or standardizing on one particular text editor isn’t usually part of a project. GNU Make on the other hand is a project build automation tool because its Makefile is tracked as part of the project. Tox is a project tool, like Make.

Tox’s original use-case

The original purpose of Tox was to run tests in isolated environments and (later) to automate running tests with multiple combinations of different versions of Python and different versions of different dependencies. It uses some more recent Tox features, but this example partial tox.ini file for Compressing a dependency matrix from Tox’s docs illustrates Tox’s originally intended use-case:

[tox]
envlist = py{27,34,36}-django{15,16}-{sqlite,mysql}

[testenv]
deps =
    django15: Django>=1.5,<1.6
    django16: Django>=1.6,<1.7
    py34-mysql: PyMySQL     ; use if both py34 and mysql are in an env name
    py27,py36: urllib3      ; use if any of py36 or py27 are in an env name
    py{27,36}-sqlite: mock  ; mocking sqlite in python 2.x

Running the command tox with this tox.ini file will generate the “cross product” of the different vetsions of Python, Django and SQLite or MySQL and run the project’s tests many times over: with Python 2.7, Django 15, and SQLite, then with Python 2.7, Django 15 and MySQL, then with Python 2.7, Django 16, and SQLite, and so on, for all combinations. More on this “generative envlist” feature below. The deps setting defines how the dependencies to be installed should change depending on the versions of Python and Django being tested and on SQLite or MySQL.

We don’t have a need at Hypothesis to test our code against many different versions like this. We’re writing a web application that gets deployed to a production environment that we control, with pinned versions of Python and all dependencies. We use Tox instead for its isolation and task automation features.

Where we’re going: our use-case for Tox

Here’s an excerpted version of Hypothesis’s tox.ini file. It defines a few different testenvs named dev (for running the dev server), tests (for running the tests), docs (for building the docs), etc. One testenv for each development task. And tells Tox what dependencies to install and what commands to run in each testenv. The rest of this tutorial will cover everything you need to understand this tox.ini file in detail.

[tox]
envlist = tests
skipsdist = true
requires = tox-pip-extensions
tox_pip_extensions_ext_venv_update = true

[testenv]
skip_install = true
deps =
    dev:   -r requirements-dev.in
    tests: -r requirements.txt
    lint:  flake8
    docs:  sphinx-autobuild
whitelist_externals =
    dev:   sh
commands =
    dev:   sh bin/hypothesis devserver
    tests: pytest -Werror tests/h/
    lint:  flake8 h
    docs:  sphinx-autobuild -BqT -b dirhtml -d {envtmpdir}/doctrees . {envtmpdir}/html

Installing Tox

Tox is like both GNU Make and your text editor in that you should install Tox system-wide and use that one single copy of Tox for each project that you work on. You don’t install Tox in your project’s virtualenv, or list it as a dependency in your project’s requirements files. Tox is the thing that creates your project virtualenvs and installs their dependencies for you.

On Ubuntu, just install Tox once system-wide and be done with it:

$ sudo pip install tox

Getting started: a minimal tox.ini file

Here’s the smallest possible tox.ini file that will run without crashing:

[tox]
skipsdist = true

If you save this as tox.ini in an empty directory and then run the command tox in that directory you should see something like this:

$ tox
python create: /home/seanh/Dropbox/Desktop/tox/.tox/python
python run-test-pre: PYTHONHASHSEED='3201412928'
__________________________________________________________________ summary ___________________________________________________________________
  python: commands succeeded
  congratulations :)
$

Tox created a virtualenv (at .tox/python) using the default version of Python on your system (Python 2.7 on my Ubuntu system). Since the tox.ini file didn’t list any dependencies or commands Tox didn’t install any dependencies or run any commands in the virtualenv. It just did nothing.

The skipsdist = true tells Tox not to try and build an sdist (source distribution) of your project. An sdist is a way of packaging up a Python project so that it can be installed on a system, published to https://pypi.org/, etc. Since our project at this point is an empty directory there’s nothing to package up. At Hypothesis we don’t build and publish installable Python packages anyway – we build web apps that we deploy to production environments – so we always use skipsdist = true.

Commands

Lets add a command to our tox.ini file using the commands setting. We’ll make it print out the version of Python being used:

[tox]
skipsdist = true

[testenv]
commands = python --version

Now if you run tox you’ll see the output of our command that it ran in the virtualenv:


$ tox
python run-test-pre: PYTHONHASHSEED='3697789845'
python runtests: commands[0] | python --version
Python 2.7.15rc1
__________________________________________________________________ summary ___________________________________________________________________
  python: commands succeeded
  congratulations :)
$

We’ve now split the tox.ini file into two sections [tox] and [testenv]. The [tox] section contains global settings that affect the global behavior of Tox itself, such as whether or not to build an sdist, the minimum version of Tox that this tox.ini file requires, etc.

The [testenv] section defines the testenv(s) that we want Tox to create for us: what versions of Python to use to create the testenvs, what dependencies to install into them, what environment variables, what commands to run in the testenvs, etc. We’re gonna cover the most important settings that you can put in the [testenv] section of your tox.ini file, starting with commands. For full documentation of all the available settings run tox --help-ini or see the docs.

External commands and whitelist_externals

One of the things that Tox does to isolate testenvs from the system is to prepend the virtuenv’s bin directory onto the PATH envvar in the subshell that the testenv’s commands are run in. You can see this by having Tox run a command that prints out the value of PATH:

[tox]
skipsdist = true

[testenv]
commands = sh -c "echo $PATH"
$ tox
python run-test-pre: PYTHONHASHSEED='3100919638'
python runtests: commands[0] | sh -c 'echo $PATH'
WARNING: test command found but not installed in testenv
  cmd: /bin/sh
  env: /home/seanh/Dropbox/Desktop/tox/.tox/python
Maybe you forgot to specify a dependency? See also the whitelist_externals envconfig setting.
/home/seanh/Dropbox/Desktop/tox/.tox/python/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
__________________________________________________________________ summary ___________________________________________________________________
  python: commands succeeded
  congratulations :)
$

The PATH envvar in the testenv’s subshell is my usual PATH envvar but with the .tox/python/bin dir prepended to the front. This is so that executables in the virtualenv’s bin are always chosen instead of any executables with the same name elsewhere on the system PATH. This is why when we’ve been putting python --version in our tox.ini file its been printing out the version of Python installed in the virtualenv instead of the system version of Python: it’s running the copy of python at .tox/python/bin/python. Similarly pip in a tox.ini file (or any file run by a tox.ini file) will be the virtualenv’s copy of pip, and the same goes for any executables that your package installs.

Notice that Tox printed out a test command found but not installed in testenv warning. This is because the tox.ini file is running the executable sh which is a system executable – there’s no sh in the virtualenv. By default Tox allows executables from the system PATH to be run if they aren’t shadowed by a virtualenv executable, but it prints this warning. We can silence it with the whitelist_externals setting, which is a list of allowed system executables:

[tox]
skipsdist = true

[testenv]
whitelist_externals = sh
commands = sh -c "echo $PATH"

Now Tox will run sh without a warning:

$ tox
python run-test-pre: PYTHONHASHSEED='3100919638'
python runtests: commands[0] | sh -c 'echo $PATH'
/home/seanh/Dropbox/Desktop/tox/.tox/python/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
__________________________________________________________________ summary ___________________________________________________________________
  python: commands succeeded
  congratulations :)
$

Running multiple commands in sequence

You can easily have Tox run a sequence of commands in order by using multiple commands in the commands setting, one command per line:

[tox]
skipsdist = true

[testenv]
whitelist_externals = echo
commands =
    echo This is the first command
    echo This is the second command
    echo This is the third command

Many settings in Tox can take a list of lines like this, including whitelist_externals, deps, passenv and setenv, etc.

Command exit codes and Tox’s exit code

If one of the commands exits with a non-zero exit code Tox will stop there and not continue on to the next commands. For example the UNIX command false exits with code 1. This tox.ini file will stop at false and not continue on to the final command:

[tox]
skipsdist = true

[testenv]
whitelist_externals =
    echo
    false
commands =
    echo This is the first command
    false
    echo This is the third command

Tox will move on to the next testenv and run that though, if multiple testenvs are given in envlist / -e / TOXENV setting.

When a command from any one of the testenvs run exits with nonzero then Tox itself will also exit with nonzero, whereas if all commands exited with zero Tox will exit with zero.

You can tell Tox to ignore the exit code of a command – continuing on to the next command even if that command exits with zero, and not failing the tox command itself if that command fails – by prepending the command with a -. This tox.ini file will run all three commands and exit with zero:

[tox]
skipsdist = true

[testenv]
whitelist_externals =
    echo
    false
commands =
    echo This is the first command
    -false
    echo This is the third command

Passing command line arguments to commands using posargs

You can use Tox’s posargs substitution to allow the user to pass command line arguments through to the command(s) that Tox is running. This is particularly useful when using Tox to run a test runner, such as pytest, that has many useful command line arguments. The user can run tox -- -x to run pytest with pytest’s -x option, for example.

Here’s a tox.ini file that installs and runs pytest and passes any command line args through:

[tox]
skipsdist = true

[testenv]
deps = pytest
commands = pytest {posargs} tests/

(Here we’re using the deps setting to tell Tox to install pytest into the virtualenv before running the command(s). More on deps later.)

If you just run tox this will have pytest run all the tests in your tests/ dir (it’ll run pytest tests/). If you want to pass any command line arguments to pytest you put them after a -- in the tox command. If no -- is given then {posargs} evaluates to the empty string. For example to have pytest print out all its command line arguments and options:

$ tox -- --help

You can also give a default value for posargs following a : within the {…}. When no -- is used on the command line the default value will be substituted in place of the {posargs:DEFAULT_VALUE}. For example this will run pytest tests/ by default but allow the user to narrow down what tests to run with a command like tox -- tests/foo/bar:

commands = pytest {posargs:tests/}

{posargs} can be used anywhere in in tox.ini and can even be used multiple times to pass the same positional arguments to multiple commands.

{posargs} is an example of a Tox substitution, more on substitution later.

envlist

So far Tox has just been creating a single virtualenv for us named python, using the system’s default version of Python. By adding the envlist setting we can get it to create virtualenvs using different versions of Python and to create and run multiple testenvs at once:

[tox]
skipsdist = true
envlist = py36

[testenv]
commands = python --version

If you run tox with this tox.ini file it’ll create a .tox/py36 virtualenv and run the commands using Python 3.6:

$ tox
py36 create: /home/seanh/Dropbox/Desktop/tox/.tox/py36
py36 run-test-pre: PYTHONHASHSEED='2938378655'
py36 runtests: commands[0] | python --version
Python 3.6.6
__________________________________________________________________ summary ___________________________________________________________________
  py36: commands succeeded
  congratulations :)
$

This requires Python 3.6 to be installed on the system where tox is run. You need Python 3.6 to create a Python 3.6 virtualenv. The envlist setting is a comma-separated list of testenvs that will be run when you run tox. If you want to run your commands in multiple versions of Python at once you can just give an envlist with multiple Pythons:

[tox]
skipsdist = true
envlist = py27,py36,py37

[testenv]
commands = python --version

tox will create py27, py36 and py37 virtualenvs in .tox and run your commands in each of them in turn (in my case the Python 3.7 testenv fails because I don’t have Python 3.7 on my system):

$ tox
py27 create: /home/seanh/Dropbox/Desktop/tox/.tox/py27
py27 run-test-pre: PYTHONHASHSEED='3564108623'
py27 runtests: commands[0] | python --version
Python 2.7.15rc1
py36 run-test-pre: PYTHONHASHSEED='3564108623'
py36 runtests: commands[0] | python --version
Python 3.6.6
py37 create: /home/seanh/Dropbox/Desktop/tox/.tox/py37
ERROR: InterpreterNotFound: python3.7
__________________________________________________________________ summary ___________________________________________________________________
  py27: commands succeeded
  py36: commands succeeded
ERROR:  py37: InterpreterNotFound: python3.7
$

The envlist setting is the default testenv(s) to run when tox is run with no -e argument. envlist can be overridden using the -e command line argument. tox -e py27,py36 will run the py27 and py36 testenvs regardless of what the envlist setting in tox.ini says.

You can also override envlist by setting the TOXENV environment variable, and you can set the envvsr TOX_SKIP_ENV to a regular expression and any testenvs that match the regex will be skipped.

py27, py36, etc are builtin “factors” that set the version of Python to use. These’ll be explained properly later, for now they’ll have to remain a little mysterious.

Dependencies

You can have Tox install the dependencies that your commands need into the virtualenvs before running the commands, using the deps setting in tox.ini. For example this tox.ini file will install pytest and all the requirements listed in your app’s requirements.txt file and then run pytest:

[tox]
skipsdist = true

[testenv]
deps =
    pytest
    -r requirements.txt
commands = pytest {posargs:tests/}

If the deps setting in your tox.ini file changes Tox will automatically recreate the virtualenv and reinstall the dependencies the next time you run it.

By default, though, Tox doesn’t detect changes to dependencies listed in files when things like -r requirements.txt are used. You would have to tell Tox to recreate the virtualenv (with the tox -r option) whenever requirements.txt changes. Fortunately the venv_update extension, part of the tox-pip-extensions package, automates this.

Environment variables

Another thing Tox does to help isolate testenvs from the system is that it removes most of your shell’s environment variables from the subshell that it runs the testenv commands in. You can see this by having a tox.ini file print out all the environment variables available in the testenv using the standard UNIX env command:

[tox]
skipsdist = true

[testenv]
whitelist_externals = env
commands = env

The output of running tox will be something like this:

$ tox
python run-test-pre: PYTHONHASHSEED='3478754311'
python runtests: commands[0] | env
LANG=en_GB.UTF-8
VIRTUAL_ENV=/home/seanh/Dropbox/Desktop/tox/.tox/python
TOX_WORK_DIR=/home/seanh/Dropbox/Desktop/tox/.tox
TOX_ENV_NAME=python
TOX_ENV_DIR=/home/seanh/Dropbox/Desktop/tox/.tox/python
PYTHONHASHSEED=3478754311
PATH=/home/seanh/Dropbox/Desktop/tox/.tox/python/bin:...
___________________________________ summary ____________________________________
  python: commands succeeded
  congratulations :)
$

A handful of environment variables are allowed to pass through by default, including PATH (with the virtuenv’s bin dir prepended). You can also see that Tox injects a few environment variables of its own that you can make use of, such as TOX_ENV_NAME (the name of the testenv being run).

If you want to pass more envvars from your shell through, for example to configure your application, you can do this using the passenv setting which is a list of additional envvars that can be set in the environment where tox is being run and will be passed in to the testenv where the commands are run:

[testenv]
passenv = 
    DATABASE_URL
    ELASTICSEARCH_URL
    PYTEST_ADDOPTS
…

You can also use setenv to set the values of additional envvars in the tox.ini file, rather than reading them from the outside environment:

[testenv]
setenv =
    DATABASE_URL = postgresql://postgres@localhost/postgres
    ELASTICSEARCH_URL = …
…

Substitutions

As well as being available to the commands that get run, environment variable values can also be used in the tox.ini file itself by using Tox substitution.

Anywhere in tox.ini (for example in a dependency, or in a command) you can use {env:ENVVAR_NAME} to substitute the value of an environment variable.

For example this tox.ini file allows you to set the command to be run by setting an environment variable named COMMAND. Test it out with a command like env COMMAND="echo hi" tox:

[tox]
skipsdist = true

[testenv]
passenv = COMMAND
commands = {env:COMMAND}

If no COMMAND envvar is present Tox will exit with an error. You can provide a default value for this case using a second ::

commands = {env:COMMAND:pytest}

Substitutions can be nested, which can allow one envvar to be used as a fallback for another. This will read the command from the COMMAND envvar or, failing that, from FALLBACK_COMMAND, or finally default to pytest if neither envvar is set:

commands = {env:COMMAND:{env:FALLBACK_COMMAND:pytest}}

As well as environment variables, Tox provides a bunch of builtin variables that can be used in substitution. For example {envtmpdir} is the path to the virtuenv’s temporary directory (which is automatically deleted after each run of tox).

This tox.ini file uses Sphinx to build a project’s documentation (located in the docs dir) and serve it locally for previewing. It uses {envtmpdir} as the directory for Sphinx’s output files (the built documentation HTML files) to avoid messing up the user’s working directory, and so that those files are automatically cleaned up each run:

[tox]
skipsdist = true

[testenv]
skip_install = true
deps =
    sphinx-autobuild
    sphinx
    sphinx_rtd_theme
changedir = docs
commands =
    sphinx-autobuild -BqT -b dirhtml -d {envtmpdir}/doctrees . {envtmpdir}/html

Factors and conditionals

Tox’s factors and the conditional settings they enable are one of the most useful and flexible features of Tox. They’re the basis of our tox.ini file at Hypothesis. But to understand them you have to grock a few Tox concepts first:

Testenvs

A Tox testenv is a bunch of settings that define how to create and run a virtuenv: what version of Python to use, what dependencies to install, what environment variables to set and to pass through, what commands to run in the venv once it's ready, etc.

Whenever you run tox you're running one or more testenvs (the testenvs listed in the envlist setting, -e command line argument, or TOXENV environment variable).

Testenv names are lists of factors
A testenv name is a minus-separated list of factors. A testenv named py27 contains a single factor, py27. A more complex testenv name like py27-tests contains two factors py27 and tests.
Factors

A factor is a collection of testenv settings. A testenv like py27-tests pulls in all of the settings from the py27 factor and the tests factor, and those settings combined become the testenv's settings.

py27, py36 etc are builtin factors. Tox comes with builtin factors for all the versions of Python, and you can just use these in your tox.ini files. These are factors that contain a single setting: the version of Python to use to create the virtualenv with (the basepython setting).

You can also define custom factors in your tox.ini file. test in the testenv name py27-test is a custom factor.

Conditionals

Whenever you give a list of things in your tox.ini file (for example the lists of environment variables to pass through, dependencies to install, and commands to run) items in the list can be made conditional on a factor by prefixing them with factorname: . For example in this tox.ini snippet:

[testenv]
deps = 
    tests: pytest

the tests: pytest implicitly creates a custom factor named tests and tells Tox to install the pytest dependency only when the tests factor is invoked. tox -e py27-tests will install pytest but tox -e py27-docs won't.

Below is a simplified version of the Hypothesis tox.ini file that uses factors and conditionals to define separate testenvs for running the dev server, running the linter, running the tests, and building the docs. This enables the following tox commands:

  • tox runs the tests in Python 2.7 (py27-tests is the default envlist)

  • You can run the tests in any version of Python (as long as you have that version of Python installed on your system) by using different testenv names: tox -e py27-tests runs the tests in Python 2.7, whereas tox -e py36-tests or tox -e py37-tests run the tests in Python 3.6 or 3.7.

  • tox -e py27-dev runs the dev server in Python 2.7. As with the tests you can run the dev server in any version of Python using commands like tox -e py36-dev.

  • tox -e py36-lint runs the linter, and tox -e py36-docs builds the docs and serves them locally.

As you can see the tox.ini file is designed to enable pyXY-COMMAND testenvs, where pyXY is any version of Python and COMMAND is any of the provided commands (tests, dev, lint, etc). It can be extended by adding as many commands as you like, automating all of your development tasks.

Here’s the tox.ini file:

[tox]
envlist = py27-tests
skipsdist = true
requires = tox-pip-extensions
tox_pip_extensions_ext_venv_update = true

[testenv]
skip_install = true
passenv =
    dev:   AUTHORITY
    dev:   BOUNCER_URL
    dev:   CLIENT_OAUTH_ID
    dev:   CLIENT_RPC_ALLOWED_ORIGINS
    dev:   CLIENT_URL
    dev:   SENTRY_DSN
    dev:   USE_HTTPS
    dev:   WEBSOCKET_URL
    tests: TEST_DATABASE_URL
    tests: ELASTICSEARCH_URL
    tests: PYTEST_ADDOPTS
deps =
    dev:   ipython
    dev:   ipdb
    dev:   -r requirements-dev.in
    lint:  flake8
    lint:  flake8-future-import
    tests: coverage
    tests: pytest
    tests: factory-boy
    tests: mock
    tests: hypothesis
    tests: -r requirements.txt
    docs:  sphinx-autobuild
    docs:  sphinx
    docs:  sphinx_rtd_theme
whitelist_externals = dev: sh
changedir = docs: docs
commands =
    python --version
    dev:   sh bin/hypothesis --dev init
    dev:   sh bin/hypothesis devserver {posargs}
    lint:  flake8 src tests
    tests: pytest {posargs:tests/h/}
    docs:  sphinx-autobuild -BqT -b dirhtml -d {envtmpdir}/doctrees . {envtmpdir}/html

To explain how this works, consider how Tox treats the commands section of the tox.ini file when you run tox -e py36-dev:

commands =
    python --version
    dev:   sh bin/hypothesis --dev init
    dev:   sh bin/hypothesis devserver {posargs}
    lint:  flake8 src tests
    tests: pytest {posargs:tests/h/}
    docs:  sphinx-autobuild -BqT -b dirhtml -d {envtmpdir}/doctrees . {envtmpdir}/html

Tox reads the commands list from top to bottom and runs only the commands that match the testenv name. python --version is an unconditional command (no factor: prefix), so that always gets run. The two dev: commands match the dev factor in py36-dev so they get run. The other commands conditions don’t match so they’re ignored. The matching commands are run in the order that they appear in the file.

This same matching procedure is applied to all lists in the tox.ini file: passenv, deps, etc. When you run tox -e py36-dev Tox first collects all the settings in the builtin py36 factor (the Python version: 3.6), then it parses the tox.ini file and collects all settings that are either unconditional or whose condition matches one of the factors py36 or dev. These collected settings form the testenv definition, and Tox runs it.

More complex conditions are also possible. The most useful of these is to reduce duplication by making a single line match multiple factors by giving a comma-separated list of factors in {…}’s. This will install requirements.txt if either the tests or the dev factor is invoked:

deps =
    {tests,dev}: -r requirements.txt

Negation, logic, etc are also possible. See Complex factor conditions in the Tox docs for details.

Debugging your tox.ini file with --showconfig

Once you’ve started adding conditionals to your tox.ini file it can be useful to have a good way to debug it. Given a tox.ini file like the one above, and a command like tox -e py27-dev, adding the --showconfig option will get Tox to print out the entire collected py27-dev testenv definition instead of running the testenv’s commands:

$ tox -e py27-dev --showconfig
tool-versions: tox-3.5.2 virtualenv-16.0.0
config-file: 
toxinipath: /home/seanh/Projects/h/tox.ini
toxinidir:  /home/seanh/Projects/h
toxworkdir: /home/seanh/Projects/h/.tox
setupdir:   /home/seanh/Projects/h
distshare:  /home/seanh/.tox/distshare
skipsdist:  True

[testenv:py27-dev]
  envdir          = /home/seanh/Projects/h/.tox/py27-dev
  setenv          = SetenvDict: {'TOX_ENV_NAME': 'py27-dev', 'TOX_ENV_DIR': '/home/seanh/Projects/h/.tox/py27-dev', 'PYTHONHASHSEED': '2540689914'}
  basepython      = python2.7
  description     = 
  envtmpdir       = /home/seanh/Projects/h/.tox/py27-dev/tmp
  envlogdir       = /home/seanh/Projects/h/.tox/py27-dev/log
  downloadcache   = None
  changedir       = /home/seanh/Projects/h
  args_are_paths  = True
  skip_install    = True
  ignore_errors   = False
  recreate        = False
  passenv         = set(['LANG', 'PIP_INDEX_URL', 'LANGUAGE', 'TOX_WORK_DIR', 'SENTRY_DSN', 'CLIENT_URL', 'AUTHORITY', 'CLIENT_RPC_ALLOWED_ORIGINS', 'WEBSOCKET_URL', 'USE_HTTPS', 'BOUNCER_URL', 'CLIENT_OAUTH_ID', 'PATH', 'LD_LIBRARY_PATH', 'TMPDIR'])
  whitelist_externals = ['sh']
  platform        = .*
  sitepackages    = False
  alwayscopy      = False
  pip_pre         = False
  usedevelop      = False
  install_command = [u'pip-faster', u'install', u'{opts}', u'{packages}', 'venv-update>=2.1.3']
  list_dependencies_command = ['python', '-m', 'pip', 'freeze']
  deps            = [ipython, ipdb, -rrequirements-dev.in]
  commands        = [['python', '--version'], ['sh', 'bin/hypothesis', '--dev', 'init'], ['sh', 'bin/hypothesis', 'devserver']]
  commands_pre    = []
  commands_post   = []
  ignore_outcome  = False
  extras          = []

Generative envlists

So far we’ve mostly ran one or two testenvs at a time with commands like tox -e py36-tests or default envlist settings like envlist = py27-tests,py36-tests. It isn’t much use to us at Hypothesis but for completeness sake Tox also has a Generative envlist feature that allows you to run many testenvs at once without having to spell out every single testenv name. This is useful for projects that need to support a lot of combinations of different versions of Python, different versions of dependencies, and different platforms at once. A single tox command can run your tests in with all the different combinations. Here’s an example from the docs:

envlist = {py27,py36}-django{15,16}, docs, flake

With this envlist setting a single tox command will generate the “cross product” of the different versions of Python and Django and run each possible combination. It will run all of these testenvs:

  • py27-django15
  • py27-django16
  • py36-django15
  • py36-django16
  • docs
  • flake

Defining testenvs using [testenv:NAME] sections

So far we’ve had a single [testenv] section in our tox.ini and we’ve used conditionals to define multiple testenvs. There’s also an entirely different way to define your testenvs in your tox.ini: using a separate [testenv:NAME] section for each testenv. Here’s an example of a tox.ini file that supports tox or tox -e py27 to run the tests in Python 2.7, and tox -e py27-dev and tox -e py36-dev to run the dev server in Python 2.7 or 3.6:

[tox]
envlist = py27-tests
skipsdist = true

[testenv]
skip_install = true

[testenv:py27-tests]
description = Run the tests in Python 2.7
deps =
    coverage
    mock
    …
    -rrequirements.txt
passenv =
    TEST_DATABASE_URL
    ELASTICSEARCH_URL
    PYTEST_ADDOPTS
commands =
    coverage run --parallel --source h,tests/h -m pytest -Werror {posargs:tests/h/}

[dev]
deps = -rrequirements-dev.in
passenv =
    ALLOWED_ORIGINS
    AUTHORITY
    BOUNCER_URL
    …
whitelist_externals = sh
commands =
    sh bin/hypothesis --dev init
    {posargs:sh bin/hypothesis devserver}

[testenv:py27-dev]
description = Run the dev server in Python 2.7
deps = {[dev]deps}
passenv = {[dev]passenv}
whitelist_externals = {[dev]whitelist_externals}
commands = {[dev]commands}

[testenv:py36-dev]
description = Run the dev server in Python 3.6
deps = {[dev]deps}
passenv = {[dev]passenv}
whitelist_externals = {[dev]whitelist_externals}
commands = {[dev]commands}

Any settings in the [testenv] section apply to all testenvs, unless overridden. You then define each testenv in its own section, for example the [testenv:py27-tests] section contains the settings for the py27-tests env (tox -e py27-tests).

The [dev] section doesn’t define a testenv and isn’t used by Tox. It’s just a bag of settings to be used in substitution by the [testenv:py27-dev] and [testenv:py36-dev] sections below. For example when [testenv:py27-dev] does deps = {[dev]deps} that means pull in all the deps from the [dev] section. You can also pull in some deps and then add more:

[testenv:py27-dev]
deps =
    {[dev]deps}
    ipython
    ipdb

These are called section substitutions. Using substitutions like this is a way to avoid duplication. It’s almost like inheritance, where [testenv:py27-dev] and [testenv:py36-dev] are child sections and [dev] is the base section they inherit from. But there’s no way to automatically inherit all the settings from a base section at once: you have to inherit each section individually with a substitution.

One advantage of writing your tox.ini file this way is that you can add a description to each testenv, and if you run tox -av it’ll list all the available testenvs and their descriptions:

$ tox -av
using tox.ini: /home/seanh/Projects/h/tox.ini
using tox-3.5.2 from /usr/local/lib/python2.7/dist-packages/tox/__init__.pyc
default environments:
py27-tests      -> Run the tests in Python 2.7

additional environments:
py27-dev        -> Run the dev server in Python 2.7
py36-dev        -> Run the dev server in Python 3.6

Overall, though, we found this way of writing tox.ini files produced much larger files that contained a lot more boilerplate and duplication, and were more difficult to read and to work with. The file above only supports running the dev server in Pythons 2.7 and 3.6, for example. To run it in Python 3.7 you’d have to add a new [testenv:py37-dev] section.

The factors and conditionals approach produces a much smaller tox.ini file while also enabling any command to be run with any version of Python.

You can also combine both approaches in a single tox.ini.

Grab bag

Some other cool stuff that Tox can do:

  • Interactive shell substitution. You can use substitutions like {tty:ON_VALUE:OFF_VALUE} to have ON_VALUE be used when Tox is run in an interactive shell (e.g. on a developer’s laptop) and OFF_VALUE be used when its run non-interactively (e.g. on CI). You could use this, for example, to pass the --pdb argument to pytest to make it drop in to a debugger prompt when a test fails, but only when Tox is being run interactively.

  • Jenkins support. You can add a [tox:jenkins] section containing global settings overrides that should be used only when Tox is running in Jenkins. See Jenkins override.

  • Force a particular version of a dependency just once with --force-dep.

All posts tagged “tox”:
All posts tagged “Hypothesis”:
New View Source Edit
×