Basic pytest Fixtures

This post covers the basics of pytest fixtures and how we use them at Hypothesis.

A “test fixture” is some code that needs to be run in order to set things up for a test. When we created the objects that we needed to pass into the method we were testing in the arrange part of the arrange, act, assert recipe, those were fixtures. Test factories are helpers for easily creating fixtures. Fixtures don’t have to be objects that we pass into the method we’re testing as arguments, they can be anything that needs to be set up for a test. For example if we want to test what a method does when there are three documents in the database, setting up the db with those three documents is a fixture.

Setup and teardown methods

Most unit testing frameworks, for example the standard unittest library that ships with Python, use the familiar setup() and teardown() methods to set up all the necessary fixtures before running a test method and, if necessary, tear things down after running the test (before running the next test). The test code might look something like this:

class MyTestClass(object):
    def setup(self):
        self.fixture_object = SomeClass()

    def test_something(self):
        # Some test code that uses self.fixture_object

The test framework runs the setup() method once before it runs each test method (and the teardown() method, if there is one, gets run once after each test).

The drawbacks of setup() methods begin to show when a test class contains different test methods that use different fixture objects or setup:

class MyTestClass(object):
    def setup(self):
        self.first_fixture_object  = SomeClass()
        self.second_fixture_object = SomeOtherClass()

    def test_something(self):
        # Some test code that uses self.first_fixture_object

    def test_something_else(self):
        # Some test code that uses self.second_fixture_object

    ...

Some of the test methods only need first_fixture_object, and some only need second_fixture_object, but both fixture objects get created for each test. In a small example this isn’t important but over a large test base unnecessarily running a lot of unused fixtures for a lot of tests can start to add significantly to how long it takes to run the tests every time any developer needs to do so.

As they get bigger and more complicated tests like this can also become hard to read. A whole lot of setup is done in the setup() method, a lot of which may be depended upon by the test method that you’re currently trying to understand, and a lot of it not. You have to scan the whole test method body looking for all the self.somethings to see what fixtures this test depends on, and then look in setup() to see what each of these fixtures is.

Worse, if a fixture sets up something in the environment, for example adding some objects to the test database, then a test method may be depending on that fixture (assuming that certain objects exist in the db) without explicitly referencing that fixture in the test body at all.

It can quickly get to the point where setup() code can be difficult to change without potentially breaking multiple test methods for unclear reasons, and where it can become hard to tell exactly what a given test method is really testing.

The problem gets worse when different test fixtures conflict with each other. This can happen when there’s a singleton resource such as a database. If one test needs to test what happens when there are no users in the database, and another needs to test what happens when there are three users in the database, well a setup() method can’t setup the database with zero users in it and also setup the same database with three users in it as the same time. Hacking around this kind of issue can lead to hard to read code and some nasty coupling between test methods.

One solution would be to break the methods up into multiple test classes. You think of each test class as its own fixture. If some tests need no users and one group in the db then that’s one fixture and one test class, if some other tests need to test what happens when there’s three users and no groups in the db then that’s another class:

class TestWithNoUsersAndOneGroup(object):
    def setup(self):
        # Add one group to the test db.
        ...

    def test_something(self):
        ...

    def test_something_else(self):
        ...

class TestWithThreeUsersAndNoGroup(object):
    def setup(self):
        # Add three users to the test db.
        ...

    def test_another_thing(self):
        ...

You may end up with some code duplication between the different setup() methods, but that can always be moved into helper methods that each of the setup() methods calls.

Clearly coming up with good names for the test classes can get pretty awkward if the different fixtures that they represent become complex and varied.

In practice, test classes hardly ever get written like this. Most commonly you see test classes named things like TestFoo containing all of the tests for the Foo class (or perhaps the foo() method), and this is how test classes are mostly used in h as well - to group together the tests for a given unit of code that’s being tested.

Another alternative is to use test helper methods…

Test helper methods

The problems with setup() methods can be worked around by simply not using setup() methods or self, and instead using simple helper methods - non-test methods in the test classes or modules that are called by each test method as needed:

class MyTestClass(object):
    def add_one_group_to_the_db(self):
        # Code that adds one group to the db and returns the group.

    def add_three_users_to_the_db(self):
        # Code that adds three users to the db and returns the users/

    def test_something(self):
        group = add_one_group_to_the_db()
        ...

    def test_something_else(self):
        group = add_one_group_to_the_db()
        ...

    def test_another_thing(self):
        users = add_three_users_to_the_db()
        ...

This works just fine:

• Each test calls only the helper methods that test needs. No helper methods are run unnecessarily for any tests that don’t need them.

• It’s easy to look at a test and see exactly what setup is done for that test and what isn’t, no more implicit dependencies on things that are done or not done in a setup() method.

• The tests are independent from each other, modifying a test or adding a new test isn’t going to break existing tests. If a new test wants to change what a helper method does, but doing so would break existing tests, you can always just add a new helper method and call that instead.

But simple helper methods like this do have a couple of drawbacks:

• Each test needs to call each helper method that it needs, which clutters up the test bodies with a lot of repetitive, boilerplate code (this gets worse when helper methods require arguments to be passed to them).

• There isn’t a good way to do any tearing down of something that was set up by a call to a helper method (you wouldn’t want each test method to have to call corresponding teardown methods).

Pytest comes with a feature that it calls pytest fixtures that solve this problem very nicely. Pytest fixtures are like test helper methods on steroids…

pytest fixtures

pytest’s fixtures are its answer to setup() and teardown() methods (pytest actually supports setup() and teardown() as well but fixtures are better). Pytest fixtures are a form of dependency injection that allow you to do something very similar to what you can do with test helper methods but without the drawbacks.

An example of a simple fixture is h’s pyramid_request fixture:

@pytest.fixture
def pyramid_request():
    """Dummy Pyramid request object."""
    request = testing.DummyRequest()
    request.auth_domain = text_type(request.domain)
    request.create_form = mock.Mock()
    request.matched_route = mock.Mock()
    request.is_xhr = False
    return request

Pyramid is the web framework that h uses to receive HTTP requests from users’ web browsers and send back HTTP responses to those requests. The request object is the object that Pyramid makes available to our code that represents the HTTP request that we’re currently responding to. Many methods in h require the request object as an argument, so tests often need a request object to pass to the method they’re testing.

The pyramid_request() function above creates a DummyRequest object, sets various fields on the request that are needed for h, and returns it.

The @pytest.fixture decorator on the top of the function registers it with pytest as a fixture function, which means that a test can make use of it by simply taking it as an argument. For example, let’s look at a test for the paginate() function:

def test_current_page_defaults_to_1(pyramid_request):
    """If there's no 'page' request param it defaults to 1."""
    pyramid_request.params = {}

    page = paginate(pyramid_request, 600, 10)

    assert page['cur'] == 1

paginate() is a function that takes the request and returns some data, including the current page, that’s used to render this pagination control at the bottom of search results pages:

The test needs a Pyramid request argument in order to call the paginator function, so it adds the pyramid_request fixture to its arguments:

def test_current_page_defaults_to_1(self, pyramid_request):
    ...

When pytest comes to run this test it’ll see the pyramid_request argument, it knows that this matches up to an @pytest.fixture function, so it first calls the pyramid_request() fixture function and then passes the value that the fixture function returns to the test as the pyramid_request argument. What pytest does is something like this:

pyramid_request_fixture = pyramid_request()
test_current_page_defaults_to_1(pyramid_request=pyramid_request_fixture)

The pyramid_request() fixture returns a Pyramid DummyRequest object all set up for use in h, and pytest passes that DummyRequest to the test as the pyramid_request argument. It’s as simple as that - all the test has to do is have an argument with the same name as the fixture function and it gets the request object passed right into it.

At its core, that’s all a pytest fixture is - a perfectly normal Python function that pytest runs before running a test, only if that test has the fixture as an argument. An @pytest.fixture decorator is needed on the top of the function to register it with pytest as a fixture.

Fixtures are no different to test helper methods, except the boilerplate of each test needing to call every helper method that it needs is avoided. It’s not often needed, but a fixture can also contain teardown code to be called after any test that used that fixture simply by using yield instead of return.

Lots of different tests can reuse the same fixture

Just like different tests can call the same helper method, once you’ve written a fixture once you can use it in as many tests as you want just by adding it as an argument to each test:

@pytest.fixture
def pyramid_request():
    ...

def test_current_page_defaults_to_1(pyramid_request):
    ...

def test_numbers_large_result_set(pyramid_request):
    ...

def test_numbers_small_result_set(pyramid_request):
    ...

One test can use lots of different fixtures

And of course just like a test can call multiple helper methods, a test can also use multiple fixtures by just having all of the fixture names as arguments.

In the factories post we looked at the factories object which can be used to easily create realistic objects for use in tests. factories is a fixture that’s used by any test that wants to use it to create test objects.

pyramid_settings is another fixture that returns the settings used to configure the h app in the test environment.

A test method can use all three fixtures, pyramid_request, factories and pyramid_settings just by taking all three as arguments by name:

def test_that_uses_multiple_fixtures(pyramid_request, factories, pyramid_settings):
    ...

Fixtures don’t have to return anything

Sometimes a test just needs a fixture function to be run before the test, because the test depends on some setup that the fixture function does, but the test doesn’t actually need to use the value that the fixture function returns. A fixture function doesn’t even have to return anything, it might just do some setup that’s needed by some tests, for example creating some entries in the database, and return nothing. If a test wants that fixture function to be run before it then it adds the fixture as an argument, if it doesn’t then it doesn’t. A good example of this is the routes fixture, which is found in many places in the h tests.

Routes are Pyramid’s way of deciding which method it should call to handle a request for a given URL. For example if a user loads the /users/seanh page then Pyramid calls the h.views.activity.UserSearchController.search() method to generate the response. The routes.py file tells Pyramid how to do this by configuring which URLs map to which methods. Routes can also be used in reverse, to generate URLs. For example if some code wants the URL for the user search page for the “seanh” user then it can get it from the route_url() method by doing request.route_url("activity.user_search", username="seanh"), which will return "https://hypothes.is/users/seanh".

Many methods in h call methods like route_url() and others that require all the routes to be setup in order to work. If the routes aren’t configured then calls to route_url() will crash with some sort of “unknown route” error. So the tests need some code to setup the routes before the test. This is what a routes fixture does, for example:

@pytest.fixture
def routes(self, pyramid_config):
    pyramid_config.add_route('activate', '/activate/{id}/{code}')

The fixture doesn’t return anything, but it adds the 'activate' route to the Pyramid config so that any calls to request.route_url("activate", ...) will now work. A test can use this fixture and then test a function that uses this route without having it crash:

def test_something(routes):
    result = some_function_that_uses_the_activate_route()

    assert result == <something>

@pytest.mark.usefixtures

The test above doesn’t actually use the routes argument in the body of the test, and the value of routes is just None anyway, it’s just there to tell pytest to call the routes() function before it calls test_something().

@pytest.mark.usefixtures() is an alternative way for a test to use a fixture when it just needs the fixture function to be run, but doesn’t actually need the fixture value in the test body:

@pytest.mark.usefixtures('routes')
def test_something():
    result = some_function_that_uses_the_activate_route()

    assert result == <something>

@pytest.mark.usefixtures("routes") tells pytest to run the routes fixture before this test, just the same as routes as an argument would do.

To use multiple fixtures this way, just pass multiple strings to usefixtures():

@pytest.mark.usefixtures("routes", "pyramid_request", "factories", "pyramid_settings")

Where to find fixtures

When you see test methods that have arguments in h, unless the test is using parametrize, those arguments are probably fixtures. But where do you find the fixture functions for the fixtures that a test uses? And where should you put any new fixtures that you want to write for your tests? Fixtures can be defined in a number of places:

1. pytest comes with some builtin fixtures.

2. In a conftest.py file such as tests/h/conftest.py (for the tests in tests/h) or tests/memex/conftest.py for the tests in tests/memex.

3. Fixture functions can go in the the test modules themselves. You’ll see this all over the place in the h tests, the fixture functions usually all go at the bottom of the test file and in alphabetical order. For example, here’s a bunch of fixtures defined at the bottom of views_test.py.

   Fixtures defined in a test file are only available to the tests in that file.

   A fixture in a test file will override any fixture with the same name in a conftest.py file.

4. Fixture functions can go in test classes. A fixture defined in a class is only available to the tests in that class, and overrides any fixture with the same name defined outside of the class or in a conftest.py file.

   We use this a lot in h as well, if a fixture is only needed by the tests in one test class then we define the fixture as a method of that test class. It keeps related code together, reducing noise and travel. It also allows other test classes to have their own fixture with the same name without conflicting.

   Like any class method these fixtures need to take self as their first argument. We usually put all the fixtures at the bottom of the class and in alphabetical order. For example, here’s some fixtures defined at the bottom of the TestAddApiView class.

Tip: You can find the definition of a fixture by using the --fixtures argument to pytest, for example:

$ tox -e py27-memex tests/memex/ -- --fixtures

This prints out the names of all of the available fixtures and the line and file at which each one is defined. You can grep the output of this command to find a particular fixture by name:

$ tox -e py27-memex tests/memex/ -- --fixtures | grep '^annotation'

If you’re only interested in the fixtures available to a particular test module, class or method you can drill down:

$ tox -e py27-h \
    tests/h/views/activity_test.py::TestSearchController::test_search_checks_for_redirects \
    -- --fixtures

Conclusion

So far we’ve seen that fixtures are a more convenient alternative to test helper methods, with less boilerplate required and with a good way to do teardown code. You define each fixture in a separate function, but rather than each test needing to call each helper method that it wants the tests can just receive the fixture objects as arguments.

Pytest fixtures also have a couple of advanced features that take them well beyond what simple helper methods can do. We’ll cover some of those in Advanced pytest Fixtures.