Changing the test database setup of pytest-django

In order for Motius to efficiently manage our talent pool, we have built a platform that is used to maintain our talent database and help us identify new potential candidates. We strive to expand the feature set of the platform and test it as much as possible in order to automate most of our processes. The test suite must execute as fast as possible and the setup of the tests must be as automatic as possible. But, that result can not always be achieved without a little bit of hacking around…

Under normal circumstances, when you run the django tests or django-pytest, the system will automatically create a test database and then run either migrations or sync the schema to fix its state. However, there was an issue with that process: we are using PostgreSQL 9.6 and we need the hstore extension installed in it before anything else happens. If hstore is not installed, then the migrations will fail and no tests will run.

For our git repo hosting and CI we are using Gitlab, so, initially we added a couple of commands in our gitlab-ci.yml to install hstore before running the tests using dbshell. So our CI pipeline was operational again. But what about the development team? In our setup we use heavily Docker Compose to bootstrap a cluster locally with all our services without requiring the engineer to have anything else installed in their system other than the Docker daemon and Docker Compose. And we wanted our devs to be able to run the tests without requiring them to be accustomed with Docker commands, but with just one command. Also, converting this process into a piece of Python code will make it more maintainable and allow us in the future to maybe expand it according to future needs like populating the database with some data before the tests begin.

So after a little bit of digging into the pytest-django docs and the code we came up with this:

import os
import pytest
import subprocess

from django.conf import settings
from django.db import connections
from django.db.utils import ConnectionDoesNotExist

def run_sql(database, sql, django_db_blocker, ignore_errors=False):
	try:
		connection = connections[database]
	except ConnectionDoesNotExist:
		raise Exception('Database {} does not exist in settings'.
	format(database))

	with django_db_blocker.unblock():
		try:
			connection.cursor().execute(sql)
		except Exception as e:
			if not ignore_errors:
				raise e
			else:
				print('Ignoring: ' + repr(e))

@pytest.fixture(scope='session')
def django_db_setup(request, django_db_blocker):

	# Install hstore in standard database just in case
	run_sql('default',
		'CREATE EXTENSION hstore;',
		django_db_blocker,
		ignore_errors=True)

	original_db_name = settings.DATABASES['default']['NAME']
	test_db_name = 'test_{}'.format(original_db_name)

	# Drop previous version of database if it exists
	run_sql('default',
	'DROP DATABASE IF EXISTS {};'.format(test_db_name),
	django_db_blocker)

	# Create an empty database
	run_sql('default',
	'CREATE DATABASE {} TEMPLATE template0;'.format(test_db_name),
	django_db_blocker)

	# Close connection to original database
	connections['default'].close()

	# Fix settings of connection to new test database
	connections['default'].settings_dict['TEST'] = {
	'COLLATION': None,
	'NAME': test_db_name,
	'CHARSET': None,
	'MIRROR': None,
	}
	connections['default'].settings_dict['NAME'] = test_db_name

	# Install hstore in test database just in case
	run_sql('default',
		'CREATE EXTENSION hstore;',
		django_db_blocker,
		ignore_errors=True)

	from pytest_django.fixtures import _disable_native_migrations
	_disable_native_migrations()

	# Apply all the migrations to the database
	with django_db_blocker.unblock():
		connections['default'].creation.create_test_db(verbosity=2,
		autoclobber=True,
		keepdb=True,
		serialize=True)

 

The algorithm is pretty close to the one used by default by pytest-django. So what’s happening here is this:

First, we ensure that hstore is installed in the standard database, because, why not? Then we drop the test database in case it has not been dropped in a previous test run. The name of the test database will be ‘test_’ + whatever the name of the actual database is. We create the test database and we close the connection to the original database. The reason for that is that once the test database has been fixed, all connections to the default database will be actually done to the test database until the end of the test run.

The next step is the most important: we install the hstore extension into the test database. Finally, we disable the migrations using a funtion from pytest-django and then call the create_test_db function. This is a function provided by Django itself and it will initialize the test_db just like the way it is initialized if we run the Django tests. One important detail in the way we call it is the keepdb=True argument. We need that set to True because we have created the test database and we don’t want that function to drop it thinking that it’s a leftover from a previous run. We have already checked for that case in our own code.

Also the reason we are disabling the migrations is that any Django project that has been in a production a couple of years and has some activity, means that it has also probably accumulated a lot of migration scripts. Imagine working on a small bug fix and running the tests after every change to make sure that it’s fine and having to run 100+ migration scripts in between each run. Not very nice huh? In our case we know that migration scripts are not a good idea, but maybe in other cases you would want to have migration scripts by default. Who knows? If you want to keep it flexible you can add the django_db_use_migrations fixture in the function and you can check its value to replicate more closely the default behavior of the django-pytest db setup.


Schreibe einen Kommentar

Deine E-Mail-Adresse wird nicht veröffentlicht. Erforderliche Felder sind mit * markiert.