Running Pre and Post Install Jobs for Your Python Packages

, a 11-minute piece by Dev Mukherjee Dev Mukherjee

We heavily use Python's Setuptools. It eases managing packages during development under and provides a way to streamline distribution. Anything that our Python code uses directly (e.g. HTML or Email templates) are distributed as part of the package. This is done by setting include_package_data argument to True in your setup definition and including references in MANIFEST.in

include our-app/templates/email/*
include our-app/templates/html/*
include our-app/templates/report/*

While maintaining Web application we found the need to distribute a number of extra along with the package. These are directly accessed by the Web server and it didn't quite feel quite right to distribute them as part of the Python package and employing an addition distribution mechanism seemed like an overkill.

To start with we included these two folders in MAINFEST.in and deployed them manually. Note: they would appear outside the Python package.

include wsgi/*
recursive-include static *

Our Python application run under a Virtualenv. The servers folder structures look something like this.

After much hunting around I figured out how to gracefully run pre or post install jobs for Python application installed via Setuptools. At the heart of it is creating a wrapper around the Setuptools installer.

If you use Virtualenv during development the biggest gotcha is that this does not work in --editable mode, which proved to be a feature since only expected to run these process in our production environments.

I read a few philosophical posts around this, arguing if these sorts of tasks were outside Setuptools's call of duty. After some quiet contemplation I summarised that it's perfectly acceptable for internally deployed projects. Anything distributed to the wider Python community (like a framework) should stay inside the confounds of site packages (unless of course you can justify it).

To get started you will need to import the following in setup.py (ensure you don't accidentally import the install class from distutils)

import os, shutil
from setuptools.command.install import install

You then define a class which inherits from install and override the run method (which is called by Setuptools) to invoke your custom code before or after you call the parent run method install.run(self).

Raising an exception will cause Setuptools to present a stack trace and halt.

In my case I wanted to ensure that the Python package isn't installed if the Web server files failed to copy (by raising an exception to stop execution) thus executing my custom code before I call the super function.

class InstallWrapper(install):
  """Provides a install wrapper for WSGI applications
  to copy additional files for Web servers to use e.g 
  static files. These were packaed using definitions 
  in MANIFEST.in and don't really belong in the 
  Python package."""

  _TARGET_ROOT_PATH = "/srv/our-app/"
  _COPY_FOLDERS = ['wsgi', 'static']

  def run(self):
    # Run this first so the install stops in case 
    # these fail otherwise the Python package is
    # successfully installed
    self._copy_web_server_files()
    # Run the standard PyPi copy
    install.run(self)

  def _copy_web_server_files(self):

    # Check to see we are running as a non-prvilleged user
    if os.geteuid() == 0:
      raise StandardError(
        "Install should be as a non-prvilleged user")

    # Check to see that the required folders exists 
    # Do this first so we don't fail half way
    for folder in self._COPY_FOLDERS:

      if not os.access(folder, os.R_OK):
        raise IOError("%s not readable from achive" % 
        folder)

    # Check to see we can write to the target
    if not os.access(self._TARGET_ROOT_PATH, os.W_OK):
      raise IOError("%s not writeable by user" % 
      self._TARGET_ROOT_PATH)

    # Clean target and copy files
    for folder in self._COPY_FOLDERS:

      target_path = os.path.join(
        self._TARGET_ROOT_PATH, folder)

      # If this exists at the target then 
      # remove it to ensure the target is clean
      if os.path.isdir(target_path):
        shutil.rmtree(target_path)

      # Copy the files from the archive
      shutil.copytree(folder, target_path)

Setuptools has various commands which you can list via python setup.py --help-commands, one of them of course being install. You can write a wrapper for all of them.

The final step is telling Setuptools to use the wrapper class instead of the standard install by overriding the install directive in the cmdclass definitions.

setup(name='our-app',
      version=__version__,
      author='Anomaly Software',
      packages=find_packages(),
      include_package_data=True,
      .....
      cmdclass={'install': InstallWrapper},
     )

I'm leaving the philosophy out of this one and recommend use with caution. It's something that we found extremely useful for our particular use case and I'm sure there are others out there.

If you don't already, I highly recommend using Virtual and Setuptools during Python development and deployment.

Next Up: a 2-minute piece by Dev Mukherjee Dev Mukherjee

Tesla Destination Charging; a Paradigm Shift for Tourism

Read more