Setuptools Surprise


I recently tripped over my reliance on a simple (and probably obscure) feature in Python’s distutils that setuptools doesn’t support. The result was that I created  a tarball for my posix_ipc module that lacked critical files. By chance, I noticed when uploading the new tarball that it was about 75% smaller than the previous version. That’s a red flag!

Fortunately, the bad tarball was only on PyPI for about 3 minutes before I noticed the problem and removed the release.

I made debugging harder on myself by stepping away from the project for a long time and forgetting what changes I’d made since the previous release.


In February 2014, I (finally) made my distribution PyPI–friendly. Prior to that I’d built my distribution tarballs with a custom script that explicitly listed each file to be included in the tarball. The typical, modern, and PyPI–friendly way to build tarballs is by writing a file that a distribution tool (like Python’s distutils) interprets into a MANIFEST file. A command like `python sdist` reads the manifest and builds the tarball.

That’s the method to which I switched in February 2014, with one exception—since my custom script already contained an explicit list of files, it was easier to write a MANIFEST file directly and skip the intermediate That works fine with distutils.

I released version 1.0.0 of posix_ipc in March of 2015, and haven’t needed to make any changes to the code until just now (the beginning of 2018). However, in February 2016, I made a small change to that I thought was harmless. (Ha!)

I added a conditional import of setuptools so that I could build wheels. (Side note: I really like wheels!) The change allows me to build posix_ipc wheels on my laptop where I can ensure setuptools is available, but otherwise falls back on Python’s distutils which works just fine for everything else I need to do, including installing from a tarball. The code looks like this —

    import setuptools as distutools
except ImportError:
    import distutils.core as distutools

The Problem

Just a few days ago, I released a maintenance release of posix_ipc, and it was then I noticed that the tarballs I built with my usual python sdist command were 75% smaller and missing several critical files. Because it had been 23 months since I made my “harmless” change to, the switch from using distutils to setuptools wasn’t exactly fresh in my mind.

However, some examination of my commit log and a realization that this was the first release I’d made after making that change gave me a suspicion, and grepping through setuptools‘ code revealed no references to MANIFEST, only

There’s also this in the setuptools documentation, if I’d bothered to read it—

[B]e sure to ignore any part of the distutils documentation that deals with MANIFEST or how it’s generated from; setuptools shields you from these issues and doesn’t work the same way in any case. Unlike the distutils, setuptools regenerates the source distribution manifest file every time you build a source distribution, and it builds it inside the project’s .egg-info directory, out of the way of your main project directory.

So that was the problem—setuptools doesn’t look for a MANIFEST file, only Since I had the former but not the latter, setuptools used its defaults instead of my list of files in MANIFEST.

The Solution

This part was easy. I converted my MANIFEST file to a which works with both setuptools and distutils. That’s probably a more robust solution than the hardcoded list in MANIFEST anyway.

I’m pleased that posix_ipc has been stable and well-behaved for such a long time, but these long breaks between releases mean a certain amount of mental rust has always accumulated when it’s time for the next one.

By the way, the source for posix_ipc is now hosted on GitHub:

3 thoughts on “Setuptools Surprise”

  1. You could try using setuptools_scm which picks up all version controlled files for your sdist, so you don’t need I do this in all my (numerous) projects. As a bonus, it can automatically generate a version based on the nearest git/hg tag.

  2. Also, conditionally import setuptools is pointless. Setuptools is everywhere, and besides, if you install the project as a wheel, isn’t even run.

  3. Thanks for the comments!

    You’re probably right about setuptools being nearly universal for most installations now. But I don’t want to assume that it’s available everywhere posix_ipc is used. Setuptools isn’t installed if you build your own Python, for instance. Since I only added setuptools to my for the specific case of building wheels on my laptop, I’m happy to remain otherwise independent of it.

    setuptools_scm is a good idea! I appreciate the tip.

Comments are closed.