I recently tripped over my reliance on a simple (and probably obscure) feature in Python’s
setuptools doesn’t support. The result was that I created a tarball for my
posix_ipc module that lacked critical files. By chance, I noticed when uploading the new tarball that it was about 75% smaller than the previous version. That’s a red flag!
Fortunately, the bad tarball was only on PyPI for about 3 minutes before I noticed the problem and removed the release.
I made debugging harder on myself by stepping away from the project for a long time and forgetting what changes I’d made since the previous release.
In February 2014, I (finally) made my distribution PyPI–friendly. Prior to that I’d built my distribution tarballs with a custom script that explicitly listed each file to be included in the tarball. The typical, modern, and PyPI–friendly way to build tarballs is by writing a
MANIFEST.in file that a distribution tool (like Python’s
distutils) interprets into a
MANIFEST file. A command like
`python setup.py sdist` reads the manifest and builds the tarball.
That’s the method to which I switched in February 2014, with one exception—since my custom script already contained an explicit list of files, it was easier to write a
MANIFEST file directly and skip the intermediate
MANIFEST.in. That works fine with
I released version 1.0.0 of
posix_ipc in March of 2015, and haven’t needed to make any changes to the code until just now (the beginning of 2018). However, in February 2016, I made a small change to
setup.py that I thought was harmless. (Ha!)
I added a conditional import of
setuptools so that I could build wheels. (Side note: I really like wheels!) The change allows me to build
posix_ipc wheels on my laptop where I can ensure
setuptools is available, but otherwise falls back on Python’s
distutils which works just fine for everything else I need
setup.py to do, including installing from a tarball. The code looks like this —
try: import setuptools as distutools except ImportError: import distutils.core as distutools
Just a few days ago, I released a maintenance release of
posix_ipc, and it was then I noticed that the tarballs I built with my usual
python setup.py sdist command were 75% smaller and missing several critical files. Because it had been 23 months since I made my “harmless” change to
setup.py, the switch from using
setuptools wasn’t exactly fresh in my mind.
However, some examination of my commit log and a realization that this was the first release I’d made after making that change gave me a suspicion, and grepping through
setuptools‘ code revealed no references to
There’s also this in the
setuptools documentation, if I’d bothered to read it—
[B]e sure to ignore any part of the distutils documentation that deals with MANIFEST or how it’s generated from MANIFEST.in; setuptools shields you from these issues and doesn’t work the same way in any case. Unlike the distutils, setuptools regenerates the source distribution manifest file every time you build a source distribution, and it builds it inside the project’s .egg-info directory, out of the way of your main project directory.
So that was the problem—
setuptools doesn’t look for a
MANIFEST file, only
MANIFEST.in. Since I had the former but not the latter,
setuptools used its defaults instead of my list of files in
This part was easy. I converted my
MANIFEST file to a
MANIFEST.in which works with both
distutils. That’s probably a more robust solution than the hardcoded list in
I’m pleased that
posix_ipc has been stable and well-behaved for such a long time, but these long breaks between releases mean a certain amount of mental rust has always accumulated when it’s time for the next one.
By the way, the source for
posix_ipc is now hosted on GitHub:
3 thoughts on “Setuptools Surprise”
You could try using setuptools_scm which picks up all version controlled files for your sdist, so you don’t need MANIFEST.in. I do this in all my (numerous) projects. As a bonus, it can automatically generate a version based on the nearest git/hg tag.
Also, conditionally import setuptools is pointless. Setuptools is everywhere, and besides, if you install the project as a wheel, setup.py isn’t even run.
Thanks for the comments!
You’re probably right about setuptools being nearly universal for most installations now. But I don’t want to assume that it’s available everywhere posix_ipc is used. Setuptools isn’t installed if you build your own Python, for instance. Since I only added setuptools to my setup.py for the specific case of building wheels on my laptop, I’m happy to remain otherwise independent of it.
setuptools_scm is a good idea! I appreciate the tip.
Comments are closed.