For this tutorial I will assume user installation of new packages by building from downloaded source, ie "python install." While this tutorial also applies to user installations with the easy_install command line tool, I reserve easy_install related notes to the "Gotcha's" section at the end of this tutorial.

This tutorial also assumes the reader is a user rather than a developer of python packages and need not worry about multiple or virtual python installations in day to day use. Of course updating to a different version [2.4,2.5,2.6,...] of python requires the user to rebuild their user library.

A word on python versions: Common python versions include 2.4 (what is on the system administered Linux computers where I work), 2.5 on Mac OS X 10.5 and 2.6 on Mac OS X 10.6. What is below is based on the official 2.6 docs, and the example uses syntax appropriate for version 2.4. It will work for all three versions. A new feature added in python 2.6 that would provide alternatives to this technique is discussed briefly in the "Gotchas" section.


A user configuration file, ~/.pydistutils.cfg, will override the internal system path for python package installations, redirecting the built libraries (lib), scripts (bin) and data (share) into user owned and specified directories. You must simply tell the python installer where theses directories are located.

The user file, ~/.pydistutils.cfg, has the following lines, using a pretty obvious syntax:

install_scripts = ~/usr/bin
install_data = ~/usr/share
install_lib = ~/usr/lib/python2.4/site-packages

This example assumes you will simply create an alternate "usr" tree in your home (~) directory, mimicking the UNIX /usr directory. There is nothing special about the names used for these directory paths, however. Generally, these user directories need not exist to specify them in ~/.pydistutils.cfg but to be safe one could simply run:

mkdir -p ~/usr/bin
mkdir -p ~/usr/share
mkdir -p ~/usr/lib/python2.4/site-packages

The location of your user Python library directory (install_lib) must be in your Python path ("import sys; print sys.path") prior to using the new modules OR installing new modules that have dependencies on other user installed modules. This requirement can be accomplished by explicitly adding the library directory to your python path. For the example above you would do this:

in bash:

export PYTHONPATH=~/usr/lib/python2.4/site-packages:${PYTHONPATH}

or csh:

setenv PYTHONPATH ~/usr/lib/python2.4/site-packages:${PYTHONPATH}

where PYTHONPATH is ordered such that earlier entries supersede latter ones and the entire $PYTHONPATH supercedes all system libraries (but see Gotcha's).

That is the gist of the solution.

Example with this configuration

With this file in place builds using syntax such as:


tar xjvf TinyUrl-0.1.0.tar.bz2

cd TinyUrl-0.1.0/

python install

will build a command line tool, ~/usr/bin/tinyurl where I can now run:


to yield

More details

To use the scripts one might want to add the install_bin directory to your unix path, $PATH.  Else one would need to explicitly call the above example as ~/usr/bin/tinyurl ..., instead.


1. Regarding default user writable installation directories: Mac

In some cases the current system python knows to check for a default user writable directory for modules in user installed packages without resorting to the $PYTHONPATH variable.  This is the case for Mac OS X (Leopard), where the directory:


is automatically searched for modules to add to the python path (whether you knew that or not might be another point of value for this tutorial!). You would instead substitute that directory,

install_lib = ~/Library/Python/2.5/site-packages

in ~/.pydistutils.cfg.   The gotcha for this is that this directory is searched *last* on the system python module path and as such the user may not be able to override system installed packages using this particular default directory. You cannot override this path ordering by adding it to your $PYTHONPATH because it is interpreted as a duplicate path and ignored.

2. easy_install

The easy_install command line tool will respect the installation instructions provided in ~/.pydistutils.cfg, however, there are gotchas related to how easy_install manipulates the module path that should be mentioned.

The most critical issue with easy_install and user installed python packages is the fact that packages can be installed in different ways that cause their modules to have priorities that appear to differ from the formal system path.  The way the python module path works, $PYTHONPATH should override all other directories (this is built into the basic architecture).   The standard path ordering would be this:

  2. System owned site-directories (e.g., /usr/local/python2.4/site-packages)
  3. ~/Library/Python/2.X/site-packages [on Macs]

Consider the case of "numpy," the widely used numeric library in python.   Lets suppose your system (root) installation includes numpy 1.3 installed in a system site directory:

cd ~
python -c "import numpy; print numpy.__version__,numpy.__file__"

# 1.3
# /usr/local/lib/python2.4/site-packages/numpy/

You then install the release version of numpy (1.4) with easy_install and ~/.pydistutils.cfg with install_lib =  ~/usr/lib/python2.4/site-packages:

easy_install -ZU numpy
python -c "import numpy; print numpy.__version__,numpy.__file__"

# 1.4
# ~/usr/local/lib/python2.4/site-packages/numpy-1.4.0-py2.4-linux-x86_64.egg/numpy/

Now you need some bleeding edge feature. So you download the source for the developer version of numpy via svn (1.5.svn) and run with the same ~/.pydistutils.cfg file:

svn co numpy
cd numpy
python install
cd ~
python -c "import numpy; print numpy.__version__,numpy.__file__"

# 1.4
# ~/usr/local/lib/python2.4/site-packages/numpy-1.4.0-py2.4-linux-x86_64.egg/numpy/

This is not intuitive unless you understand that while easy_install follows the same path ordering process, it builds its packages (called "eggs") and pre-pends them to the other paths. The module path now looks like:

  1. easy_install to $PYTHONPATH
  2. easy_install to System owned site-directories
  3. easy_install to ~/Library/Python/2.X/site-packages [on Macs]
  5. System owned site-directories
  6. ~/Library/Python/2.X/site-packages [on Macs]

And your numpy installations are thus ordered:

  1. numpy 1.4 (w/easy_install)
  2. numpy 1.5 svn (with python install to $PYTHONPATH location)
  3. numpy 1.3 system

on the python module path.  Moreover, if your system installation of numpy was built with easy_install your 1.5svn attempt falls to third place above. As such if your system owned installation used easy_install to install packages then you are basically forced to also use easy_install because the system easy_install'ed packages will supersede even your $PYTHONPATH unless you also install with easy_install.

This is not to discount easy_install. I always use it.  If you or your root install have used it, just keep always using it. For example, the way I do this with the numpy 1.5 svn versions is:

svn co numpy
cd numpy
python build
cd ..
easy_install -ZU ./numpy
cd ~
python -c "import numpy; print numpy.__version__,numpy.__file__"

# 1.5.0.dev8079
# ~/usr/local/lib/python2.4/dist-packages/numpy-1.5.0.dev8079-py2.4-linux-x86_64.egg/numpy/

Using easy_install, also makes it straightforward for me to switch versions of numpy that I have already installed.  For the working example, I have an easy_installed numpy 1.4 and numpy 1.5.0.dev8079. To switch I simply run easy_install and point at the requested version:

~/usr/bin/easy_install -Z numpy==1.5.0.dev8082 ; python -c 'import numpy; print numpy.__version__'
# 1.5.0.dev8079

~/usr/bin/easy_install -Z numpy==1.4 ; python -c 'import numpy; print numpy.__version__'
# 1.4.0

Starting from scratch with easy_install

At my institution the system administered Linux workstations use python2.4 and lack easy_install and the ipython shell. In this case I set up my ~/.pydistutils.cfg according to the example at the beginning of the tutorial, created the three install directories, and ran:


tar xzvf setuptools-0.6c11.tar.gz
python install

which installed easy_install into ~/usr/bin.  Then, without problem, I then ran

~/usr/bin/easy_install -Z nose

~/usr/bin/easy_install -Z numpy

python -c "import numpy;numpy.test()"

~/usr/bin/easy_install -Z ipython==0.9.1

~/usr/bin/easy_install -Z PIL

~/usr/bin/easy_install -Z matplotlib

3. Python 2.6+

Python 2.6 adds a default tree for user directories that is searched before system libraries and without specifying it on the user $PYTHONPATH.  This predefined tree is rooted at "~/.local" in unix environs, where user installed packages are added in "~/.local/lib/python2.6/site-packages" and executable scripts in ~/.local/bin.  You can use this directory without ~/.pydistutils.cfg by compiling source packages with:

python install --user

Note that an entry in ~/.pydistutils.cfg file, such as

install_lib = ~/usr/lib/python2.6/site-packages

will override the --user option, redirecting modules to install_lib. In this specific case "~/usr/lib/python2.6/site-packages" will not be automatically searched for modules unless it is added to $PYTHONPATH.  Further, the search path now looks like:

  2. ~/.local/lib/python2.6/site-packages
  3. System owned site-directories (e.g., /usr/local/python2.4/site-packages)
  4. ~/Library/Python/2.X/site-packages [on Macs]

Finally, to easy_install with Python 2.6's $PYTHONPATH free installation of user installed packages in ".local," users would type:

easy_install --prefix=~/.local -ZU tinyurl

for the original installation example above.  The binary is now in ./local/bin/tinyurl and the module search path with easy_install packages is now:

  1. easy_install to $PYTHONPATH
  2. easy_install to ~/.local/lib/python2.6/site-packages
  3. easy_install to System owned site-directories
  4. easy_install to ~/Library/Python/2.X/site-packages [on Macs]
  6. ~/.local/lib/python2.6/site-packages
  7. System owned site-directories
  8. ~/Library/Python/2.X/site-packages [on Macs]

    5           6