User (root/sudo free) installation of Python modules.

For this tutorial I will assume user installation of new packages by building from downloaded source, ie "python setup.py install." While this tutorial also applies to user installations with the easy_install command line tool, I reserve easy_install related notes to the "Gotcha's" section at the end of this tutorial.

This tutorial also assumes the reader is a user rather than a developer of python packages and need not worry about multiple or virtual python installations in day to day use. Of course updating to a different version [2.4,2.5,2.6,...] of python requires the user to rebuild their user library.

A word on python versions: Common python versions include 2.4 (what is on the system administered Linux computers where I work), 2.5 on Mac OS X 10.5 and 2.6 on Mac OS X 10.6. What is below is based on the official 2.6 docs, and the example uses syntax appropriate for version 2.4. It will work for all three versions. A new feature added in python 2.6 that would provide alternatives to this technique is discussed briefly in the "Gotchas" section.

Gist

A user configuration file, ~/.pydistutils.cfg, will override the internal system path for python package installations, redirecting the built libraries (lib), scripts (bin) and data (share) into user owned and specified directories. You must simply tell the python installer where theses directories are located.

The user file, ~/.pydistutils.cfg, has the following lines, using a pretty obvious syntax:


[install]
install_scripts = ~/usr/bin
install_data = ~/usr/share
install_lib = ~/usr/lib/python2.4/site-packages


This example assumes you will simply create an alternate "usr" tree in your home (~) directory, mimicking the UNIX /usr directory. There is nothing special about the names used for these directory paths, however. Generally, these user directories need not exist to specify them in ~/.pydistutils.cfg but to be safe one could simply run:


mkdir -p ~/usr/bin
mkdir -p ~/usr/share
mkdir -p ~/usr/lib/python2.4/site-packages



The location of your user Python library directory (install_lib) must be in your Python path ("import sys; print sys.path") prior to using the new modules OR installing new modules that have dependencies on other user installed modules. This requirement can be accomplished by explicitly adding the library directory to your python path. For the example above you would do this:


in bash:

export PYTHONPATH=~/usr/lib/python2.4/site-packages:${PYTHONPATH}


or csh:

setenv PYTHONPATH ~/usr/lib/python2.4/site-packages:${PYTHONPATH}


where PYTHONPATH is ordered such that earlier entries supersede latter ones and the entire $PYTHONPATH supercedes all system libraries (but see Gotcha's).

That is the gist of the solution.


Example with this configuration

With this file in place builds using syntax such as:


wget http://pypi.python.org/packages/source/T/TinyUrl/TinyUrl-0.1.0.tar.bz2#md5=a0b6a52ad898a35ea6280b43eb5c1be1

tar xjvf TinyUrl-0.1.0.tar.bz2

cd TinyUrl-0.1.0/

python setup.py install



will build a command line tool, ~/usr/bin/tinyurl where I can now run:


tinyurl http://www.astropython.org/tutorial/2010/1/User-rootsudo-free-installation-of-Python-modules



to yield


http://tinyurl.com/yhassaw


More details

To use the scripts one might want to add the install_bin directory to your unix path, $PATH.  Else one would need to explicitly call the above example as ~/usr/bin/tinyurl ..., instead.


Gotchas


1. Regarding default user writable installation directories: Mac


In some cases the current system python knows to check for a default user writable directory for modules in user installed packages without resorting to the $PYTHONPATH variable.  This is the case for Mac OS X (Leopard), where the directory:


~/Library/Python/2.5/site-packages


is automatically searched for modules to add to the python path (whether you knew that or not might be another point of value for this tutorial!). You would instead substitute that directory,


install_lib = ~/Library/Python/2.5/site-packages


in ~/.pydistutils.cfg.   The gotcha for this is that this directory is searched *last* on the system python module path and as such the user may not be able to override system installed packages using this particular default directory. You cannot override this path ordering by adding it to your $PYTHONPATH because it is interpreted as a duplicate path and ignored.


2. easy_install


The easy_install command line tool will respect the installation instructions provided in ~/.pydistutils.cfg, however, there are gotchas related to how easy_install manipulates the module path that should be mentioned.

The most critical issue with easy_install and user installed python packages is the fact that packages can be installed in different ways that cause their modules to have priorities that appear to differ from the formal system path.  The way the python module path works, $PYTHONPATH should override all other directories (this is built into the basic architecture).   The standard path ordering would be this:


  1. PYTHONPATH  
  2. System owned site-directories (e.g., /usr/local/python2.4/site-packages)
  3. ~/Library/Python/2.X/site-packages [on Macs]


Consider the case of "numpy," the widely used numeric library in python.   Lets suppose your system (root) installation includes numpy 1.3 installed in a system site directory:


cd ~
python -c "import numpy; print numpy.__version__,numpy.__file__"

# 1.3
# /usr/local/lib/python2.4/site-packages/numpy/__init__.py


You then install the release version of numpy (1.4) with easy_install and ~/.pydistutils.cfg with install_lib =  ~/usr/lib/python2.4/site-packages:


easy_install -ZU numpy
python -c "import numpy; print numpy.__version__,numpy.__file__"

# 1.4
# ~/usr/local/lib/python2.4/site-packages/numpy-1.4.0-py2.4-linux-x86_64.egg/numpy/__init__.py


Now you need some bleeding edge feature. So you download the source for the developer version of numpy via svn (1.5.svn) and run with the same ~/.pydistutils.cfg file:


svn co http://svn.scipy.org/svn/numpy/trunk numpy
cd numpy
python setup.py install
cd ~
python -c "import numpy; print numpy.__version__,numpy.__file__"

# 1.4
# ~/usr/local/lib/python2.4/site-packages/numpy-1.4.0-py2.4-linux-x86_64.egg/numpy/__init__.py


This is not intuitive unless you understand that while easy_install follows the same path ordering process, it builds its packages (called "eggs") and pre-pends them to the other paths. The module path now looks like:


  1. easy_install to $PYTHONPATH
  2. easy_install to System owned site-directories
  3. easy_install to ~/Library/Python/2.X/site-packages [on Macs]
  4. $PYTHONPATH  
  5. System owned site-directories
  6. ~/Library/Python/2.X/site-packages [on Macs]


And your numpy installations are thus ordered:


  1. numpy 1.4 (w/easy_install)
  2. numpy 1.5 svn (with python setup.py install to $PYTHONPATH location)
  3. numpy 1.3 system

on the python module path.  Moreover, if your system installation of numpy was built with easy_install your 1.5svn attempt falls to third place above. As such if your system owned installation used easy_install to install packages then you are basically forced to also use easy_install because the system easy_install'ed packages will supersede even your $PYTHONPATH unless you also install with easy_install.

This is not to discount easy_install. I always use it.  If you or your root install have used it, just keep always using it. For example, the way I do this with the numpy 1.5 svn versions is:


svn co http://svn.scipy.org/svn/numpy/trunk numpy
cd numpy
python setup.py build
cd ..
easy_install -ZU ./numpy
cd ~
python -c "import numpy; print numpy.__version__,numpy.__file__"

# 1.5.0.dev8079
# ~/usr/local/lib/python2.4/dist-packages/numpy-1.5.0.dev8079-py2.4-linux-x86_64.egg/numpy/__init__.py


Using easy_install, also makes it straightforward for me to switch versions of numpy that I have already installed.  For the working example, I have an easy_installed numpy 1.4 and numpy 1.5.0.dev8079. To switch I simply run easy_install and point at the requested version:


~/usr/bin/easy_install -Z numpy==1.5.0.dev8082 ; python -c 'import numpy; print numpy.__version__'
# 1.5.0.dev8079

~/usr/bin/easy_install -Z numpy==1.4 ; python -c 'import numpy; print numpy.__version__'
# 1.4.0


Starting from scratch with easy_install


At my institution the system administered Linux workstations use python2.4 and lack easy_install and the ipython shell. In this case I set up my ~/.pydistutils.cfg according to the example at the beginning of the tutorial, created the three install directories, and ran:


wget http://pypi.python.org/packages/source/s/setuptools/setuptools-0.6c11.tar.gz#md5=7df2a529a074f613b509fb44feefe74e

tar xzvf setuptools-0.6c11.tar.gz
python setup.py install



which installed easy_install into ~/usr/bin.  Then, without problem, I then ran


~/usr/bin/easy_install -Z nose

~/usr/bin/easy_install -Z numpy

python -c "import numpy;numpy.test()"

~/usr/bin/easy_install -Z ipython==0.9.1

~/usr/bin/easy_install -Z PIL

~/usr/bin/easy_install -Z matplotlib



3. Python 2.6+


Python 2.6 adds a default tree for user directories that is searched before system libraries and without specifying it on the user $PYTHONPATH.  This predefined tree is rooted at "~/.local" in unix environs, where user installed packages are added in "~/.local/lib/python2.6/site-packages" and executable scripts in ~/.local/bin.  You can use this directory without ~/.pydistutils.cfg by compiling source packages with:


python setup.py install --user


Note that an entry in ~/.pydistutils.cfg file, such as


install_lib = ~/usr/lib/python2.6/site-packages



will override the --user option, redirecting modules to install_lib. In this specific case "~/usr/lib/python2.6/site-packages" will not be automatically searched for modules unless it is added to $PYTHONPATH.  Further, the search path now looks like:


  1. $PYTHONPATH 
  2. ~/.local/lib/python2.6/site-packages
  3. System owned site-directories (e.g., /usr/local/python2.4/site-packages)
  4. ~/Library/Python/2.X/site-packages [on Macs]


Finally, to easy_install with Python 2.6's $PYTHONPATH free installation of user installed packages in ".local," users would type:


easy_install --prefix=~/.local -ZU tinyurl


for the original installation example above.  The binary is now in ./local/bin/tinyurl and the module search path with easy_install packages is now:


  1. easy_install to $PYTHONPATH
  2. easy_install to ~/.local/lib/python2.6/site-packages
  3. easy_install to System owned site-directories
  4. easy_install to ~/Library/Python/2.X/site-packages [on Macs]
  5. $PYTHONPATH 
  6. ~/.local/lib/python2.6/site-packages
  7. System owned site-directories
  8. ~/Library/Python/2.X/site-packages [on Macs]
Posted by virtualastronomer
Add a comment...

3 Comments

  1. Update by astropython contributor (2010-02-01)

    Significant update to tutorial to work through nuances relating to the structure of the python module path and the influence of easy_install.
  2. Re: Article by Anonymous (2013-05-22)

    Thank you for your howto ... I've a question:
    I installed spynner, but when I try to import it in python I get

    ImportError: ~/.local/lib/python2.6/site-packages/autopy-0.51-py2.6-linux-x86_64.egg/autopy/alert.so: failed to map segment from shared object: Permission denied

    Well, I have no root on that server, which is a hosted shell account. I wonder what I could do about it. I'm allowed to execute self-compiled binaries there, so I should be allowed to load .so files, right?

    Thanks in advance, I hope you have an idea :)
    King regards.

  3. Re: Article by Rahman (2013-12-18)

    Thank you so much, very helpful

Enter Your Comment