OOF2: The Manual

7.2. External Extensions

This section describes how to create an external OOF2 extension.

[Note] Note

There's no fundamental reason that external OOF2 extensions must be built using the procedure described here. Other techniques certainly exist, and users who are proficient with swig and distutils, or can do without them, should feel free to do it their own way. The methods described here should be easy for most users and should suffice in almost all cases, though.

7.2.1. Directory Structure

Figure 7.2. File Arrangement for External Extensions

File Arrangement for External Extensions

The file layout for external extensions. Names in red are directories, and lines between them indicate nested subdirectories. Blue arrows indicate the information flow during the build process. Green arrows show run-time dependencies, and are equivalent to the black arrows in Figure 7.1.

Only the files listed in bold need to be created by hand. distutils and the setup.py script creates the rest.

The top level directory, labelled Extension in Figure 7.2 can be located anywhere.

OOF2 external extension code should be organized at the top level into Python packages. Each subdirectory below the top-level Extension directory will contain the files for one package, and this directory should have the same name as the package. The distutils library installs each of these packages and an __init__.py file. The directory named package in the figure will become a package named package, with modules (corresponding to Python files in Extension/package) which can be loaded into Python with import package.module. There can be more than one package defined in an extension directory, and packages can contain subdirectories (subpackages). There must be an __init__.py file in each subdirectory.

The directories wrapper and build are created automatically by setup.py when the extension is built. build is used by distutils to store intermediate results, and its name should not be changed. wrapper stores output from swig. Its name is set in setup.py.

All of the directories under the Installation Directory on the right side of Figure 7.2 are created automatically by distutils. The installation directory itself is determined at build time, when setup.py is run.

7.2.2. The setup.py File

setup.py is a Python script that uses the distutils library to build an OOF2 extension. The distutils library is included with the Python distribution. (A working example setup.py is included in the OOF2 distribution's examples/extension directory.) The first lines (other than comments) of setup.py must be

import distutils.core
import oof2config
import oof2extutils

distutils.core is the highest level module from the Python distutils library. The oof2config and oofextutils modules are provided as part of a correct OOF2 installation. oof2config contains information on how OOF2 itself was built, and sets sys.path so that some OOF2 utility functions are available. oof2extutils contains Python functions and classes that aid the construction of OOF2 extensions.

The rest of setup.py is involved with setting up the arguments for the function distutils.core.setup, which does most of the work of building and installing an extension. distutils needs to be told how to construct the shared libraries and wrappers in Figure 7.1, and which Python files it should include.

distutils.core.setup uses a list of oof2extutils.SharedLibrary objects to determine how to build the shared libraries. setup.py must create these objects by calling the oof2extutils.SharedLibrary constructor. One SharedLibrary instance must be created for each shared library that will be built. The SharedLibrary constructor has only two required arguments, the name of the library and a list of source files. It's invoked, for example, like this:

shlib = oof2extutils.SharedLibrary(name="mylib",
                                   sources=['mysrc/a.C', 'mysrc/b.C'])

The parameter name is the name of the library (without a lib prefix or .so or other suffix). sources is a list of C or C++ files to compile. The file names should be specified relative to the current directory (the directory containing setup.py), or given as absolute path names (beginning with /).

See the oof2extutils.SharedLibrary reference page for a list of additional optional arguments.

The next step is to run swig on the swig input files and to build Python extension modules from the results. OOF2 supplies a function, oof2extutils.run_swig, that invokes swig with the appropriate arguments. The C++ files that swig produces need to be compiled into a wrapper library, and distutils needs to be told that swig's Python output files should be included in the extension package. oof2extutils contains a utility function, get_swig_ext that calls run_swig and returns both an oof2extutils.Extension object (for the wrapper library) and the name of the python package:

ext_obj, pkg = oof2extutils.get_swig_ext(srcdir = 'mysrc',
                                         srcfile = 'myext.swg',
                                         destdir = 'swigout',
                                         libraries = ['mylib'])

The parameter srcdir is the directory containing the swig source files. In this example, the swig source is in the same directory as the C++ source for the SharedLibrary, above. srcfile is the name of the swig file, and destdir is the destination directory, where the C++ and Python output files will be written. If srcfile contains a subdirectory name, the same subdirectory will be created within the destination directory. Finally, libraries is a list of libraries with which the extension module should be linked. Generally, this should be the name of the SharedLibrary object, as in the example above. The return values are in the indicated order, with ext_obj being the oof2extutils.Extension object and pkg being the name of the Python package.

The Python extension module created by the above call to get_swig_ext can be imported into any Python program as swigout.myext, combining the names of the destination directory (swigout) and source file (myext.swg).

Additional optional arguments to get_swig_ext are documented on its reference page, oof2extutils.get_swig_ext. If for some reason get_swig_ext doesn't do what you need, you'll need to read the reference page for run_swig and for the oof2extutils.Extension class.

The last thing that setup.py needs to do is to call distutils.core.setup, like this:

distutils.core.setup(name = "oof2ExtensionName",
                     version = "0.0.0",
                     author = "John Q. Oof",
                     author_email = "jqoof@address.net",
                     url = "http://www.address.net/~jqoof",
                     ext_modules = [ext_obj],
                     packages = [mysrc, pkg],
                     shlibs = [shlib])

The first five arguments are self explanatory. ext_modules is a list of the Python extension modules created by oof2extutils.get_swig_ext. packages is a list containing both the Python packages returned by get_swig_ext and any hand-written Python packages[44] (in Figure 7.1, the former correspond to the boxes labelled Python Wrapper Code and the latter to the box labelled Extension's Python Modules).

7.2.3. Building External Extensions

After creating the source files and writing setup.py, building an OOF2 extension is straightforward.

  1. Make sure that OOF2 itself is installed.

  2. Build the extension by typing

    % python setup.py build

    in a Unix shell.

  3. Install the extension by typing

    % python setup.py install

    If you aren't the superuser this command will fail because you won't have permission to write to the default installation directories. There are two solutions to this problem:

    1. Become a superuser or ask one to help you.

    2. Install the files into a directory that you own, via the --prefix command line option:

      % python setup.py install --prefix=/home/oofuser

      The example given above will put shared libraries in /home/oofuser/lib[45] and Python modules in /home/oofuser/lib/python2.4/site-packages (with the directory name python2.4 adjusted appropriately).

      If this method is used, it may be necessary to set environment variables to get OOF2 to run correctly. See Section

7.2.4. Loading External Extensions

The process described in Section 7.2.3 creates modules and libraries for OOF2, but it doesn't tell OOF2 anything about them, so the modules won't be used. Modules must be imported before they're accessible to Python. Generally, an OOF2 extension is designed so that only one of its modules needs to be imported explicitly — this main module then imports the rest of the modules.

There are three ways of importing an extension into OOF2. Assume that the extension is called ABC and its main module is contained in the file start.py. The three ways to import it are:

  1. An import command can be typed explicitly in Console Window, like this:

    import ABC.start

    This has the advantage of not requiring any premeditation, but it's awkward.

  2. The import command, above, can be placed in the .oof2rc file in the users home directory. The extension will then be loaded automatically for all future OOF2 sessions.

  3. The extension can be imported when OOF2 is started, by using the --import command line option:

    % oof2 --import ABC.start
    Environment Variables

If an OOF2 extension has been installed in a non-standard location, the Python import command may not be able to locate it, and the linker may not be able to find the shared libraries. Both of these problems will occur at run-time, and are fixed by setting shell environment variables. (The method of setting environment variables differs depending on which Unix shell you're using, and won't be discussed here.)

If you get a message like

ImportError: No module named ABC

when you try to load the extension ABC, then you need to set the PYTHONPATH variable. It should be set to the name of the directory where the OOF2 Python modules are installed. If you installed OOF2 with the command python setup.py install --prefix=/home/oofuser[45], then PYTHONPATH must be set to /home/oofuser/lib/python2.4/site-packages (but replace 2.4 by the correct Python version number). If PYTHONPATH needs to contain more than one directory (for example, if it's set to something else for another extension), then just concatenate the directory names, separated by colons.

If you get a message containing something like

ImportError: Failure linking new module: [...]


ImportError: [...]: cannot open shared object file: [...]

then you need to set LD_LIBRARY_PATH (DYLD_LIBRARY_PATH on Mac OS X) to the directory containing your extension's shared libraries. For the example above, with --prefix set to /home/oof2user, LD_LIBRARY_PATH should be set to /home/oof2user/lib.

[44] A package is a directory containing Python source files and an __init__.py file. Code in a file in the directory can be loaded into Python with import directoryname.filename. Code within __init__.py will run the first time any module from the directory is imported. __init__.py can itself import the other files in the directory, in which case the whole package can be loaded with import directoryname.

[45] On Linux and many other Unix systems, user accounts are located under the /home directory, but this practice is not universal, and on heterogeneous clusters, home directories may be soft-linked in complicated ways. In case of doubt, the value of the HOME environment variable may be used to portably refer to the user's home directory, like this:

% python setup.py install --prefix=$HOME