OOF2: The Manual

Figure 7.2. File Arrangement for External Extensions

The file layout for “external” extensions. Names in red are directories, and lines between them indicate nested subdirectories. Blue arrows indicate the information flow during the build process. Green arrows show run-time dependencies, and are equivalent to the black arrows in Figure 7.1.

Only the files listed in bold need to be created by hand. distutils and the setup.py script creates the rest.

The top level directory, labelled Extension in Figure 7.2 can be located anywhere.

OOF2 external extension code should be organized at the top level into Python packages. Each subdirectory below the top-level Extension directory will contain the files for one package, and this directory should have the same name as the package. The distutils library installs each of these packages and an __init__.py file. The directory named package in the figure will become a package named package, with modules (corresponding to Python files in Extension/package) which can be loaded into Python with import package.module. There can be more than one package defined in an extension directory, and packages can contain subdirectories (subpackages). There must be an __init__.py file in each subdirectory.

The directories wrapper and build are created automatically by setup.py when the extension is built. build is used by distutils to store intermediate results, and its name should not be changed. wrapper stores output from swig. Its name is set in setup.py.

All of the directories under the Installation Directory on the right side of Figure 7.2 are created automatically by distutils. The installation directory itself is determined at build time, when setup.py is run.

setup.py is a Python script that uses the distutils library to build an OOF2 extension. The distutils library is included with the Python distribution. (A working example setup.py is included in the OOF2 distribution's examples/extension directory.) The first lines (other than comments) of setup.py must be

import distutils.core
import oof2config
import oof2extutils
	

distutils.core is the highest level module from the Python distutils library. The oof2config and oofextutils modules are provided as part of a correct OOF2 installation. oof2config contains information on how OOF2 itself was built, and sets sys.path so that some OOF2 utility functions are available. oof2extutils contains Python functions and classes that aid the construction of OOF2 extensions.

The rest of setup.py is involved with setting up the arguments for the function distutils.core.setup, which does most of the work of building and installing an extension. distutils needs to be told how to construct the shared libraries and wrappers in Figure 7.1, and which Python files it should include.

distutils.core.setup uses a list of oof2extutils.SharedLibrary objects to determine how to build the shared libraries. setup.py must create these objects by calling the oof2extutils.SharedLibrary constructor. One SharedLibrary instance must be created for each shared library that will be built. The SharedLibrary constructor has only two required arguments, the name of the library and a list of source files. It's invoked, for example, like this:

shlib = oof2extutils.SharedLibrary(name="mylib",
                                   sources=['mysrc/a.C', 'mysrc/b.C'])

The parameter name is the name of the library (without a lib prefix or .so or other suffix). sources is a list of C or C++ files to compile. The file names should be specified relative to the current directory (the directory containing setup.py), or given as absolute path names (beginning with “/”).

See the oof2extutils.SharedLibrary reference page for a list of additional optional arguments.

The next step is to run swig on the swig input files and to build Python extension modules from the results. OOF2 supplies a function, oof2extutils.run_swig, that invokes swig with the appropriate arguments. The C++ files that swig produces need to be compiled into a wrapper library, and distutils needs to be told that swig's Python output files should be included in the extension package. oof2extutils contains a utility function, get_swig_ext that calls run_swig and returns both an oof2extutils.Extension object (for the wrapper library) and the name of the python package:

ext_obj, pkg = oof2extutils.get_swig_ext(srcdir = 'mysrc',
                                         srcfile = 'myext.swg',
                                         destdir = 'swigout',
                                         libraries = ['mylib'])

The parameter srcdir is the directory containing the swig source files. In this example, the swig source is in the same directory as the C++ source for the SharedLibrary, above. srcfile is the name of the swig file, and destdir is the destination directory, where the C++ and Python output files will be written. If srcfile contains a subdirectory name, the same subdirectory will be created within the destination directory. Finally, libraries is a list of libraries with which the extension module should be linked. Generally, this should be the name of the SharedLibrary object, as in the example above. The return values are in the indicated order, with ext_obj being the oof2extutils.Extension object and pkg being the name of the Python package.

The Python extension module created by the above call to get_swig_ext can be imported into any Python program as swigout.myext, combining the names of the destination directory (swigout) and source file (myext.swg).

Additional optional arguments to get_swig_ext are documented on its reference page, oof2extutils.get_swig_ext. If for some reason get_swig_ext doesn't do what you need, you'll need to read the reference page for run_swig and for the oof2extutils.Extension class.

The last thing that setup.py needs to do is to call distutils.core.setup, like this:

distutils.core.setup(name = "oof2ExtensionName",
                     version = "0.0.0",
                     author = "John Q. Oof",
                     author_email = "[email protected]",
                     url = "http://www.address.net/~jqoof",
                     ext_modules = [ext_obj],
                     packages = [mysrc, pkg],
                     shlibs = [shlib])
	

The first five arguments are self explanatory. ext_modules is a list of the Python extension modules created by oof2extutils.get_swig_ext. packages is a list containing both the Python packages returned by get_swig_ext and any hand-written Python packages^[43] (in Figure 7.1, the former correspond to the boxes labelled “Python Wrapper Code” and the latter to the box labelled “Extension's Python Modules”).

After creating the source files and writing setup.py, building an OOF2 extension is straightforward.

The process described in Section 7.2.3 creates modules and libraries for OOF2, but it doesn't tell OOF2 anything about them, so the modules won't be used. Modules must be imported before they're accessible to Python. Generally, an OOF2 extension is designed so that only one of its modules needs to be imported explicitly — this main module then imports the rest of the modules.

There are three ways of importing an extension into OOF2. Assume that the extension is called ABC and its main module is contained in the file start.py. The three ways to import it are:

ImportError: No module named ABC
	  

ImportError: Failure linking new module: [...]
	  

ImportError: [...]: cannot open shared object file: [...]