Extending Python with Numerical Recipes

Numerical Recipes code, or for that matter any other C++ code, can easily be invoked from within Python. For some tasks, C++ executes hugely faster than Python code, and you can also access features in Numerical Recipes (or elsewhere) that are not available in Python. You can code some parts of your project in Python and other parts in C++, and control the whole project from the Python console or a Python script.


Conventions and Overview
Compiling "Hello, world!"
Example 1: Getting and Return Scalar Variables
Example 2: Vector, Matrix, String, List, and Dict Variables
Example 3: Encapsulating a C++ Function for Other C++ Extensions
Example 4: Sending Functions to C++ Extensions as Arguments
Example 5: Persistent Objects and Object-Oriented Extensions
Example 6: Sharing Objects by Name Between Python and C++
Appendix: List of All NRpy Facilities

Conventions and Overview

Python code (or interactive input) is shown on a red background:

# Python code include somemodule somemodule.somefunction()

C++ code is shown on a green background:

#include "nr3python.h" // this is what "Hello, world" looks like static PyObject* somefunction(PyObject *self, PyObject *pyargs) { printf("Hello, world!\n"); return NRpyObject(); } // standard boilerplate (don't worry, we'll explain) static PyMethodDef somemodule_methods[] = { {"somefunction", somefunction, METH_VARARGS, "somefunction() doc string"}, {NULL, NULL, 0, NULL} }; PyMODINIT_FUNC initsomemodule(void) { import_array(); Py_InitModule("somemodule", somemodule_methods); }

A blue background is used for input to the Linux shell, or output from running code.

Hello, world!

NR always means, of course, Numerical Recipes.

By convention, identifiers beginning with "Py" are Python things, as documented in the Python C API Reference Manual. We hope to shield you from as many of those things as we can, but, if you want, you can use any or all of the Python C API interface inside of a NR-style Python extension.

Identifiers that begin with "NRpy" (we pronounce it "nurpee") are defined in our header file nr3python.h, and are designed to make it easier to go back and forth between Python and C++ (especially C++ with NR). In the code above, the only example is "NRpyObject()" which, as written, returns the more Pythonic "Py_None"

When we say "standard boilerplate", we always mean code analgous to the above for (i) a PyMethodDef array, and (ii) a PyMODINIT_FUNC function. These are just ugly little pieces of the Python C API that we can't easily shield you from. They always go at the end of your code and are completely formulaic. Get used to them.

Compiling "Hello, world!"

The hardest part of this whole tutorial is getting code like the above to compile. Once you figure out how to do this, the rest will be easy.

Copy the green box above into a file "somemodule.cpp" and try to compile it. In Windows, we do this in MS Visual Studio. In Linux we use g++ on the command line, like this:

$ g++ -fPIC -fpermissive -w -c somemodule.cpp -o somemodule.o \ -I/usr/include/python2.6 $ g++ -shared somemodule.o -o somemodule.so

(You might or might not need the "-fpermissive" and the "-w" options. You will definitely need the "-fPIC" and "-shared".)

Your first hurdle might be that the compiler can't find "nr3python.h". This means that you haven't downloaded our interface file and put it in a place that the compiler can find. Do that.

Your second hurdle might be that the compiler can't find files "Python.h" and "numpy/arrayobject.h", which are referenced in nr3python.h. These are files that come with your Python distribution. But you can't just put copies of them somewhere, because they include other files expected to be at fixed relative locations. You have to figure out where their directories are, and add these to your #include search path. (By the way, feel free to edit the first few lines of your downloaded copy of nr3python.h if that makes things easier.)

For Linux, the command lines above will produce the right kind of output file, ending in ".so". and you are done. In Windows, there is one more step: If all now goes well, your file will compile but the linker will complain that it can't find "main()". This is a good sign! You just need to change what kind of file that the compiler is producing, as described in this fine print:

You want to produce a Windows dll file, but it should have a name ending in ".pyd" instead of ".dll". In Visual Studio, you do this by opening "...Properties" on the "Project" menu. Then make two changes under Configuration Properties / General. In Project Defaults, change "Configuration Type" to "Dynamic Library". Then, in General, change "Target Extension" to ".pyd".

(By the way, Visual Studio doesn't make it easy to copy all of a project's settings to a new project. There are various commercial tools for this, so that you don't have to keep going through all the above steps for each new Python extension that you write. We use a tool called CopyWiz.)

You might have an additional hurdle that the linker can't find some expected libraries that are in the Python distribution. If so, you'll need to locate them and set your library path accordingly. Hint: If it can't find a library like "python27_d" (ending in "_d"), then you are compiling in debug mode. As normally distributed, Python doesn't have a debug mode library, so you'll have to instead compile in release (or not debug) mode.

Some users prefer to use Python's distutils package to get the compiling, linking, and installation steps right. You don't really need it for the kinds of things in this tutorial, but you can find out about it here.

When you get things right, you will have produced a ".pyd" file (Windows), or a ".so" file (Linux). Put it in your working directory, fire up the Python interpreter and try it out:

>>> import somemodule >>> somemodule.somefunction() Hello, world! >>>

It can be frustrating to get this to work the first time, but you only have to climb this particular tree once. It will be worth it! If you are stuck, try Googling "compile python extension" or "build python extension", or similar.

Example 1: Getting and Return Scalar Variables
using NRpyArgs, NRpyDoub, NRpyInt, NRpyObject, and NRpyTuple

Here is a module file that makes available three functions that can be called from Python. The first function, func, has one double precision floating point argument (Python calls this "float", while C++ calls this "double") and returns one of the same type. Numerical functions of any complexity are very slow in Python, because they (and any functions they contain) are evaluated from byte code over and over; so writing them in C++ can give an enormous speedup.

#include "nr3python.h" static PyObject* func(PyObject *self, PyObject *pyargs) { NRpyArgs args(pyargs); // unpack args Doub x = NRpyDoub(args[0]); Doub y = (x != 0. ? SQR(sin(x)/x) : 1.); return NRpyObject(y); // return a single value } // ... file continues ...

Notice the use of NRpyArgs to unpack the incoming arguments, and the use of NRpyDoub (or, below, NRpyInt) to cast them to NR's "Doub" (double) or "Int" (int) C++ types. Likewise notice the use of NRpyObject to return a single variable. (If you were returning nothing, you would return "NRpyObject()", as in the "Hello, world" example, above.)

The second function takes three arguments, two Doub and one Int, and returns three values, also two Doub and one Int. Notice how NRpyTuple is used to return more than one value. Since NRpyTuple is actually a varargs C++ function, it needs a way to recognize its last argument; hence that last argument (not returned to Python) must always be NULL, as shown.

static PyObject* gunc(PyObject *self, PyObject *pyargs) { NRpyArgs args(pyargs); // unpack args Doub x1,x2,z1,z2; Int n,m; x1 = NRpyDoub(args[0]); // use args x2 = NRpyDoub(args[1]); n = NRpyInt(args[2]); z1 = pow(x1,n); z2 = pow(x2,n); m = Int(x1+x2); return NRpyTuple( // more than one return must be a tuple NRpyObject(z1), NRpyObject(z2), NRpyObject(m), NULL // NRpyTuple *always* needs NULL as last arg ); } // ... file continues ...

Your C++ function can determine dynamically the number of arguments being sent from Python and do whatever it wants with them. Here is an example that simply sums all its Doub arguments.

static PyObject* hunc(PyObject *self, PyObject *pyargs) { NRpyArgs args(pyargs); // unpack args Doub sum=0.; Int i, n=args.size(); for (i=0;i<n;i++) { sum += NRpyDoub(args[i]); } return NRpyObject(sum); } // ...file continues...

The file ends with standard boilerplate,

// standard boilerplate static PyMethodDef example1_methods[] = { // note required name {"func", func, METH_VARARGS, "func() doc string"}, {"gunc", gunc, METH_VARARGS, "gunc() doc string"}, {"hunc", hunc, METH_VARARGS, "hunc() doc string"}, {NULL, NULL, 0, NULL} }; PyMODINIT_FUNC initexample1(void) { // note required name import_array(); Py_InitModule("example1", example1_methods); }

Notice how the "standard boilerplate" works for the whole file. Its PyMethodDef contains an entry for every function that you are making available to Python. The first (string) argument is what its name is in Python. The second argument is which function in this file it corresponds to, the rest is self-explanatory.

Notice also that the PyMODINIT_FUNC must always be named "init" concatenated with the module name, here "example1".

Now we can use all three functions from Python:

>>>import example1 >>> print example1.func(1.5) 0.4422205548 >>> a,b,c = example1.gunc(1.2, 3.4, 5) >>> print [a,b,c] [2.48832, 454.3542399999999, 4] >>> print example1.hunc(1.,2.,3.,4.,5.,6.) 21.0 >>>

Example 2: Vector, Matrix, String, List, and Dict Variables
using NRpyCharP, NRpyList, NRpyDict

For vectors and matrices on the Python side, we always use numpy, and we assume that you do, too. On the C++ side we use NR's vector and matrix classes, for example, VecDoub and MatDoub.

Our second example file exports to Python a single function "func" that does nothing useful except demonstrate how to move different kinds of objects from Python to C++ and the reverse. The file begins,

#include "nr3python.h" static PyObject* func(PyObject *self, PyObject *pyargs) { NRpyArgs args(pyargs); // unpack args const char *mystring = NRpyCharP(args[0]); // first argument is a string VecDoub myvec(args[1]); // second argument is a Doub vector MatDoub mymat(args[2]); // third argument is a Doub matrix MatInt myimat(args[3]); // fourth argument is an Int matrix NRpyList mylist(args[4]); // fifth argument is a Python list NRpyDict mydict(args[5]); // sixth argument is a Python dict // ... file continues ...

By now you get the idea: You have to know the intended type of each argument that comes in, and you call a C++ constructor that binds it to a corresponding C++ object. These constructors are all in nr3python.h. If you are a C++ wizard, you can easily add new ones to meet additional needs.

Let's look at the string argument:

// string examples Int len = strlen(mystring); // mystring[2] = 'Z'; // Error: mystring is const (and must be!) char *newstring = new char[len+1]; strcpy(newstring,mystring); // ... whatever ... delete [] newstring; // ... file continues ...

Mystring is a null-terminated string, actually a pointer to Python's internal representation of that string. Since the consequences of your altering a Python immutable object are imponderable (doing so might cause the world to end) mystring was declared as "const" on the C++ side above. Then the compiler protects us from altering it. If you want to monkey with a string's contents, make a local copy, as shown above.

Vectors and matrices are instantiated as objects whose data points back into Python. Thus, there is no unnecessary copying of large objects. You can change values in the vector or matrix, or even resize it, and the corresponding change will be reflected on the Python side. For most purposes you can think of the vector or matrix as a reference to the corresponding Python object. However, copying the vector or matrix on the C++ side produces a copy, not a reference, as shown here.

// vector and matrix examples Int v = myvec.size(); // get sizes in usual C++ way Int m = mymat.nrows(), n=mymat.ncols(); myimat[0][1] = 7; // changes a value both here and back in Python MatDoub copyofmymat = mymat; // copy values copyofmymat[1][0] = 3.; // does not change a value back in Python // ... file continues ...

We provide only a rudimentary interface to Python lists, intended for grabbing their values when you want a C++ extension to compute with them. (We also let you reset any existing value, and create a list of numerical values or strings.) Generally you'll want to do any sophisticated manipulation of lists on the Python side. If you really need to do more on the C++ side, you can of course use the Python C API directly.

// list examples Int lsize = mylist.size(); // size of top level of list Int llsize = mylist[0].size(); // size of next level at top level index 0 //PyObject_Print(mylist[0][2][2].p, stdout, 0); Int listval = NRpyInt(mylist[0][2][2]); // component must exist mylist[0][2].set(2,55.); // modifies list[0][2][2] back in Python NRpyList mynewlist(5); // length 5, available to Python only if returned mynewlist.set(0,16.6); // set component 0 mynewlist.set(1,"dog"); // Doub, Int, and char* ok for list values // ... file continues ...

Our easy interface to Python dicts (what C++ programmers would call hash memory objects) allows for values and keys that are Doub, Int, or string. If you need anything more complicated, do it on the Python side. It is sometimes a big convenience to use a dict in C++ programs, however. As shown below, you can create your own, independently of whether you intend to return it to the calling Python program.

// dict examples Doub dictval = NRpyDoub(mydict.get("somekey")); // value known to exist PyObject *testit = mydict.get("anotherkey"); // value may not exist Doub anotherval = (testit != Py_None ? NRpyDoub(testit) : 0.); mydict.set("thirdkey",523.48); // modifies dict back in Python mydict.set(54321,"stringvalue"); // Doub, Int, and char* ok on both args NRpyDict mynewdict; // new dict, available to Python only if returned mynewdict.set("A","one"); mynewdict.set("B",2); mynewdict.set("C",3.1416); Doub newdictval = NRpyInt(mynewdict.get("B")); // has value 2 // ... file continues ...

Notice above that set() takes Doub, Int, or string arguments, while get() returns a PyObject*, and must be cast to the desired type (e.g., Doub, Int, or string) by NRpyDoub() or one of its siblings. We do it this way so that you can test if the result of a get() is Py_None, meaning that there is nothing stored under that key.

Next, how do we return objects like the above to the calling Python program? Any single object can be returned by returning a single NRpyObject(). Multiple objects, as we saw in Example 1, are returned by using NRpyTuple(), as below.

// create some additional local objects as return examples VecDoub vret(6,16.); MatDoub mret(3,2,17.); // return lots of things, for example return NRpyTuple( NRpyObject(v), NRpyObject(m), NRpyObject(vret), NRpyObject(mret), NRpyObject(mynewlist), NRpyObject(mynewdict), NULL ); } // ... file continues ...

It might seem perverse that for incoming arguments we must know exactly what type to cast them to, while for returned values a single NRpyObject() function works for all. That is just an illusion. All C++ objects are in fact statically typed. NRpyObject() is just a templated function that specializes to the type of its argument -- which it knows at compile time. You'll soon get used to the oddities of such interactions between a fully dynamically typed language, like Python, and a fully statically typed one, like C++.

Finally, at the end of the file there is the standard boilerplate,

// standard boilerplate static PyMethodDef example2_methods[] = { {"func", func, METH_VARARGS, "func() doc string"}, {NULL, NULL, 0, NULL} }; PyMODINIT_FUNC initexample2(void) { import_array(); Py_InitModule("example2", example2_methods); } // ... file ends

Here is what Example 2 looks like from the Python side.

>>> from numpy import * >>> import example2 >>> arg1 = "inputstring" >>> arg2 = array([1.,2.,3.]) >>> arg3 = array([[4.,5.,6.], [7.,8.,9.]]) >>> arg4 = array([[10, 11, 12, 13], [20, 21, 22, 23]]) >>> arg5 = [ [[14],["cat"],[[15],[16],[17]]], [["toy"],["ball"]] ,[0.]] >>> arg6 = {"somekey" : 18.} >>> u,v,w,x,y,z = example2.func(arg1,arg2,arg3,arg4,arg5,arg6) >>> print u 3 >>> print v 2 >>> print w [ 16. 16. 16. 16. 16. 16.] >>> print x [[ 17. 17.] [ 17. 17.] [ 17. 17.]] >>> print y [16.6, 'dog', None, None, None] >>> print z {'A': 'one', 'C': 3.1416, 'B': 2} >>>

Example 3: Encapsulating a C++ Function for Other C++ Extensions
(also, functions with global parameters)

Example 1 already showed how export a C++ function so that it is callable from Python. But we might also want to use the fast C++ function as the argument of another C++ module, for example one that does numerical integration or solves a differential equation. This Example 3 shows how to do this. (In Example 4 we'll see what this looks like on the receiving side, in the C++ module that receives the function argument.)

#include "nr3python.h" // the function Doub thefunc(Doub x) { // if (x <= 0. || x >= 1.) NRpyException("bad arg in thefunc"); return sqrt(log(x)*log(1.-x)); } // make it callable from Python static PyObject *func(PyObject *self, PyObject *pyargs) { NRpyArgs args(pyargs); Doub ans, x = NRpyDoub(args[0]); ans = thefunc(x); return NRpyObject(ans); // send answer back to Python } // export it as a C++ function for another C++ module static PyObject *funcwrapper(PyObject *self, PyObject *pyargs) { return NRpyObject(thefunc); } // standard boilerplate static PyMethodDef example3_methods[] = { {"func", func, METH_VARARGS, "func(x), as Python function"}, {"funcwrapper", funcwrapper, METH_VARARGS, "func(x), wrapped for NRpy"}, {NULL, NULL, 0, NULL} }; PyMODINIT_FUNC initexample3(void) { import_array(); Py_InitModule("example3", example3_methods); }

So we have exported to Python two objects, one, "func" for evaluating our function "thefunc" within Python, and another, "funcwrapper", for encapsulating it for use in another C++ module.

If your function "thefunc" needed some global state, for example the values of some parameters, you could provide an additional method for setting that state, for example, adding to the above file,

Doub state; // global static PyObject* setstate(PyObject *self, PyObject *pyargs) { NRpyArgs args(pyargs); Doub state = NRpyDoub(args[0]); // first arg is the value of state return NRpyObject(); }

(Of course you also need to add "setstate" to the standard boilerplate.) The above would not be a good way to define large persistent objects, because we would not be giving Python a chance to manage their memory. For that, see Example 5. But for the occasional global parameter value it is fine (stipulating all the usual cautions about global variables).

Let's now see how Example3 is used from the Python side:

>>> import example3 >>> print example3.func(0.4) 0.684152603343 >>> thefunc = example3.funcwrapper() >>> print thefunc <capsule object NULL at 0x01021728> >>> example3.setstate(7.89) >>> print example3.func(1.4) RuntimeError: bad arg in thefunc

Example 4: Sending Functions to C++ Extensions as Arguments
using NRpyPyFunction, NRpyCFunction, NRpyAnyFunction

This is the companion example to Example 3: How do we use functions in a C++ extension when they have been sent in as arguments. They might be functions defined in Python (think if you really want to do this, since these would be slow every time they are called), or else fast functions defined in another C++ module.

The example starts with a Python-callable routine whose first argument is a Python function that is then called as a Doub(Doub) C++ function.

#include "nr3python.h" static PyObject* usepyfunc(PyObject *self, PyObject *pyargs) { NRpyArgs args(pyargs); NRpyPyFunction<Doub> func(args[0]); // specify Doub x = NRpyDoub(args[1]); Doub y = func(x); printf("x=%8.5f y=%8.5f\n",x,y); return NRpyObject(); } // ...file continues...

You always have to know the return type of a Python function from the very start -- again a consequence of C++ being a statically typed language. Notice that "NRpyPyFunction func(args[0])" is calling a constructor of the function "func", not invoking it. Then, "func(x)" is invoking it.

If the Python function referenced by "args[0]" had more than one argument, the constructor call above would be unchanged. However the invocation would change to "y = func(x,z)" or whatever. This works by overloading "func" in nr3python.h. We provide overloads for up to 4 arguments. (You can add more in nr3python.h if you want.) Checking is done to be sure that your C++ code is calling Python functions with the correct number of arguments.

If you are coding for speed, it is more likely that the function that you want to use was already coded in another C++ module, and that you brought it into Python as an encapsulated C++ function, as demonstrated in Example 3. Here is how to use such an encapsulated object in a C++ module, when it arrives as an argument.

static PyObject* usecfunc(PyObject *self, PyObject *pyargs) { NRpyArgs args(pyargs); Doub (*func)(Doub); // declare function ptr of desired type NRpyCFunction(func,args[0]); // bind incoming argument to function ptr Doub x = NRpyDoub(args[1]); Doub y = func(x); printf("x=%8.5f y=%8.5f\n",x,y); return NRpyObject(); } // ...file continues...

Now you have to know the whole function type, here Doub(Doub). Once you use NRpyCFunction to bind the capsule to the function pointer, here "func", then it really is a C++ function. Calls to it do not go through Python at all.

We have found it useful to have a way of using a function that can be either a Python function or an encapsulated C++ function, which one not known except at run time. This has some extra overhead on each function call, and nr3python.h provides the facility only for functions with a single argment. The usage is illustrated by,

static PyObject* useanyfunc(PyObject *self, PyObject *pyargs) { NRpyArgs args(pyargs); NRpyAnyFunction<Doub,Doub> func(args[0]); // specify <return, argument> type Doub x = NRpyDoub(args[1]); Doub y = func(x); printf("x=%8.5f y=%8.5f\n",x,y); return NRpyObject(); } // ...file continues...

We'll still need the standard boilerplate (below), but let's jump ahead to see how these functions look from the Python side. This piece of a .py file,

import example3 import example4 from numpy import * def thepyfunc(x): return sqrt(log(x)*log(1.-x)) print thepyfunc(0.65) print example3.func(0.65) thecfunc = example3.funcwrapper() print thecfunc example4.usepyfunc(thepyfunc,0.65) example4.usecfunc(thecfunc,0.65) example4.useanyfunc(thepyfunc,0.65) example4.useanyfunc(thecfunc,0.65)

yields this output

0.67249195993 0.67249195993 <capsule object NULL at 0x01F0FFB0> x= 0.65000 y= 0.67249 x= 0.65000 y= 0.67249 x= 0.65000 y= 0.67249 x= 0.65000 y= 0.67249

So we've evaluated the same function with six different combinations of where it is called from and whether it is evaluated in Python or C++. You might be sure you understand how each of the six is different.

On the other hand, this additional code,


produces the output,

RuntimeError: NRpyCFunction arg is not a C++ function capsule.

We finish this example with a couple of additional tricks for using Python functions within C++. First, you can set a global object on the C++ side that remembers the name of a Python function. Second, you can find out at run time, on each call, how many arguments a Python function has, and do different things accordingly. But, while these kinds of things are cute, remember that it is usually foolish to use slow Python functions within fast C++ code!

NRpyPyFunction<Doub> func; // global, Python function returning Doub static PyObject* setapyfunc(PyObject *self, PyObject *pyargs) { NRpyArgs args(pyargs); func = NRpyPyFunction<Doub>(args[0]); // set the global return NRpyObject(); } static PyObject* useapyfunc(PyObject *self, PyObject *pyargs) { Doub y; NRpyArgs args(pyargs); Int nargs = func.argcount; if (nargs == 1) y = func(NRpyDoub(args[0])); else if (nargs == 2) y = func(NRpyDoub(args[0]), NRpyDoub(args[1])); else NRpyException("bad number of args in example4"); printf("in useapyfunction got %g\n",y); return NRpyObject(); } // ...file continues...

Python input (continuing above input file):

def anotherpyfunc(x,y): return x**2 + y**3 example4.setapyfunc(thepyfunc) example4.useapyfunc(0.65) example4.setapyfunc(anotherpyfunc) example4.useapyfunc(2.,0.5)


in useapyfunction got 0.672492 in useapyfunction got 4.125

The boilerplate for all of Example 4 is

// standard boilerplate static PyMethodDef example4_methods[] = { {"usepyfunc", usepyfunc, METH_VARARGS, "usepyfunc() doc string"}, {"usecfunc", usecfunc, METH_VARARGS, "usecfunc() doc string"}, {"useanyfunc", useanyfunc, METH_VARARGS, "useanyfunc() doc string"}, {"setapyfunc", setapyfunc, METH_VARARGS, "setapyfunc() doc string"}, {"useapyfunc", useapyfunc, METH_VARARGS, "useapyfunc() doc string"}, {NULL, NULL, 0, NULL} }; PyMODINIT_FUNC initexample4(void) { import_array(); Py_InitModule("example4", example4_methods); }

Example 5: Persistent Objects and Object-Oriented Extensions

If you are a serious C++ programmer, you will probably want, a some point, to "wrap" a non-trivial C++ class (or struct) for use in Python. In particular, you may want to create more than one instance of that class, and be able to manipulate the data within each instance independently -- a key aspect of object-oriented programming.

Let's see how we might wrap the Numerical Recipes class IQagent. We want to make a Python-callable constructor and give access to the methods add(), and report(), each acting on either a float quantity or a vector of float quantities.

Here is what we want the Python to look like:

import scipy import numpy import example5 a = scipy.random.random((1000000,)) b = scipy.random.standard_normal((1000000,)) agent1 = example5.create() agent2 = example5.create() example5.addarray(agent1,a) example5.addarray(agent2,b) pvals = numpy.array([0.025,0.1667,0.5,0.8333,0.975]) print example5.reportarray(agent1, pvals) print example5.reportarray(agent2, pvals) print example5.report(agent1, 0.999) print example5.report(agent2, 0.999) del agent1 # allow memory to be reclaimed del agent2

In the above, agent1 and agent2 are two instances of IQagent (here called "example5" to fit the lesson plan). They each store their own data (set with the addarray methods) and then report back some computed quantities (by the report and reportarray methods).

The C++ code looks like this:

#include "nr3python.h" #include "sort.h" // NR includes #include "iqagent.h" struct IQagentWrapper { IQagent myagent; // an object in the wrapper // declare more objects here as needed IQagentWrapper(NRpyArgs args) { // must define even if empty // use args to instantiate objects in the wrapper // if they are pointers, use "new" } ~IQagentWrapper() { // must define even if empty // clean up, with a "delete" for every "new" } // Python access methods: PyObject *add(NRpyArgs args) { myagent.add(NRpyDoub(args[1])); // arg[0] is "this", recall return NRpyObject(); } PyObject *addarray(NRpyArgs args) { VecDoub vals(args[1]); Int i, n = vals.size(); for (i=0;i<n;i++) myagent.add(vals[i]); return NRpyObject(); } PyObject *report(NRpyArgs args) { Doub p = NRpyDoub(args[1]); return NRpyObject(myagent.report(p)); } PyObject *reportarray(NRpyArgs args) { VecDoub ps(args[1]), vals(ps.size()); Int i, n = ps.size(); for (i=0;i<n;i++) vals[i] = myagent.report(ps[i]); return NRpyObject(vals); } }; // connect each member function to an external call NRpyCONSTRUCTOR(IQagentWrapper,create); // always need this NRpyCONNECT(IQagentWrapper,add); NRpyCONNECT(IQagentWrapper,addarray); NRpyCONNECT(IQagentWrapper,report); NRpyCONNECT(IQagentWrapper,reportarray); // file continues with boilerplate ...

The idea here is that IQagentWrapper is a class that itself instantiates one instance of IQagent, internally called myagent. IQagentWrapper could additionally instantiate anything else, either as automatic variables, or else by the "new" command. We just need to be sure that the IQagentWrapper destructor properly deallocates anything that is created.

NRpyCONSTRUCTOR is a special macro that binds IQagentWrapper's constructor to (here) the Python callable function create(); and it also registers IQagentWrapper's destructor as a Python callback, so that it gets executed if you delete the instance with Python's "del" command. Similarly NRpyCONNECT is a macro that connects the other Python callable functions (add, addarray, report, and reportarray) not just to their corresponding IQagent methods in general, but to them in a particular instance of IQagent. (You can look at the code in nr3python.h to see how we do this, but you shouldn't actually need to understand it!)

The boilerplate at the end of the file is:

static PyMethodDef example5_methods[] = { {"create", create, METH_VARARGS, "create an instance"}, {"add", add, METH_VARARGS, "assimilate a value"}, {"addarray", addarray, METH_VARARGS, "assimilate an array of values"}, {"report", report, METH_VARARGS, "report a quantile"}, {"reportarray", reportarray, METH_VARARGS, "report an array of quantiles"}, {NULL, NULL, 0, NULL} }; PyMODINIT_FUNC initexample5(void) { import_array(); Py_InitModule("example5", example5_methods); }

Execution of the above Python code produces the output:

[ 0.02517341 0.16662346 0.50043114 0.83278793 0.97489099] [ -1.96224741e+00 -9.67599065e-01 -4.86543115e-04 9.69255726e-01 1.96388891e+00] 0.999002184737 3.08859930364

Example 6: Sharing Objects by Name Between Python and C++

Up to now, we have moved data between Python and C++ by the "approved" method of function arguments and return values. Actually, for vectors and matrices, we didn't really move the data. For efficiency, we moved only a reference to the data, not the data itself. Thus, changes made in a vector or matrix argument on the C++ side would also be made on the Python side. (This is a feature, not a bug!)

We can do the same kind of thing not just for function arguments, but for anything by name in Python's namespace. For integer and float scalars, this just retrieves a copy of the value. For vectors and matrices, it binds a reference, so that changes made in the C++ are also made back in Python. These changes can include not just changing data values, but also resizing the objects. The syntax is as follows:

static PyObject *func(PyObject *self, PyObject *pyargs) { // bind objects from the Python namespace Int i = NRpyInt("ii"); Doub d = NRpyDoub("dd"); static char* s = NRpyCharP("ss"); VecDoub v("vv"); MatInt m("mm"); printf("in C++ got i=%d, d=%f, s=%s, v.size=%d, m.size=(%i,%i)\n", i,d,s,v.size(),m.nrows(),m.ncols()); // file continues ...

By default, the above constructors look for the objects, by name, in Python's __main__ namespace. If you want to look in a different namespace, add an argument with the namespace name (as a string) just after the Python variable name. Note also that if you bind a Python name to a C++ global variable, then (back in Python) change what that name's reference, the C++ global will remain bound to the original reference -- which Python might deallocate. So, you should only bind by name into objects that will pass out of C++ scope when your C++ function returns to Python.

We can also do the reverse, namely to create scalars, vectors, or matrices on the C++ side, then send them back into the Python namespace. This is implemented in such a way that Python then takes over management of the data memory, so it is ok to let it pass out of scope on the C++ side. Here are examples:

// bind objects into the Python namespace Int j = 18; Doub e = 2.718; char t[] = "yourstring"; VecInt w(5,55); // length 5 of values 55 MatDoub n(4,5,6.66); // 4 by 5 of values 6.66 NRpySend(j,"jj"); NRpySend(e,"ee"); NRpySend(t,"tt"); NRpySend(w,"ww"); NRpySend(n,"nn"); return NRpyObject(); } // file continues with boilerplate ...

The Python side of this example is:

import scipy import numpy import example6 ii = 17 dd = 1.414 ss = "mystring" vv = numpy.array([1.,2.,3.,4.,5.,6.,7.,8.,9.]) mm = numpy.array([[1,2,3],[4,5,6]]) example6.func() print "in Python, got back ",[jj,ee,tt] print ww print nn

This looks odd, because it prints variables that seemingly have not been defined. But they have been defined, in Python's __main__ namespace, by the call to example6.func. The Python output is:

in C++ got i=17, d=1.414000, s=mystring, v.size=9, m.size=(2,3) in Python, got back [18, 2.718, 'yourstring'] [55 55 55 55 55] [[ 6.66 6.66 6.66 6.66 6.66] [ 6.66 6.66 6.66 6.66 6.66] [ 6.66 6.66 6.66 6.66 6.66] [ 6.66 6.66 6.66 6.66 6.66]]

The boilerplate at the end of the above C++ file is:

static PyMethodDef example6_methods[] = { {"func", func, METH_VARARGS, "func for example6"}, {NULL, NULL, 0, NULL} }; PyMODINIT_FUNC initexample6(void) { import_array(); Py_InitModule("example6", example6_methods); }

Appendix: List of All NRpy Facilities

Here is a list of all the "NRpy" facilities defined in nr3python.h.

Usage shown in the above examples:

unpack arguments from Python:
NRpyArgs(PyObject* pyaargs)

bind Python objects (by reference or name) to C++ variables:
int NRpyInt(PyObject* ob)
int NRpyInt(char *name, char *dict = NULL)
double NRpyDoub(PyObject* ob)
double NRpyDoub(char *name, char *dict = NULL)
char* NRpyCharP(PyObject *ob)
char* NRpyCharP(char *name, char *dict = NULL)
NRvector(PyObject *a) // used for VecDoub and VecInt
NRvector(char *name, char *dict = NULL)
NRmatrix(PyObject *a) // used for MatDoub and MatInt
NRmatrix(char *name, char *dict = NULL)

wrappers for Python List, Dict, Tuple:
struct NRpyDict // and various methods
struct NRpyList // and various methods
PyObject* NRpyTuple(PyObject *first, ...)

function wrappers:
void NRpyCFunction(T* &fptr, PyObject* ob)
NRpyPyFunction(PyObject *ob)
NRpyAnyFunction(PyObject *ob)

wrap C++ variables for returning to Python:
PyObject* NRpyObject(const double a)
PyObject* NRpyObject(const int a)
PyObject* NRpyObject(const bool a)
PyObject* NRpyObject(const char *a)
PyObject* NRpyObject() // Python None
PyObject* NRpyObject(NRpyList &a)
PyObject* NRpyObject(NRpyDict &a)
PyObject* NRpyObject(NRvector &a)
PyObject* NRpyObject(NRmatrix &a)

put reference to C++ variable into Python namespace:
void NRpySend(T &a, char *name, char *dict=NULL)

macros used to bind persistent objects:

Not directly used in the above examples (see nr3python.h for usage):

PyObject* NRpyException(char *str, int die=1, int val=0)
char NRpyMainName[] = "__main__";
PyObject* NRpyGetByName(char *name, char *dict = NULL)
int NRpyIsNumber(PyObject* ob)
int NRpyTypeOK(PyObject *a)
int NRpyTypeOK(PyObject *a)
int NRpyDataType()
int NRpyDataType()
double NRpyCast(PyObject *a)
int NRpyCast(PyObject *a)
char* NRpyCast(PyObject *a)