Awasu » Extending Python: Callbacks
Monday 4th November 2019 4:29 PM []

Now that we've got our extension up and running, we may well want to call Python code from our C/C++ code, perhaps to invoke callbacks[1]For example, to update the UI if the C/C++ code is doing some lengthy processing. or to do higher-level processing e.g. sorting a list.

Accepting an optional callback parameter

We'll change the add() function to accept an optional callback parameter that, if provided, will be called to log the progress of the C++ code.

First, we need to tell Python that our function accepts keyword parameters, which we do by including the METH_KEYWORDS flag:

static PyMethodDef gMethodDefs[] = {
    { "add",(PyCFunction)py_add, METH_VARARGS|METH_KEYWORDS, "Add two numbers." },
    { NULL, NULL, 0, NULL }

We then have to change our C++ function to accept an extra parameter, that will contain any keyword parameters, then use PyArg_ParseTupleAndKeywords() to extract both the positional and keyword parameters passed in to us:

py_add( PyObject* pSelf, PyObject* pArgs, PyObject* pKywdArgs )
    // extract the arguments
    int val1, val2 ;
    PyObject* pCallback = NULL ;
    const char* argNames[] = { "val1", "val2", "callback", NULL } ;
    if ( ! PyArg_ParseTupleAndKeywords( pArgs, pKywdArgs, "ii|O", (char**)argNames, &val1, &val2, &pCallback ) )
        return NULL ;

Note that we have to provide the names of all our parameters, both positional and keyword, and the format specifier we pass into PyArg_ParseTupleAndKeywords() uses a vertical bar to separate the required positional parameters from the optional keyword parameters.

It would also be prudent to check that what was passed in to us as the callback is actually callable:

    // check that the callback is callable
    if ( pCallback != NULL ) {
        if ( ! PyCallable_Check( pCallback ) ) {
            PyErr_SetString( PyExc_ValueError, "Invalid callback." ) ;
            return NULL ;

Invoking the callback

Since we will be invoking the callback in several places, we implement it as a helper function:

static void
callback( PyObject* pCallback, const string& msg )
    if ( pCallback == NULL )
        return ;

    // invoke the callback
    PyObject* pArgs = Py_BuildValue( "(s)", msg.c_str() ) ;
    PyObject* pKywdArgs = NULL ;
    PyObject* pResult = PyObject_Call( pCallback, pArgs, pKywdArgs ) ;
    Py_DECREF( pArgs ) ;
    Py_XDECREF( pKywdArgs ) ;
    Py_DECREF( pResult ) ;

First, we build a tuple containing the arguments that we will pass to the callback i.e. a progress message. Then, we invoke the callback by calling PyObject_Call() with a tuple of positional arguments, and a dictionary of keyword arguments (in this case, there are none).

Note that we have to clean up (i.e. decrement the reference counts[2]You can find out more about how Python uses reference counts to manage object lifetime in this companion tutorial.) the argument objects we created, and also the result object returned to us[3]Since ownership of this object was transferred to us..

We can now update the main C++ code to log its progress:

    // add the numbers
    callback( pCallback, MAKE_STRING( "Adding: " << val1 << " + " << val2 ) ) ;
    int result = val1 + val2 ;
    callback( pCallback, MAKE_STRING( "- result = " << result ) ) ;

I'm using the MAKE_STRING() macro from this article, that makes it easy to dynamically build strings, which is defined like this:

#define MAKE_STRING( msg )  ( ( (std::ostringstream&) ( std::ostringstream() << msg ) ).str() )

Testing the callback

After building and installing the new extension, we can see that things work as they did before, if we don't provide a callback:


But if we provide a callback, we can see it getting called as the add() function executes:


Download the source code here.

« Handling errors

Tutorial index

Managing the GIL »

   [ + ]

1. For example, to update the UI if the C/C++ code is doing some lengthy processing.
2. You can find out more about how Python uses reference counts to manage object lifetime in this companion tutorial.
3. Since ownership of this object was transferred to us.
Have your say