Python for Scientific Programming

No lengthy compilation steps. Feedback is immediately available. You can try out ideas interactively and even inject new code (or corrected code) into running programs.

A great boon to understanding.

Interactively try out all the examples that follow.

• dir(), dir(1)
• help(dir), help(range)
• type(1), type(type(1)), type(type(type(1)))

Let's explore two basic stumbling blocks: get Python to give you the real and imaginary components and the conjugate of c, where

c = 1+2j

imag(c) # NameError
c.imag
c.real
c.conjugate # This is not an error. You got the object you asked
            # for. It's a callable. If you want it to do anything, you will have to
            # call it
c.conjugate()

• Look for names in the correct namespace
• Call callables when you want their result; don't call them if you want the callable itself.
• Functions are data

• import math
• from math import sin, cos
• import math as calc
• from math import sin as cos

Ask Python for the sine of one sixth of pi.

n = 10
while n > 0:
    print n
    n -= 1

for item in sequence:
    print item

for position, item in enumerate(sequence):
    print position, item

Python's for loops abstract iteration over containers.

lambda	XXX	:	YYY
The function which takes the arguments	XXX	and returns	YYY

def add_23(n):
    return n+23

map(add_23,        range(10))
map(lambda n:n+23, range(10))

def is_odd(n):
    return n%2

def is_even(n):
    return not n%2

filter(is_odd,       range(10))
filter(lambda n:n%2, range(10))

Remember:

• Functions are data.
• Functions are objects.
• (Almost) everyhing is an object in Python!

List comprehensions

[ (x,x*x) for x in range(10) ]
[ (x, x*x) for x in range(10) if x%2 ]
[ letter*N for N in range(1,5) for letter in 'abcd' ]

Dict comprehensions (new in Python 2.7)

{ x:x*x for x in range(10) }

Set comprehensions (new in Python 2.7)

{ x*x for x in range(10) }

Generator expressions

( (x,x*x) for x in range(10) )
( (x, x*x) for x in range(10) if x%2 )
( letter*N for N in range(1,5) for letter in 'abcd' )

Lazy version of list comprehensions

An abstract data type is one defined by the operations you can carry out on it.

def make_queue():
    return []

def add_to_queue(queue, item):
    queue.append(item)

def remove_from_front_of_queue(queue):
    return queue.pop(0)

queue = make_queue()
add_to_queue(queue, 1)
add_to_queue(queue, 2)
add_to_queue(queue, 3)
remove_from_front_of_queue(queue)
remove_from_front_of_queue(queue)
remove_from_front_of_queue(queue)
remove_from_front_of_queue(queue)

class Queue(object):

    def __init__(queue):
        queue.data = []

    def add(queue, item):
        queue.data.append(item)

    def remove(queue):
        if queue.data:
            return queue.data.pop(0)
        raise Exception("Cannot remove from queue: queue is empty.")
        #raise EmptyQueue("Cannot remove from queue: queue is empty.")

queue = Queue()
queue.add(1)
queue.add(2)
queue.add(3)
queue.remove()
queue.remove()
queue.remove()
queue.remove()

NB, always use the name self for the first argument of methods: It's the strongest convention in Python, and using any other name will lead to confusion. I used the name queue instead, for didactic purposes, because it ties in with the preceding narrative. I wouldn't dream of perpetrating such crimes in production code!

• Numbers, strings, lists, dictionaries
• Functions
• Types and classes
• Modules
• Stack frames

Once you grasp the fact that functions are just objects which happen to be callable, Python will fit your brain much better.

Any objects that are callable can be thought of as functions.

l = []
store = l.append
store('apple')
store('banana')
l

map(int, '012345')

If it looks like a duck, and it quacks like a duck, then it's a duck.

What a Python object can do is far more important that its type.

I don't care who your parents are, as long as you can do the job.

Avoid checking types, check capabilities. This is often achieved by using avoiding Look Before You Leap style

if type(candidate) is not JugglingClanMember:
    print "I'm not even going to let you try"
else:
    print candidate.juggle_balls(a,b,c,d,e)

and using Easier to Ask for Forgiveness than Permission style instead

try:
    print candidate.juggle_balls(a,b,c,d,e)
except AttributeError:
    print "Hmm, you don't seem to know how to juggle at all."
except JugglingError:
    print "I'm sorry, I seem to have given you too many balls."

Python is far from perfect. Yes, there are inconsistencies, special cases, dark corners and warts. But compared to many other languages, Python is very regular, consistent and logically clean.

If you get a good grasp of certain key ideas, you will be able to understand Python much better, and will be a much more empowered Python programmer.

• Everything is an object.
• Every object is a namespace (and every namespace is an object).
• Functions are data. Call them if you want their result right now; don't call them if you need the function itself rather than its result.
• Assginment is (re-)binding, nothing more.
• Duck Typing.
• Duck Typing.
• Duck Typing!!

Unfortunately we do not have the time to (formally) cover these topics with the care they deserve.

def make_adder(n):
    def adder(m):
        return n+m
    return adder

add3 = make_adder(3)
add9 = make_adder(9)
add3(10), add9(10)

• Function factories
• Stateful functions
• Functions accepting arguments at different times

Python metaprogramming is based around two simple facts

• pretty much everything is a Python object which can be manipulated, in pure Python, at run time
• Python is very dynamic: very little is fixed at compile time; most things can be modified at run time.

We've just looked at some powerful techniques.

Some of you may find them esoteric.

Some of you may think you'll never use them. Maybe, but if you understood any of it, it will make you a better Python programmer overall.

Make sure to try IPython's edit magic, and to set your EDITOR environment variable to something you like; by default it's vi which is an experience many of you will not enjoy!

I will rely on your ability to use this in the major exercise at the end of the course.

There are many ways to initialize Numpy arrays:

[Hint: start ipython with <tt>ipython –pylab</tt> to have all the required names imported automatically.]

array([2,9,3,6])
arange(10)
linspace(-10, 10, 11)
logspace(1,10, 4, base=2)
empty(4)
zeros((4,2))
ones((4,2), complex)
ones((4,2,3))
a = empty((2,3))
ones_like(a)
a.shape
a.reshape(6,1)
a
ogrid[20:30:2]
ogrid[-10:10:5j]
ogrid[20:30:2, -10:10:5j]

and many, many more.

Using the following as our sample array

shape = (4,5,6)
a = arange(prod(shape)).reshape(shape)

try to understand the results given by the following

a[0, 0, 0]
a[-1, 1, -2]
a[1, :, 2]
a[2, 3]
a[:, 1, -2]
a[0, 2:, 1:3]
a[1:3, 2:-1, 3:-1]

Ufuncs are Numpy functions which act elementwise on the members of an array.

sin(linspace(0, pi/2, 100))
log2(logspace(0,10, 21, base=2))

[name for (name,obj) in locals().items() if type(obj) is ufunc]

Most operators are overloaded to act as ufuncs on Numpy arrays

arange(10) < 5
ogrid[-1:1:11j] ** 2
x,y = ogrid[-1:1:11j, -1:1:5j]
x
y
x+y
(x+y) ** 2
imshow((x+y) ** 2)

Beware of temporaries: the expression (x+y) ** 2 creates a temporary array containing the result of x+y, which is then used to create the final result. When dealing with large arrays, you may wish/need to avoid these temporaries.

Numexpr is a package which (among other things) automates the process of eliminating such temporaries.

To get an idea of Matlab's capabilities, use IPython's loadpy magic to run some of these examples. For instance

%loadpy http://matplotlib.sourceforge.net/examples/animation/dynamic_image.py

or

%loadpy http://matplotlib.sourceforge.net/examples/event_handling/data_browser.py

loadpy gives you a chance to edit the loaded code before executing it: if nothing seems to be happening, press ENTER once more.

Create an ordinary python module, let's call it triangle.py:

def triangle(n):
    total = 0
    for i in range(1,n+1):
        total += i
    return total

Compile it with cython:

cython triangle.py

Look at the generated file, triangle.c.

Create a file (traditionally called setup.py) containing

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

ext_modules = [Extension("triangleC", 
                         ["triangle.py"])]

setup(
    name = "Cython-test",
    cmdclass = {'build_ext':build_ext},
    ext_modules = ext_modules)

With this in place, all you need is

python setup.py build_ext --inplace

from triangleC import triangle
map(triangle, range(10))

Confirm that triangleC.triangle has identical semantics to triangle.triangle.

[Here are the files for your convenience: triangle.py, setup.py]

Test relative performance of the pure Python module and the equivalent Cython-generated extension module:

python -m timeit -s "from triangle  import triangle" "triangle(100)"
python -m timeit -s "from triangleC import triangle" "triangle(100)"

On my machine I get a speedup factor of 1.5 (2). Not bad, but not very impressive. We can do much better: The extension module still does too much unnecessary work, such as repeatedly checking the type of the data involved, even though the type will always be int.

Ask Cython to annotate your source:

cython -a triangle.py

and use a web browser to view triangle.html. The more yellow you see, the more potential there is to speed up that line.

Clicking on a source line in the annotator output will toggle display of the C code that the line generates.

Cython allows you to introduce type declarations into your program. The primary syntax for doing this is not compatible with pure Python, so we must rename the source file to triangle.pyx.

Try adding/removing the following declarations, and see how it affects the output of Cython's annotator, and how it affects the execution speed. (Don't forget to change traingle.py to triangle.pyx in setup.py.)

Declare the parameter type

def triangle(int n):
    ...

Declare the type of a local variable

def ... :
    ...
    cdef int i
    ...

Declare the return type

cpdef int triangle(int i):
    ...

Try to turn everything white. [You should be able to turn everything except the first and last lines of triangle completely white.]

On my machine timeit now reports a speedup factor of 46 (66). [Solution: triangle.pyx, setup_pyx.py ]

The triangle function receives Python objects as arguments and returns Python objects. In order to deal with these, some dynamic code must remain: this is the reason the first and last lines of triangle could not be turned white.

We can turn triangle completely white, at the cost of making it inaccessible (directly) from Python: just change the cpdef on the first line of triangle to cdef. Observe that this has two effects:

• triangle turns purely white in the annotator
• traingle disappears from the triangleC module

What is the point of having a function which is not callable from Python? You can call it indirectly, from some other function in the module which is callable from Python.

In the time available, we can only scratch the surface of Cython, but we have already seen the central ideas or which it is based:

• Write high-level Python, get low-level C
• Annotate where necessary to improve performance

Some of the features we haven't discussed include

• Cythonized classes
• C++ integration
• Fortran integration
• Numpy integration
• Automagical compilation
• Accessing low-level libraries
• Inlining C/C++

We will use Cython again in the main exercise which follows.

We need a toy program which

• is easy to implement in a short time
• does not require deep domain-sepcific expertise
• is CPU intensive
• gives easily verifiable and pretty results

We'll write a Mandelbrot set explorer.

The explorer should show a square portion of the complex plane. Points on the plane should be given a colour representing the number of iterations it took to prove that they did not belong to the set.

It should be possible to use keystrokes to perform the following actions:

• pan the view port
• zoom in and out
• change the resolution
• change the iteration cutoff

Clicking in the viewport should zoom in, recentering on the clicked point.

(Depending on how quickly we work our way through the material, I expect to skip over many of the details of the development of the larger program we are now going to write. I have tried to provide complete working examples at many intermediate stages, to help you work through the details in your own time.)

Let's start off by writing a program which displays the Mandelbrot set. We won't pay too much attention to style or efficiency for now: we just want something that works.

Run this code with ipython --pylab -i mandelbrot_0.py

Pick a value for the resolution which gives output in a reasonably quick time.

Add the following method to the Mandelbrot class

def pan(self, direction):
    dx,dy = direction
    self._x += dx * self._width
    self._y += dy * self._width
    print "Panning to", (self._x, self._y)
    self.calculate_and_draw()

You should now be able to pan the viewport by, for example, a quarter window's width to the right, by typing mandel.pan((0.25, 0)) in the ineractive session.

We can change the behaviour of an existing instance of our class.

  def pan(self, direction):
      print "Panning has been disabled"

Mandelbrot.pan = pan

mandel.pan((1,2))

The trouble with this is that the behaviour of the class instance no longer matches the source code. This is great for quickly testing ideas, but can easily lead to confusion.

• Edit the definition of Mandelbrot.pan. Give it some behaviour that you will easily recognize.
• Copy the whole Mandelbrot class definition to the clipboard
• type paste in your IPython session
• type mandel.__class__ = Mandelbrot in your IPython session
• Call mandel.pan(...) in your IPython session and confirm that the new behaviour is active.

Note that we changed the type of mandel from the old version of the class, to the new one.

Here is how you might track down all existing instances of the Mandelbrot class:

import gc
need_update = [ instance for instance in gc.get_objects() 
                if type(instance) is Mandelbrot ]

Now use IPython's edit magic to change the definition of a method in the Mandelbrot class. Then you can make all the instances you found earlier pick up the new definition, by changing their class to the new version of Mandelbrot.

for instance in need_update:
    instance.__class__ = Mandelbrot

Write a class decorator, based on the last idea, which automatically updates the instances of the decorated class, when that class is redefined.

Solution

This is useful for trying out new ideas in your classes without losing the existing state of your instances: An interactive development aid.

Here is how we might use it with our Mandelbrot class. Use it to add zoom, change_resoluion and max_iterations methods. Test them and fix any problems interactively, reusing the same single instance of Mandelbrot.

Hints:

def zoom(self, factor=2):
    self._width /= factor
    print "Zooming to width", self._width
    self.calculate_and_draw()

def change_resolution(self, factor=1.1):
    self._resolution = int(factor * self._resolution)
    print "Changing resolution to", self._resolution
    self.calculate_and_draw()

def max_iterations(self, new_value=None):
    if new_value:
        self._max_iterations = new_value
        print "Setting maximum iterations to", self._max_iterations
        self.calculate_and_draw()
    return self._max_iterations

Solution

As we increase the resolution and zoom in (requiring an increase in iteration limit), the time needed to calculate the image grows rapidly. Deeper exploration of the fractal becomes impractical at this speed. We need to optimize.

Before optimizing always profile. Let's use IPython's prun magic to help us. Interactively change the image parameters until the image update takes a few seconds, then profile with

prun mandel.update_image()

On my machine this suggests that around 93% of the time is spent in the escape_time function (and the abs function called inside it). Let's use Cython to speed up escape_time, while keeping as much of the rest of the code as possible in dynamic, flexible, pure Python.

Simply moving out escape_time into a separate module, compiling it with Cython and importing it, results in a speedup factor of around 1.1 on my machine.

Use the Cython annotator to guide you in adding type declarations to escape_time. After doing this, I get a speedup factor of almost 8 WRT the mandelbrot_3.py.

The use of abs seems to be the major cause of remaining yellowness. Let's hand code something equivalent. I now get a speedup factor close to 16 WRT mandelbrot_3.py.

[Note, the speedup factor is highly dependent on the value of max_iterations: while max_iterations = 1000 gives me a speedup factor of 16, max_iterations = 100 gives me a factor of only 2. It is also strongly affected by the region of the complex plane that is being analysed.]

Solution: mandelbrot_4.py, cython_mandel4.pyx, setup_4.py

Mandelbrot.update_image calls escape_time in a loop. On each iteration, an unboxed complex number is extracted from a Numpy array placed into a Python object, which is then sent into escape_time where it is unboxed again and dealt with at a low level. Each point is (pointlessly) hopping back and forth accross the Python-C boundary.

It would be much better if we could push that whole loop down into Cython.

Solution: mandelbrot_5.py, cython_mandel5.pyx, setup_5.py

On my machine, this gives a speedup factor of almost 270 WRT mandelbrot_3.py.

Swap the order of the nested loops in escape_time and see what effect it has on performance. For large arrays you should find that one version is appreciably faster than the other. Why?

Having to type code at the IPython prompt, in order to navigate the Mandelbrot set, is not very convenient. We'll add the ability to navigate with keystrokes and mouse clicks in the viewer.

Look trough the documentation on Matplotlib event handling.

Add click-to-zoom support for the viewer thus:

class Mandelbrot(object):

    def __init__(*args, **kwds):
        # ...
        fig = figure()
        fig.canvas.mpl_connect('button_press_event', self.on_click)
    # ...

    def on_click(self, event):
        # Don't do anything if the click lies outside the axes
        if event.xdata is None or event.ydata is None:
            return
        # Convert axis coordinates into image coordinates and recenter
        # image on the clicked point.
        self._x += self._pixel_to_position(event.xdata, 'x')
        self._y += self._pixel_to_position(event.ydata, 'y')

        print "Event coords:", event.xdata, event.ydata
        print "Image coordsA", self._x, self._y

        # Mouse buttons which zoom after recentering
        if event.button != 1:
            # As you view the fractal boundary in greater detail, you
            # need more iterations to distinguish the features.
            self._max_iterations *= 1.25
            # Not calling self.max_iterations() as this would
            # recalculate the image unnecessarily, *before* we zoom.
            print "Setting maximum iterations to", self._max_iterations
            self.zoom()

        # Any other mouse buttons only recentre
        else:
            print "Recentering on", (self._x, self._y)
            self.calculate_and_draw()

    def _pixel_to_position(self, pixel, axis):
        # In Matplotlib coordinates, the y-axis points downwards, so
        # we reverse the result for the y-axis
        result = (float(pixel) / self._resolution - 0.5) * self._width
        return -result if axis == 'y' else result

Solution

Add a handler which allows you to navigate with keystrokes. For example

• arrow keys to pan the viewer
• + / - to zoom in/out
• R / r to increase/decrease the resolution
• Sequence of digits followed by ENTER to set the iteration limit.

Solution

Let's take a step back to think about how we developed this program.

1. Quick and dirty first version. Easily done with Python's expressive nature, interactive development, and helpful packages such as IPython, Numpy and Matplotlib.
2. Used Cython to optimize just the tiny portion of the code that needed to go faster. (We are running some very efficient low-level code, both in the libraries that we use, and in the Cython-generated portion that we wrote: but we have been spared most of the pain of dealing with the low-level details ourselves.)
3. Continued flexible development with most code in high-level Python.

I submit that this style of development works well in many situations, especially ones where the nature of the code and what it is applied to are exploratory and where rapid interactive feedback is valuable, as is often the case in scientific programming.