6
2010-03-16
Snippet: Parallel map using multiprocessing
This module (courtesy Brian Refsdal at SAO) implements a parallelized version of the native Python map function that utilizes the Python multiprocessing module to divide and conquer an iterable. It takes advantage of the nifty NumPy function array_split() to divide an input iterable into approximately equal chunks. It also demonstrates the use of the multiprocessing Manager class and how to use Queues in multiprocessing.
The updated version below ...
2009-12-27
Blog: MPI Cluster Programming with Python and Amazon EC2
Here is an interesting talk from PyCon 2008 by Peter Skomoroch (Juice Analytics). Has anyone tried EC2? In our local discussions it was said that EC2 is relatively expensive compared to other options, any comments? I'm working on a smallish problem that would benefit from having a couple hundred processors for interactive analysis. From the talk this can be done with ipython1 and ElasticWulf. Here is the ...
