About

Welcome to Panela, Matt Harrison's take on mostly Open Source, Linux, Python, innovation in those areas, other buzzwords and Dick Proenneke. It comes complete with the illustrations as needed. Note the opinions expressed here are merely my opinions and not the opinions of my employer.

about Matt

Calendar

««Dec 2010»»
SMTWTFS
    1234
567891011
12131415
16
1718
19202122232425
262728293031

Mailing List

My RSS Feeds








[ANN] coreit - helper to throw Python code onto cores

« H E python :: linux :: open source :: email
posted 2010.07.01 Thu

I've got some Python batch processes that run sequentially and take up 100% of a core. I've got a 4 core machine that was begging me to use it. So I pulled out multiprocessing and wrote a little wrapper. I'm calling it coreit. Here's the code. End result was that it ran 3 times faster (only used 3 of the cores).

This probably exists elsewhere. It's pretty basic but gets the job done. Feedback is welcome.

tags:      

links: digg this    del.icio.us    reddit


comments (2)


1. Artur left...
2010.07.01 Thu 1:56 pm

But... This is already done in stdlib:

p = multiprocessing.Pool(4) res = p.map(slowfun, args)


2. Matt left...
2010.07.01 Thu 2:16 pm

Artur- Thanks for the comment. Obviously I need to RTM ;) It looks like the only advantage to my code is if you might have different functions that you need to run concurrently. map only supports one function, mine supports multiple callables.