ℹ️ Skipped - page is already crawled
| Filter | Status | Condition | Details |
|---|---|---|---|
| HTTP status | PASS | download_http_code = 200 | HTTP 200 |
| Age cutoff | PASS | download_stamp > now() - 6 MONTH | 0 months ago |
| History drop | PASS | isNull(history_drop_reason) | No drop reason |
| Spam/ban | PASS | fh_dont_index != 1 AND ml_spam_score = 0 | ml_spam_score=0 |
| Canonical | PASS | meta_canonical IS NULL OR = '' OR = src_unparsed | Not set |
| Property | Value |
|---|---|
| URL | https://chryswoods.com/parallel_python/futures_part2.html |
| Last Crawled | 2026-04-06 03:50:17 (16 hours ago) |
| First Indexed | 2020-11-25 08:22:05 (5 years ago) |
| HTTP Status Code | 200 |
| Meta Title | chryswoods.com | Part 2: Asynchronous Functions and Futures |
| Meta Description | null |
| Meta Canonical | null |
| Boilerpipe Text | The
Pool.map
function allows you to map a single function across an entire list of data. But what if you want to apply lots of different functions? The solution is to tell individual workers to run different functions, by applying functions to workers.
The
Pool
class comes with the function
apply
. This is used to tell one process in the worker pool to run a specified function. For example, create a new script called
poolapply.py
and type into it;
import
time
from
multiprocessing
import
Pool, current_process
def
slow_function(nsecs):
"""
Function that sleeps for 'nsecs' seconds, returning
the number of seconds that it slept
"""
print
(
"Process
%s
going to sleep for
%s
second(s)"
%
(current_process().pid, nsecs))
# use the time.sleep function to sleep for nsecs seconds
time.sleep(nsecs)
print
(
"Process
%s
waking up"
%
current_process().pid)
return
nsecs
if
__name__
==
"__main__"
:
print
(
"Master process is PID
%s
"
%
current_process().pid)
with
Pool()
as
pool:
r
=
pool.
apply
(slow_function, [
5
])
print
(
"Result is
%s
"
%
r)
Run this script using
python poolapply.py
. You should see the something like this printed to the screen
Master process is PID 8691
Process 8692 going to sleep for 5 second(s)
Process 8692 waking up
Result is 5
(with a delay of five seconds when the worker process sleeps)
The key line in this script is
r
=
pool.
apply
(slow_function, [
5
])
The
pool.apply
function will request that one of the workers in the pool should run the passed function (in this case
slow_function
), with the arguments passed to the function held in the list (in this case
[5]
). The
pool.apply
function will wait until the passed function has finished, and will return the result of that function (here copied into
r
).
The arguments to the applied function must be placed into a list. This is the case, even if the applied function has just a single argument (i.e. this is why we had to write
[5]
rather than just
5
. The list of arguments must contain the same number of arguments as needed by the applied function, in the same order as declared in the function. For example, edit your
poolapply.py
function to read;
import
time
from
multiprocessing
import
Pool, current_process
def
slow_add(nsecs, x, y):
"""
Function that sleeps for 'nsecs' seconds, and
then returns the sum of x and y
"""
print
(
"Process
%s
going to sleep for
%s
second(s)"
%
(current_process().pid,nsecs))
time.sleep(nsecs)
print
(
"Process
%s
waking up"
%
current_process().pid)
return
x
+
y
if
__name__
==
"__main__"
:
print
(
"Master process is PID
%s
"
%
current_process().pid)
with
Pool()
as
pool:
r
=
pool.
apply
(slow_add, [
1
,
6
,
7
])
print
(
"Result is
%s
"
%
r)
Here we have edited
slow_function
to
slow_add
, with this function accepting three arguments. These three arguments are passed using the list in
pool.apply(slow_add, [1, 6, 7])
.
Running this script using
python poolapply.py
should give output similar to
Master process is PID 8893
Process 8895 going to sleep for 1 second(s)
Process 8895 waking up
Result is 13
Asynchronous Functions
A major problem of
Pool.apply
is that the master process is blocked until the worker has finished processing the applied function. This is obviously an issue if you want to run multiple applied functions in parallel!
Fortunately,
Pool
has an
apply_async
function. This is an asynchronous version of
apply
that applies the function in a worker process, but without blocking the master. Create a new python script called
applyasync.py
and copy into it;
import
time
from
multiprocessing
import
Pool, current_process
def
slow_add(nsecs, x, y):
"""
Function that sleeps for 'nsecs' seconds, and
then returns the sum of x and y
"""
print
(
"Process
%s
going to sleep for
%s
second(s)"
%
(current_process().pid,nsecs))
time.sleep(nsecs)
print
(
"Process
%s
waking up"
%
current_process().pid)
return
x
+
y
if
__name__
==
"__main__"
:
print
(
"Master process is PID
%s
"
%
current_process().pid)
with
Pool()
as
pool:
r1
=
pool.apply_async(slow_add, [
1
,
6
,
7
])
r2
=
pool.apply_async(slow_add, [
1
,
2
,
3
])
r1.wait()
print
(
"Result one is
%s
"
%
r1.get())
r2.wait()
print
(
"Result two is
%s
"
%
r2.get())
Run this script using
python applyasync.py
and you should see results similar to
Master process is PID 8717
Process 8719 going to sleep for 1 second(s)
Process 8720 going to sleep for 1 second(s)
Process 8719 waking up
Process 8720 waking up
Result one is 13
Result two is 5
The keys lines of this script are
r1 = pool.apply_async(slow_add, [1, 6, 7])
r2 = pool.apply_async(slow_add, [1, 2, 3])
The
apply_async
function is identical to
apply
, except that it returns control to the master process immediately. This means that the master process is free to continue working (e.g. here, it
apply_async
s a second
slow_add
function). In this case, it allows us to run two
slow_sums
in parallel. Most noticeably here, even though each function call took one second to run, the whole program did not take two seconds. Due to running them in parallel, it finished the whole program in just over one second.
Futures
An issue with running a function asynchronously is that the return value of the function is not available immediately. This means that, when running an asynchronous function, you don’t get the return value directly. Instead,
apply_async
returns a placeholder for the return value. This placeholder is called a “future”, and is a variable that in the future will be given the result of the function.
Futures are a very common variable type in parallel programming across many languages. Futures provide several common functions;
Block (wait) until the result is available. In
multiprocessing
, this is via the
.wait()
function, e.g.
r1.wait()
in the above script.
Retrieve the result when it is available (blocking until it is available). This is the
.get()
function, e.g.
r1.get()
.
Test whether or not the result is available. This is the
.ready()
function, which returns
True
when the asynchronous function has finished and the result is available via
.get()
.
Test whether or not the function was a success, e.g. whether or not an exception was raised when running the function. This is the
.successful()
function, which returns
True
if the asynchronous function completed without raising an exception. Note that this function should only be called after the result is available (e.g. when
.ready()
returns
True
).
In the above example,
r1
and
r2
were both futures for the results of the two asynchronous calls of
slow_sum
. The two
slow_sum
calls were processed by two worker processes. The master process was then blocked using
r1.wait()
to wait for the result of the first call, and then blocked using
r2.wait()
to wait or the result of the second call.
(note that we had to wait for all of the results to be delivered to our futures before we exited the
with
block, or else the pool of workers could be destroyed before the functions have completed and the results are available)
We can explore this more using the following example. Create a script called
future.py
and copy into it;
import
time
from
multiprocessing
import
Pool
def
slow_add(nsecs, x, y):
"""
Function that sleeps for 'nsecs' seconds, and
then returns the sum of x and y
"""
time.sleep(nsecs)
return
x
+
y
def
slow_diff(nsecs, x, y):
"""
Function that sleeps for 'nsecs' seconds, and
then retruns the difference of x and y
"""
time.sleep(nsecs)
return
x
-
y
def
broken_function(nsecs):
"""Function that deliberately raises an AssertationError"""
time.sleep(nsecs)
raise
ValueError
(
"Called broken function"
)
if
__name__
==
"__main__"
:
futures
=
[]
with
Pool()
as
pool:
futures.append(pool.apply_async(slow_add, [
3
,
6
,
7
]))
futures.append(pool.apply_async(slow_diff, [
2
,
5
,
2
]))
futures.append(pool.apply_async(slow_add, [
1
,
8
,
1
]))
futures.append(pool.apply_async(slow_diff, [
5
,
9
,
2
]))
futures.append(pool.apply_async(broken_function, [
4
]))
while
True
:
all_finished
=
True
print
(
"
\n
Have the workers finished?"
)
for
i, future
in
enumerate
(futures):
if
future.ready():
print
(
"Process
%s
has finished"
%
i)
else
:
all_finished
=
False
print
(
"Process
%s
is running..."
%
i)
if
all_finished:
break
time.sleep(
1
)
print
(
"
\n
Here are the results."
)
for
i, future
in
enumerate
(futures):
if
future.successful():
print
(
"Process
%s
was successful. Result is
%s
"
%
(i, future.get()))
else
:
print
(
"Process
%s
failed!"
%
i)
try
:
future.get()
except
Exception
as
e:
print
(
" Error =
%s
:
%s
"
%
(
type
(e), e))
Running this script using
python future.py
should give results similar to
Have the workers finished?
Process 0 is running...
Process 1 is running...
Process 2 is running...
Process 3 is running...
Process 4 is running...
Have the workers finished?
Process 0 is running...
Process 1 is running...
Process 2 has finished
Process 3 is running...
Process 4 is running...
Have the workers finished?
Process 0 is running...
Process 1 has finished
Process 2 has finished
Process 3 is running...
Process 4 is running...
Have the workers finished?
Process 0 has finished
Process 1 has finished
Process 2 has finished
Process 3 is running...
Process 4 is running...
Have the workers finished?
Process 0 has finished
Process 1 has finished
Process 2 has finished
Process 3 is running...
Process 4 is running...
Have the workers finished?
Process 0 has finished
Process 1 has finished
Process 2 has finished
Process 3 has finished
Process 4 has finished
Here are the results.
Process 0 was successful. Result is 13
Process 1 was successful. Result is 3
Process 2 was successful. Result is 9
Process 3 was successful. Result is 7
Process 4 failed!
Error = <class 'ValueError'> : Called broken function
Is this output that you expected? Note that the exception raised by
broken_function
is held safely in its associated future. This is indicated by
.successful()
returning
False
, thereby allowing us to handle the exception in a
try...except
block that is put around the
.get()
function (if you
.get()
a future that contains an exception, then that exception is raised).
Exercise
Edit the
future.py
script so that you can control the number of workers in the pool using a command line argument (e.g. using
Pool(processes=int(sys.argv[1]))
rather than
Pool()
).
Edit the script to add calls to more asynchronous functions.
Then experiment with running the script with different numbers of processes in the pool and with different numbers of asynchronous function calls.
How are the asynchronous function calls distributed across the pool of worker processes?
Previous
Up
Next |
| Markdown | - [Home](https://chryswoods.com/index.html)
- [Research](https://chryswoods.com/research/README.html)
- [What is good research software engineering?](https://chryswoods.com/research/good_rse.html)
- [How can the RSE group work with you?](https://chryswoods.com/research/collaboration.html)
- [RSE Group news](https://chryswoods.com/research/news.html)
- [Jobs](https://chryswoods.com/research/jobs.html)
- [Training Resources](https://chryswoods.com/main/courses.html)
- [Courses](https://chryswoods.com/main/courses.html)
- [Python and Data](https://chryswoods.com/python_and_data/README.html)
- [Beginning Perl](https://chryswoods.com/beginning_perl/README.html)
- [Perl Basics](https://chryswoods.com/beginning_perl/basics.html)
- [Loops](https://chryswoods.com/beginning_perl/loops.html)
- [Arguments\!](https://chryswoods.com/beginning_perl/arguments.html)
- [Conditions](https://chryswoods.com/beginning_perl/conditions.html)
- [Files](https://chryswoods.com/beginning_perl/files.html)
- [Writing Files](https://chryswoods.com/beginning_perl/writing.html)
- [Splitting Lines](https://chryswoods.com/beginning_perl/splitting.html)
- [Searching Files](https://chryswoods.com/beginning_perl/searching.html)
- [Search and Replace](https://chryswoods.com/beginning_perl/replacing.html)
- [Running Programs](https://chryswoods.com/beginning_perl/running.html)
- [Job Scripts](https://chryswoods.com/beginning_perl/jobs.html)
- [What Next?](https://chryswoods.com/beginning_perl/whatnext.html)
- [Beginning Python](https://chryswoods.com/beginning_python/README.html)
- [Python Basics](https://chryswoods.com/beginning_python/basics.html)
- [Loops](https://chryswoods.com/beginning_python/loops.html)
- [Arguments\!](https://chryswoods.com/beginning_python/arguments.html)
- [Conditions](https://chryswoods.com/beginning_python/conditions.html)
- [Files](https://chryswoods.com/beginning_python/files.html)
- [Writing Files](https://chryswoods.com/beginning_python/writing.html)
- [Splitting Lines](https://chryswoods.com/beginning_python/splitting.html)
- [Searching Files](https://chryswoods.com/beginning_python/searching.html)
- [Search and Replace](https://chryswoods.com/beginning_python/replacing.html)
- [Running Programs](https://chryswoods.com/beginning_python/running.html)
- [Job Scripts](https://chryswoods.com/beginning_python/jobs.html)
- [What Next?](https://chryswoods.com/beginning_python/whatnext.html)
- [Intermediate Python](https://chryswoods.com/intermediate_python/README.html)
- [Lists](https://chryswoods.com/intermediate_python/lists.html)
- [Dictionaries](https://chryswoods.com/intermediate_python/dictionaries.html)
- [Functions](https://chryswoods.com/intermediate_python/functions.html)
- [Modules](https://chryswoods.com/intermediate_python/modules.html)
- [Documenting Code](https://chryswoods.com/intermediate_python/documenting.html)
- [Objects and Classes](https://chryswoods.com/intermediate_python/objects.html)
- [Testing](https://chryswoods.com/intermediate_python/testing.html)
- [Regular Expressions in Python](https://chryswoods.com/intermediate_python/regexp.html)
- [What next?](https://chryswoods.com/intermediate_python/whatnext.html)
- [Parallel Programming with Python](https://chryswoods.com/parallel_python/README.html)
- [Part 1: Functional Programming](https://chryswoods.com/parallel_python/part1.html)
- [Functions as Objects](https://chryswoods.com/parallel_python/functions.html)
- [Mapping Functions](https://chryswoods.com/parallel_python/map.html)
- [Reduction](https://chryswoods.com/parallel_python/reduce.html)
- [Anonymous Functions (lambda)](https://chryswoods.com/parallel_python/lambda.html)
- [Part 2: Multicore (local) Parallel Programming](https://chryswoods.com/parallel_python/part2.html)
- [Multiprocessing](https://chryswoods.com/parallel_python/multiprocessing.html)
- [Pool](https://chryswoods.com/parallel_python/pool_part2.html)
- [Parallel map/reduce](https://chryswoods.com/parallel_python/mapreduce_part2.html)
- [Asynchronous Functions and Futures](https://chryswoods.com/parallel_python/futures_part2.html)
- [Asynchronous Mapping](https://chryswoods.com/parallel_python/async_map.html)
- [Part 3: Multinode (distributed/cluster) Parallel Programming](https://chryswoods.com/parallel_python/part3.html)
- [Scoop](https://chryswoods.com/parallel_python/scoop.html)
- [Distributed map/reduce](https://chryswoods.com/parallel_python/mapreduce_part3.html)
- [Running Scoop on a Cluster](https://chryswoods.com/parallel_python/cluster.html)
- [What Next?](https://chryswoods.com/parallel_python/whatnext.html)
- [Epilogue](https://chryswoods.com/parallel_python/epilogue.html)
- [Changes from Python 2 to 3](https://chryswoods.com/parallel_python/python2to3.html)
- [Global Interpreter Lock (GIL)](https://chryswoods.com/parallel_python/gil.html)
- [Beginning R](https://chryswoods.com/beginning_r/index.html)
- [Intermediate R](https://chryswoods.com/intermediate_r/index.html)
- [Introduction to Data Analysis in R](https://chryswoods.com/data_analysis_r/index.html)
- [Beginning C++](https://chryswoods.com/beginning_c++/README.html)
- [Why C++?](https://chryswoods.com/beginning_c++/why.html)
- [C++ Basics](https://chryswoods.com/beginning_c++/basics.html)
- [Syntax Compared to Python](https://chryswoods.com/beginning_c++/syntax.html)
- [Types, Scopes and Auto](https://chryswoods.com/beginning_c++/typing.html)
- [Lists and Dictionaries](https://chryswoods.com/beginning_c++/lists.html)
- [Objects and Classes](https://chryswoods.com/beginning_c++/objects.html)
- [Concepts, Default Arguments and Operators](https://chryswoods.com/beginning_c++/operators.html)
- [What next?](https://chryswoods.com/beginning_c++/whatnext.html)
- [Parallel Programming with C++](https://chryswoods.com/parallel_c++/README.html)
- [Part 1: Functional Programming](https://chryswoods.com/parallel_c++/part1.html)
- [Functions as Objects](https://chryswoods.com/parallel_c++/functions.html)
- [Mapping Functions](https://chryswoods.com/parallel_c++/map.html)
- [Reduction](https://chryswoods.com/parallel_c++/reduce.html)
- [Anonymous Functions (lambda)](https://chryswoods.com/parallel_c++/lambda.html)
- [Map/Reduce](https://chryswoods.com/parallel_c++/mapreduce.html)
- [Part 2: Parallel Programming Using Intel Threading Building Blocks](https://chryswoods.com/parallel_c++/part2.html)
- [tbb::parallel\_for](https://chryswoods.com/parallel_c++/parallel_for.html)
- [tbb::parallel\_reduce](https://chryswoods.com/parallel_c++/parallel_reduce.html)
- [Writing a parallel map/reduce](https://chryswoods.com/parallel_c++/parallel_mapreduce.html)
- [What Next?](https://chryswoods.com/parallel_c++/whatnext.html)
- [Efficient Vectorisation with C++](https://chryswoods.com/vector_c++/README.html)
- [Part 1: Introduction to Vectorisation](https://chryswoods.com/vector_c++/part1.html)
- [What is Vectorisation?](https://chryswoods.com/vector_c++/vectorisation.html)
- [How to Vectorise (omp simd)](https://chryswoods.com/vector_c++/simd.html)
- [omp simd features](https://chryswoods.com/vector_c++/features.html)
- [Memory Layout](https://chryswoods.com/vector_c++/memory.html)
- [omp simd limitations](https://chryswoods.com/vector_c++/limitations.html)
- [Part 2: Vectorisation using Intrinsics](https://chryswoods.com/vector_c++/part2.html)
- [SSE Intrinsics](https://chryswoods.com/vector_c++/emmintrin.html)
- [AVX Intrinsics](https://chryswoods.com/vector_c++/immintrin.html)
- [Portable Vectorisation](https://chryswoods.com/vector_c++/portable.html)
- [What Next?](https://chryswoods.com/vector_c++/whatnext.html)
- [Parallel Programming with OpenMP](https://chryswoods.com/beginning_openmp/README.html)
- [Basics](https://chryswoods.com/beginning_openmp/basics.html)
- [Compiler Directives / Pragmas](https://chryswoods.com/beginning_openmp/directives.html)
- [Sections](https://chryswoods.com/beginning_openmp/sections.html)
- [Loops](https://chryswoods.com/beginning_openmp/loops.html)
- [Critical Code](https://chryswoods.com/beginning_openmp/critical.html)
- [Reduction](https://chryswoods.com/beginning_openmp/reduction.html)
- [Map / Reduce](https://chryswoods.com/beginning_openmp/mapreduce.html)
- [Maximising Performance](https://chryswoods.com/beginning_openmp/performance.html)
- [Case Study](https://chryswoods.com/beginning_openmp/casestudy.html)
- [What Next?](https://chryswoods.com/beginning_openmp/whatnext.html)
- [Parallel Programming with MPI](https://chryswoods.com/beginning_mpi/README.html)
- [Basics](https://chryswoods.com/beginning_mpi/basics.html)
- [MPI Functions](https://chryswoods.com/beginning_mpi/functions.html)
- [Sections](https://chryswoods.com/beginning_mpi/sections.html)
- [Loops](https://chryswoods.com/beginning_mpi/loops.html)
- [Messages](https://chryswoods.com/beginning_mpi/messages.html)
- [Reduction](https://chryswoods.com/beginning_mpi/reduction.html)
- [Map / Reduce](https://chryswoods.com/beginning_mpi/mapreduce.html)
- [Maximising Performance](https://chryswoods.com/beginning_mpi/performance.html)
- [What Next?](https://chryswoods.com/beginning_mpi/whatnext.html)
- [Version Control with Git](https://chryswoods.com/beginning_git/README.html)
- [Git Basics](https://chryswoods.com/beginning_git/basics.html)
- [Adding Files](https://chryswoods.com/beginning_git/adding.html)
- [Committing Changes](https://chryswoods.com/beginning_git/committing.html)
- [Diffing (seeing what has changed)](https://chryswoods.com/beginning_git/diffing.html)
- [Changing Versions](https://chryswoods.com/beginning_git/versions.html)
- [Branching](https://chryswoods.com/beginning_git/branching.html)
- [Renaming and Removing Files](https://chryswoods.com/beginning_git/renaming.html)
- [Subdirectories and Ignoring Files](https://chryswoods.com/beginning_git/subdirs.html)
- [Git in the Cloud](https://chryswoods.com/beginning_git/github.html)
- [Pushing to the Cloud](https://chryswoods.com/beginning_git/push.html)
- [Markdown](https://chryswoods.com/beginning_git/markdown.html)
- [Cloning a Repository](https://chryswoods.com/beginning_git/cloning.html)
- [Merging](https://chryswoods.com/beginning_git/merging.html)
- [Pull Requests](https://chryswoods.com/beginning_git/pull.html)
- [Continuous Integration](https://chryswoods.com/beginning_git/ci.html)
- [What next?](https://chryswoods.com/beginning_git/whatnext.html)
- [JupyterHub and Kubernetes](https://chryswoods.com/inception_workshop/README.html)
- [Creating the workshop](https://chryswoods.com/inception_workshop/course/part01.html)
- [Finding all your dependencies](https://chryswoods.com/inception_workshop/course/part02.html)
- [Building the docker image](https://chryswoods.com/inception_workshop/course/part03.html)
- [JupyterHub](https://chryswoods.com/inception_workshop/course/part04.html)
- [Uploading to the cloud](https://chryswoods.com/inception_workshop/course/part05.html)
- [Kubernetes](https://chryswoods.com/inception_workshop/course/part06.html)
- [Helm](https://chryswoods.com/inception_workshop/course/part07.html)
- [Configuring JupyterHub](https://chryswoods.com/inception_workshop/course/part08.html)
- [What next?](https://chryswoods.com/inception_workshop/course/whatnext.html)
- [Introduction to Monte Carlo](https://chryswoods.com/intro_to_mc/README.html)
- [Part 1](https://chryswoods.com/intro_to_mc/part1/README.html)
- [Introduction to Monte Carlo](https://chryswoods.com/intro_to_mc/part1/intro.html)
- [Software](https://chryswoods.com/intro_to_mc/part1/software.html)
- [Metropolis Monte Carlo](https://chryswoods.com/intro_to_mc/part1/metropolis.html)
- [Running metropolis.py](https://chryswoods.com/intro_to_mc/part1/running.html)
- [Control Variables](https://chryswoods.com/intro_to_mc/part1/control.html)
- [Phase Changes](https://chryswoods.com/intro_to_mc/part1/phase.html)
- [Phase Space and Ensembles](https://chryswoods.com/intro_to_mc/part1/ensemble.html)
- [Volume Moves](https://chryswoods.com/intro_to_mc/part1/volume.html)
- [NPT Simulations](https://chryswoods.com/intro_to_mc/part1/npt.html)
- [Summary](https://chryswoods.com/intro_to_mc/part1/summary.html)
- [Part 2](https://chryswoods.com/intro_to_mc/part2/README.html)
- [Introduction and Software](https://chryswoods.com/intro_to_mc/part2/intro.html)
- [Sampling the Solvent - Rigid Body Moves](https://chryswoods.com/intro_to_mc/part2/rigid.html)
- [Sampling the Ligand - Intramolecular Moves](https://chryswoods.com/intro_to_mc/part2/intra.html)
- [Sampling the Protein - Backbone Moves](https://chryswoods.com/intro_to_mc/part2/backbone.html)
- [Sampling it all - Weighting Moves](https://chryswoods.com/intro_to_mc/part2/weight.html)
- [Summary](https://chryswoods.com/intro_to_mc/part2/whatnext.html)
- [Molecular Visualisation, Modelling and Dynamics](https://chryswoods.com/dynamics/README.html)
- [Part 1: Molecular Visualisation](https://chryswoods.com/dynamics/visualisation/README.html)
- [1a: Opening Files](https://chryswoods.com/dynamics/visualisation/opening_files.html)
- [1b: Manipulating the View](https://chryswoods.com/dynamics/visualisation/mouse.html)
- [1c: Graphical Representations](https://chryswoods.com/dynamics/visualisation/representations.html)
- [1d: Selecting Atoms](https://chryswoods.com/dynamics/visualisation/selection.html)
- [1e: Complex Selections](https://chryswoods.com/dynamics/visualisation/complex_selection.html)
- [1f: Rendering](https://chryswoods.com/dynamics/visualisation/rendering.html)
- [1g: Movies](https://chryswoods.com/dynamics/visualisation/movies.html)
- [1h: Picking Atoms](https://chryswoods.com/dynamics/visualisation/picking.html)
- [1i: Comparing Trajectories](https://chryswoods.com/dynamics/visualisation/comparing.html)
- [1j: What Next?](https://chryswoods.com/dynamics/visualisation/whatnext.html)
- [Part 2: Molecular Dynamics](https://chryswoods.com/dynamics/dynamics/README.html)
- [2a: Getting Started](https://chryswoods.com/dynamics/dynamics/getting_started.html)
- [2b: Theory of MD](https://chryswoods.com/dynamics/dynamics/theory.html)
- [2c: Changing Time](https://chryswoods.com/dynamics/dynamics/time.html)
- [2d: Shake (Rattle and Roll)](https://chryswoods.com/dynamics/dynamics/shake.html)
- [2e: Periodic Boundary Conditions](https://chryswoods.com/dynamics/dynamics/protein.html)
- [2f: Under Pressure](https://chryswoods.com/dynamics/dynamics/pressure.html)
- [2g: Running the Simulation](https://chryswoods.com/dynamics/dynamics/simulation.html)
- [2h: What Next?](https://chryswoods.com/dynamics/dynamics/whatnext.html)
- [Part 3: Mutation Studies](https://chryswoods.com/dynamics/mutation/README.html)
- [3a: Getting Started](https://chryswoods.com/dynamics/mutation/gettingstarted.html)
- [3b: Mutating the Protein](https://chryswoods.com/dynamics/mutation/mutation.html)
- [3c: Solvating the Protein](https://chryswoods.com/dynamics/mutation/solvation.html)
- [3d: Minimising the System](https://chryswoods.com/dynamics/mutation/minimisation.html)
- [3e: Heating the System](https://chryswoods.com/dynamics/mutation/heating.html)
- [3f: Equilibrating the System](https://chryswoods.com/dynamics/mutation/equilibration.html)
- [3g: Running the Simulation](https://chryswoods.com/dynamics/mutation/simulation.html)
- [3h: Comparing Trajectories](https://chryswoods.com/dynamics/mutation/compare.html)
- [3i: What Next?](https://chryswoods.com/dynamics/mutation/whatnext.html)
- [QM/MM Monte Carlo](https://chryswoods.com/embo2014/Practical.html)
- [Software Carpentry](https://chryswoods.com/main/softwarecarpentry.html)
- [Bath (July 2013) Workshop](http://tinyurl.com/swcbath)
- [Bristol (September 2013) Workshop](http://tinyurl.com/swcbristol)
- [Exeter (November 2013) Workshop](http://tinyurl.com/swcexeter)
- [Software](https://chryswoods.com/main/software.html)
- [Sire Molecular Simulation Framework](https://siremol.org/)
- [ProtoMS Monte Carlo Package](http://protoms.org/)
- [Publications](https://chryswoods.com/main/publications.html)
- [Talks](https://chryswoods.com/talks/README.html)
- [Useful Links](https://chryswoods.com/main/links.html)
- [Books that are Worth Reading](https://chryswoods.com/main/reading.html)
- [Contact Information](https://chryswoods.com/main/contact.html)
- [About this Website](https://chryswoods.com/main/website.html)
- [ chryswoods.com](https://chryswoods.com/parallel_python/futures_part2.html#main_menu)
# Part 2: Asynchronous Functions and Futures
The `Pool.map` function allows you to map a single function across an entire list of data. But what if you want to apply lots of different functions? The solution is to tell individual workers to run different functions, by applying functions to workers.
The `Pool` class comes with the function `apply`. This is used to tell one process in the worker pool to run a specified function. For example, create a new script called `poolapply.py` and type into it;
```
import time
from multiprocessing import Pool, current_process
def slow_function(nsecs):
"""
Function that sleeps for 'nsecs' seconds, returning
the number of seconds that it slept
"""
print("Process %s going to sleep for %s second(s)" % (current_process().pid, nsecs))
# use the time.sleep function to sleep for nsecs seconds
time.sleep(nsecs)
print("Process %s waking up" % current_process().pid)
return nsecs
if __name__ == "__main__":
print("Master process is PID %s" % current_process().pid)
with Pool() as pool:
r = pool.apply(slow_function, [5])
print("Result is %s" % r)
```
Run this script using `python poolapply.py`. You should see the something like this printed to the screen
```
Master process is PID 8691
Process 8692 going to sleep for 5 second(s)
Process 8692 waking up
Result is 5
```
(with a delay of five seconds when the worker process sleeps)
The key line in this script is
```
r = pool.apply(slow_function, [5])
```
The `pool.apply` function will request that one of the workers in the pool should run the passed function (in this case `slow_function`), with the arguments passed to the function held in the list (in this case `[5]`). The `pool.apply` function will wait until the passed function has finished, and will return the result of that function (here copied into `r`).
The arguments to the applied function must be placed into a list. This is the case, even if the applied function has just a single argument (i.e. this is why we had to write `[5]` rather than just `5`. The list of arguments must contain the same number of arguments as needed by the applied function, in the same order as declared in the function. For example, edit your `poolapply.py` function to read;
```
import time
from multiprocessing import Pool, current_process
def slow_add(nsecs, x, y):
"""
Function that sleeps for 'nsecs' seconds, and
then returns the sum of x and y
"""
print("Process %s going to sleep for %s second(s)" % (current_process().pid,nsecs))
time.sleep(nsecs)
print("Process %s waking up" % current_process().pid)
return x + y
if __name__ == "__main__":
print("Master process is PID %s" % current_process().pid)
with Pool() as pool:
r = pool.apply(slow_add, [1, 6, 7])
print("Result is %s" % r)
```
Here we have edited `slow_function` to `slow_add`, with this function accepting three arguments. These three arguments are passed using the list in `pool.apply(slow_add, [1, 6, 7])`.
Running this script using `python poolapply.py` should give output similar to
```
Master process is PID 8893
Process 8895 going to sleep for 1 second(s)
Process 8895 waking up
Result is 13
```
## Asynchronous Functions
A major problem of `Pool.apply` is that the master process is blocked until the worker has finished processing the applied function. This is obviously an issue if you want to run multiple applied functions in parallel\!
Fortunately, `Pool` has an `apply_async` function. This is an asynchronous version of `apply` that applies the function in a worker process, but without blocking the master. Create a new python script called `applyasync.py` and copy into it;
```
import time
from multiprocessing import Pool, current_process
def slow_add(nsecs, x, y):
"""
Function that sleeps for 'nsecs' seconds, and
then returns the sum of x and y
"""
print("Process %s going to sleep for %s second(s)" % (current_process().pid,nsecs))
time.sleep(nsecs)
print("Process %s waking up" % current_process().pid)
return x + y
if __name__ == "__main__":
print("Master process is PID %s" % current_process().pid)
with Pool() as pool:
r1 = pool.apply_async(slow_add, [1, 6, 7])
r2 = pool.apply_async(slow_add, [1, 2, 3])
r1.wait()
print("Result one is %s" % r1.get())
r2.wait()
print("Result two is %s" % r2.get())
```
Run this script using `python applyasync.py` and you should see results similar to
```
Master process is PID 8717
Process 8719 going to sleep for 1 second(s)
Process 8720 going to sleep for 1 second(s)
Process 8719 waking up
Process 8720 waking up
Result one is 13
Result two is 5
```
The keys lines of this script are
```
r1 = pool.apply_async(slow_add, [1, 6, 7])
r2 = pool.apply_async(slow_add, [1, 2, 3])
```
The `apply_async` function is identical to `apply`, except that it returns control to the master process immediately. This means that the master process is free to continue working (e.g. here, it `apply_async`s a second `slow_add` function). In this case, it allows us to run two `slow_sums` in parallel. Most noticeably here, even though each function call took one second to run, the whole program did not take two seconds. Due to running them in parallel, it finished the whole program in just over one second.
## Futures
An issue with running a function asynchronously is that the return value of the function is not available immediately. This means that, when running an asynchronous function, you don’t get the return value directly. Instead, `apply_async` returns a placeholder for the return value. This placeholder is called a “future”, and is a variable that in the future will be given the result of the function.
Futures are a very common variable type in parallel programming across many languages. Futures provide several common functions;
- Block (wait) until the result is available. In `multiprocessing`, this is via the `.wait()` function, e.g. `r1.wait()` in the above script.
- Retrieve the result when it is available (blocking until it is available). This is the `.get()` function, e.g. `r1.get()`.
- Test whether or not the result is available. This is the `.ready()` function, which returns `True` when the asynchronous function has finished and the result is available via `.get()`.
- Test whether or not the function was a success, e.g. whether or not an exception was raised when running the function. This is the `.successful()` function, which returns `True` if the asynchronous function completed without raising an exception. Note that this function should only be called after the result is available (e.g. when `.ready()` returns `True`).
In the above example, `r1` and `r2` were both futures for the results of the two asynchronous calls of `slow_sum`. The two `slow_sum` calls were processed by two worker processes. The master process was then blocked using `r1.wait()` to wait for the result of the first call, and then blocked using `r2.wait()` to wait or the result of the second call.
(note that we had to wait for all of the results to be delivered to our futures before we exited the `with` block, or else the pool of workers could be destroyed before the functions have completed and the results are available)
We can explore this more using the following example. Create a script called `future.py` and copy into it;
```
import time
from multiprocessing import Pool
def slow_add(nsecs, x, y):
"""
Function that sleeps for 'nsecs' seconds, and
then returns the sum of x and y
"""
time.sleep(nsecs)
return x + y
def slow_diff(nsecs, x, y):
"""
Function that sleeps for 'nsecs' seconds, and
then retruns the difference of x and y
"""
time.sleep(nsecs)
return x - y
def broken_function(nsecs):
"""Function that deliberately raises an AssertationError"""
time.sleep(nsecs)
raise ValueError("Called broken function")
if __name__ == "__main__":
futures = []
with Pool() as pool:
futures.append(pool.apply_async(slow_add, [3, 6, 7]))
futures.append(pool.apply_async(slow_diff, [2, 5, 2]))
futures.append(pool.apply_async(slow_add, [1, 8, 1]))
futures.append(pool.apply_async(slow_diff, [5, 9, 2]))
futures.append(pool.apply_async(broken_function, [4]))
while True:
all_finished = True
print("\nHave the workers finished?")
for i, future in enumerate(futures):
if future.ready():
print("Process %s has finished" % i)
else:
all_finished = False
print("Process %s is running..." % i)
if all_finished:
break
time.sleep(1)
print("\nHere are the results.")
for i, future in enumerate(futures):
if future.successful():
print("Process %s was successful. Result is %s" % (i, future.get()))
else:
print("Process %s failed!" % i)
try:
future.get()
except Exception as e:
print(" Error = %s : %s" % (type(e), e))
```
Running this script using `python future.py` should give results similar to
```
Have the workers finished?
Process 0 is running...
Process 1 is running...
Process 2 is running...
Process 3 is running...
Process 4 is running...
Have the workers finished?
Process 0 is running...
Process 1 is running...
Process 2 has finished
Process 3 is running...
Process 4 is running...
Have the workers finished?
Process 0 is running...
Process 1 has finished
Process 2 has finished
Process 3 is running...
Process 4 is running...
Have the workers finished?
Process 0 has finished
Process 1 has finished
Process 2 has finished
Process 3 is running...
Process 4 is running...
Have the workers finished?
Process 0 has finished
Process 1 has finished
Process 2 has finished
Process 3 is running...
Process 4 is running...
Have the workers finished?
Process 0 has finished
Process 1 has finished
Process 2 has finished
Process 3 has finished
Process 4 has finished
Here are the results.
Process 0 was successful. Result is 13
Process 1 was successful. Result is 3
Process 2 was successful. Result is 9
Process 3 was successful. Result is 7
Process 4 failed!
Error = <class 'ValueError'> : Called broken function
```
Is this output that you expected? Note that the exception raised by `broken_function` is held safely in its associated future. This is indicated by `.successful()` returning `False`, thereby allowing us to handle the exception in a `try...except` block that is put around the `.get()` function (if you `.get()` a future that contains an exception, then that exception is raised).
***
## Exercise
Edit the `future.py` script so that you can control the number of workers in the pool using a command line argument (e.g. using `Pool(processes=int(sys.argv[1]))` rather than `Pool()`).
Edit the script to add calls to more asynchronous functions.
Then experiment with running the script with different numbers of processes in the pool and with different numbers of asynchronous function calls.
How are the asynchronous function calls distributed across the pool of worker processes?
***
# [Previous](https://chryswoods.com/parallel_python/mapreduce_part2.html) [Up](https://chryswoods.com/parallel_python/part2.html) [Next](https://chryswoods.com/parallel_python/async_map.html)
Quick Links \| [Home](https://chryswoods.com/index.html) \| [Courses](https://chryswoods.com/main/courses.html) \| [Software](https://chryswoods.com/main/software.html) \| [Contact](https://chryswoods.com/main/contact.html)
[Copyright Information](https://chryswoods.com/main/copyright.html) \| [Report a problem](<mailto:chryswoods@gmail.com?subject=There is a problem with your website&body=Hi, I found a problem on the page parallel_python/futures_part2.html>) I [Privacy](https://chryswoods.com/main/cookies.html) |
| Readable Markdown | null |
| Shard | 133 (laksa) |
| Root Hash | 1202965565032433533 |
| Unparsed URL | com,chryswoods!/parallel_python/futures_part2.html s443 |