🕷️ Crawler Inspector

URL Lookup

Direct Parameter Lookup

Raw Queries and Responses

1. Shard Calculation

Query:

Response:

Calculated Shard: 133 (from laksa194)

2. Crawled Status Check

Query:

curl -X POST \
  'http://laksa133.int.ahrefs:8124/' \
  -H 'Content-Type: text/plain' \
  -H 'X-ClickHouse-Database: crawler3' \
  -H 'Authorization: Basic YXBpOg==' \
  -d 'SELECT getAhrefsURLFromUnparsed(src_unparsed) AS found_url, ifNull(toUnixTimestamp(download_stamp), 0) AS crawl_time, ifNull(toUnixTimestamp(props_url_first_seen), 0) AS first_indexed_time, download_http_code AS http_code, src_unparsed AS src_unparsed, src_root_hash AS src_root_hash, history_drop_reason AS history_drop_reason, meta_title AS meta_title, meta_descriptions AS meta_descriptions, attrs_boilerpipe_text AS attrs_boilerpipe_text, attrs_markdown AS attrs_markdown, attrs_readable_markdown AS attrs_readable_markdown, meta_canonical AS meta_canonical FROM crawler3.page_info_local FINAL PREWHERE (src_root_hash, src_unparsed) IN ((getAhrefsRootHashFromUnparsed(getAhrefsUnparsedNoserviceFromURL(\'https://chryswoods.com/parallel_python/async_map.html\')), getAhrefsUnparsedNoserviceFromURL(\'https://chryswoods.com/parallel_python/async_map.html\'))) FORMAT JSONEachRow'

Response:

{"found_url":"https:\/\/chryswoods.com\/parallel_python\/async_map.html","crawl_time":1775453199,"first_indexed_time":1606292525,"http_code":200,"src_unparsed":"com,chryswoods!\/parallel_python\/async_map.html s443","src_root_hash":"1202965565032433533","history_drop_reason":null,"meta_title":"chryswoods.com | Part 2: Asynchronous Mapping","meta_descriptions":[],"attrs_boilerpipe_text":"Asynchronous functions allow you to give different tasks to different members of the\nmultiprocessing.Pool\n. However, giving functions one by one is not very efficient. It would be good to be able to combine mapping with asynchronous functions, i.e. be able to give different mapping tasks simultanously to the pool of workers. Fortunately,\nPool.map_async\nprovides exactly that - an asynchronous parallel map.\nCreate a new python script called\nasyncmap.py\nand copy into it\nfrom\nfunctools\nimport\nreduce\nfrom\nmultiprocessing\nimport\nPool, current_process\nimport\ntime\ndef\nadd(x, y):\n\"\"\"Return the sum of the arguments\"\"\"\nprint\n(\n\"Worker\n%s\nis processing add(\n%s\n,\n%s\n)\"\n%\n(current_process().pid, x, y))\ntime.sleep(\n1\n)\nreturn\nx\n+\ny\ndef\nproduct(x, y):\n\"\"\"Return the product of the arguments\"\"\"\nprint\n(\n\"Worker\n%s\nis processing product(\n%s\n,\n%s\n)\"\n%\n(current_process().pid, x, y))\ntime.sleep(\n1\n)\nreturn\nx\n*\ny\nif\n__name__\n==\n\"__main__\"\n:\na\n=\n[\n1\n,\n2\n,\n3\n,\n4\n,\n5\n,\n6\n,\n7\n,\n8\n,\n9\n,\n10\n]\nb\n=\n[\n11\n,\n12\n,\n13\n,\n14\n,\n15\n,\n16\n,\n17\n,\n18\n,\n19\n,\n20\n]\n# Now create a Pool of workers\nwith\nPool()\nas\npool:\nsum_future\n=\npool.starmap_async(add,\nzip\n(a,b))\nproduct_future\n=\npool.starmap_async(product,\nzip\n(a,b))\nsum_future.wait()\nproduct_future.wait()\ntotal_sum\n=\nreduce\n(\nlambda\nx, y: x\n+\ny, sum_future.get())\ntotal_product\n=\nreduce\n(\nlambda\nx, y: x\n+\ny, product_future.get())\nprint\n(\n\"Sum of sums of 'a' and 'b' is\n%s\n\"\n%\ntotal_sum)\nprint\n(\n\"Sum of products of 'a' and 'b' is\n%s\n\"\n%\ntotal_product)\nRunning this script, e.g. via\npython asyncmap.py\nshould result in something like\nWorker 32722 is processing add(1, 11)\nWorker 32723 is processing add(2, 12)\nWorker 32724 is processing add(3, 13)\nWorker 32725 is processing add(4, 14)\nWorker 32722 is processing add(5, 15)\nWorker 32724 is processing add(6, 16)\nWorker 32725 is processing add(7, 17)\nWorker 32723 is processing add(8, 18)\nWorker 32722 is processing add(9, 19)\nWorker 32724 is processing add(10, 20)\nWorker 32725 is processing product(1, 11)\nWorker 32723 is processing product(2, 12)\nWorker 32722 is processing product(3, 13)\nWorker 32723 is processing product(4, 14)\nWorker 32724 is processing product(5, 15)\nWorker 32725 is processing product(6, 16)\nWorker 32722 is processing product(7, 17)\nWorker 32725 is processing product(8, 18)\nWorker 32723 is processing product(9, 19)\nWorker 32724 is processing product(10, 20)\nSum of sums of 'a' and 'b' is 210\nSum of products of 'a' and 'b' is 935\nThis script provides two functions,\nadd\nand\nproduct\n, which are mapped asynchronously using the\nPool.map_async\nfunction. This is identical to the\nPool.map\nfunction that you used before, except now the map is performed asynchronously. This means that the resulting list is returned in a future (in this case, the futures\nsum_future\nand\nproduct_future\n. The results are waited for using the\n.wait()\nfunctions, remembering to make sure that we don’t exit the\nwith\nblock until all results are available. Then, the results of mapping are retrieved using the\n.get()\nfunction of the futures.\nChunking\nBy default, the\nPool.map\nfunction divides the work over the pool of workers by assiging pieces of work one by one. In the example above, the work to be performed was;\nadd(1, 11)\nadd(2, 12)\nadd(3, 13)\netc.\nadd(10,20)\nproduct(1, 11)\nproduct(2, 12)\nproduct(3, 13)\netc.\nproduct(10, 20)\nThe work was assigned one by one to the four workers on my computer, i.e. the first worker process was given\nadd(1, 11)\n, the second\nadd(2, 12)\n, the third\nadd(3, 13)\nthe then the fourth\nadd(4, 14)\n. The first worker to finish was then given\nadd(5, 15)\n, then the next given\nadd(6, 16)\netc. etc.\nGiving work one by one can be very inefficient for quick tasks, as the time needed by a worker process to stop and get new work can be longer than it takes to actually complete the task. To solve this problem, you can control how many work items are handed out to each worker process at a time. This is known as chunking, and the number of work items is known as the chunk of work to perform.\nYou can control the number of work items to perform per worker (the chunk size) by setting the\nchunksize\nargument, e.g.\nsum_future\n=\npool.starmap_async(add,\nzip\n(a, b), chunksize\n=\n5\n)\nwould suggest to\npool\nthat each worker be given a chunk of five pieces of work. Note that this is just a suggestion, and\npool\nmay decide to use a slightly smaller or larger chunk size depending on the amount of work and the number of workers available.\nModify your\nasyncmap.py\nscript and set the\nchunksize\nto 5 for both of the asynchronous maps for\nadd\nand\nproduct\n. Re-run your script. You should see something like;\nWorker 658 is processing add(1, 11)\nWorker 659 is processing add(6, 16)\nWorker 660 is processing product(1, 11)\nWorker 661 is processing product(6, 16)\nWorker 659 is processing add(7, 17)\nWorker 660 is processing product(2, 12)\nWorker 661 is processing product(7, 17)\nWorker 658 is processing add(2, 12)\nWorker 660 is processing product(3, 13)\nWorker 659 is processing add(8, 18)\nWorker 661 is processing product(8, 18)\nWorker 658 is processing add(3, 13)\nWorker 660 is processing product(4, 14)\nWorker 659 is processing add(9, 19)\nWorker 661 is processing product(9, 19)\nWorker 658 is processing add(4, 14)\nWorker 659 is processing add(10, 20)\nWorker 660 is processing product(5, 15)\nWorker 661 is processing product(10, 20)\nWorker 658 is processing add(5, 15)\nSum of sums of 'a' and 'b' is 210\nSum of products of 'a' and 'b' is 935\nMy laptop has four workers. The first worker is assigned the first five items of work, i.e.\nadd(1, 11)\nto\nadd(5, 15)\n, and it starts by running\nadd(1, 11)\n, hence why\nadd(1, 11)\nis printed first.\nThe next worker is given the next five items of work, i.e.\nadd(6, 16)\nto\nadd(10,20)\n, and starts by running\nadd(6, 16)\n, hence why\nadd(6, 16)\nis printed second.\nThe next worker is given the next five items of work, i.e.\nproduct(1, 11)\nto\nproduct(5, 15)\n, and it starts by running\nproduct(1, 11)\n, hence why this is printed third.\nThe last worker is given the next five items of work, i.e.\nproduct(6, 16)\nto\nproduct(10, 20)\n, and it starts by running\nproduct(6, 16)\n, hence why this is printed fourth.\nOnce each worker has finished its first item of work, it moves onto its second. This is why\nadd(2, 12)\n,\nadd(7, 17)\n,\nproduct(2, 12)\nand\nproduct(7, 17)\nare printed next. Then, each worker moves onto its third piece of work etc. etc.\nIf you don’t specify the\nchunksize\nthen it is equal to\n1\n. When writing a new script you should experiment with different values of\nchunksize\nto find the value that gives best performance.\nExercise\nEdit your script written in answer to\nexercise 2 of Parallel Map\/Reduce\n, in which you count all of the words used in all Shakespeare plays (e.g. an example answer\nis here\n).\nEdit the script so that you use an asynchronous map to distribute the work over the pool. This will free up the master process to give feedback to the user of the script, e.g. to print a progress or status message while the work is running to reassure the user that the script has not frozen. For example\nwhile\nnot\nfuture.ready():\nprint\n(\n\"Work is in progress...\"\n)\ntime.sleep(\n0.1\n)\nAdd a status message to your script to reassure the user that your script hasn’t frozen while it is processing.\n(note that you can call your script using\npython -u countwords.py shakespeare\/*\nto use the\n-u\nargument to stop Python from buffering text written to standard output)\nIf you get stuck or want inspiration, a possible answer is given\nhere\n.\nPrevious\nUp\nNext","attrs_markdown":"- [Home](https:\/\/chryswoods.com\/index.html)\n- [Research](https:\/\/chryswoods.com\/research\/README.html)\n  - [What is good research software engineering?](https:\/\/chryswoods.com\/research\/good_rse.html)\n  - [How can the RSE group work with you?](https:\/\/chryswoods.com\/research\/collaboration.html)\n  - [RSE Group news](https:\/\/chryswoods.com\/research\/news.html)\n  - [Jobs](https:\/\/chryswoods.com\/research\/jobs.html)\n  - [Training Resources](https:\/\/chryswoods.com\/main\/courses.html)\n- [Courses](https:\/\/chryswoods.com\/main\/courses.html)\n  - [Python and Data](https:\/\/chryswoods.com\/python_and_data\/README.html)\n  - [Beginning Perl](https:\/\/chryswoods.com\/beginning_perl\/README.html)\n    - [Perl Basics](https:\/\/chryswoods.com\/beginning_perl\/basics.html)\n    - [Loops](https:\/\/chryswoods.com\/beginning_perl\/loops.html)\n    - [Arguments\\!](https:\/\/chryswoods.com\/beginning_perl\/arguments.html)\n    - [Conditions](https:\/\/chryswoods.com\/beginning_perl\/conditions.html)\n    - [Files](https:\/\/chryswoods.com\/beginning_perl\/files.html)\n    - [Writing Files](https:\/\/chryswoods.com\/beginning_perl\/writing.html)\n    - [Splitting Lines](https:\/\/chryswoods.com\/beginning_perl\/splitting.html)\n    - [Searching Files](https:\/\/chryswoods.com\/beginning_perl\/searching.html)\n    - [Search and Replace](https:\/\/chryswoods.com\/beginning_perl\/replacing.html)\n    - [Running Programs](https:\/\/chryswoods.com\/beginning_perl\/running.html)\n    - [Job Scripts](https:\/\/chryswoods.com\/beginning_perl\/jobs.html)\n    - [What Next?](https:\/\/chryswoods.com\/beginning_perl\/whatnext.html)\n  - [Beginning Python](https:\/\/chryswoods.com\/beginning_python\/README.html)\n    - [Python Basics](https:\/\/chryswoods.com\/beginning_python\/basics.html)\n    - [Loops](https:\/\/chryswoods.com\/beginning_python\/loops.html)\n    - [Arguments\\!](https:\/\/chryswoods.com\/beginning_python\/arguments.html)\n    - [Conditions](https:\/\/chryswoods.com\/beginning_python\/conditions.html)\n    - [Files](https:\/\/chryswoods.com\/beginning_python\/files.html)\n    - [Writing Files](https:\/\/chryswoods.com\/beginning_python\/writing.html)\n    - [Splitting Lines](https:\/\/chryswoods.com\/beginning_python\/splitting.html)\n    - [Searching Files](https:\/\/chryswoods.com\/beginning_python\/searching.html)\n    - [Search and Replace](https:\/\/chryswoods.com\/beginning_python\/replacing.html)\n    - [Running Programs](https:\/\/chryswoods.com\/beginning_python\/running.html)\n    - [Job Scripts](https:\/\/chryswoods.com\/beginning_python\/jobs.html)\n    - [What Next?](https:\/\/chryswoods.com\/beginning_python\/whatnext.html)\n  - [Intermediate Python](https:\/\/chryswoods.com\/intermediate_python\/README.html)\n    - [Lists](https:\/\/chryswoods.com\/intermediate_python\/lists.html)\n    - [Dictionaries](https:\/\/chryswoods.com\/intermediate_python\/dictionaries.html)\n    - [Functions](https:\/\/chryswoods.com\/intermediate_python\/functions.html)\n    - [Modules](https:\/\/chryswoods.com\/intermediate_python\/modules.html)\n    - [Documenting Code](https:\/\/chryswoods.com\/intermediate_python\/documenting.html)\n    - [Objects and Classes](https:\/\/chryswoods.com\/intermediate_python\/objects.html)\n    - [Testing](https:\/\/chryswoods.com\/intermediate_python\/testing.html)\n    - [Regular Expressions in Python](https:\/\/chryswoods.com\/intermediate_python\/regexp.html)\n    - [What next?](https:\/\/chryswoods.com\/intermediate_python\/whatnext.html)\n  - [Parallel Programming with Python](https:\/\/chryswoods.com\/parallel_python\/README.html)\n    - [Part 1: Functional Programming](https:\/\/chryswoods.com\/parallel_python\/part1.html)\n      - [Functions as Objects](https:\/\/chryswoods.com\/parallel_python\/functions.html)\n      - [Mapping Functions](https:\/\/chryswoods.com\/parallel_python\/map.html)\n      - [Reduction](https:\/\/chryswoods.com\/parallel_python\/reduce.html)\n      - [Anonymous Functions (lambda)](https:\/\/chryswoods.com\/parallel_python\/lambda.html)\n    - [Part 2: Multicore (local) Parallel Programming](https:\/\/chryswoods.com\/parallel_python\/part2.html)\n      - [Multiprocessing](https:\/\/chryswoods.com\/parallel_python\/multiprocessing.html)\n      - [Pool](https:\/\/chryswoods.com\/parallel_python\/pool_part2.html)\n      - [Parallel map\/reduce](https:\/\/chryswoods.com\/parallel_python\/mapreduce_part2.html)\n      - [Asynchronous Functions and Futures](https:\/\/chryswoods.com\/parallel_python\/futures_part2.html)\n      - [Asynchronous Mapping](https:\/\/chryswoods.com\/parallel_python\/async_map.html)\n    - [Part 3: Multinode (distributed\/cluster) Parallel Programming](https:\/\/chryswoods.com\/parallel_python\/part3.html)\n      - [Scoop](https:\/\/chryswoods.com\/parallel_python\/scoop.html)\n      - [Distributed map\/reduce](https:\/\/chryswoods.com\/parallel_python\/mapreduce_part3.html)\n      - [Running Scoop on a Cluster](https:\/\/chryswoods.com\/parallel_python\/cluster.html)\n      - [What Next?](https:\/\/chryswoods.com\/parallel_python\/whatnext.html)\n    - [Epilogue](https:\/\/chryswoods.com\/parallel_python\/epilogue.html)\n      - [Changes from Python 2 to 3](https:\/\/chryswoods.com\/parallel_python\/python2to3.html)\n      - [Global Interpreter Lock (GIL)](https:\/\/chryswoods.com\/parallel_python\/gil.html)\n  - [Beginning R](https:\/\/chryswoods.com\/beginning_r\/index.html)\n  - [Intermediate R](https:\/\/chryswoods.com\/intermediate_r\/index.html)\n  - [Introduction to Data Analysis in R](https:\/\/chryswoods.com\/data_analysis_r\/index.html)\n  - [Beginning C++](https:\/\/chryswoods.com\/beginning_c++\/README.html)\n    - [Why C++?](https:\/\/chryswoods.com\/beginning_c++\/why.html)\n    - [C++ Basics](https:\/\/chryswoods.com\/beginning_c++\/basics.html)\n    - [Syntax Compared to Python](https:\/\/chryswoods.com\/beginning_c++\/syntax.html)\n    - [Types, Scopes and Auto](https:\/\/chryswoods.com\/beginning_c++\/typing.html)\n    - [Lists and Dictionaries](https:\/\/chryswoods.com\/beginning_c++\/lists.html)\n    - [Objects and Classes](https:\/\/chryswoods.com\/beginning_c++\/objects.html)\n    - [Concepts, Default Arguments and Operators](https:\/\/chryswoods.com\/beginning_c++\/operators.html)\n    - [What next?](https:\/\/chryswoods.com\/beginning_c++\/whatnext.html)\n  - [Parallel Programming with C++](https:\/\/chryswoods.com\/parallel_c++\/README.html)\n    - [Part 1: Functional Programming](https:\/\/chryswoods.com\/parallel_c++\/part1.html)\n      - [Functions as Objects](https:\/\/chryswoods.com\/parallel_c++\/functions.html)\n      - [Mapping Functions](https:\/\/chryswoods.com\/parallel_c++\/map.html)\n      - [Reduction](https:\/\/chryswoods.com\/parallel_c++\/reduce.html)\n      - [Anonymous Functions (lambda)](https:\/\/chryswoods.com\/parallel_c++\/lambda.html)\n      - [Map\/Reduce](https:\/\/chryswoods.com\/parallel_c++\/mapreduce.html)\n    - [Part 2: Parallel Programming Using Intel Threading Building Blocks](https:\/\/chryswoods.com\/parallel_c++\/part2.html)\n      - [tbb::parallel\\_for](https:\/\/chryswoods.com\/parallel_c++\/parallel_for.html)\n      - [tbb::parallel\\_reduce](https:\/\/chryswoods.com\/parallel_c++\/parallel_reduce.html)\n      - [Writing a parallel map\/reduce](https:\/\/chryswoods.com\/parallel_c++\/parallel_mapreduce.html)\n      - [What Next?](https:\/\/chryswoods.com\/parallel_c++\/whatnext.html)\n  - [Efficient Vectorisation with C++](https:\/\/chryswoods.com\/vector_c++\/README.html)\n    - [Part 1: Introduction to Vectorisation](https:\/\/chryswoods.com\/vector_c++\/part1.html)\n      - [What is Vectorisation?](https:\/\/chryswoods.com\/vector_c++\/vectorisation.html)\n      - [How to Vectorise (omp simd)](https:\/\/chryswoods.com\/vector_c++\/simd.html)\n      - [omp simd features](https:\/\/chryswoods.com\/vector_c++\/features.html)\n      - [Memory Layout](https:\/\/chryswoods.com\/vector_c++\/memory.html)\n      - [omp simd limitations](https:\/\/chryswoods.com\/vector_c++\/limitations.html)\n    - [Part 2: Vectorisation using Intrinsics](https:\/\/chryswoods.com\/vector_c++\/part2.html)\n      - [SSE Intrinsics](https:\/\/chryswoods.com\/vector_c++\/emmintrin.html)\n      - [AVX Intrinsics](https:\/\/chryswoods.com\/vector_c++\/immintrin.html)\n      - [Portable Vectorisation](https:\/\/chryswoods.com\/vector_c++\/portable.html)\n      - [What Next?](https:\/\/chryswoods.com\/vector_c++\/whatnext.html)\n  - [Parallel Programming with OpenMP](https:\/\/chryswoods.com\/beginning_openmp\/README.html)\n    - [Basics](https:\/\/chryswoods.com\/beginning_openmp\/basics.html)\n    - [Compiler Directives \/ Pragmas](https:\/\/chryswoods.com\/beginning_openmp\/directives.html)\n    - [Sections](https:\/\/chryswoods.com\/beginning_openmp\/sections.html)\n    - [Loops](https:\/\/chryswoods.com\/beginning_openmp\/loops.html)\n    - [Critical Code](https:\/\/chryswoods.com\/beginning_openmp\/critical.html)\n    - [Reduction](https:\/\/chryswoods.com\/beginning_openmp\/reduction.html)\n    - [Map \/ Reduce](https:\/\/chryswoods.com\/beginning_openmp\/mapreduce.html)\n    - [Maximising Performance](https:\/\/chryswoods.com\/beginning_openmp\/performance.html)\n    - [Case Study](https:\/\/chryswoods.com\/beginning_openmp\/casestudy.html)\n    - [What Next?](https:\/\/chryswoods.com\/beginning_openmp\/whatnext.html)\n  - [Parallel Programming with MPI](https:\/\/chryswoods.com\/beginning_mpi\/README.html)\n    - [Basics](https:\/\/chryswoods.com\/beginning_mpi\/basics.html)\n    - [MPI Functions](https:\/\/chryswoods.com\/beginning_mpi\/functions.html)\n    - [Sections](https:\/\/chryswoods.com\/beginning_mpi\/sections.html)\n    - [Loops](https:\/\/chryswoods.com\/beginning_mpi\/loops.html)\n    - [Messages](https:\/\/chryswoods.com\/beginning_mpi\/messages.html)\n    - [Reduction](https:\/\/chryswoods.com\/beginning_mpi\/reduction.html)\n    - [Map \/ Reduce](https:\/\/chryswoods.com\/beginning_mpi\/mapreduce.html)\n    - [Maximising Performance](https:\/\/chryswoods.com\/beginning_mpi\/performance.html)\n    - [What Next?](https:\/\/chryswoods.com\/beginning_mpi\/whatnext.html)\n  - [Version Control with Git](https:\/\/chryswoods.com\/beginning_git\/README.html)\n    - [Git Basics](https:\/\/chryswoods.com\/beginning_git\/basics.html)\n    - [Adding Files](https:\/\/chryswoods.com\/beginning_git\/adding.html)\n    - [Committing Changes](https:\/\/chryswoods.com\/beginning_git\/committing.html)\n    - [Diffing (seeing what has changed)](https:\/\/chryswoods.com\/beginning_git\/diffing.html)\n    - [Changing Versions](https:\/\/chryswoods.com\/beginning_git\/versions.html)\n    - [Branching](https:\/\/chryswoods.com\/beginning_git\/branching.html)\n    - [Renaming and Removing Files](https:\/\/chryswoods.com\/beginning_git\/renaming.html)\n    - [Subdirectories and Ignoring Files](https:\/\/chryswoods.com\/beginning_git\/subdirs.html)\n    - [Git in the Cloud](https:\/\/chryswoods.com\/beginning_git\/github.html)\n    - [Pushing to the Cloud](https:\/\/chryswoods.com\/beginning_git\/push.html)\n    - [Markdown](https:\/\/chryswoods.com\/beginning_git\/markdown.html)\n    - [Cloning a Repository](https:\/\/chryswoods.com\/beginning_git\/cloning.html)\n    - [Merging](https:\/\/chryswoods.com\/beginning_git\/merging.html)\n    - [Pull Requests](https:\/\/chryswoods.com\/beginning_git\/pull.html)\n    - [Continuous Integration](https:\/\/chryswoods.com\/beginning_git\/ci.html)\n    - [What next?](https:\/\/chryswoods.com\/beginning_git\/whatnext.html)\n  - [JupyterHub and Kubernetes](https:\/\/chryswoods.com\/inception_workshop\/README.html)\n    - [Creating the workshop](https:\/\/chryswoods.com\/inception_workshop\/course\/part01.html)\n    - [Finding all your dependencies](https:\/\/chryswoods.com\/inception_workshop\/course\/part02.html)\n    - [Building the docker image](https:\/\/chryswoods.com\/inception_workshop\/course\/part03.html)\n    - [JupyterHub](https:\/\/chryswoods.com\/inception_workshop\/course\/part04.html)\n    - [Uploading to the cloud](https:\/\/chryswoods.com\/inception_workshop\/course\/part05.html)\n    - [Kubernetes](https:\/\/chryswoods.com\/inception_workshop\/course\/part06.html)\n    - [Helm](https:\/\/chryswoods.com\/inception_workshop\/course\/part07.html)\n    - [Configuring JupyterHub](https:\/\/chryswoods.com\/inception_workshop\/course\/part08.html)\n    - [What next?](https:\/\/chryswoods.com\/inception_workshop\/course\/whatnext.html)\n  - [Introduction to Monte Carlo](https:\/\/chryswoods.com\/intro_to_mc\/README.html)\n    - [Part 1](https:\/\/chryswoods.com\/intro_to_mc\/part1\/README.html)\n      - [Introduction to Monte Carlo](https:\/\/chryswoods.com\/intro_to_mc\/part1\/intro.html)\n      - [Software](https:\/\/chryswoods.com\/intro_to_mc\/part1\/software.html)\n      - [Metropolis Monte Carlo](https:\/\/chryswoods.com\/intro_to_mc\/part1\/metropolis.html)\n      - [Running metropolis.py](https:\/\/chryswoods.com\/intro_to_mc\/part1\/running.html)\n      - [Control Variables](https:\/\/chryswoods.com\/intro_to_mc\/part1\/control.html)\n      - [Phase Changes](https:\/\/chryswoods.com\/intro_to_mc\/part1\/phase.html)\n      - [Phase Space and Ensembles](https:\/\/chryswoods.com\/intro_to_mc\/part1\/ensemble.html)\n      - [Volume Moves](https:\/\/chryswoods.com\/intro_to_mc\/part1\/volume.html)\n      - [NPT Simulations](https:\/\/chryswoods.com\/intro_to_mc\/part1\/npt.html)\n      - [Summary](https:\/\/chryswoods.com\/intro_to_mc\/part1\/summary.html)\n    - [Part 2](https:\/\/chryswoods.com\/intro_to_mc\/part2\/README.html)\n      - [Introduction and Software](https:\/\/chryswoods.com\/intro_to_mc\/part2\/intro.html)\n      - [Sampling the Solvent - Rigid Body Moves](https:\/\/chryswoods.com\/intro_to_mc\/part2\/rigid.html)\n      - [Sampling the Ligand - Intramolecular Moves](https:\/\/chryswoods.com\/intro_to_mc\/part2\/intra.html)\n      - [Sampling the Protein - Backbone Moves](https:\/\/chryswoods.com\/intro_to_mc\/part2\/backbone.html)\n      - [Sampling it all - Weighting Moves](https:\/\/chryswoods.com\/intro_to_mc\/part2\/weight.html)\n      - [Summary](https:\/\/chryswoods.com\/intro_to_mc\/part2\/whatnext.html)\n  - [Molecular Visualisation, Modelling and Dynamics](https:\/\/chryswoods.com\/dynamics\/README.html)\n    - [Part 1: Molecular Visualisation](https:\/\/chryswoods.com\/dynamics\/visualisation\/README.html)\n      - [1a: Opening Files](https:\/\/chryswoods.com\/dynamics\/visualisation\/opening_files.html)\n      - [1b: Manipulating the View](https:\/\/chryswoods.com\/dynamics\/visualisation\/mouse.html)\n      - [1c: Graphical Representations](https:\/\/chryswoods.com\/dynamics\/visualisation\/representations.html)\n      - [1d: Selecting Atoms](https:\/\/chryswoods.com\/dynamics\/visualisation\/selection.html)\n      - [1e: Complex Selections](https:\/\/chryswoods.com\/dynamics\/visualisation\/complex_selection.html)\n      - [1f: Rendering](https:\/\/chryswoods.com\/dynamics\/visualisation\/rendering.html)\n      - [1g: Movies](https:\/\/chryswoods.com\/dynamics\/visualisation\/movies.html)\n      - [1h: Picking Atoms](https:\/\/chryswoods.com\/dynamics\/visualisation\/picking.html)\n      - [1i: Comparing Trajectories](https:\/\/chryswoods.com\/dynamics\/visualisation\/comparing.html)\n      - [1j: What Next?](https:\/\/chryswoods.com\/dynamics\/visualisation\/whatnext.html)\n    - [Part 2: Molecular Dynamics](https:\/\/chryswoods.com\/dynamics\/dynamics\/README.html)\n      - [2a: Getting Started](https:\/\/chryswoods.com\/dynamics\/dynamics\/getting_started.html)\n      - [2b: Theory of MD](https:\/\/chryswoods.com\/dynamics\/dynamics\/theory.html)\n      - [2c: Changing Time](https:\/\/chryswoods.com\/dynamics\/dynamics\/time.html)\n      - [2d: Shake (Rattle and Roll)](https:\/\/chryswoods.com\/dynamics\/dynamics\/shake.html)\n      - [2e: Periodic Boundary Conditions](https:\/\/chryswoods.com\/dynamics\/dynamics\/protein.html)\n      - [2f: Under Pressure](https:\/\/chryswoods.com\/dynamics\/dynamics\/pressure.html)\n      - [2g: Running the Simulation](https:\/\/chryswoods.com\/dynamics\/dynamics\/simulation.html)\n      - [2h: What Next?](https:\/\/chryswoods.com\/dynamics\/dynamics\/whatnext.html)\n    - [Part 3: Mutation Studies](https:\/\/chryswoods.com\/dynamics\/mutation\/README.html)\n      - [3a: Getting Started](https:\/\/chryswoods.com\/dynamics\/mutation\/gettingstarted.html)\n      - [3b: Mutating the Protein](https:\/\/chryswoods.com\/dynamics\/mutation\/mutation.html)\n      - [3c: Solvating the Protein](https:\/\/chryswoods.com\/dynamics\/mutation\/solvation.html)\n      - [3d: Minimising the System](https:\/\/chryswoods.com\/dynamics\/mutation\/minimisation.html)\n      - [3e: Heating the System](https:\/\/chryswoods.com\/dynamics\/mutation\/heating.html)\n      - [3f: Equilibrating the System](https:\/\/chryswoods.com\/dynamics\/mutation\/equilibration.html)\n      - [3g: Running the Simulation](https:\/\/chryswoods.com\/dynamics\/mutation\/simulation.html)\n      - [3h: Comparing Trajectories](https:\/\/chryswoods.com\/dynamics\/mutation\/compare.html)\n      - [3i: What Next?](https:\/\/chryswoods.com\/dynamics\/mutation\/whatnext.html)\n  - [QM\/MM Monte Carlo](https:\/\/chryswoods.com\/embo2014\/Practical.html)\n  - [Software Carpentry](https:\/\/chryswoods.com\/main\/softwarecarpentry.html)\n    - [Bath (July 2013) Workshop](http:\/\/tinyurl.com\/swcbath)\n    - [Bristol (September 2013) Workshop](http:\/\/tinyurl.com\/swcbristol)\n    - [Exeter (November 2013) Workshop](http:\/\/tinyurl.com\/swcexeter)\n- [Software](https:\/\/chryswoods.com\/main\/software.html)\n  - [Sire Molecular Simulation Framework](https:\/\/siremol.org\/)\n  - [ProtoMS Monte Carlo Package](http:\/\/protoms.org\/)\n- [Publications](https:\/\/chryswoods.com\/main\/publications.html)\n- [Talks](https:\/\/chryswoods.com\/talks\/README.html)\n- [Useful Links](https:\/\/chryswoods.com\/main\/links.html)\n- [Books that are Worth Reading](https:\/\/chryswoods.com\/main\/reading.html)\n- [Contact Information](https:\/\/chryswoods.com\/main\/contact.html)\n- [About this Website](https:\/\/chryswoods.com\/main\/website.html)\n\n- [![](https:\/\/chryswoods.com\/images\/hamburger.png) chryswoods.com](https:\/\/chryswoods.com\/parallel_python\/async_map.html#main_menu)\n\n# Part 2: Asynchronous Mapping\nAsynchronous functions allow you to give different tasks to different members of the `multiprocessing.Pool`. However, giving functions one by one is not very efficient. It would be good to be able to combine mapping with asynchronous functions, i.e. be able to give different mapping tasks simultanously to the pool of workers. Fortunately, `Pool.map_async` provides exactly that - an asynchronous parallel map.\n\nCreate a new python script called `asyncmap.py` and copy into it\n```\nfrom functools import reduce\nfrom multiprocessing import Pool, current_process\nimport time\n\ndef add(x, y):\n    \"\"\"Return the sum of the arguments\"\"\"\n    print(\"Worker %s is processing add(%s, %s)\" % (current_process().pid, x, y))\n    time.sleep(1)\n    return x + y\n\ndef product(x, y):\n    \"\"\"Return the product of the arguments\"\"\"\n    print(\"Worker %s is processing product(%s, %s)\" % (current_process().pid, x, y))\n    time.sleep(1)\n    return x * y\n\nif __name__ == \"__main__\":\n\n    a = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]\n    b = [11, 12, 13, 14, 15, 16, 17, 18, 19, 20]\n\n    # Now create a Pool of workers\n    with Pool() as pool:\n        sum_future = pool.starmap_async(add, zip(a,b))\n        product_future = pool.starmap_async(product, zip(a,b))\n\n        sum_future.wait()\n        product_future.wait()\n\n    total_sum = reduce(lambda x, y: x + y, sum_future.get())\n    total_product = reduce(lambda x, y: x + y, product_future.get())\n\n    print(\"Sum of sums of 'a' and 'b' is %s\" % total_sum)\n    print(\"Sum of products of 'a' and 'b' is %s\" % total_product)\n```\nRunning this script, e.g. via `python asyncmap.py` should result in something like\n```\nWorker 32722 is processing add(1, 11)\nWorker 32723 is processing add(2, 12)\nWorker 32724 is processing add(3, 13)\nWorker 32725 is processing add(4, 14)\nWorker 32722 is processing add(5, 15)\nWorker 32724 is processing add(6, 16)\nWorker 32725 is processing add(7, 17)\nWorker 32723 is processing add(8, 18)\nWorker 32722 is processing add(9, 19)\nWorker 32724 is processing add(10, 20)\nWorker 32725 is processing product(1, 11)\nWorker 32723 is processing product(2, 12)\nWorker 32722 is processing product(3, 13)\nWorker 32723 is processing product(4, 14)\nWorker 32724 is processing product(5, 15)\nWorker 32725 is processing product(6, 16)\nWorker 32722 is processing product(7, 17)\nWorker 32725 is processing product(8, 18)\nWorker 32723 is processing product(9, 19)\nWorker 32724 is processing product(10, 20)\nSum of sums of 'a' and 'b' is 210\nSum of products of 'a' and 'b' is 935\n```\nThis script provides two functions, `add` and `product`, which are mapped asynchronously using the `Pool.map_async` function. This is identical to the `Pool.map` function that you used before, except now the map is performed asynchronously. This means that the resulting list is returned in a future (in this case, the futures `sum_future` and `product_future`. The results are waited for using the `.wait()` functions, remembering to make sure that we don’t exit the `with` block until all results are available. Then, the results of mapping are retrieved using the `.get()` function of the futures.\n\n## Chunking\nBy default, the `Pool.map` function divides the work over the pool of workers by assiging pieces of work one by one. In the example above, the work to be performed was;\n```\nadd(1, 11)\nadd(2, 12)\nadd(3, 13)\netc.\nadd(10,20)\nproduct(1, 11)\nproduct(2, 12)\nproduct(3, 13)\netc.\nproduct(10, 20)\n```\nThe work was assigned one by one to the four workers on my computer, i.e. the first worker process was given `add(1, 11)`, the second `add(2, 12)`, the third `add(3, 13)` the then the fourth `add(4, 14)`. The first worker to finish was then given `add(5, 15)`, then the next given `add(6, 16)` etc. etc.\n\nGiving work one by one can be very inefficient for quick tasks, as the time needed by a worker process to stop and get new work can be longer than it takes to actually complete the task. To solve this problem, you can control how many work items are handed out to each worker process at a time. This is known as chunking, and the number of work items is known as the chunk of work to perform.\n\nYou can control the number of work items to perform per worker (the chunk size) by setting the `chunksize` argument, e.g.\n```\nsum_future = pool.starmap_async(add, zip(a, b), chunksize=5)\n```\nwould suggest to `pool` that each worker be given a chunk of five pieces of work. Note that this is just a suggestion, and `pool` may decide to use a slightly smaller or larger chunk size depending on the amount of work and the number of workers available.\n\nModify your `asyncmap.py` script and set the `chunksize` to 5 for both of the asynchronous maps for `add` and `product`. Re-run your script. You should see something like;\n```\nWorker 658 is processing add(1, 11)\nWorker 659 is processing add(6, 16)\nWorker 660 is processing product(1, 11)\nWorker 661 is processing product(6, 16)\nWorker 659 is processing add(7, 17)\nWorker 660 is processing product(2, 12)\nWorker 661 is processing product(7, 17)\nWorker 658 is processing add(2, 12)\nWorker 660 is processing product(3, 13)\nWorker 659 is processing add(8, 18)\nWorker 661 is processing product(8, 18)\nWorker 658 is processing add(3, 13)\nWorker 660 is processing product(4, 14)\nWorker 659 is processing add(9, 19)\nWorker 661 is processing product(9, 19)\nWorker 658 is processing add(4, 14)\nWorker 659 is processing add(10, 20)\nWorker 660 is processing product(5, 15)\nWorker 661 is processing product(10, 20)\nWorker 658 is processing add(5, 15)\nSum of sums of 'a' and 'b' is 210\nSum of products of 'a' and 'b' is 935\n```\nMy laptop has four workers. The first worker is assigned the first five items of work, i.e. `add(1, 11)` to `add(5, 15)`, and it starts by running `add(1, 11)`, hence why `add(1, 11)` is printed first.\n\nThe next worker is given the next five items of work, i.e. `add(6, 16)` to `add(10,20)`, and starts by running `add(6, 16)`, hence why `add(6, 16)` is printed second.\n\nThe next worker is given the next five items of work, i.e. `product(1, 11)` to `product(5, 15)`, and it starts by running `product(1, 11)`, hence why this is printed third.\n\nThe last worker is given the next five items of work, i.e. `product(6, 16)` to `product(10, 20)`, and it starts by running `product(6, 16)`, hence why this is printed fourth.\n\nOnce each worker has finished its first item of work, it moves onto its second. This is why `add(2, 12)`, `add(7, 17)`, `product(2, 12)` and `product(7, 17)` are printed next. Then, each worker moves onto its third piece of work etc. etc.\n\nIf you don’t specify the `chunksize` then it is equal to `1`. When writing a new script you should experiment with different values of `chunksize` to find the value that gives best performance.\n***\n## Exercise\nEdit your script written in answer to [exercise 2 of Parallel Map\/Reduce](https:\/\/chryswoods.com\/parallel_python\/mapreduce_part2.html), in which you count all of the words used in all Shakespeare plays (e.g. an example answer [is here](https:\/\/chryswoods.com\/parallel_python\/mapreduce2_answer2.html)).\n\nEdit the script so that you use an asynchronous map to distribute the work over the pool. This will free up the master process to give feedback to the user of the script, e.g. to print a progress or status message while the work is running to reassure the user that the script has not frozen. For example\n```\nwhile not future.ready():\n    print(\"Work is in progress...\")\n    time.sleep(0.1)\n```\nAdd a status message to your script to reassure the user that your script hasn’t frozen while it is processing.\n\n(note that you can call your script using `python -u countwords.py shakespeare\/*` to use the `-u` argument to stop Python from buffering text written to standard output)\n\nIf you get stuck or want inspiration, a possible answer is given [here](https:\/\/chryswoods.com\/parallel_python\/async_map_answer1.html).\n***\n# [Previous](https:\/\/chryswoods.com\/parallel_python\/futures_part2.html) [Up](https:\/\/chryswoods.com\/parallel_python\/part2.html) [Next](https:\/\/chryswoods.com\/parallel_python\/part3.html)\n\nQuick Links \\| [Home](https:\/\/chryswoods.com\/index.html) \\| [Courses](https:\/\/chryswoods.com\/main\/courses.html) \\| [Software](https:\/\/chryswoods.com\/main\/software.html) \\| [Contact](https:\/\/chryswoods.com\/main\/contact.html)\n\n[Copyright Information](https:\/\/chryswoods.com\/main\/copyright.html) \\| [Report a problem](<mailto:chryswoods@gmail.com?subject=There is a problem with your website&body=Hi, I found a problem on the page parallel_python\/async_map.html>) I [Privacy](https:\/\/chryswoods.com\/main\/cookies.html)","attrs_readable_markdown":null,"meta_canonical":null}

3. Robots.txt Check

Query:

Response:

4. Spam/Ban Check

Query:

Response:

5. Seen Status Check

ℹ️ Skipped - page is already crawled

📄

INDEXABLE

✅

CRAWLED

17 hours ago

🤖

ROBOTS ALLOWED

Page Info Filters

Filter	Status	Condition	Details
HTTP status	PASS	`download_http_code = 200`	HTTP 200
Age cutoff	PASS	`download_stamp > now() - 6 MONTH`	0 months ago
History drop	PASS	`isNull(history_drop_reason)`	No drop reason
Spam/ban	PASS	`fh_dont_index != 1 AND ml_spam_score = 0`	ml_spam_score=0
Canonical	PASS	`meta_canonical IS NULL OR = '' OR = src_unparsed`	Not set

Page Details

Property	Value
URL	https://chryswoods.com/parallel_python/async_map.html
Last Crawled	2026-04-06 05:26:39 (17 hours ago)
First Indexed	2020-11-25 08:22:05 (5 years ago)
HTTP Status Code	200
Meta Title	chryswoods.com \| Part 2: Asynchronous Mapping
Meta Description	null
Meta Canonical	null
Boilerpipe Text	Asynchronous functions allow you to give different tasks to different members of the multiprocessing.Pool . However, giving functions one by one is not very efficient. It would be good to be able to combine mapping with asynchronous functions, i.e. be able to give different mapping tasks simultanously to the pool of workers. Fortunately, Pool.map_async provides exactly that - an asynchronous parallel map. Create a new python script called asyncmap.py and copy into it from functools import reduce from multiprocessing import Pool, current_process import time def add(x, y): """Return the sum of the arguments""" print ( "Worker %s is processing add( %s , %s )" % (current_process().pid, x, y)) time.sleep( 1 ) return x + y def product(x, y): """Return the product of the arguments""" print ( "Worker %s is processing product( %s , %s )" % (current_process().pid, x, y)) time.sleep( 1 ) return x * y if __name__ == "__main__" : a = [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 ] b = [ 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 ] # Now create a Pool of workers with Pool() as pool: sum_future = pool.starmap_async(add, zip (a,b)) product_future = pool.starmap_async(product, zip (a,b)) sum_future.wait() product_future.wait() total_sum = reduce ( lambda x, y: x + y, sum_future.get()) total_product = reduce ( lambda x, y: x + y, product_future.get()) print ( "Sum of sums of 'a' and 'b' is %s " % total_sum) print ( "Sum of products of 'a' and 'b' is %s " % total_product) Running this script, e.g. via python asyncmap.py should result in something like Worker 32722 is processing add(1, 11) Worker 32723 is processing add(2, 12) Worker 32724 is processing add(3, 13) Worker 32725 is processing add(4, 14) Worker 32722 is processing add(5, 15) Worker 32724 is processing add(6, 16) Worker 32725 is processing add(7, 17) Worker 32723 is processing add(8, 18) Worker 32722 is processing add(9, 19) Worker 32724 is processing add(10, 20) Worker 32725 is processing product(1, 11) Worker 32723 is processing product(2, 12) Worker 32722 is processing product(3, 13) Worker 32723 is processing product(4, 14) Worker 32724 is processing product(5, 15) Worker 32725 is processing product(6, 16) Worker 32722 is processing product(7, 17) Worker 32725 is processing product(8, 18) Worker 32723 is processing product(9, 19) Worker 32724 is processing product(10, 20) Sum of sums of 'a' and 'b' is 210 Sum of products of 'a' and 'b' is 935 This script provides two functions, add and product , which are mapped asynchronously using the Pool.map_async function. This is identical to the Pool.map function that you used before, except now the map is performed asynchronously. This means that the resulting list is returned in a future (in this case, the futures sum_future and product_future . The results are waited for using the .wait() functions, remembering to make sure that we don’t exit the with block until all results are available. Then, the results of mapping are retrieved using the .get() function of the futures. Chunking By default, the Pool.map function divides the work over the pool of workers by assiging pieces of work one by one. In the example above, the work to be performed was; add(1, 11) add(2, 12) add(3, 13) etc. add(10,20) product(1, 11) product(2, 12) product(3, 13) etc. product(10, 20) The work was assigned one by one to the four workers on my computer, i.e. the first worker process was given add(1, 11) , the second add(2, 12) , the third add(3, 13) the then the fourth add(4, 14) . The first worker to finish was then given add(5, 15) , then the next given add(6, 16) etc. etc. Giving work one by one can be very inefficient for quick tasks, as the time needed by a worker process to stop and get new work can be longer than it takes to actually complete the task. To solve this problem, you can control how many work items are handed out to each worker process at a time. This is known as chunking, and the number of work items is known as the chunk of work to perform. You can control the number of work items to perform per worker (the chunk size) by setting the chunksize argument, e.g. sum_future = pool.starmap_async(add, zip (a, b), chunksize = 5 ) would suggest to pool that each worker be given a chunk of five pieces of work. Note that this is just a suggestion, and pool may decide to use a slightly smaller or larger chunk size depending on the amount of work and the number of workers available. Modify your asyncmap.py script and set the chunksize to 5 for both of the asynchronous maps for add and product . Re-run your script. You should see something like; Worker 658 is processing add(1, 11) Worker 659 is processing add(6, 16) Worker 660 is processing product(1, 11) Worker 661 is processing product(6, 16) Worker 659 is processing add(7, 17) Worker 660 is processing product(2, 12) Worker 661 is processing product(7, 17) Worker 658 is processing add(2, 12) Worker 660 is processing product(3, 13) Worker 659 is processing add(8, 18) Worker 661 is processing product(8, 18) Worker 658 is processing add(3, 13) Worker 660 is processing product(4, 14) Worker 659 is processing add(9, 19) Worker 661 is processing product(9, 19) Worker 658 is processing add(4, 14) Worker 659 is processing add(10, 20) Worker 660 is processing product(5, 15) Worker 661 is processing product(10, 20) Worker 658 is processing add(5, 15) Sum of sums of 'a' and 'b' is 210 Sum of products of 'a' and 'b' is 935 My laptop has four workers. The first worker is assigned the first five items of work, i.e. add(1, 11) to add(5, 15) , and it starts by running add(1, 11) , hence why add(1, 11) is printed first. The next worker is given the next five items of work, i.e. add(6, 16) to add(10,20) , and starts by running add(6, 16) , hence why add(6, 16) is printed second. The next worker is given the next five items of work, i.e. product(1, 11) to product(5, 15) , and it starts by running product(1, 11) , hence why this is printed third. The last worker is given the next five items of work, i.e. product(6, 16) to product(10, 20) , and it starts by running product(6, 16) , hence why this is printed fourth. Once each worker has finished its first item of work, it moves onto its second. This is why add(2, 12) , add(7, 17) , product(2, 12) and product(7, 17) are printed next. Then, each worker moves onto its third piece of work etc. etc. If you don’t specify the chunksize then it is equal to 1 . When writing a new script you should experiment with different values of chunksize to find the value that gives best performance. Exercise Edit your script written in answer to exercise 2 of Parallel Map/Reduce , in which you count all of the words used in all Shakespeare plays (e.g. an example answer is here ). Edit the script so that you use an asynchronous map to distribute the work over the pool. This will free up the master process to give feedback to the user of the script, e.g. to print a progress or status message while the work is running to reassure the user that the script has not frozen. For example while not future.ready(): print ( "Work is in progress..." ) time.sleep( 0.1 ) Add a status message to your script to reassure the user that your script hasn’t frozen while it is processing. (note that you can call your script using python -u countwords.py shakespeare/* to use the -u argument to stop Python from buffering text written to standard output) If you get stuck or want inspiration, a possible answer is given here . Previous Up Next
Markdown	- [Home](https://chryswoods.com/index.html) - [Research](https://chryswoods.com/research/README.html) - [What is good research software engineering?](https://chryswoods.com/research/good_rse.html) - [How can the RSE group work with you?](https://chryswoods.com/research/collaboration.html) - [RSE Group news](https://chryswoods.com/research/news.html) - [Jobs](https://chryswoods.com/research/jobs.html) - [Training Resources](https://chryswoods.com/main/courses.html) - [Courses](https://chryswoods.com/main/courses.html) - [Python and Data](https://chryswoods.com/python_and_data/README.html) - [Beginning Perl](https://chryswoods.com/beginning_perl/README.html) - [Perl Basics](https://chryswoods.com/beginning_perl/basics.html) - [Loops](https://chryswoods.com/beginning_perl/loops.html) - [Arguments\!](https://chryswoods.com/beginning_perl/arguments.html) - [Conditions](https://chryswoods.com/beginning_perl/conditions.html) - [Files](https://chryswoods.com/beginning_perl/files.html) - [Writing Files](https://chryswoods.com/beginning_perl/writing.html) - [Splitting Lines](https://chryswoods.com/beginning_perl/splitting.html) - [Searching Files](https://chryswoods.com/beginning_perl/searching.html) - [Search and Replace](https://chryswoods.com/beginning_perl/replacing.html) - [Running Programs](https://chryswoods.com/beginning_perl/running.html) - [Job Scripts](https://chryswoods.com/beginning_perl/jobs.html) - [What Next?](https://chryswoods.com/beginning_perl/whatnext.html) - [Beginning Python](https://chryswoods.com/beginning_python/README.html) - [Python Basics](https://chryswoods.com/beginning_python/basics.html) - [Loops](https://chryswoods.com/beginning_python/loops.html) - [Arguments\!](https://chryswoods.com/beginning_python/arguments.html) - [Conditions](https://chryswoods.com/beginning_python/conditions.html) - [Files](https://chryswoods.com/beginning_python/files.html) - [Writing Files](https://chryswoods.com/beginning_python/writing.html) - [Splitting Lines](https://chryswoods.com/beginning_python/splitting.html) - [Searching Files](https://chryswoods.com/beginning_python/searching.html) - [Search and Replace](https://chryswoods.com/beginning_python/replacing.html) - [Running Programs](https://chryswoods.com/beginning_python/running.html) - [Job Scripts](https://chryswoods.com/beginning_python/jobs.html) - [What Next?](https://chryswoods.com/beginning_python/whatnext.html) - [Intermediate Python](https://chryswoods.com/intermediate_python/README.html) - [Lists](https://chryswoods.com/intermediate_python/lists.html) - [Dictionaries](https://chryswoods.com/intermediate_python/dictionaries.html) - [Functions](https://chryswoods.com/intermediate_python/functions.html) - [Modules](https://chryswoods.com/intermediate_python/modules.html) - [Documenting Code](https://chryswoods.com/intermediate_python/documenting.html) - [Objects and Classes](https://chryswoods.com/intermediate_python/objects.html) - [Testing](https://chryswoods.com/intermediate_python/testing.html) - [Regular Expressions in Python](https://chryswoods.com/intermediate_python/regexp.html) - [What next?](https://chryswoods.com/intermediate_python/whatnext.html) - [Parallel Programming with Python](https://chryswoods.com/parallel_python/README.html) - [Part 1: Functional Programming](https://chryswoods.com/parallel_python/part1.html) - [Functions as Objects](https://chryswoods.com/parallel_python/functions.html) - [Mapping Functions](https://chryswoods.com/parallel_python/map.html) - [Reduction](https://chryswoods.com/parallel_python/reduce.html) - [Anonymous Functions (lambda)](https://chryswoods.com/parallel_python/lambda.html) - [Part 2: Multicore (local) Parallel Programming](https://chryswoods.com/parallel_python/part2.html) - [Multiprocessing](https://chryswoods.com/parallel_python/multiprocessing.html) - [Pool](https://chryswoods.com/parallel_python/pool_part2.html) - [Parallel map/reduce](https://chryswoods.com/parallel_python/mapreduce_part2.html) - [Asynchronous Functions and Futures](https://chryswoods.com/parallel_python/futures_part2.html) - [Asynchronous Mapping](https://chryswoods.com/parallel_python/async_map.html) - [Part 3: Multinode (distributed/cluster) Parallel Programming](https://chryswoods.com/parallel_python/part3.html) - [Scoop](https://chryswoods.com/parallel_python/scoop.html) - [Distributed map/reduce](https://chryswoods.com/parallel_python/mapreduce_part3.html) - [Running Scoop on a Cluster](https://chryswoods.com/parallel_python/cluster.html) - [What Next?](https://chryswoods.com/parallel_python/whatnext.html) - [Epilogue](https://chryswoods.com/parallel_python/epilogue.html) - [Changes from Python 2 to 3](https://chryswoods.com/parallel_python/python2to3.html) - [Global Interpreter Lock (GIL)](https://chryswoods.com/parallel_python/gil.html) - [Beginning R](https://chryswoods.com/beginning_r/index.html) - [Intermediate R](https://chryswoods.com/intermediate_r/index.html) - [Introduction to Data Analysis in R](https://chryswoods.com/data_analysis_r/index.html) - [Beginning C++](https://chryswoods.com/beginning_c++/README.html) - [Why C++?](https://chryswoods.com/beginning_c++/why.html) - [C++ Basics](https://chryswoods.com/beginning_c++/basics.html) - [Syntax Compared to Python](https://chryswoods.com/beginning_c++/syntax.html) - [Types, Scopes and Auto](https://chryswoods.com/beginning_c++/typing.html) - [Lists and Dictionaries](https://chryswoods.com/beginning_c++/lists.html) - [Objects and Classes](https://chryswoods.com/beginning_c++/objects.html) - [Concepts, Default Arguments and Operators](https://chryswoods.com/beginning_c++/operators.html) - [What next?](https://chryswoods.com/beginning_c++/whatnext.html) - [Parallel Programming with C++](https://chryswoods.com/parallel_c++/README.html) - [Part 1: Functional Programming](https://chryswoods.com/parallel_c++/part1.html) - [Functions as Objects](https://chryswoods.com/parallel_c++/functions.html) - [Mapping Functions](https://chryswoods.com/parallel_c++/map.html) - [Reduction](https://chryswoods.com/parallel_c++/reduce.html) - [Anonymous Functions (lambda)](https://chryswoods.com/parallel_c++/lambda.html) - [Map/Reduce](https://chryswoods.com/parallel_c++/mapreduce.html) - [Part 2: Parallel Programming Using Intel Threading Building Blocks](https://chryswoods.com/parallel_c++/part2.html) - [tbb::parallel\_for](https://chryswoods.com/parallel_c++/parallel_for.html) - [tbb::parallel\_reduce](https://chryswoods.com/parallel_c++/parallel_reduce.html) - [Writing a parallel map/reduce](https://chryswoods.com/parallel_c++/parallel_mapreduce.html) - [What Next?](https://chryswoods.com/parallel_c++/whatnext.html) - [Efficient Vectorisation with C++](https://chryswoods.com/vector_c++/README.html) - [Part 1: Introduction to Vectorisation](https://chryswoods.com/vector_c++/part1.html) - [What is Vectorisation?](https://chryswoods.com/vector_c++/vectorisation.html) - [How to Vectorise (omp simd)](https://chryswoods.com/vector_c++/simd.html) - [omp simd features](https://chryswoods.com/vector_c++/features.html) - [Memory Layout](https://chryswoods.com/vector_c++/memory.html) - [omp simd limitations](https://chryswoods.com/vector_c++/limitations.html) - [Part 2: Vectorisation using Intrinsics](https://chryswoods.com/vector_c++/part2.html) - [SSE Intrinsics](https://chryswoods.com/vector_c++/emmintrin.html) - [AVX Intrinsics](https://chryswoods.com/vector_c++/immintrin.html) - [Portable Vectorisation](https://chryswoods.com/vector_c++/portable.html) - [What Next?](https://chryswoods.com/vector_c++/whatnext.html) - [Parallel Programming with OpenMP](https://chryswoods.com/beginning_openmp/README.html) - [Basics](https://chryswoods.com/beginning_openmp/basics.html) - [Compiler Directives / Pragmas](https://chryswoods.com/beginning_openmp/directives.html) - [Sections](https://chryswoods.com/beginning_openmp/sections.html) - [Loops](https://chryswoods.com/beginning_openmp/loops.html) - [Critical Code](https://chryswoods.com/beginning_openmp/critical.html) - [Reduction](https://chryswoods.com/beginning_openmp/reduction.html) - [Map / Reduce](https://chryswoods.com/beginning_openmp/mapreduce.html) - [Maximising Performance](https://chryswoods.com/beginning_openmp/performance.html) - [Case Study](https://chryswoods.com/beginning_openmp/casestudy.html) - [What Next?](https://chryswoods.com/beginning_openmp/whatnext.html) - [Parallel Programming with MPI](https://chryswoods.com/beginning_mpi/README.html) - [Basics](https://chryswoods.com/beginning_mpi/basics.html) - [MPI Functions](https://chryswoods.com/beginning_mpi/functions.html) - [Sections](https://chryswoods.com/beginning_mpi/sections.html) - [Loops](https://chryswoods.com/beginning_mpi/loops.html) - [Messages](https://chryswoods.com/beginning_mpi/messages.html) - [Reduction](https://chryswoods.com/beginning_mpi/reduction.html) - [Map / Reduce](https://chryswoods.com/beginning_mpi/mapreduce.html) - [Maximising Performance](https://chryswoods.com/beginning_mpi/performance.html) - [What Next?](https://chryswoods.com/beginning_mpi/whatnext.html) - [Version Control with Git](https://chryswoods.com/beginning_git/README.html) - [Git Basics](https://chryswoods.com/beginning_git/basics.html) - [Adding Files](https://chryswoods.com/beginning_git/adding.html) - [Committing Changes](https://chryswoods.com/beginning_git/committing.html) - [Diffing (seeing what has changed)](https://chryswoods.com/beginning_git/diffing.html) - [Changing Versions](https://chryswoods.com/beginning_git/versions.html) - [Branching](https://chryswoods.com/beginning_git/branching.html) - [Renaming and Removing Files](https://chryswoods.com/beginning_git/renaming.html) - [Subdirectories and Ignoring Files](https://chryswoods.com/beginning_git/subdirs.html) - [Git in the Cloud](https://chryswoods.com/beginning_git/github.html) - [Pushing to the Cloud](https://chryswoods.com/beginning_git/push.html) - [Markdown](https://chryswoods.com/beginning_git/markdown.html) - [Cloning a Repository](https://chryswoods.com/beginning_git/cloning.html) - [Merging](https://chryswoods.com/beginning_git/merging.html) - [Pull Requests](https://chryswoods.com/beginning_git/pull.html) - [Continuous Integration](https://chryswoods.com/beginning_git/ci.html) - [What next?](https://chryswoods.com/beginning_git/whatnext.html) - [JupyterHub and Kubernetes](https://chryswoods.com/inception_workshop/README.html) - [Creating the workshop](https://chryswoods.com/inception_workshop/course/part01.html) - [Finding all your dependencies](https://chryswoods.com/inception_workshop/course/part02.html) - [Building the docker image](https://chryswoods.com/inception_workshop/course/part03.html) - [JupyterHub](https://chryswoods.com/inception_workshop/course/part04.html) - [Uploading to the cloud](https://chryswoods.com/inception_workshop/course/part05.html) - [Kubernetes](https://chryswoods.com/inception_workshop/course/part06.html) - [Helm](https://chryswoods.com/inception_workshop/course/part07.html) - [Configuring JupyterHub](https://chryswoods.com/inception_workshop/course/part08.html) - [What next?](https://chryswoods.com/inception_workshop/course/whatnext.html) - [Introduction to Monte Carlo](https://chryswoods.com/intro_to_mc/README.html) - [Part 1](https://chryswoods.com/intro_to_mc/part1/README.html) - [Introduction to Monte Carlo](https://chryswoods.com/intro_to_mc/part1/intro.html) - [Software](https://chryswoods.com/intro_to_mc/part1/software.html) - [Metropolis Monte Carlo](https://chryswoods.com/intro_to_mc/part1/metropolis.html) - [Running metropolis.py](https://chryswoods.com/intro_to_mc/part1/running.html) - [Control Variables](https://chryswoods.com/intro_to_mc/part1/control.html) - [Phase Changes](https://chryswoods.com/intro_to_mc/part1/phase.html) - [Phase Space and Ensembles](https://chryswoods.com/intro_to_mc/part1/ensemble.html) - [Volume Moves](https://chryswoods.com/intro_to_mc/part1/volume.html) - [NPT Simulations](https://chryswoods.com/intro_to_mc/part1/npt.html) - [Summary](https://chryswoods.com/intro_to_mc/part1/summary.html) - [Part 2](https://chryswoods.com/intro_to_mc/part2/README.html) - [Introduction and Software](https://chryswoods.com/intro_to_mc/part2/intro.html) - [Sampling the Solvent - Rigid Body Moves](https://chryswoods.com/intro_to_mc/part2/rigid.html) - [Sampling the Ligand - Intramolecular Moves](https://chryswoods.com/intro_to_mc/part2/intra.html) - [Sampling the Protein - Backbone Moves](https://chryswoods.com/intro_to_mc/part2/backbone.html) - [Sampling it all - Weighting Moves](https://chryswoods.com/intro_to_mc/part2/weight.html) - [Summary](https://chryswoods.com/intro_to_mc/part2/whatnext.html) - [Molecular Visualisation, Modelling and Dynamics](https://chryswoods.com/dynamics/README.html) - [Part 1: Molecular Visualisation](https://chryswoods.com/dynamics/visualisation/README.html) - [1a: Opening Files](https://chryswoods.com/dynamics/visualisation/opening_files.html) - [1b: Manipulating the View](https://chryswoods.com/dynamics/visualisation/mouse.html) - [1c: Graphical Representations](https://chryswoods.com/dynamics/visualisation/representations.html) - [1d: Selecting Atoms](https://chryswoods.com/dynamics/visualisation/selection.html) - [1e: Complex Selections](https://chryswoods.com/dynamics/visualisation/complex_selection.html) - [1f: Rendering](https://chryswoods.com/dynamics/visualisation/rendering.html) - [1g: Movies](https://chryswoods.com/dynamics/visualisation/movies.html) - [1h: Picking Atoms](https://chryswoods.com/dynamics/visualisation/picking.html) - [1i: Comparing Trajectories](https://chryswoods.com/dynamics/visualisation/comparing.html) - [1j: What Next?](https://chryswoods.com/dynamics/visualisation/whatnext.html) - [Part 2: Molecular Dynamics](https://chryswoods.com/dynamics/dynamics/README.html) - [2a: Getting Started](https://chryswoods.com/dynamics/dynamics/getting_started.html) - [2b: Theory of MD](https://chryswoods.com/dynamics/dynamics/theory.html) - [2c: Changing Time](https://chryswoods.com/dynamics/dynamics/time.html) - [2d: Shake (Rattle and Roll)](https://chryswoods.com/dynamics/dynamics/shake.html) - [2e: Periodic Boundary Conditions](https://chryswoods.com/dynamics/dynamics/protein.html) - [2f: Under Pressure](https://chryswoods.com/dynamics/dynamics/pressure.html) - [2g: Running the Simulation](https://chryswoods.com/dynamics/dynamics/simulation.html) - [2h: What Next?](https://chryswoods.com/dynamics/dynamics/whatnext.html) - [Part 3: Mutation Studies](https://chryswoods.com/dynamics/mutation/README.html) - [3a: Getting Started](https://chryswoods.com/dynamics/mutation/gettingstarted.html) - [3b: Mutating the Protein](https://chryswoods.com/dynamics/mutation/mutation.html) - [3c: Solvating the Protein](https://chryswoods.com/dynamics/mutation/solvation.html) - [3d: Minimising the System](https://chryswoods.com/dynamics/mutation/minimisation.html) - [3e: Heating the System](https://chryswoods.com/dynamics/mutation/heating.html) - [3f: Equilibrating the System](https://chryswoods.com/dynamics/mutation/equilibration.html) - [3g: Running the Simulation](https://chryswoods.com/dynamics/mutation/simulation.html) - [3h: Comparing Trajectories](https://chryswoods.com/dynamics/mutation/compare.html) - [3i: What Next?](https://chryswoods.com/dynamics/mutation/whatnext.html) - [QM/MM Monte Carlo](https://chryswoods.com/embo2014/Practical.html) - [Software Carpentry](https://chryswoods.com/main/softwarecarpentry.html) - [Bath (July 2013) Workshop](http://tinyurl.com/swcbath) - [Bristol (September 2013) Workshop](http://tinyurl.com/swcbristol) - [Exeter (November 2013) Workshop](http://tinyurl.com/swcexeter) - [Software](https://chryswoods.com/main/software.html) - [Sire Molecular Simulation Framework](https://siremol.org/) - [ProtoMS Monte Carlo Package](http://protoms.org/) - [Publications](https://chryswoods.com/main/publications.html) - [Talks](https://chryswoods.com/talks/README.html) - [Useful Links](https://chryswoods.com/main/links.html) - [Books that are Worth Reading](https://chryswoods.com/main/reading.html) - [Contact Information](https://chryswoods.com/main/contact.html) - [About this Website](https://chryswoods.com/main/website.html) - [![](https://chryswoods.com/images/hamburger.png) chryswoods.com](https://chryswoods.com/parallel_python/async_map.html#main_menu) # Part 2: Asynchronous Mapping Asynchronous functions allow you to give different tasks to different members of the `multiprocessing.Pool`. However, giving functions one by one is not very efficient. It would be good to be able to combine mapping with asynchronous functions, i.e. be able to give different mapping tasks simultanously to the pool of workers. Fortunately, `Pool.map_async` provides exactly that - an asynchronous parallel map. Create a new python script called `asyncmap.py` and copy into it ``` from functools import reduce from multiprocessing import Pool, current_process import time def add(x, y): """Return the sum of the arguments""" print("Worker %s is processing add(%s, %s)" % (current_process().pid, x, y)) time.sleep(1) return x + y def product(x, y): """Return the product of the arguments""" print("Worker %s is processing product(%s, %s)" % (current_process().pid, x, y)) time.sleep(1) return x * y if __name__ == "__main__": a = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] b = [11, 12, 13, 14, 15, 16, 17, 18, 19, 20] # Now create a Pool of workers with Pool() as pool: sum_future = pool.starmap_async(add, zip(a,b)) product_future = pool.starmap_async(product, zip(a,b)) sum_future.wait() product_future.wait() total_sum = reduce(lambda x, y: x + y, sum_future.get()) total_product = reduce(lambda x, y: x + y, product_future.get()) print("Sum of sums of 'a' and 'b' is %s" % total_sum) print("Sum of products of 'a' and 'b' is %s" % total_product) ``` Running this script, e.g. via `python asyncmap.py` should result in something like ``` Worker 32722 is processing add(1, 11) Worker 32723 is processing add(2, 12) Worker 32724 is processing add(3, 13) Worker 32725 is processing add(4, 14) Worker 32722 is processing add(5, 15) Worker 32724 is processing add(6, 16) Worker 32725 is processing add(7, 17) Worker 32723 is processing add(8, 18) Worker 32722 is processing add(9, 19) Worker 32724 is processing add(10, 20) Worker 32725 is processing product(1, 11) Worker 32723 is processing product(2, 12) Worker 32722 is processing product(3, 13) Worker 32723 is processing product(4, 14) Worker 32724 is processing product(5, 15) Worker 32725 is processing product(6, 16) Worker 32722 is processing product(7, 17) Worker 32725 is processing product(8, 18) Worker 32723 is processing product(9, 19) Worker 32724 is processing product(10, 20) Sum of sums of 'a' and 'b' is 210 Sum of products of 'a' and 'b' is 935 ``` This script provides two functions, `add` and `product`, which are mapped asynchronously using the `Pool.map_async` function. This is identical to the `Pool.map` function that you used before, except now the map is performed asynchronously. This means that the resulting list is returned in a future (in this case, the futures `sum_future` and `product_future`. The results are waited for using the `.wait()` functions, remembering to make sure that we don’t exit the `with` block until all results are available. Then, the results of mapping are retrieved using the `.get()` function of the futures. ## Chunking By default, the `Pool.map` function divides the work over the pool of workers by assiging pieces of work one by one. In the example above, the work to be performed was; ``` add(1, 11) add(2, 12) add(3, 13) etc. add(10,20) product(1, 11) product(2, 12) product(3, 13) etc. product(10, 20) ``` The work was assigned one by one to the four workers on my computer, i.e. the first worker process was given `add(1, 11)`, the second `add(2, 12)`, the third `add(3, 13)` the then the fourth `add(4, 14)`. The first worker to finish was then given `add(5, 15)`, then the next given `add(6, 16)` etc. etc. Giving work one by one can be very inefficient for quick tasks, as the time needed by a worker process to stop and get new work can be longer than it takes to actually complete the task. To solve this problem, you can control how many work items are handed out to each worker process at a time. This is known as chunking, and the number of work items is known as the chunk of work to perform. You can control the number of work items to perform per worker (the chunk size) by setting the `chunksize` argument, e.g. ``` sum_future = pool.starmap_async(add, zip(a, b), chunksize=5) ``` would suggest to `pool` that each worker be given a chunk of five pieces of work. Note that this is just a suggestion, and `pool` may decide to use a slightly smaller or larger chunk size depending on the amount of work and the number of workers available. Modify your `asyncmap.py` script and set the `chunksize` to 5 for both of the asynchronous maps for `add` and `product`. Re-run your script. You should see something like; ``` Worker 658 is processing add(1, 11) Worker 659 is processing add(6, 16) Worker 660 is processing product(1, 11) Worker 661 is processing product(6, 16) Worker 659 is processing add(7, 17) Worker 660 is processing product(2, 12) Worker 661 is processing product(7, 17) Worker 658 is processing add(2, 12) Worker 660 is processing product(3, 13) Worker 659 is processing add(8, 18) Worker 661 is processing product(8, 18) Worker 658 is processing add(3, 13) Worker 660 is processing product(4, 14) Worker 659 is processing add(9, 19) Worker 661 is processing product(9, 19) Worker 658 is processing add(4, 14) Worker 659 is processing add(10, 20) Worker 660 is processing product(5, 15) Worker 661 is processing product(10, 20) Worker 658 is processing add(5, 15) Sum of sums of 'a' and 'b' is 210 Sum of products of 'a' and 'b' is 935 ``` My laptop has four workers. The first worker is assigned the first five items of work, i.e. `add(1, 11)` to `add(5, 15)`, and it starts by running `add(1, 11)`, hence why `add(1, 11)` is printed first. The next worker is given the next five items of work, i.e. `add(6, 16)` to `add(10,20)`, and starts by running `add(6, 16)`, hence why `add(6, 16)` is printed second. The next worker is given the next five items of work, i.e. `product(1, 11)` to `product(5, 15)`, and it starts by running `product(1, 11)`, hence why this is printed third. The last worker is given the next five items of work, i.e. `product(6, 16)` to `product(10, 20)`, and it starts by running `product(6, 16)`, hence why this is printed fourth. Once each worker has finished its first item of work, it moves onto its second. This is why `add(2, 12)`, `add(7, 17)`, `product(2, 12)` and `product(7, 17)` are printed next. Then, each worker moves onto its third piece of work etc. etc. If you don’t specify the `chunksize` then it is equal to `1`. When writing a new script you should experiment with different values of `chunksize` to find the value that gives best performance. *** ## Exercise Edit your script written in answer to [exercise 2 of Parallel Map/Reduce](https://chryswoods.com/parallel_python/mapreduce_part2.html), in which you count all of the words used in all Shakespeare plays (e.g. an example answer [is here](https://chryswoods.com/parallel_python/mapreduce2_answer2.html)). Edit the script so that you use an asynchronous map to distribute the work over the pool. This will free up the master process to give feedback to the user of the script, e.g. to print a progress or status message while the work is running to reassure the user that the script has not frozen. For example ``` while not future.ready(): print("Work is in progress...") time.sleep(0.1) ``` Add a status message to your script to reassure the user that your script hasn’t frozen while it is processing. (note that you can call your script using `python -u countwords.py shakespeare/` to use the `-u` argument to stop Python from buffering text written to standard output) If you get stuck or want inspiration, a possible answer is given [here](https://chryswoods.com/parallel_python/async_map_answer1.html). ** # [Previous](https://chryswoods.com/parallel_python/futures_part2.html) [Up](https://chryswoods.com/parallel_python/part2.html) [Next](https://chryswoods.com/parallel_python/part3.html) Quick Links \\| [Home](https://chryswoods.com/index.html) \\| [Courses](https://chryswoods.com/main/courses.html) \\| [Software](https://chryswoods.com/main/software.html) \\| [Contact](https://chryswoods.com/main/contact.html) [Copyright Information](https://chryswoods.com/main/copyright.html) \\| [Report a problem](<mailto:chryswoods@gmail.com?subject=There is a problem with your website&body=Hi, I found a problem on the page parallel_python/async_map.html>) I [Privacy](https://chryswoods.com/main/cookies.html)
Readable Markdown	null
Shard	133 (laksa)
Root Hash	1202965565032433533
Unparsed URL	com,chryswoods!/parallel_python/async_map.html s443