🕷️ Crawler Inspector

[Skip to main content](https://stackoverflow.com/questions/45603116/async-multiprocessing-python#content) 1. [About](https://stackoverflow.co/) 2. Products 3. [For Teams](https://stackoverflow.co/internal/) 4. Try new site Try BETA 1. [Stack Internal Implement a knowledge platform layer to power your enterprise and AI tools.](https://stackoverflow.co/internal/?utm_medium=referral&utm_source=stackoverflow-community&utm_campaign=top-nav&utm_content=stack-overflow-for-teams) 2. [Stack Data Licensing Get access to top-class technical expertise with trusted & attributed content.](https://stackoverflow.co/data-licensing/?utm_medium=referral&utm_source=stackoverflow-community&utm_campaign=top-nav&utm_content=overflow-api) 3. [Stack Ads Connect your brand to the world’s most trusted technologist communities.](https://stackoverflow.co/advertising/?utm_medium=referral&utm_source=stackoverflow-community&utm_campaign=top-nav&utm_content=stack-overflow-advertising) 4. [Releases Keep up-to-date on features we add to Stack Overflow and Stack Internal.](https://stackoverflow.blog/releases/?utm_medium=referral&utm_source=stackoverflow-community&utm_campaign=top-nav&utm_content=releases) 5. [About the company](https://stackoverflow.co/?utm_medium=referral&utm_source=stackoverflow-community&utm_campaign=top-nav&utm_content=about-the-company) [Visit the blog](https://stackoverflow.blog/?utm_medium=referral&utm_source=stackoverflow-community&utm_campaign=top-nav&utm_content=blog) 1. ### [current community](https://stackoverflow.com/) - [Stack Overflow](https://stackoverflow.com/) [help](https://stackoverflow.com/help) [chat](https://chat.stackoverflow.com/?tab=explore) - [Meta Stack Overflow](https://meta.stackoverflow.com/) ### your communities [Sign up](https://stackoverflow.com/users/signup?ssrc=site_switcher&returnurl=https%3A%2F%2Fstackoverflow.com%2Fquestions%2F45603116%2Fasync-multiprocessing-python) or [log in](https://stackoverflow.com/users/login?ssrc=site_switcher&returnurl=https%3A%2F%2Fstackoverflow.com%2Fquestions%2F45603116%2Fasync-multiprocessing-python) to customize your list. ### [more stack exchange communities](https://stackexchange.com/sites) [company blog](https://stackoverflow.blog/) 2. [Log in](https://stackoverflow.com/users/login?ssrc=head&returnurl=https%3A%2F%2Fstackoverflow.com%2Fquestions%2F45603116%2Fasync-multiprocessing-python) 3. [Sign up](https://stackoverflow.com/users/signup?ssrc=head&returnurl=https%3A%2F%2Fstackoverflow.com%2Fquestions%2F45603116%2Fasync-multiprocessing-python) 1. 1. [Home](https://stackoverflow.com/) 2. [Questions](https://stackoverflow.com/questions) 3. [AI Assist](https://stackoverflow.com/ai-assist) 4. [Tags](https://stackoverflow.com/tags) 5. [Challenges](https://stackoverflow.com/beta/challenges) 6. [Chat](https://chat.stackoverflow.com/?tab=explore) 7. [Articles](https://stackoverflow.blog/contributed?utm_medium=referral&utm_source=stackoverflow-community&utm_campaign=so-blog&utm_content=experiment-articles) 8. [Users](https://stackoverflow.com/users) 9. [Companies](https://stackoverflow.com/jobs/companies?so_medium=stackoverflow&so_source=SiteNav) 10. [Collectives]() 11. Communities for your favorite technologies. [Explore all Collectives](https://stackoverflow.com/collectives-all) 2. Stack Internal Stack Overflow for Teams is now called **Stack Internal**. Bring the best of human thought and AI automation together at your work. [Try for free](https://stackoverflowteams.com/teams/create/free/?utm_medium=referral&utm_source=stackoverflow-community&utm_campaign=side-bar&utm_content=explore-teams) [Learn more](https://stackoverflow.co/internal/?utm_medium=referral&utm_source=stackoverflow-community&utm_campaign=side-bar&utm_content=explore-teams) 3. [Stack Internal]() 4. Bring the best of human thought and AI automation together at your work. [Learn more](https://stackoverflow.co/internal/?utm_medium=referral&utm_source=stackoverflow-community&utm_campaign=side-bar&utm_content=explore-teams-compact) ##### Collectives™ on Stack Overflow Find centralized, trusted content and collaborate around the technologies you use most. [Learn more about Collectives](https://stackoverflow.com/collectives) **Stack Internal** Knowledge at work Bring the best of human thought and AI automation together at your work. [Explore Stack Internal](https://stackoverflow.co/internal/?utm_medium=referral&utm_source=stackoverflow-community&utm_campaign=side-bar&utm_content=explore-teams-compact-popover) # [Async multiprocessing python](https://stackoverflow.com/questions/45603116/async-multiprocessing-python) [Ask Question](https://stackoverflow.com/questions/ask) Asked 8 years, 7 months ago Modified [8 years, 7 months ago](https://stackoverflow.com/questions/45603116/async-multiprocessing-python?lastactivity "2017-08-10 13:28:09Z") Viewed 25k times 13 So I've read this nice article about asynch threads in python. Tough, the last one have some troubles with the [GIL](http://www.dabeaz.com/GIL/) and threads are not as effective as it may seems. Luckily python incorporates Multiprocessing which are designed to be not affected by this trouble. I'd like to understand how to implement a multiprocessing queue (with Pipe open for each process) in an async manner so it wouldn't hang a running [async webserver](http://aiohttp.readthedocs.io/en/stable/) . I've read this [topic](https://stackoverflow.com/questions/32955846/in-python-is-there-an-async-equivalent-to-multiprocessing-or-concurrent-futures) however I'm not looking for performance but rather boxing out a big calculation that hangs my webserver. Those calculations require pictures so they might have a significant i/o exchange but in my understanding this is something that is pretty well handled by async. All the calcs are separate from each other so they are not meant to be mixed. I'm trying to build this in front of a ws handler. If you hint heresy in this please let me know as well :) - [python](https://stackoverflow.com/questions/tagged/python "show questions tagged 'python'") - [multithreading](https://stackoverflow.com/questions/tagged/multithreading "show questions tagged 'multithreading'") - [asynchronous](https://stackoverflow.com/questions/tagged/asynchronous "show questions tagged 'asynchronous'") [Share](https://stackoverflow.com/q/45603116 "Short permalink to this question") [Improve this question](https://stackoverflow.com/posts/45603116/edit) Follow asked Aug 10, 2017 at 1:42 [](https://stackoverflow.com/users/8208594/pawy) [Pawy](https://stackoverflow.com/users/8208594/pawy) 27911 gold badge33 silver badges66 bronze badges 4 - Is there a reason why you wouldn't be interested in python hreading module ([docs.python.org/3/library/threading.html](https://docs.python.org/3/library/threading.html))? Alceste\_ – [Alceste\_](https://stackoverflow.com/users/8155816/alceste "624 reputation") 2017-08-10 02:03:05 +00:00 Commented Aug 10, 2017 at 2:03 - 1 Yes, CPU threads have a tendency to lock the GIL which makes the hole infra slower. edit : explained here [dabeaz.blogspot.fr/2010/02/…](http://dabeaz.blogspot.fr/2010/02/revisiting-thread-priorities-and-new.html) Pawy – [Pawy](https://stackoverflow.com/users/8208594/pawy "279 reputation") 2017-08-10 09:07:59 +00:00 Commented Aug 10, 2017 at 9:07 - Someone nice on \#python gave me hints about async executor ; After some research it seems the full answer is here [pythonadventures.wordpress.com/tag/processpoolexecutor](https://pythonadventures.wordpress.com/tag/processpoolexecutor/) Pawy – [Pawy](https://stackoverflow.com/users/8208594/pawy "279 reputation") 2017-08-10 12:05:44 +00:00 Commented Aug 10, 2017 at 12:05 - You can make it an answer and accept it so your thread get solved. ;) Alceste\_ – [Alceste\_](https://stackoverflow.com/users/8155816/alceste "624 reputation") 2017-08-10 12:09:28 +00:00 Commented Aug 10, 2017 at 12:09 [Add a comment](https://stackoverflow.com/questions/45603116/async-multiprocessing-python "Use comments to ask for more information or suggest improvements. Avoid answering questions in comments.") \| ## 1 Answer 1 Sorted by: [Reset to default](https://stackoverflow.com/questions/45603116/async-multiprocessing-python?answertab=scoredesc#tab-top) 14 *This is re-sourced from a article after someone nice on \#python irc hinted me on async executors, and another answer on reddit :* (2) Using ProcessPoolExecutor “The ProcessPoolExecutor class is an Executor subclass that uses a pool of processes to execute calls asynchronously. ProcessPoolExecutor uses the multiprocessing module, which allows it to side-step the Global Interpreter Lock but also means that only picklable objects can be executed and returned.” ``` import asyncio from concurrent.futures import ProcessPoolExecutor def cpu_heavy(num): print('entering cpu_heavy', num) import time time.sleep(10) print('leaving cpu_heavy', num) return num async def main(loop): print('entering main') executor = ProcessPoolExecutor(max_workers=3) data = await asyncio.gather(*(loop.run_in_executor(executor, cpu_heavy, num) for num in range(3))) print('got result', data) print('leaving main') loop = asyncio.get_event_loop() loop.run_until_complete(main(loop)) ``` And this from another nice guy on reddit ;) [Share](https://stackoverflow.com/a/45615121 "Short permalink to this answer") [Improve this answer](https://stackoverflow.com/posts/45615121/edit) Follow answered Aug 10, 2017 at 13:28 [](https://stackoverflow.com/users/8208594/pawy) [Pawy](https://stackoverflow.com/users/8208594/pawy) 27911 gold badge33 silver badges66 bronze badges Sign up to request clarification or add additional context in comments. ## 3 Comments Add a comment [](https://stackoverflow.com/users/8208594/pawy) Pawy [Pawy](https://stackoverflow.com/users/8208594/pawy) [Over a year ago](https://stackoverflow.com/questions/45603116/async-multiprocessing-python#comment78189367_45615121) I also wanted to point to this question which might be better formulated and contains another "version" of the answer. [stackoverflow.com/questions/27290656/…](https://stackoverflow.com/questions/27290656/should-i-use-two-asyncio-event-loops-in-one-program/27298880#27298880 "should i use two asyncio event loops in one program") ; Difference being in one spawning spawning only one process at the time whereas another spawns bunch of them. So depending on yours needs you might want to use run\_in\_executor alone, or if you want to split out a calc, I guess you'll be using asyncio.gather as well 2017-08-10T13:37:42.48Z+00:00 1 Reply - Copy link [](https://stackoverflow.com/users/5860340/an-se) An Se [An Se](https://stackoverflow.com/users/5860340/an-se) [Over a year ago](https://stackoverflow.com/questions/45603116/async-multiprocessing-python#comment115456512_45615121) g e n i u s. simply genius. 2020-12-15T14:09:14.59Z+00:00 0 Reply - Copy link [](https://stackoverflow.com/users/1794230/%EA%B9%80%EB%AF%BC%EC%A4%80) 김민준 [김민준](https://stackoverflow.com/users/1794230/%EA%B9%80%EB%AF%BC%EC%A4%80) [Over a year ago](https://stackoverflow.com/questions/45603116/async-multiprocessing-python#comment121390711_45615121) Note that you'll need to run last two lines in `if __name__ == '__main__':` clause or your spawned processes will also try to start processes, causing `concurrent.futures.process.BrokenProcessPool` error. 2021-08-07T01:14:59.873Z+00:00 0 Reply - Copy link Add a comment Start asking to get answers Find the answer to your question by asking. [Ask question](https://stackoverflow.com/questions/ask) Explore related questions - [python](https://stackoverflow.com/questions/tagged/python "show questions tagged 'python'") - [multithreading](https://stackoverflow.com/questions/tagged/multithreading "show questions tagged 'multithreading'") - [asynchronous](https://stackoverflow.com/questions/tagged/asynchronous "show questions tagged 'asynchronous'") See similar questions with these tags. - The Overflow Blog - [Domain expertise still wanted: the latest trends in AI-assisted knowledge for...](https://stackoverflow.blog/2026/03/16/domain-expertise-still-wanted-the-latest-trends-in-ai/ "Domain expertise still wanted: the latest trends in AI-assisted knowledge for developers") - [Keeping the lights on for open source](https://stackoverflow.blog/2026/03/17/keeping-the-lights-on-for-open-source/) - Featured on Meta - [Logo updates to Stack Overflow's visual identity](https://meta.stackexchange.com/questions/417394/logo-updates-to-stack-overflows-visual-identity) - [Policy: Generative AI (e.g., ChatGPT) is banned](https://meta.stackoverflow.com/questions/421831/policy-generative-ai-e-g-chatgpt-is-banned) - [Release notes and bug fixes for beta.stackoverflow.com](https://meta.stackoverflow.com/questions/438499/release-notes-and-bug-fixes-for-beta-stackoverflow-com) - [I’m Jody, the Chief Product and Technology Officer at Stack Overflow. Let’s...](https://meta.stackoverflow.com/questions/438369/i-m-jody-the-chief-product-and-technology-officer-at-stack-overflow-let-s-talk "I’m Jody, the Chief Product and Technology Officer at Stack Overflow. Let’s talk about the site redesign") #### Linked [34](https://stackoverflow.com/questions/32955846/in-python-is-there-an-async-equivalent-to-multiprocessing-or-concurrent-futures "Question score (upvotes - downvotes)") [In Python, is there an async equivalent to multiprocessing or concurrent.futures?](https://stackoverflow.com/questions/32955846/in-python-is-there-an-async-equivalent-to-multiprocessing-or-concurrent-futures?noredirect=1) [13](https://stackoverflow.com/questions/27290656/should-i-use-two-asyncio-event-loops-in-one-program "Question score (upvotes - downvotes)") [Should I use two asyncio event loops in one program?](https://stackoverflow.com/questions/27290656/should-i-use-two-asyncio-event-loops-in-one-program?noredirect=1) [1](https://stackoverflow.com/questions/68150129/how-to-nest-a-function-that-responds-to-outside-of-function-call "Question score (upvotes - downvotes)") [How to nest a function that responds to outside of function call?](https://stackoverflow.com/questions/68150129/how-to-nest-a-function-that-responds-to-outside-of-function-call?noredirect=1) #### Related [1](https://stackoverflow.com/questions/41258775/python-multiprocessing-with-shared-variables-and-asynchronous "Question score (upvotes - downvotes)") [python multiprocessing with shared variables and asynchronous?](https://stackoverflow.com/questions/41258775/python-multiprocessing-with-shared-variables-and-asynchronous) [2](https://stackoverflow.com/questions/43408235/multi-task-at-once-in-python "Question score (upvotes - downvotes)") [multi task at once in python](https://stackoverflow.com/questions/43408235/multi-task-at-once-in-python) [2](https://stackoverflow.com/questions/47951760/how-to-run-multiple-asynchronous-processes-in-python-using-multiprocessing "Question score (upvotes - downvotes)") [How to run multiple asynchronous processes in Python using multiprocessing?](https://stackoverflow.com/questions/47951760/how-to-run-multiple-asynchronous-processes-in-python-using-multiprocessing) [0](https://stackoverflow.com/questions/49042917/run-multiple-tasks-asynchronously-in-python "Question score (upvotes - downvotes)") [Run Multiple Tasks Asynchronously in Python](https://stackoverflow.com/questions/49042917/run-multiple-tasks-asynchronously-in-python) [0](https://stackoverflow.com/questions/64943709/needs-assistance-with-async-or-multithreading-task "Question score (upvotes - downvotes)") [Needs assistance with Async or Multithreading task](https://stackoverflow.com/questions/64943709/needs-assistance-with-async-or-multithreading-task) [1](https://stackoverflow.com/questions/66469289/python-multithreading-how-to-run-multiple-async-threads-in-a-loop "Question score (upvotes - downvotes)") [Python multithreading: How to run multiple async threads in a loop](https://stackoverflow.com/questions/66469289/python-multithreading-how-to-run-multiple-async-threads-in-a-loop) [0](https://stackoverflow.com/questions/67754516/python-thread-running-asynchronously "Question score (upvotes - downvotes)") [Python Thread running asynchronously](https://stackoverflow.com/questions/67754516/python-thread-running-asynchronously) [0](https://stackoverflow.com/questions/67840850/asynchronous-threading-problem-in-python "Question score (upvotes - downvotes)") [Asynchronous, Threading problem in python](https://stackoverflow.com/questions/67840850/asynchronous-threading-problem-in-python) [0](https://stackoverflow.com/questions/70757290/how-to-write-a-simple-python-multi-thread-task "Question score (upvotes - downvotes)") [how to write a simple Python Multi-thread task](https://stackoverflow.com/questions/70757290/how-to-write-a-simple-python-multi-thread-task) [0](https://stackoverflow.com/questions/74179655/running-multiple-processses-asynchronously "Question score (upvotes - downvotes)") [Running multiple processses asynchronously](https://stackoverflow.com/questions/74179655/running-multiple-processses-asynchronously) #### [Hot Network Questions](https://stackexchange.com/questions?tab=hot) - [Decomposing finite groups into unions of transversal normal subgroups](https://mathoverflow.net/questions/509210/decomposing-finite-groups-into-unions-of-transversal-normal-subgroups) - [Is there any way to find the file offset of the corresponding buffer position?](https://emacs.stackexchange.com/questions/85601/is-there-any-way-to-find-the-file-offset-of-the-corresponding-buffer-position) - [What is the history of name changes for D\&D 5.5e / D\&D 2024?](https://rpg.stackexchange.com/questions/218950/what-is-the-history-of-name-changes-for-dd-5-5e-dd-2024) - [How does this FM modulated 123 Hz oscillator work?](https://electronics.stackexchange.com/questions/767052/how-does-this-fm-modulated-123-hz-oscillator-work) - [Can we respond to "way bigger" with "How way bigger?"](https://english.stackexchange.com/questions/639246/can-we-respond-to-way-bigger-with-how-way-bigger) - [PubChem Service no longer supports a list of CompoundID specifications](https://mathematica.stackexchange.com/questions/319030/pubchem-service-no-longer-supports-a-list-of-compoundid-specifications) - [Simple Boost::Asio Server and Receiver in one class](https://codereview.stackexchange.com/questions/301626/simple-boostasio-server-and-receiver-in-one-class) - [PI on medical leave: ethics and etiquette](https://academia.stackexchange.com/questions/226203/pi-on-medical-leave-ethics-and-etiquette) - [Is any countably infinite set meager?](https://math.stackexchange.com/questions/5129101/is-any-countably-infinite-set-meager) - [Rulers, bots, cards, spots. Which country am I from?](https://puzzling.stackexchange.com/questions/137440/rulers-bots-cards-spots-which-country-am-i-from) - [Left and right Kan extensions of a presheaf on open subsets of Cartesian spaces along the canonical inclusion into the category of manifolds](https://math.stackexchange.com/questions/5129129/left-and-right-kan-extensions-of-a-presheaf-on-open-subsets-of-cartesian-spaces) - [plain TeX to LaTeX conversion](https://tex.stackexchange.com/questions/760993/plain-tex-to-latex-conversion) - [Would like to install a new dimmer light switch to replace really old rotary style dimmer switch](https://diy.stackexchange.com/questions/329638/would-like-to-install-a-new-dimmer-light-switch-to-replace-really-old-rotary-sty) - [Do I top up the same amount of new engine oil as much as I drained the old one?](https://mechanics.stackexchange.com/questions/102007/do-i-top-up-the-same-amount-of-new-engine-oil-as-much-as-i-drained-the-old-one) - [When was Krypton first depicted as a rigidly stratified caste society?](https://scifi.stackexchange.com/questions/303714/when-was-krypton-first-depicted-as-a-rigidly-stratified-caste-society) - [Is the modal logic totally reducible to the first order logic/propositional algebra?](https://philosophy.stackexchange.com/questions/137128/is-the-modal-logic-totally-reducible-to-the-first-order-logic-propositional-alge) - [Multiligual accessible PDF - Language not recognized by screen readers](https://tex.stackexchange.com/questions/760997/multiligual-accessible-pdf-language-not-recognized-by-screen-readers) - [BCsort: A fast, statistical, in-place ternary distribution sort](https://codereview.stackexchange.com/questions/301618/bcsort-a-fast-statistical-in-place-ternary-distribution-sort) - [/usr/lib/x86\_64-linux-gnu/gimp/3.0/plug-ins/file-jpeg/file-jpeg: fatal error: Segmentation fault](https://askubuntu.com/questions/1564938/usr-lib-x86-64-linux-gnu-gimp-3-0-plug-ins-file-jpeg-file-jpeg-fatal-error-se) - [Is there currently a way to use custom lists in slides with ltx-talk class?](https://tex.stackexchange.com/questions/761002/is-there-currently-a-way-to-use-custom-lists-in-slides-with-ltx-talk-class) - [The assumptions for causal inference seem obvious - why state them?](https://stats.stackexchange.com/questions/675196/the-assumptions-for-causal-inference-seem-obvious-why-state-them) - [Different staves of the same measure have different time signatures](https://music.stackexchange.com/questions/143374/different-staves-of-the-same-measure-have-different-time-signatures) - [Choosing an approach to collaborative inter-process resource allocation](https://softwareengineering.stackexchange.com/questions/460993/choosing-an-approach-to-collaborative-inter-process-resource-allocation) - [The Designation of 'Fundamental': Historical Origins of 'Fundamental Theorems' in Mathematics](https://hsm.stackexchange.com/questions/19296/the-designation-of-fundamental-historical-origins-of-fundamental-theorems-i) [more hot questions](https://stackoverflow.com/questions/45603116/async-multiprocessing-python) [Question feed](https://stackoverflow.com/feeds/question/45603116 "Feed of this question and its answers") # Subscribe to RSS Question feed To subscribe to this RSS feed, copy and paste this URL into your RSS reader.  lang-py ##### [Stack Overflow](https://stackoverflow.com/) - [Questions](https://stackoverflow.com/questions) - [Help](https://stackoverflow.com/help) - [Chat](https://chat.stackoverflow.com/?tab=explore) ##### [Business](https://stackoverflow.co/) - [Stack Internal](https://stackoverflow.co/internal/?utm_medium=referral&utm_source=stackoverflow-community&utm_campaign=footer&utm_content=teams) - [Stack Data Licensing](https://stackoverflow.co/data-licensing/?utm_medium=referral&utm_source=stackoverflow-community&utm_campaign=footer&utm_content=data-licensing) - [Stack Ads](https://stackoverflow.co/advertising/?utm_medium=referral&utm_source=stackoverflow-community&utm_campaign=footer&utm_content=advertising) ##### [Company](https://stackoverflow.co/) - [About](https://stackoverflow.co/) - [Press](https://stackoverflow.co/company/press/) - [Work Here](https://stackoverflow.co/company/work-here/) - [Legal](https://stackoverflow.com/legal) - [Privacy Policy](https://stackoverflow.com/legal/privacy-policy) - [Terms of Service](https://stackoverflow.com/legal/terms-of-service/public) - [Contact Us](https://stackoverflow.com/contact) - Cookie Settings - [Cookie Policy](https://policies.stackoverflow.co/stack-overflow/cookie-policy) ##### [Stack Exchange Network](https://stackexchange.com/) - [Technology](https://stackexchange.com/sites#technology) - [Culture & recreation](https://stackexchange.com/sites#culturerecreation) - [Life & arts](https://stackexchange.com/sites#lifearts) - [Science](https://stackexchange.com/sites#science) - [Professional](https://stackexchange.com/sites#professional) - [Business](https://stackexchange.com/sites#business) - [API](https://api.stackexchange.com/) - [Data](https://data.stackexchange.com/) - [Blog](https://stackoverflow.blog/?blb=1) - [Facebook](https://www.facebook.com/officialstackoverflow/) - [Twitter](https://twitter.com/stackoverflow) - [LinkedIn](https://linkedin.com/company/stack-overflow) - [Instagram](https://www.instagram.com/thestackoverflow) Site design / logo © 2026 Stack Exchange Inc; user contributions licensed under [CC BY-SA](https://stackoverflow.com/help/licensing) . rev 2026.3.18.41202

13 So I've read this nice article about asynch threads in python. Tough, the last one have some troubles with the [GIL](http://www.dabeaz.com/GIL/) and threads are not as effective as it may seems. Luckily python incorporates Multiprocessing which are designed to be not affected by this trouble. I'd like to understand how to implement a multiprocessing queue (with Pipe open for each process) in an async manner so it wouldn't hang a running [async webserver](http://aiohttp.readthedocs.io/en/stable/) . I've read this [topic](https://stackoverflow.com/questions/32955846/in-python-is-there-an-async-equivalent-to-multiprocessing-or-concurrent-futures) however I'm not looking for performance but rather boxing out a big calculation that hangs my webserver. Those calculations require pictures so they might have a significant i/o exchange but in my understanding this is something that is pretty well handled by async. All the calcs are separate from each other so they are not meant to be mixed. I'm trying to build this in front of a ws handler. If you hint heresy in this please let me know as well :) asked Aug 10, 2017 at 1:42 [](https://stackoverflow.com/users/8208594/pawy) 4 14 *This is re-sourced from a article after someone nice on \#python irc hinted me on async executors, and another answer on reddit :* (2) Using ProcessPoolExecutor “The ProcessPoolExecutor class is an Executor subclass that uses a pool of processes to execute calls asynchronously. ProcessPoolExecutor uses the multiprocessing module, which allows it to side-step the Global Interpreter Lock but also means that only picklable objects can be executed and returned.” ``` import asyncio from concurrent.futures import ProcessPoolExecutor def cpu_heavy(num): print('entering cpu_heavy', num) import time time.sleep(10) print('leaving cpu_heavy', num) return num async def main(loop): print('entering main') executor = ProcessPoolExecutor(max_workers=3) data = await asyncio.gather(*(loop.run_in_executor(executor, cpu_heavy, num) for num in range(3))) print('got result', data) print('leaving main') loop = asyncio.get_event_loop() loop.run_until_complete(main(loop)) ``` And this from another nice guy on reddit ;) answered Aug 10, 2017 at 13:28 [](https://stackoverflow.com/users/8208594/pawy) 3 Comments [](https://stackoverflow.com/users/8208594/pawy) I also wanted to point to this question which might be better formulated and contains another "version" of the answer. [stackoverflow.com/questions/27290656/…](https://stackoverflow.com/questions/27290656/should-i-use-two-asyncio-event-loops-in-one-program/27298880#27298880 "should i use two asyncio event loops in one program") ; Difference being in one spawning spawning only one process at the time whereas another spawns bunch of them. So depending on yours needs you might want to use run\_in\_executor alone, or if you want to split out a calc, I guess you'll be using asyncio.gather as well 2017-08-10T13:37:42.48Z+00:00 [](https://stackoverflow.com/users/5860340/an-se) g e n i u s. simply genius. 2020-12-15T14:09:14.59Z+00:00 [](https://stackoverflow.com/users/1794230/%EA%B9%80%EB%AF%BC%EC%A4%80) Note that you'll need to run last two lines in `if __name__ == '__main__':` clause or your spawned processes will also try to start processes, causing `concurrent.futures.process.BrokenProcessPool` error. 2021-08-07T01:14:59.873Z+00:00 Start asking to get answers Find the answer to your question by asking. [Ask question](https://stackoverflow.com/questions/ask) Explore related questions See similar questions with these tags.