🕷️ Crawler Inspector

URL Lookup

Direct Parameter Lookup

Raw Queries and Responses

1. Shard Calculation

Query:

Response:

Calculated Shard: 174 (from laksa134)

2. Crawled Status Check

Query:

curl -X POST \
  'http://laksa174.int.ahrefs:8124/' \
  -H 'Content-Type: text/plain' \
  -H 'X-ClickHouse-Database: crawler3' \
  -H 'Authorization: Basic YXBpOg==' \
  -d 'SELECT getAhrefsURLFromUnparsed(src_unparsed) AS found_url, ifNull(toUnixTimestamp(download_stamp), 0) AS crawl_time, ifNull(toUnixTimestamp(props_url_first_seen), 0) AS first_indexed_time, download_http_code AS http_code, src_unparsed AS src_unparsed, src_root_hash AS src_root_hash, history_drop_reason AS history_drop_reason, meta_title AS meta_title, meta_descriptions AS meta_descriptions, attrs_boilerpipe_text AS attrs_boilerpipe_text, attrs_markdown AS attrs_markdown, attrs_readable_markdown AS attrs_readable_markdown, meta_canonical AS meta_canonical FROM crawler3.page_info_local FINAL PREWHERE (src_root_hash, src_unparsed) IN ((getAhrefsRootHashFromUnparsed(getAhrefsUnparsedNoserviceFromURL(\'https://github.com/dano/aioprocessing\')), getAhrefsUnparsedNoserviceFromURL(\'https://github.com/dano/aioprocessing\'))) FORMAT JSONEachRow'

Response:

{"found_url":"https:\/\/github.com\/dano\/aioprocessing","crawl_time":1772714087,"first_indexed_time":1414269914,"http_code":200,"src_unparsed":"com,github!\/dano\/aioprocessing s443","src_root_hash":"6325672905007345774","history_drop_reason":null,"meta_title":"GitHub - dano\/aioprocessing: A Python 3.5+ library that integrates the multiprocessing module with asyncio · GitHub","meta_descriptions":["A Python 3.5+ library that integrates the multiprocessing module with asyncio - dano\/aioprocessing"],"attrs_boilerpipe_text":"aioprocessing\nprovides asynchronous,\nasyncio\ncompatible, coroutine\nversions of many blocking instance methods on objects in the\nmultiprocessing\nlibrary. To use\ndill\nfor universal pickling, install using\npip install aioprocessing[dill]\n. Here's an example demonstrating the\naioprocessing\nversions of\nEvent\n,\nQueue\n, and\nLock\n:\nimport\ntime\nimport\nasyncio\nimport\naioprocessing\ndef\nfunc\n(\nqueue\n,\nevent\n,\nlock\n,\nitems\n):\n\"\"\" Demo worker function.\nThis worker function runs in its own process, and uses\nnormal blocking calls to aioprocessing objects, exactly\nthe way you would use oridinary multiprocessing objects.\n\"\"\"\nwith\nlock\n:\nevent\n.\nset\n()\nfor\nitem\nin\nitems\n:\ntime\n.\nsleep\n(\n3\n)\nqueue\n.\nput\n(\nitem\n+\n5\n)\nqueue\n.\nclose\n()\nasync\ndef\nexample\n(\nqueue\n,\nevent\n,\nlock\n):\nl\n=\n[\n1\n,\n2\n,\n3\n,\n4\n,\n5\n]\np\n=\naioprocessing\n.\nAioProcess\n(\ntarget\n=\nfunc\n,\nargs\n=\n(\nqueue\n,\nevent\n,\nlock\n,\nl\n))\np\n.\nstart\n()\nwhile\nTrue\n:\nresult\n=\nawait\nqueue\n.\ncoro_get\n()\nif\nresult\nis\nNone\n:\nbreak\nprint\n(\n\"Got result {}\"\n.\nformat\n(\nresult\n))\nawait\np\n.\ncoro_join\n()\nasync\ndef\nexample2\n(\nqueue\n,\nevent\n,\nlock\n):\nawait\nevent\n.\ncoro_wait\n()\nasync\nwith\nlock\n:\nawait\nqueue\n.\ncoro_put\n(\n78\n)\nawait\nqueue\n.\ncoro_put\n(\nNone\n)\n# Shut down the worker\nif\n__name__\n==\n\"__main__\"\n:\nloop\n=\nasyncio\n.\nget_event_loop\n()\nqueue\n=\naioprocessing\n.\nAioQueue\n()\nlock\n=\naioprocessing\n.\nAioLock\n()\nevent\n=\naioprocessing\n.\nAioEvent\n()\ntasks\n=\n[\nasyncio\n.\nensure_future\n(\nexample\n(\nqueue\n,\nevent\n,\nlock\n)),\nasyncio\n.\nensure_future\n(\nexample2\n(\nqueue\n,\nevent\n,\nlock\n)),\n    ]\nloop\n.\nrun_until_complete\n(\nasyncio\n.\nwait\n(\ntasks\n))\nloop\n.\nclose\n()\nThe aioprocessing objects can be used just like their multiprocessing\nequivalents - as they are in\nfunc\nabove - but they can also be\nseamlessly used inside of\nasyncio\ncoroutines, without ever blocking\nthe event loop.\nWhat's new\nv2.0.1\nFixed a bug that kept the\nAioBarrier\nand\nAioEvent\nproxies returned from\nAioManager\ninstances from working. Thanks to Giorgos Apostolopoulos for the fix.\nv2.0.0\nAdd support for universal pickling using\ndill\n, installable with\npip install aioprocessing[dill]\n. The library will now attempt to import\nmultiprocess\n, falling back to stdlib\nmultiprocessing\n. Force stdlib behaviour by setting a non-empty environment variable\nAIOPROCESSING_DILL_DISABLED=1\n. This can be used to avoid\nerrors\nwhen attempting to combine\naioprocessing[dill]\nwith stdlib\nmultiprocessing\nbased objects like\nconcurrent.futures.ProcessPoolExecutor\n.\nHow does it work?\nIn most cases, this library makes blocking calls to\nmultiprocessing\nmethods\nasynchronous by executing the call in a\nThreadPoolExecutor\n, using\nasyncio.run_in_executor()\n.\nIt does\nnot\nre-implement multiprocessing using asynchronous I\/O. This means\nthere is extra overhead added when you use\naioprocessing\nobjects instead of\nmultiprocessing\nobjects, because each one is generally introducing a\nThreadPoolExecutor\ncontaining at least one\nthreading.Thread\n. It also means\nthat all the normal risks you get when you mix threads with fork apply here, too\n(See\nhttp:\/\/bugs.python.org\/issue6721\nfor more info).\nThe one exception to this is\naioprocessing.AioPool\n, which makes use of the\nexisting\ncallback\nand\nerror_callback\nkeyword arguments in the various\nPool.*_async\nmethods to run them as\nasyncio\ncoroutines. Note that\nmultiprocessing.Pool\nis actually using threads internally, so the thread\/fork\nmixing caveat still applies.\nEach\nmultiprocessing\nclass is replaced by an equivalent\naioprocessing\nclass,\ndistinguished by the\nAio\nprefix. So,\nPool\nbecomes\nAioPool\n, etc. All methods\nthat could block on I\/O also have a coroutine version that can be used with\nasyncio\n. For example,\nmultiprocessing.Lock.acquire()\ncan be replaced with\naioprocessing.AioLock.coro_acquire()\n. You can pass an\nasyncio\nEventLoop object to any\ncoro_*\nmethod using the\nloop\nkeyword argument. For example,\nlock.coro_acquire(loop=my_loop)\n.\nNote that you can also use the\naioprocessing\nsynchronization primitives as replacements\nfor their equivalent\nthreading\nprimitives, in single-process, multi-threaded programs\nthat use\nasyncio\n.\nWhat parts of multiprocessing are supported?\nMost of them! All methods that could do blocking I\/O in the following objects\nhave equivalent versions in\naioprocessing\nthat extend the\nmultiprocessing\nversions by adding coroutine versions of all the blocking methods.\nPool\nProcess\nPipe\nLock\nRLock\nSemaphore\nBoundedSemaphore\nEvent\nCondition\nBarrier\nconnection.Connection\nconnection.Listener\nconnection.Client\nQueue\nJoinableQueue\nSimpleQueue\nAll\nmanagers.SyncManager\nProxy\nversions of the items above (\nSyncManager.Queue\n,\nSyncManager.Lock()\n, etc.).\nWhat versions of Python are compatible?\naioprocessing\nwill work out of the box on Python 3.5+.\nGotchas\nKeep in mind that, while the API exposes coroutines for interacting with\nmultiprocessing\nAPIs, internally they are almost always being delegated\nto a\nThreadPoolExecutor\n, this means the caveats that apply with using\nThreadPoolExecutor\nwith\nasyncio\napply: namely, you won't be able to\ncancel any of the coroutines, because the work being done in the worker\nthread can't be interrupted.","attrs_markdown":"[Skip to content](https:\/\/github.com\/dano\/aioprocessing#start-of-content)\n\n## Navigation Menu\nToggle navigation\n\n[Sign in](https:\/\/github.com\/login?return_to=https%3A%2F%2Fgithub.com%2Fdano%2Faioprocessing)\n\nAppearance settings\n\n- Platform\n  \n  - AI CODE CREATION\n    \n    - [GitHub CopilotWrite better code with AI](https:\/\/github.com\/features\/copilot)\n    - [GitHub SparkBuild and deploy intelligent apps](https:\/\/github.com\/features\/spark)\n    - [GitHub ModelsManage and compare prompts](https:\/\/github.com\/features\/models)\n    - [MCP RegistryNewIntegrate external tools](https:\/\/github.com\/mcp)\n  - DEVELOPER WORKFLOWS\n    \n    - [ActionsAutomate any workflow](https:\/\/github.com\/features\/actions)\n    - [CodespacesInstant dev environments](https:\/\/github.com\/features\/codespaces)\n    - [IssuesPlan and track work](https:\/\/github.com\/features\/issues)\n    - [Code ReviewManage code changes](https:\/\/github.com\/features\/code-review)\n  - APPLICATION SECURITY\n    \n    - [GitHub Advanced SecurityFind and fix vulnerabilities](https:\/\/github.com\/security\/advanced-security)\n    - [Code securitySecure your code as you build](https:\/\/github.com\/security\/advanced-security\/code-security)\n    - [Secret protectionStop leaks before they start](https:\/\/github.com\/security\/advanced-security\/secret-protection)\n  - EXPLORE\n    \n    - [Why GitHub](https:\/\/github.com\/why-github)\n    - [Documentation](https:\/\/docs.github.com\/)\n    - [Blog](https:\/\/github.blog\/)\n    - [Changelog](https:\/\/github.blog\/changelog)\n    - [Marketplace](https:\/\/github.com\/marketplace)\n  \n  [View all features](https:\/\/github.com\/features)\n- Solutions\n  \n  - BY COMPANY SIZE\n    \n    - [Enterprises](https:\/\/github.com\/enterprise)\n    - [Small and medium teams](https:\/\/github.com\/team)\n    - [Startups](https:\/\/github.com\/enterprise\/startups)\n    - [Nonprofits](https:\/\/github.com\/solutions\/industry\/nonprofits)\n  - BY USE CASE\n    \n    - [App Modernization](https:\/\/github.com\/solutions\/use-case\/app-modernization)\n    - [DevSecOps](https:\/\/github.com\/solutions\/use-case\/devsecops)\n    - [DevOps](https:\/\/github.com\/solutions\/use-case\/devops)\n    - [CI\/CD](https:\/\/github.com\/solutions\/use-case\/ci-cd)\n    - [View all use cases](https:\/\/github.com\/solutions\/use-case)\n  - BY INDUSTRY\n    \n    - [Healthcare](https:\/\/github.com\/solutions\/industry\/healthcare)\n    - [Financial services](https:\/\/github.com\/solutions\/industry\/financial-services)\n    - [Manufacturing](https:\/\/github.com\/solutions\/industry\/manufacturing)\n    - [Government](https:\/\/github.com\/solutions\/industry\/government)\n    - [View all industries](https:\/\/github.com\/solutions\/industry)\n  \n  [View all solutions](https:\/\/github.com\/solutions)\n- Resources\n  \n  - EXPLORE BY TOPIC\n    \n    - [AI](https:\/\/github.com\/resources\/articles?topic=ai)\n    - [Software Development](https:\/\/github.com\/resources\/articles?topic=software-development)\n    - [DevOps](https:\/\/github.com\/resources\/articles?topic=devops)\n    - [Security](https:\/\/github.com\/resources\/articles?topic=security)\n    - [View all topics](https:\/\/github.com\/resources\/articles)\n  - EXPLORE BY TYPE\n    \n    - [Customer stories](https:\/\/github.com\/customer-stories)\n    - [Events & webinars](https:\/\/github.com\/resources\/events)\n    - [Ebooks & reports](https:\/\/github.com\/resources\/whitepapers)\n    - [Business insights](https:\/\/github.com\/solutions\/executive-insights)\n    - [GitHub Skills](https:\/\/skills.github.com\/)\n  - SUPPORT & SERVICES\n    \n    - [Documentation](https:\/\/docs.github.com\/)\n    - [Customer support](https:\/\/support.github.com\/)\n    - [Community forum](https:\/\/github.com\/orgs\/community\/discussions)\n    - [Trust center](https:\/\/github.com\/trust-center)\n    - [Partners](https:\/\/github.com\/partners)\n  \n  [View all resources](https:\/\/github.com\/resources)\n- Open Source\n  \n  - COMMUNITY\n    \n    - [GitHub SponsorsFund open source developers](https:\/\/github.com\/sponsors)\n  - PROGRAMS\n    \n    - [Security Lab](https:\/\/securitylab.github.com\/)\n    - [Maintainer Community](https:\/\/maintainers.github.com\/)\n    - [Accelerator](https:\/\/github.com\/accelerator)\n    - [Archive Program](https:\/\/archiveprogram.github.com\/)\n  - REPOSITORIES\n    \n    - [Topics](https:\/\/github.com\/topics)\n    - [Trending](https:\/\/github.com\/trending)\n    - [Collections](https:\/\/github.com\/collections)\n- Enterprise\n  \n  - ENTERPRISE SOLUTIONS\n    \n    - [Enterprise platformAI-powered developer platform](https:\/\/github.com\/enterprise)\n  - AVAILABLE ADD-ONS\n    \n    - [GitHub Advanced SecurityEnterprise-grade security features](https:\/\/github.com\/security\/advanced-security)\n    - [Copilot for BusinessEnterprise-grade AI features](https:\/\/github.com\/features\/copilot\/copilot-business)\n    - [Premium SupportEnterprise-grade 24\/7 support](https:\/\/github.com\/premium-support)\n- [Pricing](https:\/\/github.com\/pricing)\n\nSearch or jump to...\n\n# Search code, repositories, users, issues, pull requests...\n[Search syntax tips](https:\/\/docs.github.com\/search-github\/github-code-search\/understanding-github-code-search-syntax)\n\n# Provide feedback\nCancel\n\nSubmit feedback\n\n# Saved searches\n## Use saved searches to filter your results more quickly\n\nCancel\n\nCreate saved search\n\n[Sign in](https:\/\/github.com\/login?return_to=https%3A%2F%2Fgithub.com%2Fdano%2Faioprocessing)\n\n[Sign up](https:\/\/github.com\/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E&source=header-repo&source_repo=dano%2Faioprocessing)\n\nAppearance settings\n\nResetting focus\n\nYou signed in with another tab or window. [Reload](https:\/\/github.com\/dano\/aioprocessing) to refresh your session. You signed out in another tab or window. [Reload](https:\/\/github.com\/dano\/aioprocessing) to refresh your session. You switched accounts on another tab or window. [Reload](https:\/\/github.com\/dano\/aioprocessing) to refresh your session.\n\nDismiss alert\n\n{{ message }}\n\n[dano](https:\/\/github.com\/dano)   \/  **[aioprocessing](https:\/\/github.com\/dano\/aioprocessing)**  Public\n\n- [Notifications](https:\/\/github.com\/login?return_to=%2Fdano%2Faioprocessing)\n  You must be signed in to change notification settings\n- [Fork 37](https:\/\/github.com\/login?return_to=%2Fdano%2Faioprocessing)\n- [Star 662](https:\/\/github.com\/login?return_to=%2Fdano%2Faioprocessing)\n\n- [Code](https:\/\/github.com\/dano\/aioprocessing)\n- [Issues 7](https:\/\/github.com\/dano\/aioprocessing\/issues)\n- [Pull requests 0](https:\/\/github.com\/dano\/aioprocessing\/pulls)\n- [Actions](https:\/\/github.com\/dano\/aioprocessing\/actions)\n- [Projects](https:\/\/github.com\/dano\/aioprocessing\/projects)\n- [Wiki](https:\/\/github.com\/dano\/aioprocessing\/wiki)\n- [Security 0](https:\/\/github.com\/dano\/aioprocessing\/security)\n- [Insights](https:\/\/github.com\/dano\/aioprocessing\/pulse)\n\nAdditional navigation options\n\n- [Code](https:\/\/github.com\/dano\/aioprocessing)\n- [Issues](https:\/\/github.com\/dano\/aioprocessing\/issues)\n- [Pull requests](https:\/\/github.com\/dano\/aioprocessing\/pulls)\n- [Actions](https:\/\/github.com\/dano\/aioprocessing\/actions)\n- [Projects](https:\/\/github.com\/dano\/aioprocessing\/projects)\n- [Wiki](https:\/\/github.com\/dano\/aioprocessing\/wiki)\n- [Security](https:\/\/github.com\/dano\/aioprocessing\/security)\n- [Insights](https:\/\/github.com\/dano\/aioprocessing\/pulse)\n\n# dano\/aioprocessing\nmaster\n\n[**3** Branches](https:\/\/github.com\/dano\/aioprocessing\/branches)\n\n[**3** Tags](https:\/\/github.com\/dano\/aioprocessing\/tags)\n\nGo to file\n\nCode\n\nOpen more actions menu\n\n## Folders and files\n| Name | Name | Last commit message | Last commit date |\n|---|---|---|---|\n| Latest commit [![dano](https:\/\/avatars.githubusercontent.com\/u\/38807?v=4&size=40)](https:\/\/github.com\/dano)[dano](https:\/\/github.com\/dano\/aioprocessing\/commits?author=dano) [Fix flake issue](https:\/\/github.com\/dano\/aioprocessing\/commit\/39e84ac016aa21d93835b44ca134b0791c8c13ba) Open commit details Sep 16, 2022 [39e84ac](https:\/\/github.com\/dano\/aioprocessing\/commit\/39e84ac016aa21d93835b44ca134b0791c8c13ba) · Sep 16, 2022 History [148 Commits](https:\/\/github.com\/dano\/aioprocessing\/commits\/master\/) Open commit details 148 Commits |   |   |   |\n\n## Repository files navigation\n- [README](https:\/\/github.com\/dano\/aioprocessing)\n- [License](https:\/\/github.com\/dano\/aioprocessing)\n\n# aioprocessing\n[![Build Status](https:\/\/github.com\/dano\/aioprocessing\/workflows\/aioprocessing%20tests\/badge.svg?branch=master)](https:\/\/github.com\/dano\/aioprocessing\/actions)\n\n`aioprocessing` provides asynchronous, [`asyncio`](https:\/\/docs.python.org\/3\/library\/asyncio.html) compatible, coroutine versions of many blocking instance methods on objects in the [`multiprocessing`](https:\/\/docs.python.org\/3\/library\/multiprocessing.html) library. To use [`dill`](https:\/\/pypi.org\/project\/dill) for universal pickling, install using `pip install aioprocessing[dill]`. Here's an example demonstrating the `aioprocessing` versions of `Event`, `Queue`, and `Lock`:\n```\nimport time\nimport asyncio\nimport aioprocessing\n\n\ndef func(queue, event, lock, items):\n    \"\"\" Demo worker function.\n\n    This worker function runs in its own process, and uses\n    normal blocking calls to aioprocessing objects, exactly \n    the way you would use oridinary multiprocessing objects.\n\n    \"\"\"\n    with lock:\n        event.set()\n        for item in items:\n            time.sleep(3)\n            queue.put(item+5)\n    queue.close()\n\n\nasync def example(queue, event, lock):\n    l = [1,2,3,4,5]\n    p = aioprocessing.AioProcess(target=func, args=(queue, event, lock, l))\n    p.start()\n    while True:\n        result = await queue.coro_get()\n        if result is None:\n            break\n        print(\"Got result {}\".format(result))\n    await p.coro_join()\n\nasync def example2(queue, event, lock):\n    await event.coro_wait()\n    async with lock:\n        await queue.coro_put(78)\n        await queue.coro_put(None) # Shut down the worker\n\nif __name__ == \"__main__\":\n    loop = asyncio.get_event_loop()\n    queue = aioprocessing.AioQueue()\n    lock = aioprocessing.AioLock()\n    event = aioprocessing.AioEvent()\n    tasks = [\n        asyncio.ensure_future(example(queue, event, lock)), \n        asyncio.ensure_future(example2(queue, event, lock)),\n    ]\n    loop.run_until_complete(asyncio.wait(tasks))\n    loop.close()\n```\nThe aioprocessing objects can be used just like their multiprocessing equivalents - as they are in `func` above - but they can also be seamlessly used inside of `asyncio` coroutines, without ever blocking the event loop.\n\n## What's new\n`v2.0.1`\n\n- Fixed a bug that kept the `AioBarrier` and `AioEvent` proxies returned from `AioManager` instances from working. Thanks to Giorgos Apostolopoulos for the fix.\n\n`v2.0.0`\n\n- Add support for universal pickling using [`dill`](https:\/\/github.com\/uqfoundation\/dill), installable with `pip install aioprocessing[dill]`. The library will now attempt to import [`multiprocess`](https:\/\/github.com\/uqfoundation\/multiprocess), falling back to stdlib `multiprocessing`. Force stdlib behaviour by setting a non-empty environment variable `AIOPROCESSING_DILL_DISABLED=1`. This can be used to avoid [errors](https:\/\/github.com\/dano\/aioprocessing\/pull\/36#discussion_r631178933) when attempting to combine `aioprocessing[dill]` with stdlib `multiprocessing` based objects like `concurrent.futures.ProcessPoolExecutor`.\n## How does it work?\nIn most cases, this library makes blocking calls to `multiprocessing` methods asynchronous by executing the call in a [`ThreadPoolExecutor`](https:\/\/docs.python.org\/3\/library\/concurrent.futures.html#threadpoolexecutor), using [`asyncio.run_in_executor()`](https:\/\/docs.python.org\/3\/library\/asyncio-eventloop.html#asyncio.BaseEventLoop.run_in_executor). It does *not* re-implement multiprocessing using asynchronous I\/O. This means there is extra overhead added when you use `aioprocessing` objects instead of `multiprocessing` objects, because each one is generally introducing a `ThreadPoolExecutor` containing at least one [`threading.Thread`](https:\/\/docs.python.org\/2\/library\/threading.html#thread-objects). It also means that all the normal risks you get when you mix threads with fork apply here, too (See <http:\/\/bugs.python.org\/issue6721> for more info).\n\nThe one exception to this is `aioprocessing.AioPool`, which makes use of the existing `callback` and `error_callback` keyword arguments in the various [`Pool.*_async`](https:\/\/docs.python.org\/3\/library\/multiprocessing.html#multiprocessing.pool.Pool.apply_async) methods to run them as `asyncio` coroutines. Note that `multiprocessing.Pool` is actually using threads internally, so the thread\/fork mixing caveat still applies.\n\nEach `multiprocessing` class is replaced by an equivalent `aioprocessing` class, distinguished by the `Aio` prefix. So, `Pool` becomes `AioPool`, etc. All methods that could block on I\/O also have a coroutine version that can be used with `asyncio`. For example, `multiprocessing.Lock.acquire()` can be replaced with `aioprocessing.AioLock.coro_acquire()`. You can pass an `asyncio` EventLoop object to any `coro_*` method using the `loop` keyword argument. For example, `lock.coro_acquire(loop=my_loop)`.\n\nNote that you can also use the `aioprocessing` synchronization primitives as replacements for their equivalent `threading` primitives, in single-process, multi-threaded programs that use `asyncio`.\n\n## What parts of multiprocessing are supported?\nMost of them! All methods that could do blocking I\/O in the following objects have equivalent versions in `aioprocessing` that extend the `multiprocessing` versions by adding coroutine versions of all the blocking methods.\n\n- `Pool`\n- `Process`\n- `Pipe`\n- `Lock`\n- `RLock`\n- `Semaphore`\n- `BoundedSemaphore`\n- `Event`\n- `Condition`\n- `Barrier`\n- `connection.Connection`\n- `connection.Listener`\n- `connection.Client`\n- `Queue`\n- `JoinableQueue`\n- `SimpleQueue`\n- All `managers.SyncManager` `Proxy` versions of the items above (`SyncManager.Queue`, `SyncManager.Lock()`, etc.).\n## What versions of Python are compatible?\n`aioprocessing` will work out of the box on Python 3.5+.\n\n## Gotchas\nKeep in mind that, while the API exposes coroutines for interacting with `multiprocessing` APIs, internally they are almost always being delegated to a `ThreadPoolExecutor`, this means the caveats that apply with using `ThreadPoolExecutor` with `asyncio` apply: namely, you won't be able to cancel any of the coroutines, because the work being done in the worker thread can't be interrupted.\n\n## About\nA Python 3.5+ library that integrates the multiprocessing module with asyncio\n\n### Topics\n[python](https:\/\/github.com\/topics\/python \"Topic: python\")   [multiprocessing](https:\/\/github.com\/topics\/multiprocessing \"Topic: multiprocessing\")   [coroutines](https:\/\/github.com\/topics\/coroutines \"Topic: coroutines\")   [asyncio](https:\/\/github.com\/topics\/asyncio \"Topic: asyncio\")\n\n### Resources\n[Readme](https:\/\/github.com\/dano\/aioprocessing#readme-ov-file)\n\n### License\n[View license](https:\/\/github.com\/dano\/aioprocessing#License-1-ov-file)\n\n### Uh oh\\!\nThere was an error while loading. [Please reload this page](https:\/\/github.com\/dano\/aioprocessing).\n\n[Activity](https:\/\/github.com\/dano\/aioprocessing\/activity)\n\n### Stars\n[**662** stars](https:\/\/github.com\/dano\/aioprocessing\/stargazers)\n\n### Watchers\n[**18** watching](https:\/\/github.com\/dano\/aioprocessing\/watchers)\n\n### Forks\n[**37** forks](https:\/\/github.com\/dano\/aioprocessing\/forks)\n\n[Report repository](https:\/\/github.com\/contact\/report-content?content_url=https%3A%2F%2Fgithub.com%2Fdano%2Faioprocessing&report=dano+%28user%29)\n\n## [Releases 3](https:\/\/github.com\/dano\/aioprocessing\/releases)\n[v2.0.1 Latest Sep 15, 2022](https:\/\/github.com\/dano\/aioprocessing\/releases\/tag\/v2.0.1)\n\n[\\+ 2 releases](https:\/\/github.com\/dano\/aioprocessing\/releases)\n\n## [Packages 0](https:\/\/github.com\/users\/dano\/packages?repo_name=aioprocessing)\nNo packages published\n\n## [Used by 570](https:\/\/github.com\/dano\/aioprocessing\/network\/dependents)\n[![@lucasmarcelin0](https:\/\/avatars.githubusercontent.com\/u\/66127687?s=64&v=4) ![@bleu](https:\/\/avatars.githubusercontent.com\/u\/116457461?s=64&v=4) ![@rehmantechwork-code](https:\/\/avatars.githubusercontent.com\/u\/262094790?s=64&v=4) ![@rooksec-security-lab](https:\/\/avatars.githubusercontent.com\/u\/251647220?s=64&v=4) ![@lunatechza](https:\/\/avatars.githubusercontent.com\/u\/8457050?s=64&v=4) ![@Danny-Dasilva](https:\/\/avatars.githubusercontent.com\/u\/32214239?s=64&v=4) ![@1jamesthompson1](https:\/\/avatars.githubusercontent.com\/u\/103026808?s=64&v=4) ![@BITS-Rohit](https:\/\/avatars.githubusercontent.com\/u\/125949183?s=64&v=4) + 562](https:\/\/github.com\/dano\/aioprocessing\/network\/dependents)\n\n## [Contributors](https:\/\/github.com\/dano\/aioprocessing\/graphs\/contributors)\n### Uh oh\\!\nThere was an error while loading. [Please reload this page](https:\/\/github.com\/dano\/aioprocessing).\n\n## Languages\n- [Python 100.0%](https:\/\/github.com\/dano\/aioprocessing\/search?l=python)\n\n## Footer\n© 2026 GitHub, Inc.\n\n### Footer navigation\n- [Terms](https:\/\/docs.github.com\/site-policy\/github-terms\/github-terms-of-service)\n- [Privacy](https:\/\/docs.github.com\/site-policy\/privacy-policies\/github-privacy-statement)\n- [Security](https:\/\/github.com\/security)\n- [Status](https:\/\/www.githubstatus.com\/)\n- [Community](https:\/\/github.community\/)\n- [Docs](https:\/\/docs.github.com\/)\n- [Contact](https:\/\/support.github.com\/?tags=dotcom-footer)\n- Manage cookies\n- Do not share my personal information\n\nYou can’t perform that action at this time.","attrs_readable_markdown":"[![Build Status](https:\/\/github.com\/dano\/aioprocessing\/workflows\/aioprocessing%20tests\/badge.svg?branch=master)](https:\/\/github.com\/dano\/aioprocessing\/actions)\n\n`aioprocessing` provides asynchronous, [`asyncio`](https:\/\/docs.python.org\/3\/library\/asyncio.html) compatible, coroutine versions of many blocking instance methods on objects in the [`multiprocessing`](https:\/\/docs.python.org\/3\/library\/multiprocessing.html) library. To use [`dill`](https:\/\/pypi.org\/project\/dill) for universal pickling, install using `pip install aioprocessing[dill]`. Here's an example demonstrating the `aioprocessing` versions of `Event`, `Queue`, and `Lock`:\n```\nimport time\nimport asyncio\nimport aioprocessing\n\n\ndef func(queue, event, lock, items):\n    \"\"\" Demo worker function.\n\n    This worker function runs in its own process, and uses\n    normal blocking calls to aioprocessing objects, exactly \n    the way you would use oridinary multiprocessing objects.\n\n    \"\"\"\n    with lock:\n        event.set()\n        for item in items:\n            time.sleep(3)\n            queue.put(item+5)\n    queue.close()\n\n\nasync def example(queue, event, lock):\n    l = [1,2,3,4,5]\n    p = aioprocessing.AioProcess(target=func, args=(queue, event, lock, l))\n    p.start()\n    while True:\n        result = await queue.coro_get()\n        if result is None:\n            break\n        print(\"Got result {}\".format(result))\n    await p.coro_join()\n\nasync def example2(queue, event, lock):\n    await event.coro_wait()\n    async with lock:\n        await queue.coro_put(78)\n        await queue.coro_put(None) # Shut down the worker\n\nif __name__ == \"__main__\":\n    loop = asyncio.get_event_loop()\n    queue = aioprocessing.AioQueue()\n    lock = aioprocessing.AioLock()\n    event = aioprocessing.AioEvent()\n    tasks = [\n        asyncio.ensure_future(example(queue, event, lock)), \n        asyncio.ensure_future(example2(queue, event, lock)),\n    ]\n    loop.run_until_complete(asyncio.wait(tasks))\n    loop.close()\n```\nThe aioprocessing objects can be used just like their multiprocessing equivalents - as they are in `func` above - but they can also be seamlessly used inside of `asyncio` coroutines, without ever blocking the event loop.\n\nWhat's new\n\n`v2.0.1`\n\n- Fixed a bug that kept the `AioBarrier` and `AioEvent` proxies returned from `AioManager` instances from working. Thanks to Giorgos Apostolopoulos for the fix.\n\n`v2.0.0`\n\n- Add support for universal pickling using [`dill`](https:\/\/github.com\/uqfoundation\/dill), installable with `pip install aioprocessing[dill]`. The library will now attempt to import [`multiprocess`](https:\/\/github.com\/uqfoundation\/multiprocess), falling back to stdlib `multiprocessing`. Force stdlib behaviour by setting a non-empty environment variable `AIOPROCESSING_DILL_DISABLED=1`. This can be used to avoid [errors](https:\/\/github.com\/dano\/aioprocessing\/pull\/36#discussion_r631178933) when attempting to combine `aioprocessing[dill]` with stdlib `multiprocessing` based objects like `concurrent.futures.ProcessPoolExecutor`.\n\nHow does it work?\n\nIn most cases, this library makes blocking calls to `multiprocessing` methods asynchronous by executing the call in a [`ThreadPoolExecutor`](https:\/\/docs.python.org\/3\/library\/concurrent.futures.html#threadpoolexecutor), using [`asyncio.run_in_executor()`](https:\/\/docs.python.org\/3\/library\/asyncio-eventloop.html#asyncio.BaseEventLoop.run_in_executor). It does *not* re-implement multiprocessing using asynchronous I\/O. This means there is extra overhead added when you use `aioprocessing` objects instead of `multiprocessing` objects, because each one is generally introducing a `ThreadPoolExecutor` containing at least one [`threading.Thread`](https:\/\/docs.python.org\/2\/library\/threading.html#thread-objects). It also means that all the normal risks you get when you mix threads with fork apply here, too (See <http:\/\/bugs.python.org\/issue6721> for more info).\n\nThe one exception to this is `aioprocessing.AioPool`, which makes use of the existing `callback` and `error_callback` keyword arguments in the various [`Pool.*_async`](https:\/\/docs.python.org\/3\/library\/multiprocessing.html#multiprocessing.pool.Pool.apply_async) methods to run them as `asyncio` coroutines. Note that `multiprocessing.Pool` is actually using threads internally, so the thread\/fork mixing caveat still applies.\n\nEach `multiprocessing` class is replaced by an equivalent `aioprocessing` class, distinguished by the `Aio` prefix. So, `Pool` becomes `AioPool`, etc. All methods that could block on I\/O also have a coroutine version that can be used with `asyncio`. For example, `multiprocessing.Lock.acquire()` can be replaced with `aioprocessing.AioLock.coro_acquire()`. You can pass an `asyncio` EventLoop object to any `coro_*` method using the `loop` keyword argument. For example, `lock.coro_acquire(loop=my_loop)`.\n\nNote that you can also use the `aioprocessing` synchronization primitives as replacements for their equivalent `threading` primitives, in single-process, multi-threaded programs that use `asyncio`.\n\nWhat parts of multiprocessing are supported?\n\nMost of them! All methods that could do blocking I\/O in the following objects have equivalent versions in `aioprocessing` that extend the `multiprocessing` versions by adding coroutine versions of all the blocking methods.\n\n- `Pool`\n- `Process`\n- `Pipe`\n- `Lock`\n- `RLock`\n- `Semaphore`\n- `BoundedSemaphore`\n- `Event`\n- `Condition`\n- `Barrier`\n- `connection.Connection`\n- `connection.Listener`\n- `connection.Client`\n- `Queue`\n- `JoinableQueue`\n- `SimpleQueue`\n- All `managers.SyncManager` `Proxy` versions of the items above (`SyncManager.Queue`, `SyncManager.Lock()`, etc.).\n\nWhat versions of Python are compatible?\n\n`aioprocessing` will work out of the box on Python 3.5+.\n\nGotchas\n\nKeep in mind that, while the API exposes coroutines for interacting with `multiprocessing` APIs, internally they are almost always being delegated to a `ThreadPoolExecutor`, this means the caveats that apply with using `ThreadPoolExecutor` with `asyncio` apply: namely, you won't be able to cancel any of the coroutines, because the work being done in the worker thread can't be interrupted.","meta_canonical":null}

3. Robots.txt Check

Query:

Response:

4. Spam/Ban Check

Query:

Response:

5. Seen Status Check

ℹ️ Skipped - page is already crawled

📄

INDEXABLE

✅

CRAWLED

1 month ago

🤖

ROBOTS ALLOWED

Page Info Filters

Filter	Status	Condition	Details
HTTP status	PASS	`download_http_code = 200`	HTTP 200
Age cutoff	PASS	`download_stamp > now() - 6 MONTH`	1.1 months ago (distributed domain, exempt)
History drop	PASS	`isNull(history_drop_reason)`	No drop reason
Spam/ban	PASS	`fh_dont_index != 1 AND ml_spam_score = 0`	ml_spam_score=0
Canonical	PASS	`meta_canonical IS NULL OR = '' OR = src_unparsed`	Not set

Page Details

Property	Value
URL	https://github.com/dano/aioprocessing
Last Crawled	2026-03-05 12:34:47 (1 month ago)
First Indexed	2014-10-25 20:45:14 (11 years ago)
HTTP Status Code	200
Meta Title	GitHub - dano/aioprocessing: A Python 3.5+ library that integrates the multiprocessing module with asyncio · GitHub
Meta Description	A Python 3.5+ library that integrates the multiprocessing module with asyncio - dano/aioprocessing
Meta Canonical	null
Boilerpipe Text	aioprocessing provides asynchronous, asyncio compatible, coroutine versions of many blocking instance methods on objects in the multiprocessing library. To use dill for universal pickling, install using pip install aioprocessing[dill] . Here's an example demonstrating the aioprocessing versions of Event , Queue , and Lock : import time import asyncio import aioprocessing def func ( queue , event , lock , items ): """ Demo worker function. This worker function runs in its own process, and uses normal blocking calls to aioprocessing objects, exactly the way you would use oridinary multiprocessing objects. """ with lock : event . set () for item in items : time . sleep ( 3 ) queue . put ( item + 5 ) queue . close () async def example ( queue , event , lock ): l = [ 1 , 2 , 3 , 4 , 5 ] p = aioprocessing . AioProcess ( target = func , args = ( queue , event , lock , l )) p . start () while True : result = await queue . coro_get () if result is None : break print ( "Got result {}" . format ( result )) await p . coro_join () async def example2 ( queue , event , lock ): await event . coro_wait () async with lock : await queue . coro_put ( 78 ) await queue . coro_put ( None ) # Shut down the worker if __name__ == "__main__" : loop = asyncio . get_event_loop () queue = aioprocessing . AioQueue () lock = aioprocessing . AioLock () event = aioprocessing . AioEvent () tasks = [ asyncio . ensure_future ( example ( queue , event , lock )), asyncio . ensure_future ( example2 ( queue , event , lock )), ] loop . run_until_complete ( asyncio . wait ( tasks )) loop . close () The aioprocessing objects can be used just like their multiprocessing equivalents - as they are in func above - but they can also be seamlessly used inside of asyncio coroutines, without ever blocking the event loop. What's new v2.0.1 Fixed a bug that kept the AioBarrier and AioEvent proxies returned from AioManager instances from working. Thanks to Giorgos Apostolopoulos for the fix. v2.0.0 Add support for universal pickling using dill , installable with pip install aioprocessing[dill] . The library will now attempt to import multiprocess , falling back to stdlib multiprocessing . Force stdlib behaviour by setting a non-empty environment variable AIOPROCESSING_DILL_DISABLED=1 . This can be used to avoid errors when attempting to combine aioprocessing[dill] with stdlib multiprocessing based objects like concurrent.futures.ProcessPoolExecutor . How does it work? In most cases, this library makes blocking calls to multiprocessing methods asynchronous by executing the call in a ThreadPoolExecutor , using asyncio.run_in_executor() . It does not re-implement multiprocessing using asynchronous I/O. This means there is extra overhead added when you use aioprocessing objects instead of multiprocessing objects, because each one is generally introducing a ThreadPoolExecutor containing at least one threading.Thread . It also means that all the normal risks you get when you mix threads with fork apply here, too (See http://bugs.python.org/issue6721 for more info). The one exception to this is aioprocessing.AioPool , which makes use of the existing callback and error_callback keyword arguments in the various Pool._async methods to run them as asyncio coroutines. Note that multiprocessing.Pool is actually using threads internally, so the thread/fork mixing caveat still applies. Each multiprocessing class is replaced by an equivalent aioprocessing class, distinguished by the Aio prefix. So, Pool becomes AioPool , etc. All methods that could block on I/O also have a coroutine version that can be used with asyncio . For example, multiprocessing.Lock.acquire() can be replaced with aioprocessing.AioLock.coro_acquire() . You can pass an asyncio EventLoop object to any coro_ method using the loop keyword argument. For example, lock.coro_acquire(loop=my_loop) . Note that you can also use the aioprocessing synchronization primitives as replacements for their equivalent threading primitives, in single-process, multi-threaded programs that use asyncio . What parts of multiprocessing are supported? Most of them! All methods that could do blocking I/O in the following objects have equivalent versions in aioprocessing that extend the multiprocessing versions by adding coroutine versions of all the blocking methods. Pool Process Pipe Lock RLock Semaphore BoundedSemaphore Event Condition Barrier connection.Connection connection.Listener connection.Client Queue JoinableQueue SimpleQueue All managers.SyncManager Proxy versions of the items above ( SyncManager.Queue , SyncManager.Lock() , etc.). What versions of Python are compatible? aioprocessing will work out of the box on Python 3.5+. Gotchas Keep in mind that, while the API exposes coroutines for interacting with multiprocessing APIs, internally they are almost always being delegated to a ThreadPoolExecutor , this means the caveats that apply with using ThreadPoolExecutor with asyncio apply: namely, you won't be able to cancel any of the coroutines, because the work being done in the worker thread can't be interrupted.
Markdown	[Skip to content](https://github.com/dano/aioprocessing#start-of-content) ## Navigation Menu Toggle navigation [Sign in](https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fdano%2Faioprocessing) Appearance settings - Platform - AI CODE CREATION - [GitHub CopilotWrite better code with AI](https://github.com/features/copilot) - [GitHub SparkBuild and deploy intelligent apps](https://github.com/features/spark) - [GitHub ModelsManage and compare prompts](https://github.com/features/models) - [MCP RegistryNewIntegrate external tools](https://github.com/mcp) - DEVELOPER WORKFLOWS - [ActionsAutomate any workflow](https://github.com/features/actions) - [CodespacesInstant dev environments](https://github.com/features/codespaces) - [IssuesPlan and track work](https://github.com/features/issues) - [Code ReviewManage code changes](https://github.com/features/code-review) - APPLICATION SECURITY - [GitHub Advanced SecurityFind and fix vulnerabilities](https://github.com/security/advanced-security) - [Code securitySecure your code as you build](https://github.com/security/advanced-security/code-security) - [Secret protectionStop leaks before they start](https://github.com/security/advanced-security/secret-protection) - EXPLORE - [Why GitHub](https://github.com/why-github) - [Documentation](https://docs.github.com/) - [Blog](https://github.blog/) - [Changelog](https://github.blog/changelog) - [Marketplace](https://github.com/marketplace) [View all features](https://github.com/features) - Solutions - BY COMPANY SIZE - [Enterprises](https://github.com/enterprise) - [Small and medium teams](https://github.com/team) - [Startups](https://github.com/enterprise/startups) - [Nonprofits](https://github.com/solutions/industry/nonprofits) - BY USE CASE - [App Modernization](https://github.com/solutions/use-case/app-modernization) - [DevSecOps](https://github.com/solutions/use-case/devsecops) - [DevOps](https://github.com/solutions/use-case/devops) - [CI/CD](https://github.com/solutions/use-case/ci-cd) - [View all use cases](https://github.com/solutions/use-case) - BY INDUSTRY - [Healthcare](https://github.com/solutions/industry/healthcare) - [Financial services](https://github.com/solutions/industry/financial-services) - [Manufacturing](https://github.com/solutions/industry/manufacturing) - [Government](https://github.com/solutions/industry/government) - [View all industries](https://github.com/solutions/industry) [View all solutions](https://github.com/solutions) - Resources - EXPLORE BY TOPIC - [AI](https://github.com/resources/articles?topic=ai) - [Software Development](https://github.com/resources/articles?topic=software-development) - [DevOps](https://github.com/resources/articles?topic=devops) - [Security](https://github.com/resources/articles?topic=security) - [View all topics](https://github.com/resources/articles) - EXPLORE BY TYPE - [Customer stories](https://github.com/customer-stories) - [Events & webinars](https://github.com/resources/events) - [Ebooks & reports](https://github.com/resources/whitepapers) - [Business insights](https://github.com/solutions/executive-insights) - [GitHub Skills](https://skills.github.com/) - SUPPORT & SERVICES - [Documentation](https://docs.github.com/) - [Customer support](https://support.github.com/) - [Community forum](https://github.com/orgs/community/discussions) - [Trust center](https://github.com/trust-center) - [Partners](https://github.com/partners) [View all resources](https://github.com/resources) - Open Source - COMMUNITY - [GitHub SponsorsFund open source developers](https://github.com/sponsors) - PROGRAMS - [Security Lab](https://securitylab.github.com/) - [Maintainer Community](https://maintainers.github.com/) - [Accelerator](https://github.com/accelerator) - [Archive Program](https://archiveprogram.github.com/) - REPOSITORIES - [Topics](https://github.com/topics) - [Trending](https://github.com/trending) - [Collections](https://github.com/collections) - Enterprise - ENTERPRISE SOLUTIONS - [Enterprise platformAI-powered developer platform](https://github.com/enterprise) - AVAILABLE ADD-ONS - [GitHub Advanced SecurityEnterprise-grade security features](https://github.com/security/advanced-security) - [Copilot for BusinessEnterprise-grade AI features](https://github.com/features/copilot/copilot-business) - [Premium SupportEnterprise-grade 24/7 support](https://github.com/premium-support) - [Pricing](https://github.com/pricing) Search or jump to... # Search code, repositories, users, issues, pull requests... [Search syntax tips](https://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax) # Provide feedback Cancel Submit feedback # Saved searches ## Use saved searches to filter your results more quickly Cancel Create saved search [Sign in](https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fdano%2Faioprocessing) [Sign up](https://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E&source=header-repo&source_repo=dano%2Faioprocessing) Appearance settings Resetting focus You signed in with another tab or window. [Reload](https://github.com/dano/aioprocessing) to refresh your session. You signed out in another tab or window. [Reload](https://github.com/dano/aioprocessing) to refresh your session. You switched accounts on another tab or window. [Reload](https://github.com/dano/aioprocessing) to refresh your session. Dismiss alert {{ message }} [dano](https://github.com/dano) / [aioprocessing](https://github.com/dano/aioprocessing) Public - [Notifications](https://github.com/login?return_to=%2Fdano%2Faioprocessing) You must be signed in to change notification settings - [Fork 37](https://github.com/login?return_to=%2Fdano%2Faioprocessing) - [Star 662](https://github.com/login?return_to=%2Fdano%2Faioprocessing) - [Code](https://github.com/dano/aioprocessing) - [Issues 7](https://github.com/dano/aioprocessing/issues) - [Pull requests 0](https://github.com/dano/aioprocessing/pulls) - [Actions](https://github.com/dano/aioprocessing/actions) - [Projects](https://github.com/dano/aioprocessing/projects) - [Wiki](https://github.com/dano/aioprocessing/wiki) - [Security 0](https://github.com/dano/aioprocessing/security) - [Insights](https://github.com/dano/aioprocessing/pulse) Additional navigation options - [Code](https://github.com/dano/aioprocessing) - [Issues](https://github.com/dano/aioprocessing/issues) - [Pull requests](https://github.com/dano/aioprocessing/pulls) - [Actions](https://github.com/dano/aioprocessing/actions) - [Projects](https://github.com/dano/aioprocessing/projects) - [Wiki](https://github.com/dano/aioprocessing/wiki) - [Security](https://github.com/dano/aioprocessing/security) - [Insights](https://github.com/dano/aioprocessing/pulse) # dano/aioprocessing master [3 Branches](https://github.com/dano/aioprocessing/branches) [3 Tags](https://github.com/dano/aioprocessing/tags) Go to file Code Open more actions menu ## Folders and files \| Name \| Name \| Last commit message \| Last commit date \| \|---\|---\|---\|---\| \| Latest commit [![dano](https://avatars.githubusercontent.com/u/38807?v=4&size=40)](https://github.com/dano)[dano](https://github.com/dano/aioprocessing/commits?author=dano) [Fix flake issue](https://github.com/dano/aioprocessing/commit/39e84ac016aa21d93835b44ca134b0791c8c13ba) Open commit details Sep 16, 2022 [39e84ac](https://github.com/dano/aioprocessing/commit/39e84ac016aa21d93835b44ca134b0791c8c13ba) · Sep 16, 2022 History [148 Commits](https://github.com/dano/aioprocessing/commits/master/) Open commit details 148 Commits \| \| \| \| ## Repository files navigation - [README](https://github.com/dano/aioprocessing) - [License](https://github.com/dano/aioprocessing) # aioprocessing [![Build Status](https://github.com/dano/aioprocessing/workflows/aioprocessing%20tests/badge.svg?branch=master)](https://github.com/dano/aioprocessing/actions) `aioprocessing` provides asynchronous, [`asyncio`](https://docs.python.org/3/library/asyncio.html) compatible, coroutine versions of many blocking instance methods on objects in the [`multiprocessing`](https://docs.python.org/3/library/multiprocessing.html) library. To use [`dill`](https://pypi.org/project/dill) for universal pickling, install using `pip install aioprocessing[dill]`. Here's an example demonstrating the `aioprocessing` versions of `Event`, `Queue`, and `Lock`: ``` import time import asyncio import aioprocessing def func(queue, event, lock, items): """ Demo worker function. This worker function runs in its own process, and uses normal blocking calls to aioprocessing objects, exactly the way you would use oridinary multiprocessing objects. """ with lock: event.set() for item in items: time.sleep(3) queue.put(item+5) queue.close() async def example(queue, event, lock): l = [1,2,3,4,5] p = aioprocessing.AioProcess(target=func, args=(queue, event, lock, l)) p.start() while True: result = await queue.coro_get() if result is None: break print("Got result {}".format(result)) await p.coro_join() async def example2(queue, event, lock): await event.coro_wait() async with lock: await queue.coro_put(78) await queue.coro_put(None) # Shut down the worker if __name__ == "__main__": loop = asyncio.get_event_loop() queue = aioprocessing.AioQueue() lock = aioprocessing.AioLock() event = aioprocessing.AioEvent() tasks = [ asyncio.ensure_future(example(queue, event, lock)), asyncio.ensure_future(example2(queue, event, lock)), ] loop.run_until_complete(asyncio.wait(tasks)) loop.close() ``` The aioprocessing objects can be used just like their multiprocessing equivalents - as they are in `func` above - but they can also be seamlessly used inside of `asyncio` coroutines, without ever blocking the event loop. ## What's new `v2.0.1` - Fixed a bug that kept the `AioBarrier` and `AioEvent` proxies returned from `AioManager` instances from working. Thanks to Giorgos Apostolopoulos for the fix. `v2.0.0` - Add support for universal pickling using [`dill`](https://github.com/uqfoundation/dill), installable with `pip install aioprocessing[dill]`. The library will now attempt to import [`multiprocess`](https://github.com/uqfoundation/multiprocess), falling back to stdlib `multiprocessing`. Force stdlib behaviour by setting a non-empty environment variable `AIOPROCESSING_DILL_DISABLED=1`. This can be used to avoid [errors](https://github.com/dano/aioprocessing/pull/36#discussion_r631178933) when attempting to combine `aioprocessing[dill]` with stdlib `multiprocessing` based objects like `concurrent.futures.ProcessPoolExecutor`. ## How does it work? In most cases, this library makes blocking calls to `multiprocessing` methods asynchronous by executing the call in a [`ThreadPoolExecutor`](https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor), using [`asyncio.run_in_executor()`](https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.BaseEventLoop.run_in_executor). It does not re-implement multiprocessing using asynchronous I/O. This means there is extra overhead added when you use `aioprocessing` objects instead of `multiprocessing` objects, because each one is generally introducing a `ThreadPoolExecutor` containing at least one [`threading.Thread`](https://docs.python.org/2/library/threading.html#thread-objects). It also means that all the normal risks you get when you mix threads with fork apply here, too (See <http://bugs.python.org/issue6721> for more info). The one exception to this is `aioprocessing.AioPool`, which makes use of the existing `callback` and `error_callback` keyword arguments in the various [`Pool._async`](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.apply_async) methods to run them as `asyncio` coroutines. Note that `multiprocessing.Pool` is actually using threads internally, so the thread/fork mixing caveat still applies. Each `multiprocessing` class is replaced by an equivalent `aioprocessing` class, distinguished by the `Aio` prefix. So, `Pool` becomes `AioPool`, etc. All methods that could block on I/O also have a coroutine version that can be used with `asyncio`. For example, `multiprocessing.Lock.acquire()` can be replaced with `aioprocessing.AioLock.coro_acquire()`. You can pass an `asyncio` EventLoop object to any `coro_` method using the `loop` keyword argument. For example, `lock.coro_acquire(loop=my_loop)`. Note that you can also use the `aioprocessing` synchronization primitives as replacements for their equivalent `threading` primitives, in single-process, multi-threaded programs that use `asyncio`. ## What parts of multiprocessing are supported? Most of them! All methods that could do blocking I/O in the following objects have equivalent versions in `aioprocessing` that extend the `multiprocessing` versions by adding coroutine versions of all the blocking methods. - `Pool` - `Process` - `Pipe` - `Lock` - `RLock` - `Semaphore` - `BoundedSemaphore` - `Event` - `Condition` - `Barrier` - `connection.Connection` - `connection.Listener` - `connection.Client` - `Queue` - `JoinableQueue` - `SimpleQueue` - All `managers.SyncManager` `Proxy` versions of the items above (`SyncManager.Queue`, `SyncManager.Lock()`, etc.). ## What versions of Python are compatible? `aioprocessing` will work out of the box on Python 3.5+. ## Gotchas Keep in mind that, while the API exposes coroutines for interacting with `multiprocessing` APIs, internally they are almost always being delegated to a `ThreadPoolExecutor`, this means the caveats that apply with using `ThreadPoolExecutor` with `asyncio` apply: namely, you won't be able to cancel any of the coroutines, because the work being done in the worker thread can't be interrupted. ## About A Python 3.5+ library that integrates the multiprocessing module with asyncio ### Topics [python](https://github.com/topics/python "Topic: python") [multiprocessing](https://github.com/topics/multiprocessing "Topic: multiprocessing") [coroutines](https://github.com/topics/coroutines "Topic: coroutines") [asyncio](https://github.com/topics/asyncio "Topic: asyncio") ### Resources [Readme](https://github.com/dano/aioprocessing#readme-ov-file) ### License [View license](https://github.com/dano/aioprocessing#License-1-ov-file) ### Uh oh\! There was an error while loading. [Please reload this page](https://github.com/dano/aioprocessing). [Activity](https://github.com/dano/aioprocessing/activity) ### Stars [662 stars](https://github.com/dano/aioprocessing/stargazers) ### Watchers [18 watching](https://github.com/dano/aioprocessing/watchers) ### Forks [37 forks](https://github.com/dano/aioprocessing/forks) [Report repository](https://github.com/contact/report-content?content_url=https%3A%2F%2Fgithub.com%2Fdano%2Faioprocessing&report=dano+%28user%29) ## [Releases 3](https://github.com/dano/aioprocessing/releases) [v2.0.1 Latest Sep 15, 2022](https://github.com/dano/aioprocessing/releases/tag/v2.0.1) [\+ 2 releases](https://github.com/dano/aioprocessing/releases) ## [Packages 0](https://github.com/users/dano/packages?repo_name=aioprocessing) No packages published ## [Used by 570](https://github.com/dano/aioprocessing/network/dependents) [![@lucasmarcelin0](https://avatars.githubusercontent.com/u/66127687?s=64&v=4) ![@bleu](https://avatars.githubusercontent.com/u/116457461?s=64&v=4) ![@rehmantechwork-code](https://avatars.githubusercontent.com/u/262094790?s=64&v=4) ![@rooksec-security-lab](https://avatars.githubusercontent.com/u/251647220?s=64&v=4) ![@lunatechza](https://avatars.githubusercontent.com/u/8457050?s=64&v=4) ![@Danny-Dasilva](https://avatars.githubusercontent.com/u/32214239?s=64&v=4) ![@1jamesthompson1](https://avatars.githubusercontent.com/u/103026808?s=64&v=4) ![@BITS-Rohit](https://avatars.githubusercontent.com/u/125949183?s=64&v=4) + 562](https://github.com/dano/aioprocessing/network/dependents) ## [Contributors](https://github.com/dano/aioprocessing/graphs/contributors) ### Uh oh\! There was an error while loading. [Please reload this page](https://github.com/dano/aioprocessing). ## Languages - [Python 100.0%](https://github.com/dano/aioprocessing/search?l=python) ## Footer © 2026 GitHub, Inc. ### Footer navigation - [Terms](https://docs.github.com/site-policy/github-terms/github-terms-of-service) - [Privacy](https://docs.github.com/site-policy/privacy-policies/github-privacy-statement) - [Security](https://github.com/security) - [Status](https://www.githubstatus.com/) - [Community](https://github.community/) - [Docs](https://docs.github.com/) - [Contact](https://support.github.com/?tags=dotcom-footer) - Manage cookies - Do not share my personal information You can’t perform that action at this time.
Readable Markdown	[![Build Status](https://github.com/dano/aioprocessing/workflows/aioprocessing%20tests/badge.svg?branch=master)](https://github.com/dano/aioprocessing/actions) `aioprocessing` provides asynchronous, [`asyncio`](https://docs.python.org/3/library/asyncio.html) compatible, coroutine versions of many blocking instance methods on objects in the [`multiprocessing`](https://docs.python.org/3/library/multiprocessing.html) library. To use [`dill`](https://pypi.org/project/dill) for universal pickling, install using `pip install aioprocessing[dill]`. Here's an example demonstrating the `aioprocessing` versions of `Event`, `Queue`, and `Lock`: ``` import time import asyncio import aioprocessing def func(queue, event, lock, items): """ Demo worker function. This worker function runs in its own process, and uses normal blocking calls to aioprocessing objects, exactly the way you would use oridinary multiprocessing objects. """ with lock: event.set() for item in items: time.sleep(3) queue.put(item+5) queue.close() async def example(queue, event, lock): l = [1,2,3,4,5] p = aioprocessing.AioProcess(target=func, args=(queue, event, lock, l)) p.start() while True: result = await queue.coro_get() if result is None: break print("Got result {}".format(result)) await p.coro_join() async def example2(queue, event, lock): await event.coro_wait() async with lock: await queue.coro_put(78) await queue.coro_put(None) # Shut down the worker if __name__ == "__main__": loop = asyncio.get_event_loop() queue = aioprocessing.AioQueue() lock = aioprocessing.AioLock() event = aioprocessing.AioEvent() tasks = [ asyncio.ensure_future(example(queue, event, lock)), asyncio.ensure_future(example2(queue, event, lock)), ] loop.run_until_complete(asyncio.wait(tasks)) loop.close() ``` The aioprocessing objects can be used just like their multiprocessing equivalents - as they are in `func` above - but they can also be seamlessly used inside of `asyncio` coroutines, without ever blocking the event loop. What's new `v2.0.1` - Fixed a bug that kept the `AioBarrier` and `AioEvent` proxies returned from `AioManager` instances from working. Thanks to Giorgos Apostolopoulos for the fix. `v2.0.0` - Add support for universal pickling using [`dill`](https://github.com/uqfoundation/dill), installable with `pip install aioprocessing[dill]`. The library will now attempt to import [`multiprocess`](https://github.com/uqfoundation/multiprocess), falling back to stdlib `multiprocessing`. Force stdlib behaviour by setting a non-empty environment variable `AIOPROCESSING_DILL_DISABLED=1`. This can be used to avoid [errors](https://github.com/dano/aioprocessing/pull/36#discussion_r631178933) when attempting to combine `aioprocessing[dill]` with stdlib `multiprocessing` based objects like `concurrent.futures.ProcessPoolExecutor`. How does it work? In most cases, this library makes blocking calls to `multiprocessing` methods asynchronous by executing the call in a [`ThreadPoolExecutor`](https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor), using [`asyncio.run_in_executor()`](https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.BaseEventLoop.run_in_executor). It does not re-implement multiprocessing using asynchronous I/O. This means there is extra overhead added when you use `aioprocessing` objects instead of `multiprocessing` objects, because each one is generally introducing a `ThreadPoolExecutor` containing at least one [`threading.Thread`](https://docs.python.org/2/library/threading.html#thread-objects). It also means that all the normal risks you get when you mix threads with fork apply here, too (See <http://bugs.python.org/issue6721> for more info). The one exception to this is `aioprocessing.AioPool`, which makes use of the existing `callback` and `error_callback` keyword arguments in the various [`Pool._async`](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.apply_async) methods to run them as `asyncio` coroutines. Note that `multiprocessing.Pool` is actually using threads internally, so the thread/fork mixing caveat still applies. Each `multiprocessing` class is replaced by an equivalent `aioprocessing` class, distinguished by the `Aio` prefix. So, `Pool` becomes `AioPool`, etc. All methods that could block on I/O also have a coroutine version that can be used with `asyncio`. For example, `multiprocessing.Lock.acquire()` can be replaced with `aioprocessing.AioLock.coro_acquire()`. You can pass an `asyncio` EventLoop object to any `coro_` method using the `loop` keyword argument. For example, `lock.coro_acquire(loop=my_loop)`. Note that you can also use the `aioprocessing` synchronization primitives as replacements for their equivalent `threading` primitives, in single-process, multi-threaded programs that use `asyncio`. What parts of multiprocessing are supported? Most of them! All methods that could do blocking I/O in the following objects have equivalent versions in `aioprocessing` that extend the `multiprocessing` versions by adding coroutine versions of all the blocking methods. - `Pool` - `Process` - `Pipe` - `Lock` - `RLock` - `Semaphore` - `BoundedSemaphore` - `Event` - `Condition` - `Barrier` - `connection.Connection` - `connection.Listener` - `connection.Client` - `Queue` - `JoinableQueue` - `SimpleQueue` - All `managers.SyncManager` `Proxy` versions of the items above (`SyncManager.Queue`, `SyncManager.Lock()`, etc.). What versions of Python are compatible? `aioprocessing` will work out of the box on Python 3.5+. Gotchas Keep in mind that, while the API exposes coroutines for interacting with `multiprocessing` APIs, internally they are almost always being delegated to a `ThreadPoolExecutor`, this means the caveats that apply with using `ThreadPoolExecutor` with `asyncio` apply: namely, you won't be able to cancel any of the coroutines, because the work being done in the worker thread can't be interrupted.
Shard	174 (laksa)
Root Hash	6325672905007345774
Unparsed URL	com,github!/dano/aioprocessing s443