🕷️ Crawler Inspector

URL Lookup

Direct Parameter Lookup

Raw Queries and Responses

1. Shard Calculation

Query:

Response:

Calculated Shard: 83 (from laksa049)

2. Crawled Status Check

Query:

curl -X POST \
  'http://laksa083.int.ahrefs:8124/' \
  -H 'Content-Type: text/plain' \
  -H 'X-ClickHouse-Database: crawler3' \
  -H 'Authorization: Basic YXBpOg==' \
  -d 'SELECT getAhrefsURLFromUnparsed(src_unparsed) AS found_url, ifNull(toUnixTimestamp(download_stamp), 0) AS crawl_time, ifNull(toUnixTimestamp(props_url_first_seen), 0) AS first_indexed_time, download_http_code AS http_code, src_unparsed AS src_unparsed, src_root_hash AS src_root_hash, history_drop_reason AS history_drop_reason, meta_title AS meta_title, meta_descriptions AS meta_descriptions, attrs_boilerpipe_text AS attrs_boilerpipe_text, attrs_markdown AS attrs_markdown, attrs_readable_markdown AS attrs_readable_markdown, meta_canonical AS meta_canonical FROM crawler3.page_info_local FINAL PREWHERE (src_root_hash, src_unparsed) IN ((getAhrefsRootHashFromUnparsed(getAhrefsUnparsedNoserviceFromURL(\'https://www.digitalocean.com/community/tutorials/python-multiprocessing-example\')), getAhrefsUnparsedNoserviceFromURL(\'https://www.digitalocean.com/community/tutorials/python-multiprocessing-example\'))) FORMAT JSONEachRow'

Response:

{"found_url":"https:\/\/www.digitalocean.com\/community\/tutorials\/python-multiprocessing-example","crawl_time":1774477767,"first_indexed_time":1659690331,"http_code":200,"src_unparsed":"com,digitalocean!www,\/community\/tutorials\/python-multiprocessing-example s443","src_root_hash":"13428457316079428483","history_drop_reason":null,"meta_title":"Python Multiprocessing Example | DigitalOcean","meta_descriptions":["Speed up your Python code with multiprocessing. Learn how to parallelize tasks for faster results."],"attrs_boilerpipe_text":"In our previous tutorial, we learned about\nPython CSV Example\n. In this tutorial we are going to learn Python Multiprocessing with examples.\nPython Multiprocessing\nParallel processing is getting more attention nowadays. If you still don’t know about the parallel processing, learn from\nwikipedia\n. As CPU manufacturers start adding more and more cores to their processors, creating parallel code is a great way to improve performance. Python introduced\nmultiprocessing\nmodule to let us write parallel code. To understand the main motivation of this module, we have to know some basics about parallel programming. After reading this article, we hope that, you would be able to gather some knowledge on this topic.\nPython Multiprocessing Process, Queue and Locks\nThere are plenty of classes in python multiprocessing module for building a parallel program. Among them, three basic classes are\nProcess\n,\nQueue\nand\nLock\n. These classes will help you to build a parallel program. But before describing about those, let us initiate this topic with simple code. To make a parallel program useful, you have to know how many cores are there in you pc. Python Multiprocessing module enables you to know that. The following simple code will print the number of cores in your pc.\nimport\nmultiprocessing\nprint\n(\n\"Number of cpu : \"\n,\nmultiprocessing\n.\ncpu_count\n(\n)\n)\nThe following output may vary for your pc. For me, number of cores is 8.\nPython multiprocessing Process class\nPython multiprocessing\nProcess\nclass is an abstraction that sets up another Python process, provides it to run code and a way for the parent application to control execution. There are two important functions that belongs to the Process class -\nstart()\nand\njoin()\nfunction. At first, we need to write a function, that will be run by the process. Then, we need to instantiate a process object. If we create a process object, nothing will happen until we tell it to start processing via\nstart()\nfunction. Then, the process will run and return its result. After that we tell the process to complete via\njoin()\nfunction. Without\njoin()\nfunction call, process will remain idle and won’t terminate. So if you create many processes and don’t terminate them, you may face scarcity of resources. Then you may need to kill them manually. One important thing is, if you want to pass any argument through the process you need to use\nargs\nkeyword argument. The following code will be helpful to understand the usage of Process class.\nfrom\nmultiprocessing\nimport\nProcess\ndef\nprint_func\n(\ncontinent\n=\n'Asia'\n)\n:\nprint\n(\n'The name of continent is : '\n,\ncontinent\n)\nif\n__name__\n==\n\"__main__\"\n:\n# confirms that the code is under main function\nnames\n=\n[\n'America'\n,\n'Europe'\n,\n'Africa'\n]\nprocs\n=\n[\n]\nproc\n=\nProcess\n(\ntarget\n=\nprint_func\n)\n# instantiating without any argument\nprocs\n.\nappend\n(\nproc\n)\nproc\n.\nstart\n(\n)\n# instantiating process with arguments\nfor\nname\nin\nnames\n:\n# print(name)\nproc\n=\nProcess\n(\ntarget\n=\nprint_func\n,\nargs\n=\n(\nname\n,\n)\n)\nprocs\n.\nappend\n(\nproc\n)\nproc\n.\nstart\n(\n)\n# complete the processes\nfor\nproc\nin\nprocs\n:\nproc\n.\njoin\n(\n)\nThe output of the following code will be:\nPython multiprocessing Queue class\nYou have basic knowledge about computer data-structure, you probably know about Queue. Python Multiprocessing modules provides\nQueue\nclass that is exactly a\nFirst-In-First-Out\ndata structure. They can store any pickle Python object (though simple ones are best) and are extremely useful for sharing data between processes. Queues are specially useful when passed as a parameter to a Process’ target function to enable the Process to consume data. By using\nput()\nfunction we can insert data to then queue and using\nget()\nwe can get items from queues. See the following code for a quick example.\nfrom\nmultiprocessing\nimport\nQueue\n\ncolors\n=\n[\n'red'\n,\n'green'\n,\n'blue'\n,\n'black'\n]\ncnt\n=\n1\n# instantiating a queue object\nqueue\n=\nQueue\n(\n)\nprint\n(\n'pushing items to queue:'\n)\nfor\ncolor\nin\ncolors\n:\nprint\n(\n'item no: '\n,\ncnt\n,\n' '\n,\ncolor\n)\nqueue\n.\nput\n(\ncolor\n)\ncnt\n+=\n1\nprint\n(\n'\\npopping items from queue:'\n)\ncnt\n=\n0\nwhile\nnot\nqueue\n.\nempty\n(\n)\n:\nprint\n(\n'item no: '\n,\ncnt\n,\n' '\n,\nqueue\n.\nget\n(\n)\n)\ncnt\n+=\n1\nPython multiprocessing Lock Class\nThe task of Lock class is quite simple. It allows code to claim lock so that no other process can execute the similar code until the lock has be released. So the task of Lock class is mainly two. One is to claim lock and other is to release the lock. To claim lock the,\nacquire()\nfunction is used and to release lock\nrelease()\nfunction is used.\nPython multiprocessing example\nIn this Python multiprocessing example, we will merge all our knowledge together. Suppose we have some tasks to accomplish. To get that task done, we will use several processes. So, we will maintain two queue. One will contain the tasks and the other will contain the log of completed task. Then we instantiate the processes to complete the task. Note that the python Queue class is already synchronized. That means, we don’t need to use the Lock class to block multiple process to access the same queue object. That’s why, we don’t need to use Lock class in this case. Below is the implementation where we are adding tasks to the queue, then creating processes and starting them, then using\njoin()\nto complete the processes. Finally we are printing the log from the second queue.\nfrom\nmultiprocessing\nimport\nLock\n,\nProcess\n,\nQueue\n,\ncurrent_process\nimport\ntime\nimport\nqueue\n# imported for using queue.Empty exception\ndef\ndo_job\n(\ntasks_to_accomplish\n,\ntasks_that_are_done\n)\n:\nwhile\nTrue\n:\ntry\n:\n'''\n                try to get task from the queue. get_nowait() function will \n                raise queue.Empty exception if the queue is empty. \n                queue(False) function would do the same task also.\n            '''\ntask\n=\ntasks_to_accomplish\n.\nget_nowait\n(\n)\nexcept\nqueue\n.\nEmpty\n:\nbreak\nelse\n:\n'''\n                if no exception has been raised, add the task completion \n                message to task_that_are_done queue\n            '''\nprint\n(\ntask\n)\ntasks_that_are_done\n.\nput\n(\ntask\n+\n' is done by '\n+\ncurrent_process\n(\n)\n.\nname\n)\ntime\n.\nsleep\n(\n.5\n)\nreturn\nTrue\ndef\nmain\n(\n)\n:\nnumber_of_task\n=\n10\nnumber_of_processes\n=\n4\ntasks_to_accomplish\n=\nQueue\n(\n)\ntasks_that_are_done\n=\nQueue\n(\n)\nprocesses\n=\n[\n]\nfor\ni\nin\nrange\n(\nnumber_of_task\n)\n:\ntasks_to_accomplish\n.\nput\n(\n\"Task no \"\n+\nstr\n(\ni\n)\n)\n# creating processes\nfor\nw\nin\nrange\n(\nnumber_of_processes\n)\n:\np\n=\nProcess\n(\ntarget\n=\ndo_job\n,\nargs\n=\n(\ntasks_to_accomplish\n,\ntasks_that_are_done\n)\n)\nprocesses\n.\nappend\n(\np\n)\np\n.\nstart\n(\n)\n# completing process\nfor\np\nin\nprocesses\n:\np\n.\njoin\n(\n)\n# print the output\nwhile\nnot\ntasks_that_are_done\n.\nempty\n(\n)\n:\nprint\n(\ntasks_that_are_done\n.\nget\n(\n)\n)\nreturn\nTrue\nif\n__name__\n==\n'__main__'\n:\nmain\n(\n)\nDepending on the number of task, the code will take some time to show you the output. The output of the following code will vary from time to time.\nPython multiprocessing Pool\nPython multiprocessing Pool can be used for parallel execution of a function across multiple input values, distributing the input data across processes (data parallelism). Below is a simple Python multiprocessing Pool example.\nfrom\nmultiprocessing\nimport\nPool\nimport\ntime\n\nwork\n=\n(\n[\n\"A\"\n,\n5\n]\n,\n[\n\"B\"\n,\n2\n]\n,\n[\n\"C\"\n,\n1\n]\n,\n[\n\"D\"\n,\n3\n]\n)\ndef\nwork_log\n(\nwork_data\n)\n:\nprint\n(\n\" Process %s waiting %s seconds\"\n%\n(\nwork_data\n[\n0\n]\n,\nwork_data\n[\n1\n]\n)\n)\ntime\n.\nsleep\n(\nint\n(\nwork_data\n[\n1\n]\n)\n)\nprint\n(\n\" Process %s Finished.\"\n%\nwork_data\n[\n0\n]\n)\ndef\npool_handler\n(\n)\n:\np\n=\nPool\n(\n2\n)\np\n.\nmap\n(\nwork_log\n,\nwork\n)\nif\n__name__\n==\n'__main__'\n:\npool_handler\n(\n)\nBelow image shows the output of the above program. Notice that pool size is 2, so two executions of\nwork_log\nfunction is happening in parallel. When one of the function processing finishes, it picks the next argument and so on.\nSo, that’s all for python multiprocessing module. Reference:\nOfficial Documentation","attrs_markdown":"[Now Available: OpenAI GPT-5.4 on the Gradient™ AI Platform](https:\/\/www.digitalocean.com\/blog\/whats-new-on-gradient-ai-platform#week-of-march-2)\n\n[Now Available: OpenAI GPT-5.3 Codex on the Gradient™ AI Platform](https:\/\/www.digitalocean.com\/blog\/whats-new-on-gradient-ai-platform#week-of-february-23rd)\n\n[Now Available: Claude Sonnet 4.6 on the Gradient™ AI Platform](https:\/\/www.digitalocean.com\/blog\/whats-new-on-gradient-ai-platform#week-of-february-17th)\n\n[Heroku is shifting to sustaining engineering. Migrate your workloads to DigitalOcean](https:\/\/www.digitalocean.com\/blog\/migrate-heroku-to-digitalocean)\n\n[Now Available: Claude Opus 4.6 on Gradient™ AI Platform](https:\/\/www.digitalocean.com\/blog\/claude-opus-4-6-gradient-ai-platform)\n\n[🔥 Run OpenClaw on DigitalOcean App Platform](https:\/\/www.digitalocean.com\/blog\/openclaw-digitalocean-app-platform)\n\n[Join us for Deploy in San Francisco, on April 28, 2026. Register now](https:\/\/www.digitalocean.com\/deploy)\n\n- [Blog](https:\/\/www.digitalocean.com\/blog)\n- [Docs](https:\/\/docs.digitalocean.com\/products)\n- [Careers](https:\/\/www.digitalocean.com\/careers)\n- [Get Support](https:\/\/www.digitalocean.com\/support)\n- [Contact Sales](https:\/\/www.digitalocean.com\/company\/contact\/sales?referrer=tophat)\n\n[DigitalOcean](https:\/\/www.digitalocean.com\/)\n\n- Products\n  - Featured Products\n    - [Droplets Scalable virtual machines](https:\/\/www.digitalocean.com\/products\/droplets)\n    - [Kubernetes Scale more effectively](https:\/\/www.digitalocean.com\/products\/kubernetes)\n    - [Gradient™ AI Inference Cloud Build and scale with AI](https:\/\/www.digitalocean.com\/products\/gradient)\n    - [Cloudways Managed cloud hosting](https:\/\/www.digitalocean.com\/products\/cloudways)\n    - [App Platform Get apps to market faster](https:\/\/www.digitalocean.com\/products\/app-platform)\n    - [Managed Databases Fully-managed database hosting](https:\/\/www.digitalocean.com\/products\/managed-databases)\n  \n  [See all products](https:\/\/www.digitalocean.com\/products)\n- Solutions\n  - - - [AI and Machine Learning](https:\/\/www.digitalocean.com\/products\/gradient)\n        \n        Develop, train, and deploy AI apps\n        - [GPUs](https:\/\/www.digitalocean.com\/products\/gradient\/gpu-droplets)\n        - [Platform](https:\/\/www.digitalocean.com\/products\/gradient\/platform)\n        - [1-Click Models](https:\/\/www.digitalocean.com\/products\/gradient\/1-click-models)\n        - [HR Knowledge Assistant](https:\/\/www.digitalocean.com\/solutions\/ai-hr-knowledge-assistant)\n        - [Code Copilot](https:\/\/www.digitalocean.com\/solutions\/ai-code-copilot)\n        - [Support Ticket Triage](https:\/\/www.digitalocean.com\/solutions\/ai-support-ticket-triage)\n        - [Recommendation Engine](https:\/\/www.digitalocean.com\/solutions\/ai-recommendation-engine)\n      - [Blockchain](https:\/\/www.digitalocean.com\/solutions\/blockchain)\n        \n        Infrastructure for decentralized apps\n      - [Blogs, Forums and Content Websites](https:\/\/www.digitalocean.com\/solutions\/content-hosting)\n        \n        Lightning-fast, reliable CMS hosting\n        - [Wordpress](https:\/\/www.digitalocean.com\/solutions\/wordpress-hosting)\n        - [Ghost](https:\/\/marketplace.digitalocean.com\/apps\/ghost)\n        - [Mastodon](https:\/\/marketplace.digitalocean.com\/apps\/mastodon)\n      - [Data Analytics](https:\/\/www.digitalocean.com\/solutions\/data-analytics)\n        \n        Real-time data processing at scale\n        - [Data Streaming](https:\/\/www.digitalocean.com\/solutions\/data-streaming-cloud)\n        - [AdTech & Martech](https:\/\/www.digitalocean.com\/solutions\/adtech-martech)\n        - [Kafka](https:\/\/www.digitalocean.com\/products\/managed-databases-kafka?utm_source=top-nav-data-analytics)\n    - - [Developer Tools](https:\/\/www.digitalocean.com\/solutions\/developer-tools)\n        \n        DevOps and CI\/CD solutions\n        - [CI\/CD](https:\/\/www.digitalocean.com\/solutions\/cicd-pipelines)\n        - [Prototyping](https:\/\/www.digitalocean.com\/solutions\/prototype-hosting)\n      - [Digital Marketing Agencies](https:\/\/www.digitalocean.com\/solutions\/digital-marketing-agencies)\n        \n        Power your clients’ websites and campaigns\n        - [Freelancer](https:\/\/www.digitalocean.com\/solutions\/freelancer-website-hosting)\n        - [IT Consulting](https:\/\/www.digitalocean.com\/solutions\/it-consulting)\n      - [Ecommerce](https:\/\/www.digitalocean.com\/solutions\/ecommerce-hosting)\n        \n        Build beautiful online storefronts\n        - [Dropshipping](https:\/\/www.digitalocean.com\/solutions\/dropshipping-hosting)\n        - [WooCommerce](https:\/\/marketplace.digitalocean.com\/apps\/wordpress-woocommerce)\n        - [Magento](https:\/\/marketplace.digitalocean.com\/apps\/magento-2-open-source)\n      - [Game Development](https:\/\/www.digitalocean.com\/solutions\/gaming-development)\n        \n        Low-latency multiplayer servers\n        - [Minecraft Hosting](https:\/\/marketplace.digitalocean.com\/apps\/minecraft-java-edition-server)\n    - - [IoT](https:\/\/www.digitalocean.com\/solutions\/iot-cloud)\n        \n        Connect to the power of the cloud\n        - [Kafka](https:\/\/www.digitalocean.com\/products\/managed-databases-kafka?utm_source=top-nav-iot)\n      - [ISVs](https:\/\/www.digitalocean.com\/solutions\/isv)\n        \n        Streamlined ISV application development\n      - [Secure Web Hosting](https:\/\/www.digitalocean.com\/solutions\/secure-web-hosting)\n        \n        Powerful protection from DDoS and more\n        - [Private VPN](https:\/\/www.digitalocean.com\/solutions\/vpn)\n      - [Startup Cloud Hosting](https:\/\/www.digitalocean.com\/solutions\/startups)\n        \n        Scalable, cost-effective infrastructure\n        - [Small Business](https:\/\/www.digitalocean.com\/solutions\/small-business-website-hosting)\n      - [Video Streaming](https:\/\/www.digitalocean.com\/solutions\/streaming)\n        \n        High-bandwidth, low-latency delivery\n        - [Kafka](https:\/\/www.digitalocean.com\/products\/managed-databases-kafka?utm_source=top-nav-video-streaming)\n    - - [Web and Mobile Apps](https:\/\/www.digitalocean.com\/solutions\/web-mobile-apps)\n        \n        Simple cross-platform app hosting\n        - [cPanel](https:\/\/www.digitalocean.com\/solutions\/cpanel-hosting)\n        - [Docker](https:\/\/www.digitalocean.com\/solutions\/docker-hosting)\n        - [Next.js](https:\/\/www.digitalocean.com\/solutions\/nextjs-hosting)\n        - [Node.js](https:\/\/www.digitalocean.com\/solutions\/nodejs-hosting)\n      - [Website Hosting](https:\/\/www.digitalocean.com\/solutions\/website-hosting)\n        \n        Fast page loads and reliable site uptime\n        - [VPS Hosting](https:\/\/www.digitalocean.com\/solutions\/vps-hosting)\n        - [Virtual Machines](https:\/\/www.digitalocean.com\/solutions\/virtual-machines)\n  \n  [See all solutions](https:\/\/www.digitalocean.com\/business)\n- Developers\n  - Our Community\n    - [Community Home DevOps and development guides](https:\/\/www.digitalocean.com\/community)\n    - [CSS-Tricks All things web design](https:\/\/css-tricks.com\/)\n    - [The Wave Content to level up your business.](https:\/\/www.digitalocean.com\/resources)\n- Partners\n  - DigitalOcean Partner Programs\n    - [Become a Partner](https:\/\/www.digitalocean.com\/partners\/pod)\n    - [Partner Services Program](https:\/\/www.digitalocean.com\/partners\/services)\n    - [DigitalOcean AI Partner Program](https:\/\/www.digitalocean.com\/partners\/ai-partner-program)\n    - [Marketplace](https:\/\/marketplace.digitalocean.com\/)\n    - [DigitalOcean Startups](https:\/\/www.digitalocean.com\/startups)\n    - [Connect with a Partner](https:\/\/www.digitalocean.com\/partners\/directory)\n- [Pricing](https:\/\/www.digitalocean.com\/pricing)\n\n- Log in\n  - [Community](https:\/\/www.digitalocean.com\/api\/dynamic-content\/v1\/login?success_redirect=https%3A%2F%2Fwww.digitalocean.com%2Fcommunity%2Ftutorials%2Fpython-multiprocessing-example&error_redirect=https%3A%2F%2Fwww.digitalocean.com%2Fauth-error&type=login)\n  - [DigitalOcean](https:\/\/cloud.digitalocean.com\/login)\n- Sign up\n  - [Community](https:\/\/www.digitalocean.com\/api\/dynamic-content\/v1\/login?success_redirect=https%3A%2F%2Fwww.digitalocean.com%2Fcommunity%2Ftutorials%2Fpython-multiprocessing-example&error_redirect=https%3A%2F%2Fwww.digitalocean.com%2Fauth-error&type=register)\n  - [DigitalOcean](https:\/\/cloud.digitalocean.com\/registrations\/new)\n\n- [Blog](https:\/\/www.digitalocean.com\/blog)\n- [Docs](https:\/\/docs.digitalocean.com\/products)\n- [Careers](https:\/\/www.digitalocean.com\/careers)\n- [Get Support](https:\/\/www.digitalocean.com\/support)\n- [Contact Sales](https:\/\/www.digitalocean.com\/company\/contact\/sales?referrer=tophat)\n\n- Log in\n  - [Community](https:\/\/www.digitalocean.com\/api\/dynamic-content\/v1\/login?success_redirect=https%3A%2F%2Fwww.digitalocean.com%2Fcommunity%2Ftutorials%2Fpython-multiprocessing-example&error_redirect=https%3A%2F%2Fwww.digitalocean.com%2Fauth-error&type=login)\n  - [DigitalOcean](https:\/\/cloud.digitalocean.com\/login)\n- Sign up\n  - [Community](https:\/\/www.digitalocean.com\/api\/dynamic-content\/v1\/login?success_redirect=https%3A%2F%2Fwww.digitalocean.com%2Fcommunity%2Ftutorials%2Fpython-multiprocessing-example&error_redirect=https%3A%2F%2Fwww.digitalocean.com%2Fauth-error&type=register)\n  - [DigitalOcean](https:\/\/cloud.digitalocean.com\/registrations\/new)\n\n- [Tutorials](https:\/\/www.digitalocean.com\/community\/tutorials)\n- [Questions](https:\/\/www.digitalocean.com\/community\/questions)\n- [Product Docs](https:\/\/docs.digitalocean.com\/)\n- Search Community\n\n## Report this\nWhat is the reason for this report?\n\nThis undefined is spam\n\nThis undefined is offensive\n\nThis undefined is off-topic\n\nThis undefined is other\n\nSubmit\n\n## Table of contents\n1. [Python Multiprocessing](https:\/\/www.digitalocean.com\/community\/tutorials\/python-multiprocessing-example#python-multiprocessing)\n\n1. [Tutorials](https:\/\/www.digitalocean.com\/community\/tutorials?subtype=tutorial)\n2. [Python](https:\/\/www.digitalocean.com\/community\/tags\/python)\n3. Python Multiprocessing Example\n\n[Tutorial](https:\/\/www.digitalocean.com\/community\/tutorials?subtype=tutorial)\n\n# Python Multiprocessing Example\nPublished on August 3, 2022\n\n[Python](https:\/\/www.digitalocean.com\/community\/tags\/python)\n\n![Anish Singh Walia](https:\/\/www.gravatar.com\/avatar\/c91039d94d0ab6e82a10aaf08e3c1c341d5744c8533ff6d077ee4a765246d8d3?default=retro&size=256)![Pankaj Kumar](https:\/\/www.gravatar.com\/avatar\/dcb62410bd20fd354c21af393bec6e67b235a09758848745cfd4405e05bba1f4?default=retro&size=256)\n\nBy [Anish Singh Walia](https:\/\/www.digitalocean.com\/community\/users\/asinghwalia) and [Pankaj Kumar](https:\/\/www.digitalocean.com\/community\/users\/pankajkumarjournaldev)\n\n![Python Multiprocessing Example](https:\/\/www.digitalocean.com\/api\/static-content\/v1\/images?src=%2F_next%2Fstatic%2Fmedia%2Fintro-to-cloud.d49bc5f7.jpeg&width=1920)\n\nTable of contents\n\nPopular topics\n\nIn our previous tutorial, we learned about [Python CSV Example](https:\/\/www.digitalocean.com\/community\/tutorials\/parse-csv-files-in-python). In this tutorial we are going to learn Python Multiprocessing with examples.\n\n## [Python Multiprocessing](https:\/\/www.digitalocean.com\/community\/tutorials\/python-multiprocessing-example#python-multiprocessing)\nParallel processing is getting more attention nowadays. If you still don’t know about the parallel processing, learn from [wikipedia](https:\/\/en.wikipedia.org\/wiki\/Parallel_processing). As CPU manufacturers start adding more and more cores to their processors, creating parallel code is a great way to improve performance. Python introduced **multiprocessing** module to let us write parallel code. To understand the main motivation of this module, we have to know some basics about parallel programming. After reading this article, we hope that, you would be able to gather some knowledge on this topic.\n\n### [Python Multiprocessing Process, Queue and Locks](https:\/\/www.digitalocean.com\/community\/tutorials\/python-multiprocessing-example#python-multiprocessing-process-queue-and-locks)\nThere are plenty of classes in python multiprocessing module for building a parallel program. Among them, three basic classes are `Process`, `Queue` and `Lock`. These classes will help you to build a parallel program. But before describing about those, let us initiate this topic with simple code. To make a parallel program useful, you have to know how many cores are there in you pc. Python Multiprocessing module enables you to know that. The following simple code will print the number of cores in your pc.\n\n```\nimport multiprocessing\n\nprint(\"Number of cpu : \", multiprocessing.cpu_count())\n```\nCopy\n\nThe following output may vary for your pc. For me, number of cores is 8. ![python multiprocessing example cpu count](https:\/\/journaldev.nyc3.cdn.digitaloceanspaces.com\/2017\/09\/python-multiprocessing-cpu-count.png)\n\n### [Python multiprocessing Process class](https:\/\/www.digitalocean.com\/community\/tutorials\/python-multiprocessing-example#python-multiprocessing-process-class)\nPython multiprocessing `Process` class is an abstraction that sets up another Python process, provides it to run code and a way for the parent application to control execution. There are two important functions that belongs to the Process class - `start()` and `join()` function. At first, we need to write a function, that will be run by the process. Then, we need to instantiate a process object. If we create a process object, nothing will happen until we tell it to start processing via `start()` function. Then, the process will run and return its result. After that we tell the process to complete via `join()` function. Without `join()` function call, process will remain idle and won’t terminate. So if you create many processes and don’t terminate them, you may face scarcity of resources. Then you may need to kill them manually. One important thing is, if you want to pass any argument through the process you need to use `args` keyword argument. The following code will be helpful to understand the usage of Process class.\n\n```\nfrom multiprocessing import Process\n\n\ndef print_func(continent='Asia'):\n    print('The name of continent is : ', continent)\n\nif __name__ == \"__main__\":  # confirms that the code is under main function\n    names = ['America', 'Europe', 'Africa']\n    procs = []\n    proc = Process(target=print_func)  # instantiating without any argument\n    procs.append(proc)\n    proc.start()\n\n    # instantiating process with arguments\n    for name in names:\n        # print(name)\n        proc = Process(target=print_func, args=(name,))\n        procs.append(proc)\n        proc.start()\n\n    # complete the processes\n    for proc in procs:\n        proc.join()\n```\nCopy\n\nThe output of the following code will be: ![python multiprocessing example process class](https:\/\/journaldev.nyc3.cdn.digitaloceanspaces.com\/2017\/09\/python-multiprocessing-process.png)\n\n### [Python multiprocessing Queue class](https:\/\/www.digitalocean.com\/community\/tutorials\/python-multiprocessing-example#python-multiprocessing-queue-class)\nYou have basic knowledge about computer data-structure, you probably know about Queue. Python Multiprocessing modules provides `Queue` class that is exactly a **First-In-First-Out** data structure. They can store any pickle Python object (though simple ones are best) and are extremely useful for sharing data between processes. Queues are specially useful when passed as a parameter to a Process’ target function to enable the Process to consume data. By using `put()` function we can insert data to then queue and using `get()` we can get items from queues. See the following code for a quick example.\n\n```\nfrom multiprocessing import Queue\n\ncolors = ['red', 'green', 'blue', 'black']\ncnt = 1\n# instantiating a queue object\nqueue = Queue()\nprint('pushing items to queue:')\nfor color in colors:\n    print('item no: ', cnt, ' ', color)\n    queue.put(color)\n    cnt += 1\n\nprint('\\npopping items from queue:')\ncnt = 0\nwhile not queue.empty():\n    print('item no: ', cnt, ' ', queue.get())\n    cnt += 1\n```\nCopy\n\n![python multiprocessing queue](https:\/\/journaldev.nyc3.cdn.digitaloceanspaces.com\/2017\/09\/python-multiprocessing-queue.png)\n\n### [Python multiprocessing Lock Class](https:\/\/www.digitalocean.com\/community\/tutorials\/python-multiprocessing-example#python-multiprocessing-lock-class)\nThe task of Lock class is quite simple. It allows code to claim lock so that no other process can execute the similar code until the lock has be released. So the task of Lock class is mainly two. One is to claim lock and other is to release the lock. To claim lock the, `acquire()` function is used and to release lock `release()` function is used.\n\n### [Python multiprocessing example](https:\/\/www.digitalocean.com\/community\/tutorials\/python-multiprocessing-example#python-multiprocessing-example)\nIn this Python multiprocessing example, we will merge all our knowledge together. Suppose we have some tasks to accomplish. To get that task done, we will use several processes. So, we will maintain two queue. One will contain the tasks and the other will contain the log of completed task. Then we instantiate the processes to complete the task. Note that the python Queue class is already synchronized. That means, we don’t need to use the Lock class to block multiple process to access the same queue object. That’s why, we don’t need to use Lock class in this case. Below is the implementation where we are adding tasks to the queue, then creating processes and starting them, then using `join()` to complete the processes. Finally we are printing the log from the second queue.\n\n```\nfrom multiprocessing import Lock, Process, Queue, current_process\nimport time\nimport queue # imported for using queue.Empty exception\n\n\ndef do_job(tasks_to_accomplish, tasks_that_are_done):\n    while True:\n        try:\n            '''\n                try to get task from the queue. get_nowait() function will \n                raise queue.Empty exception if the queue is empty. \n                queue(False) function would do the same task also.\n            '''\n            task = tasks_to_accomplish.get_nowait()\n        except queue.Empty:\n\n            break\n        else:\n            '''\n                if no exception has been raised, add the task completion \n                message to task_that_are_done queue\n            '''\n            print(task)\n            tasks_that_are_done.put(task + ' is done by ' + current_process().name)\n            time.sleep(.5)\n    return True\n\n\ndef main():\n    number_of_task = 10\n    number_of_processes = 4\n    tasks_to_accomplish = Queue()\n    tasks_that_are_done = Queue()\n    processes = []\n\n    for i in range(number_of_task):\n        tasks_to_accomplish.put(\"Task no \" + str(i))\n\n    # creating processes\n    for w in range(number_of_processes):\n        p = Process(target=do_job, args=(tasks_to_accomplish, tasks_that_are_done))\n        processes.append(p)\n        p.start()\n\n    # completing process\n    for p in processes:\n        p.join()\n\n    # print the output\n    while not tasks_that_are_done.empty():\n        print(tasks_that_are_done.get())\n\n    return True\n\n\nif __name__ == '__main__':\n    main()\n```\nCopy\n\nDepending on the number of task, the code will take some time to show you the output. The output of the following code will vary from time to time. ![python multiprocessing example](https:\/\/journaldev.nyc3.cdn.digitaloceanspaces.com\/2017\/09\/python-multiprocessing-example-1.png)\n\n### [Python multiprocessing Pool](https:\/\/www.digitalocean.com\/community\/tutorials\/python-multiprocessing-example#python-multiprocessing-pool)\nPython multiprocessing Pool can be used for parallel execution of a function across multiple input values, distributing the input data across processes (data parallelism). Below is a simple Python multiprocessing Pool example.\n\n```\nfrom multiprocessing import Pool\n\nimport time\n\nwork = ([\"A\", 5], [\"B\", 2], [\"C\", 1], [\"D\", 3])\n\n\ndef work_log(work_data):\n    print(\" Process %s waiting %s seconds\" % (work_data[0], work_data[1]))\n    time.sleep(int(work_data[1]))\n    print(\" Process %s Finished.\" % work_data[0])\n\n\ndef pool_handler():\n    p = Pool(2)\n    p.map(work_log, work)\n\n\nif __name__ == '__main__':\n    pool_handler()\n```\nCopy\n\nBelow image shows the output of the above program. Notice that pool size is 2, so two executions of `work_log` function is happening in parallel. When one of the function processing finishes, it picks the next argument and so on. ![python multiprocessing pool](https:\/\/journaldev.nyc3.cdn.digitaloceanspaces.com\/2017\/09\/python-multiprocessing-pool.png) So, that’s all for python multiprocessing module. Reference: [Official Documentation](https:\/\/docs.python.org\/3.6\/library\/multiprocessing.html)\n\nThanks for learning with the DigitalOcean Community. Check out our offerings for compute, storage, networking, and managed databases.\n\n[Learn more about our products](https:\/\/www.digitalocean.com\/products \"Learn more about our products\")\n\n### About the author(s)\n![Anish Singh Walia](https:\/\/www.gravatar.com\/avatar\/c91039d94d0ab6e82a10aaf08e3c1c341d5744c8533ff6d077ee4a765246d8d3?default=retro&size=256)\n\nAnish Singh Walia\n\nAuthor\n\nSr Technical Writer and Team Lead\n\n[See author profile](https:\/\/www.digitalocean.com\/community\/users\/asinghwalia)\n\nI help Businesses scale with AI x SEO x (authentic) Content that revives traffic and keeps leads flowing \\| 3,000,000+ Average monthly readers on Medium \\| Sr Technical Writer(Team Lead) @ DigitalOcean \\| Ex-Cloud Consultant @ AMEX \\| Ex-Site Reliability Engineer(DevOps)@Nutanix\n\n[See author profile](https:\/\/www.digitalocean.com\/community\/users\/asinghwalia)\n\n![Pankaj Kumar](https:\/\/www.gravatar.com\/avatar\/dcb62410bd20fd354c21af393bec6e67b235a09758848745cfd4405e05bba1f4?default=retro&size=256)\n\nPankaj Kumar\n\nAuthor\n\n[See author profile](https:\/\/www.digitalocean.com\/community\/users\/pankajkumarjournaldev)\n\nJava and Python Developer for 20+ years, Open Source Enthusiast, Founder of https:\/\/www.askpython.com\/, https:\/\/www.linuxfordevices.com\/, and JournalDev.com (acquired by DigitalOcean). Passionate about writing technical articles and sharing knowledge with others. Love Java, Python, Unix and related technologies. Follow my X @PankajWebDev\n\n[See author profile](https:\/\/www.digitalocean.com\/community\/users\/pankajkumarjournaldev)\n\nCategory:\n\n[Tutorial](https:\/\/www.digitalocean.com\/community\/tutorials?subtype=tutorial)\n\nTags:\n\n[Python](https:\/\/www.digitalocean.com\/community\/tags\/python)\n\nWhile we believe that this content benefits our community, we have not yet thoroughly reviewed it. If you have any suggestions for improvements, please let us know by clicking the “report an issue“ button at the bottom of the tutorial.\n\n#### Still looking for an answer?\n[Ask a question](https:\/\/www.digitalocean.com\/community\/questions)[Search for more help](https:\/\/www.digitalocean.com\/community)\n\nWas this helpful?\n\nYes\n\nNo\n\nComments(10)\n\nFollow-up questions(0)\n\n![JournalDev](https:\/\/www.gravatar.com\/avatar\/7c7a4546c8430a5a91a338a177bf8e7e8e153fe7eaee076acfa6d46029efef49?default=retro)\n\n[JournalDev](https:\/\/www.digitalocean.com\/community\/users\/journaldev)![DigitalOcean Employee badge](https:\/\/www.digitalocean.com\/api\/static-content\/v1\/images?src=%2F_next%2Fstatic%2Fmedia%2Fdo-logo.15453cf1.svg&width=32)\n\n[July 15, 2018](https:\/\/www.digitalocean.com\/community\/tutorials\/python-multiprocessing-example?comment=177730)\n\nShow less\n\nHi You wrote “Without join() function call, process will remain idle and won’t terminate”. That is not correct., the process will terminate with or without join(). The difference is: -With join(): the program will wait for every process to finish -Without join(): each process will run without waiting for other processes to finish’ Thks\n\n\\- namta\n\n(1) replies\n\n![JournalDev](https:\/\/www.gravatar.com\/avatar\/7c7a4546c8430a5a91a338a177bf8e7e8e153fe7eaee076acfa6d46029efef49?default=retro)\n\n[JournalDev](https:\/\/www.digitalocean.com\/community\/users\/journaldev)![DigitalOcean Employee badge](https:\/\/www.digitalocean.com\/api\/static-content\/v1\/images?src=%2F_next%2Fstatic%2Fmedia%2Fdo-logo.15453cf1.svg&width=32)\n\n[September 27, 2018](https:\/\/www.digitalocean.com\/community\/tutorials\/python-multiprocessing-example?comment=177731)\n\nShow less\n\nWhy does the start() must be called within main function ?\n\n\\- xram\n\n![JournalDev](https:\/\/www.gravatar.com\/avatar\/7c7a4546c8430a5a91a338a177bf8e7e8e153fe7eaee076acfa6d46029efef49?default=retro)\n\n[JournalDev](https:\/\/www.digitalocean.com\/community\/users\/journaldev)![DigitalOcean Employee badge](https:\/\/www.digitalocean.com\/api\/static-content\/v1\/images?src=%2F_next%2Fstatic%2Fmedia%2Fdo-logo.15453cf1.svg&width=32)\n\n[February 7, 2019](https:\/\/www.digitalocean.com\/community\/tutorials\/python-multiprocessing-example?comment=177732)\n\nShow less\n\nIn the do\\_job function ,if there other method to replace time.sleep? For some function ,you may need not to know the possible time to sleep.\n\n\\- RUI ZHANG\n\n![JournalDev](https:\/\/www.gravatar.com\/avatar\/7c7a4546c8430a5a91a338a177bf8e7e8e153fe7eaee076acfa6d46029efef49?default=retro)\n\n[JournalDev](https:\/\/www.digitalocean.com\/community\/users\/journaldev)![DigitalOcean Employee badge](https:\/\/www.digitalocean.com\/api\/static-content\/v1\/images?src=%2F_next%2Fstatic%2Fmedia%2Fdo-logo.15453cf1.svg&width=32)\n\n[February 11, 2019](https:\/\/www.digitalocean.com\/community\/tutorials\/python-multiprocessing-example?comment=177733)\n\nShow less\n\nThank you but I think there isn’t enough explanation.\n\n\\- Emre ATAKLI\n\n![JournalDev](https:\/\/www.gravatar.com\/avatar\/7c7a4546c8430a5a91a338a177bf8e7e8e153fe7eaee076acfa6d46029efef49?default=retro)\n\n[JournalDev](https:\/\/www.digitalocean.com\/community\/users\/journaldev)![DigitalOcean Employee badge](https:\/\/www.digitalocean.com\/api\/static-content\/v1\/images?src=%2F_next%2Fstatic%2Fmedia%2Fdo-logo.15453cf1.svg&width=32)\n\n[June 11, 2019](https:\/\/www.digitalocean.com\/community\/tutorials\/python-multiprocessing-example?comment=177734)\n\nShow less\n\nHow to Kill the Proc? is Proc.terminate() works best\n\n\\- Senthilkumar Rajendran\n\n![JournalDev](https:\/\/www.gravatar.com\/avatar\/7c7a4546c8430a5a91a338a177bf8e7e8e153fe7eaee076acfa6d46029efef49?default=retro)\n\n[JournalDev](https:\/\/www.digitalocean.com\/community\/users\/journaldev)![DigitalOcean Employee badge](https:\/\/www.digitalocean.com\/api\/static-content\/v1\/images?src=%2F_next%2Fstatic%2Fmedia%2Fdo-logo.15453cf1.svg&width=32)\n\n[June 20, 2019](https:\/\/www.digitalocean.com\/community\/tutorials\/python-multiprocessing-example?comment=177735)\n\nShow less\n\nGreat article man ! Simple and very informative. Even official python documentation for multiprocessing is not this simple and easy to read.\n\n\\- Ashish Dhiman\n\n![JournalDev](https:\/\/www.gravatar.com\/avatar\/7c7a4546c8430a5a91a338a177bf8e7e8e153fe7eaee076acfa6d46029efef49?default=retro)\n\n[JournalDev](https:\/\/www.digitalocean.com\/community\/users\/journaldev)![DigitalOcean Employee badge](https:\/\/www.digitalocean.com\/api\/static-content\/v1\/images?src=%2F_next%2Fstatic%2Fmedia%2Fdo-logo.15453cf1.svg&width=32)\n\n[July 12, 2019](https:\/\/www.digitalocean.com\/community\/tutorials\/python-multiprocessing-example?comment=177736)\n\nShow less\n\nVery easy to understand and informative! Thanks a lot man\\!\n\n\\- Kenny\n\n![JournalDev](https:\/\/www.gravatar.com\/avatar\/7c7a4546c8430a5a91a338a177bf8e7e8e153fe7eaee076acfa6d46029efef49?default=retro)\n\n[JournalDev](https:\/\/www.digitalocean.com\/community\/users\/journaldev)![DigitalOcean Employee badge](https:\/\/www.digitalocean.com\/api\/static-content\/v1\/images?src=%2F_next%2Fstatic%2Fmedia%2Fdo-logo.15453cf1.svg&width=32)\n\n[February 5, 2020](https:\/\/www.digitalocean.com\/community\/tutorials\/python-multiprocessing-example?comment=177737)\n\nShow less\n\nThanks, your two way communication example helped me finally close in on my bug. One that your code will have as well if number or size of messages in the queue fills the buffer used by it. ‘.get()’ needs to be called on the Queue being used to send messages back from the sub-process (child) to the main process (parent) before the ‘.join()’ call that blocks main until the sub-process has completed. What is not seen from glancing at the Python code is that underneath, the Queue depends on a limited size buffer, even if no maximum count is set. Because of that, if the get() and join() are out of order a deadlock can happen where the sub-process cannot exit because it cannot finish putting things in the queue, and the main process cannot continue to where it removes things from the queue because it is waiting on the sub-process to exit. “…you need to make sure that all items which have been put on the queue will eventually be removed before the process is joined.” - <https:\/\/docs.python.org\/3.8\/library\/multiprocessing.html#programming-guidelines>\n\n\\- Kevin Whalen\n\n![JournalDev](https:\/\/www.gravatar.com\/avatar\/7c7a4546c8430a5a91a338a177bf8e7e8e153fe7eaee076acfa6d46029efef49?default=retro)\n\n[JournalDev](https:\/\/www.digitalocean.com\/community\/users\/journaldev)![DigitalOcean Employee badge](https:\/\/www.digitalocean.com\/api\/static-content\/v1\/images?src=%2F_next%2Fstatic%2Fmedia%2Fdo-logo.15453cf1.svg&width=32)\n\n[May 8, 2020](https:\/\/www.digitalocean.com\/community\/tutorials\/python-multiprocessing-example?comment=177738)\n\nShow less\n\nThanks was helpful as always. The simple explanation is the strong point.\n\n\\- Sukesh Suvarna\n\n![JournalDev](https:\/\/www.gravatar.com\/avatar\/7c7a4546c8430a5a91a338a177bf8e7e8e153fe7eaee076acfa6d46029efef49?default=retro)\n\n[JournalDev](https:\/\/www.digitalocean.com\/community\/users\/journaldev)![DigitalOcean Employee badge](https:\/\/www.digitalocean.com\/api\/static-content\/v1\/images?src=%2F_next%2Fstatic%2Fmedia%2Fdo-logo.15453cf1.svg&width=32)\n\n[June 6, 2020](https:\/\/www.digitalocean.com\/community\/tutorials\/python-multiprocessing-example?comment=177739)\n\nShow less\n\nPlease help me to resolve this issue <https:\/\/stackoverflow.com\/q\/62237516\/13193575>\n\n\\- Lovely\n\nLoad more comments\n\n[![Creative Commons](https:\/\/www.digitalocean.com\/api\/static-content\/v1\/images?src=%2F_next%2Fstatic%2Fmedia%2Fcreativecommons.c0a877f1.png&width=384)](https:\/\/creativecommons.org\/licenses\/by-nc-sa\/4.0\/)This work is licensed under a Creative Commons Attribution-NonCommercial- ShareAlike 4.0 International License.\n\n## Deploy on DigitalOcean\nClick below to sign up for DigitalOcean's virtual machines, Databases, and AIML products.\n\n[Sign up](https:\/\/cloud.digitalocean.com\/registrations\/new?refcode=f6fcd01aaffb)\n\n## Popular Topics\n1. [AI\/ML](https:\/\/www.digitalocean.com\/community\/tags\/ai-ml)\n2. [Ubuntu](https:\/\/www.digitalocean.com\/community\/tags\/ubuntu)\n3. [Linux Basics](https:\/\/www.digitalocean.com\/community\/tags\/linux-basics)\n4. [JavaScript](https:\/\/www.digitalocean.com\/community\/tags\/javascript)\n5. [Python](https:\/\/www.digitalocean.com\/community\/tags\/python)\n6. [MySQL](https:\/\/www.digitalocean.com\/community\/tags\/mysql)\n7. [Docker](https:\/\/www.digitalocean.com\/community\/tags\/docker)\n8. [Kubernetes](https:\/\/www.digitalocean.com\/community\/tags\/kubernetes)\n9. [All tutorials](https:\/\/www.digitalocean.com\/community\/tutorials)\n10. [Talk to an expert](https:\/\/www.digitalocean.com\/company\/contact\/sales?referrer=tutorials)\n\n### Connect on Discord\nJoin the conversation in our Discord to connect with fellow developers\n\n[Visit Discord](https:\/\/discord.gg\/digitalocean)\n\n## Featured tutorials\n1. [SOLID Design Principles Explained: Building Better Software Architecture](https:\/\/www.digitalocean.com\/community\/tutorials\/s-o-l-i-d-the-first-five-principles-of-object-oriented-design)\n2. [How To Remove Docker Images, Containers, and Volumes](https:\/\/www.digitalocean.com\/community\/tutorials\/how-to-remove-docker-images-containers-and-volumes)\n3. [How to Create a MySQL User and Grant Privileges (Step-by-Step)](https:\/\/www.digitalocean.com\/community\/tutorials\/how-to-create-a-new-user-and-grant-permissions-in-mysql)\n\n- [All tutorials](https:\/\/www.digitalocean.com\/community\/tutorials)\n- [All topic tags](https:\/\/www.digitalocean.com\/community\/tags)\n##### Join the Tech Talk\n- [Python Multiprocessing](https:\/\/www.digitalocean.com\/community\/tutorials\/python-multiprocessing-example#python-multiprocessing)\n\n- ## Deploy on DigitalOcean\n  Click below to sign up for DigitalOcean's virtual machines, Databases, and AIML products.\n  \n  [Sign up](https:\/\/cloud.digitalocean.com\/registrations\/new?refcode=f6fcd01aaffb)\n  \n  ## Popular Topics\n  1. [AI\/ML](https:\/\/www.digitalocean.com\/community\/tags\/ai-ml)\n  2. [Ubuntu](https:\/\/www.digitalocean.com\/community\/tags\/ubuntu)\n  3. [Linux Basics](https:\/\/www.digitalocean.com\/community\/tags\/linux-basics)\n  4. [JavaScript](https:\/\/www.digitalocean.com\/community\/tags\/javascript)\n  5. [Python](https:\/\/www.digitalocean.com\/community\/tags\/python)\n  6. [MySQL](https:\/\/www.digitalocean.com\/community\/tags\/mysql)\n  7. [Docker](https:\/\/www.digitalocean.com\/community\/tags\/docker)\n  8. [Kubernetes](https:\/\/www.digitalocean.com\/community\/tags\/kubernetes)\n  9. [All tutorials](https:\/\/www.digitalocean.com\/community\/tutorials)\n  10. [Talk to an expert](https:\/\/www.digitalocean.com\/company\/contact\/sales?referrer=tutorials)\n  \n  ### Connect on Discord\n  Join the conversation in our Discord to connect with fellow developers\n  \n  [Visit Discord](https:\/\/discord.gg\/digitalocean)\n  \n  ## Featured tutorials\n  1. [SOLID Design Principles Explained: Building Better Software Architecture](https:\/\/www.digitalocean.com\/community\/tutorials\/s-o-l-i-d-the-first-five-principles-of-object-oriented-design)\n  2. [How To Remove Docker Images, Containers, and Volumes](https:\/\/www.digitalocean.com\/community\/tutorials\/how-to-remove-docker-images-containers-and-volumes)\n  3. [How to Create a MySQL User and Grant Privileges (Step-by-Step)](https:\/\/www.digitalocean.com\/community\/tutorials\/how-to-create-a-new-user-and-grant-permissions-in-mysql)\n  \n  - [All tutorials](https:\/\/www.digitalocean.com\/community\/tutorials)\n  - [All topic tags](https:\/\/www.digitalocean.com\/community\/tags)\n\n![](https:\/\/www.digitalocean.com\/api\/static-content\/v1\/images?src=%2F_next%2Fstatic%2Fmedia%2Ftutorials-2-tulip.764b9f59.svg&width=1920)\n\n## Become a contributor for community\nGet paid to write technical tutorials and select a tech-focused charity to receive a matching donation.\n\n[Sign Up](https:\/\/www.digitalocean.com\/community\/pages\/write-for-digitalocean)\n\n![](https:\/\/www.digitalocean.com\/api\/static-content\/v1\/images?src=%2F_next%2Fstatic%2Fmedia%2Fdocs-2-kiwi.239a03ef.svg&width=1920)\n\n## DigitalOcean Documentation\nFull documentation for every DigitalOcean product.\n\n[Learn more](https:\/\/docs.digitalocean.com\/)\n\n![](https:\/\/www.digitalocean.com\/api\/static-content\/v1\/images?src=%2F_next%2Fstatic%2Fmedia%2Fblogs-1-lavender.495d1f00.svg&width=1920)\n\n## Resources for startups and AI-native businesses\nThe Wave has everything you need to know about building a business, from raising funding to marketing your product.\n\n[Learn more](https:\/\/www.digitalocean.com\/resources)\n\n## Get our newsletter\nStay up to date by signing up for DigitalOcean’s Infrastructure as a Newsletter.\n\nNew accounts only. By submitting your email you agree to our [Privacy Policy](https:\/\/www.digitalocean.com\/legal\/privacy-policy)\n\n## The developer cloud\nScale up as you grow — whether you're running one virtual machine or ten thousand.\n\n[View all products](https:\/\/www.digitalocean.com\/products)\n\n![](https:\/\/www.digitalocean.com\/api\/static-content\/v1\/images?src=%2F_next%2Fstatic%2Fmedia%2Fclouds-mobile.5d14bead.svg&width=3840)\n\n## Get started for free\nSign up and get \\$200 in credit for your first 60 days with DigitalOcean.\\*\n\n[Get started](https:\/\/cloud.digitalocean.com\/registrations\/new?refcode=f6fcd01aaffb)\n\n![](https:\/\/www.digitalocean.com\/api\/static-content\/v1\/images?src=%2F_next%2Fstatic%2Fmedia%2Fwaves-mobile.a054c63e.svg&width=3840)\n\n\\*This promotional offer applies to new accounts only.\n\n## Company\n- [About](https:\/\/www.digitalocean.com\/about)\n- [Leadership](https:\/\/www.digitalocean.com\/leadership\/executive-management)\n- [Blog](https:\/\/www.digitalocean.com\/blog)\n- [Careers](https:\/\/www.digitalocean.com\/careers)\n- [Customers](https:\/\/www.digitalocean.com\/customers)\n- [Partners](https:\/\/www.digitalocean.com\/partners)\n- [Referral Program](https:\/\/www.digitalocean.com\/referral-program)\n- [Affiliate Program](https:\/\/www.digitalocean.com\/affiliates)\n- [Press](https:\/\/www.digitalocean.com\/press)\n- [Legal](https:\/\/www.digitalocean.com\/legal)\n- [Privacy Policy](https:\/\/www.digitalocean.com\/legal\/privacy-policy)\n- [Security](https:\/\/www.digitalocean.com\/security)\n- [Investor Relations](https:\/\/investors.digitalocean.com\/)\n\n## Products\n- [Overview](https:\/\/www.digitalocean.com\/products)\n- [Droplets](https:\/\/www.digitalocean.com\/products\/droplets)\n- [Kubernetes](https:\/\/www.digitalocean.com\/products\/kubernetes)\n- [Functions](https:\/\/www.digitalocean.com\/products\/functions)\n- [App Platform](https:\/\/www.digitalocean.com\/products\/app-platform)\n- [Gradient™ AI GPU Droplets](https:\/\/www.digitalocean.com\/products\/gradient\/gpu-droplets)\n- [Gradient™ AI Bare Metal GPUs](https:\/\/www.digitalocean.com\/products\/gradient\/bare-metal-gpus)\n- [Gradient™ AI 1-Click Models](https:\/\/www.digitalocean.com\/products\/gradient\/1-click-models)\n- [Gradient™ AI Platform](https:\/\/www.digitalocean.com\/products\/gradient\/platform)\n- [Load Balancers](https:\/\/www.digitalocean.com\/products\/load-balancers)\n- [Managed Databases](https:\/\/www.digitalocean.com\/products\/managed-databases)\n- [Spaces](https:\/\/www.digitalocean.com\/products\/spaces)\n- [Block Storage](https:\/\/www.digitalocean.com\/products\/block-storage)\n- [Network File Storage](https:\/\/www.digitalocean.com\/products\/storage\/network-file-storage)\n- [API](https:\/\/docs.digitalocean.com\/reference\/api)\n- [Uptime](https:\/\/www.digitalocean.com\/products\/uptime-monitoring)\n- [Identity and Access Management](https:\/\/www.digitalocean.com\/products\/identity-access-management)\n- [Cloudways](https:\/\/www.digitalocean.com\/products\/cloudways)\n\n## Resources\n- [Community Tutorials](https:\/\/www.digitalocean.com\/community\/tutorials)\n- [Community Q\\&A](https:\/\/www.digitalocean.com\/community\/questions)\n- [CSS-Tricks](https:\/\/css-tricks.com\/)\n- [Write for DOnations](https:\/\/www.digitalocean.com\/community\/pages\/write-for-digitalocean)\n- [Currents Research](https:\/\/www.digitalocean.com\/currents)\n- [DigitalOcean Startups](https:\/\/www.digitalocean.com\/startups)\n- [Wavemakers Program](https:\/\/www.digitalocean.com\/wavemakers)\n- [Compass Council](https:\/\/www.digitalocean.com\/research)\n- [Open Source](https:\/\/www.digitalocean.com\/open-source)\n- [Newsletter Signup](https:\/\/www.digitalocean.com\/community#iaan)\n- [Marketplace](https:\/\/www.digitalocean.com\/products\/marketplace)\n- [Pricing](https:\/\/www.digitalocean.com\/pricing)\n- [Pricing Calculator](https:\/\/www.digitalocean.com\/pricing\/calculator)\n- [Documentation](https:\/\/docs.digitalocean.com\/)\n- [Release Notes](https:\/\/docs.digitalocean.com\/release-notes)\n- [Code of Conduct](https:\/\/www.digitalocean.com\/community\/pages\/code-of-conduct)\n- [Shop Swag](http:\/\/store.digitalocean.com\/)\n\n## Solutions\n- [Website Hosting](https:\/\/www.digitalocean.com\/solutions\/website-hosting)\n- [VPS Hosting](https:\/\/www.digitalocean.com\/solutions\/vps-hosting)\n- [Web & Mobile Apps](https:\/\/www.digitalocean.com\/solutions\/web-mobile-apps)\n- [Game Development](https:\/\/www.digitalocean.com\/solutions\/gaming-development)\n- [Streaming](https:\/\/www.digitalocean.com\/solutions\/streaming)\n- [VPN](https:\/\/www.digitalocean.com\/solutions\/vpn)\n- [SaaS Platforms](https:\/\/www.digitalocean.com\/solutions\/saas)\n- [Cloud Hosting for Blockchain](https:\/\/www.digitalocean.com\/solutions\/blockchain)\n- [Startup Resources](https:\/\/www.digitalocean.com\/resources)\n- [Migration Assistance](https:\/\/www.digitalocean.com\/migrate)\n\n## Contact\n- [Support](https:\/\/www.digitalocean.com\/support)\n- [Sales](https:\/\/www.digitalocean.com\/company\/contact\/sales?referrer=footer)\n- [Report Abuse](https:\/\/www.digitalocean.com\/company\/contact\/abuse)\n- [System Status](https:\/\/status.digitalocean.com\/)\n- [Share your ideas](https:\/\/ideas.digitalocean.com\/)\n\n## Company\n- [About](https:\/\/www.digitalocean.com\/about)\n- [Leadership](https:\/\/www.digitalocean.com\/leadership\/executive-management)\n- [Blog](https:\/\/www.digitalocean.com\/blog)\n- [Careers](https:\/\/www.digitalocean.com\/careers)\n- [Customers](https:\/\/www.digitalocean.com\/customers)\n- [Partners](https:\/\/www.digitalocean.com\/partners)\n- [Referral Program](https:\/\/www.digitalocean.com\/referral-program)\n- [Affiliate Program](https:\/\/www.digitalocean.com\/affiliates)\n- [Press](https:\/\/www.digitalocean.com\/press)\n- [Legal](https:\/\/www.digitalocean.com\/legal)\n- [Privacy Policy](https:\/\/www.digitalocean.com\/legal\/privacy-policy)\n- [Security](https:\/\/www.digitalocean.com\/security)\n- [Investor Relations](https:\/\/investors.digitalocean.com\/)\n\n## Products\n- [Overview](https:\/\/www.digitalocean.com\/products)\n- [Droplets](https:\/\/www.digitalocean.com\/products\/droplets)\n- [Kubernetes](https:\/\/www.digitalocean.com\/products\/kubernetes)\n- [Functions](https:\/\/www.digitalocean.com\/products\/functions)\n- [App Platform](https:\/\/www.digitalocean.com\/products\/app-platform)\n- [Gradient™ AI GPU Droplets](https:\/\/www.digitalocean.com\/products\/gradient\/gpu-droplets)\n- [Gradient™ AI Bare Metal GPUs](https:\/\/www.digitalocean.com\/products\/gradient\/bare-metal-gpus)\n- [Gradient™ AI 1-Click Models](https:\/\/www.digitalocean.com\/products\/gradient\/1-click-models)\n- [Gradient™ AI Platform](https:\/\/www.digitalocean.com\/products\/gradient\/platform)\n- [Load Balancers](https:\/\/www.digitalocean.com\/products\/load-balancers)\n- [Managed Databases](https:\/\/www.digitalocean.com\/products\/managed-databases)\n- [Spaces](https:\/\/www.digitalocean.com\/products\/spaces)\n- [Block Storage](https:\/\/www.digitalocean.com\/products\/block-storage)\n- [Network File Storage](https:\/\/www.digitalocean.com\/products\/storage\/network-file-storage)\n- [API](https:\/\/docs.digitalocean.com\/reference\/api)\n- [Uptime](https:\/\/www.digitalocean.com\/products\/uptime-monitoring)\n- [Identity and Access Management](https:\/\/www.digitalocean.com\/products\/identity-access-management)\n- [Cloudways](https:\/\/www.digitalocean.com\/products\/cloudways)\n\n## Resources\n- [Community Tutorials](https:\/\/www.digitalocean.com\/community\/tutorials)\n- [Community Q\\&A](https:\/\/www.digitalocean.com\/community\/questions)\n- [CSS-Tricks](https:\/\/css-tricks.com\/)\n- [Write for DOnations](https:\/\/www.digitalocean.com\/community\/pages\/write-for-digitalocean)\n- [Currents Research](https:\/\/www.digitalocean.com\/currents)\n- [DigitalOcean Startups](https:\/\/www.digitalocean.com\/startups)\n- [Wavemakers Program](https:\/\/www.digitalocean.com\/wavemakers)\n- [Compass Council](https:\/\/www.digitalocean.com\/research)\n- [Open Source](https:\/\/www.digitalocean.com\/open-source)\n- [Newsletter Signup](https:\/\/www.digitalocean.com\/community#iaan)\n- [Marketplace](https:\/\/www.digitalocean.com\/products\/marketplace)\n- [Pricing](https:\/\/www.digitalocean.com\/pricing)\n- [Pricing Calculator](https:\/\/www.digitalocean.com\/pricing\/calculator)\n- [Documentation](https:\/\/docs.digitalocean.com\/)\n- [Release Notes](https:\/\/docs.digitalocean.com\/release-notes)\n- [Code of Conduct](https:\/\/www.digitalocean.com\/community\/pages\/code-of-conduct)\n- [Shop Swag](http:\/\/store.digitalocean.com\/)\n\n## Solutions\n- [Website Hosting](https:\/\/www.digitalocean.com\/solutions\/website-hosting)\n- [VPS Hosting](https:\/\/www.digitalocean.com\/solutions\/vps-hosting)\n- [Web & Mobile Apps](https:\/\/www.digitalocean.com\/solutions\/web-mobile-apps)\n- [Game Development](https:\/\/www.digitalocean.com\/solutions\/gaming-development)\n- [Streaming](https:\/\/www.digitalocean.com\/solutions\/streaming)\n- [VPN](https:\/\/www.digitalocean.com\/solutions\/vpn)\n- [SaaS Platforms](https:\/\/www.digitalocean.com\/solutions\/saas)\n- [Cloud Hosting for Blockchain](https:\/\/www.digitalocean.com\/solutions\/blockchain)\n- [Startup Resources](https:\/\/www.digitalocean.com\/resources)\n- [Migration Assistance](https:\/\/www.digitalocean.com\/migrate)\n\n## Contact\n- [Support](https:\/\/www.digitalocean.com\/support)\n- [Sales](https:\/\/www.digitalocean.com\/company\/contact\/sales?referrer=footer)\n- [Report Abuse](https:\/\/www.digitalocean.com\/company\/contact\/abuse)\n- [System Status](https:\/\/status.digitalocean.com\/)\n- [Share your ideas](https:\/\/ideas.digitalocean.com\/)\n\n© 2026 DigitalOcean, LLC.[Sitemap](https:\/\/www.digitalocean.com\/sitemap).[Cookie Preferences]()\n\nThis site uses cookies and related technologies, as described in our [privacy policy](https:\/\/www.digitalocean.com\/legal\/privacy-policy\/), for purposes that may include site operation, analytics, enhanced user experience, or advertising. You may choose to consent to our use of these technologies, or manage your own preferences. Please visit our [cookie policy](https:\/\/www.digitalocean.com\/legal\/cookie-policy) for more information.\n\nAgree & Proceed\n\nDecline All\n\nManage Choices\n\nLoading...\n\n## Community\n## Product Docs\n## Marketplace\n## DigitalOcean Blog\n\nnavigate\n\ngo\n\nexit","attrs_readable_markdown":"In our previous tutorial, we learned about [Python CSV Example](https:\/\/www.digitalocean.com\/community\/tutorials\/parse-csv-files-in-python). In this tutorial we are going to learn Python Multiprocessing with examples.\n\n## [Python Multiprocessing](https:\/\/www.digitalocean.com\/community\/tutorials\/python-multiprocessing-example#python-multiprocessing)\nParallel processing is getting more attention nowadays. If you still don’t know about the parallel processing, learn from [wikipedia](https:\/\/en.wikipedia.org\/wiki\/Parallel_processing). As CPU manufacturers start adding more and more cores to their processors, creating parallel code is a great way to improve performance. Python introduced **multiprocessing** module to let us write parallel code. To understand the main motivation of this module, we have to know some basics about parallel programming. After reading this article, we hope that, you would be able to gather some knowledge on this topic.\n\n### [Python Multiprocessing Process, Queue and Locks](https:\/\/www.digitalocean.com\/community\/tutorials\/python-multiprocessing-example#python-multiprocessing-process-queue-and-locks)\nThere are plenty of classes in python multiprocessing module for building a parallel program. Among them, three basic classes are `Process`, `Queue` and `Lock`. These classes will help you to build a parallel program. But before describing about those, let us initiate this topic with simple code. To make a parallel program useful, you have to know how many cores are there in you pc. Python Multiprocessing module enables you to know that. The following simple code will print the number of cores in your pc.\n```\nimport multiprocessing\n\nprint(\"Number of cpu : \", multiprocessing.cpu_count())\n```\nThe following output may vary for your pc. For me, number of cores is 8. ![python multiprocessing example cpu count](https:\/\/journaldev.nyc3.cdn.digitaloceanspaces.com\/2017\/09\/python-multiprocessing-cpu-count.png)\n\n### [Python multiprocessing Process class](https:\/\/www.digitalocean.com\/community\/tutorials\/python-multiprocessing-example#python-multiprocessing-process-class)\nPython multiprocessing `Process` class is an abstraction that sets up another Python process, provides it to run code and a way for the parent application to control execution. There are two important functions that belongs to the Process class - `start()` and `join()` function. At first, we need to write a function, that will be run by the process. Then, we need to instantiate a process object. If we create a process object, nothing will happen until we tell it to start processing via `start()` function. Then, the process will run and return its result. After that we tell the process to complete via `join()` function. Without `join()` function call, process will remain idle and won’t terminate. So if you create many processes and don’t terminate them, you may face scarcity of resources. Then you may need to kill them manually. One important thing is, if you want to pass any argument through the process you need to use `args` keyword argument. The following code will be helpful to understand the usage of Process class.\n```\nfrom multiprocessing import Process\n\n\ndef print_func(continent='Asia'):\n    print('The name of continent is : ', continent)\n\nif __name__ == \"__main__\":  # confirms that the code is under main function\n    names = ['America', 'Europe', 'Africa']\n    procs = []\n    proc = Process(target=print_func)  # instantiating without any argument\n    procs.append(proc)\n    proc.start()\n\n    # instantiating process with arguments\n    for name in names:\n        # print(name)\n        proc = Process(target=print_func, args=(name,))\n        procs.append(proc)\n        proc.start()\n\n    # complete the processes\n    for proc in procs:\n        proc.join()\n```\nThe output of the following code will be: ![python multiprocessing example process class](https:\/\/journaldev.nyc3.cdn.digitaloceanspaces.com\/2017\/09\/python-multiprocessing-process.png)\n\n### [Python multiprocessing Queue class](https:\/\/www.digitalocean.com\/community\/tutorials\/python-multiprocessing-example#python-multiprocessing-queue-class)\nYou have basic knowledge about computer data-structure, you probably know about Queue. Python Multiprocessing modules provides `Queue` class that is exactly a **First-In-First-Out** data structure. They can store any pickle Python object (though simple ones are best) and are extremely useful for sharing data between processes. Queues are specially useful when passed as a parameter to a Process’ target function to enable the Process to consume data. By using `put()` function we can insert data to then queue and using `get()` we can get items from queues. See the following code for a quick example.\n```\nfrom multiprocessing import Queue\n\ncolors = ['red', 'green', 'blue', 'black']\ncnt = 1\n# instantiating a queue object\nqueue = Queue()\nprint('pushing items to queue:')\nfor color in colors:\n    print('item no: ', cnt, ' ', color)\n    queue.put(color)\n    cnt += 1\n\nprint('\\npopping items from queue:')\ncnt = 0\nwhile not queue.empty():\n    print('item no: ', cnt, ' ', queue.get())\n    cnt += 1\n```\n![python multiprocessing queue](https:\/\/journaldev.nyc3.cdn.digitaloceanspaces.com\/2017\/09\/python-multiprocessing-queue.png)\n\n### [Python multiprocessing Lock Class](https:\/\/www.digitalocean.com\/community\/tutorials\/python-multiprocessing-example#python-multiprocessing-lock-class)\nThe task of Lock class is quite simple. It allows code to claim lock so that no other process can execute the similar code until the lock has be released. So the task of Lock class is mainly two. One is to claim lock and other is to release the lock. To claim lock the, `acquire()` function is used and to release lock `release()` function is used.\n\n### [Python multiprocessing example](https:\/\/www.digitalocean.com\/community\/tutorials\/python-multiprocessing-example#python-multiprocessing-example)\nIn this Python multiprocessing example, we will merge all our knowledge together. Suppose we have some tasks to accomplish. To get that task done, we will use several processes. So, we will maintain two queue. One will contain the tasks and the other will contain the log of completed task. Then we instantiate the processes to complete the task. Note that the python Queue class is already synchronized. That means, we don’t need to use the Lock class to block multiple process to access the same queue object. That’s why, we don’t need to use Lock class in this case. Below is the implementation where we are adding tasks to the queue, then creating processes and starting them, then using `join()` to complete the processes. Finally we are printing the log from the second queue.\n```\nfrom multiprocessing import Lock, Process, Queue, current_process\nimport time\nimport queue # imported for using queue.Empty exception\n\n\ndef do_job(tasks_to_accomplish, tasks_that_are_done):\n    while True:\n        try:\n            '''\n                try to get task from the queue. get_nowait() function will \n                raise queue.Empty exception if the queue is empty. \n                queue(False) function would do the same task also.\n            '''\n            task = tasks_to_accomplish.get_nowait()\n        except queue.Empty:\n\n            break\n        else:\n            '''\n                if no exception has been raised, add the task completion \n                message to task_that_are_done queue\n            '''\n            print(task)\n            tasks_that_are_done.put(task + ' is done by ' + current_process().name)\n            time.sleep(.5)\n    return True\n\n\ndef main():\n    number_of_task = 10\n    number_of_processes = 4\n    tasks_to_accomplish = Queue()\n    tasks_that_are_done = Queue()\n    processes = []\n\n    for i in range(number_of_task):\n        tasks_to_accomplish.put(\"Task no \" + str(i))\n\n    # creating processes\n    for w in range(number_of_processes):\n        p = Process(target=do_job, args=(tasks_to_accomplish, tasks_that_are_done))\n        processes.append(p)\n        p.start()\n\n    # completing process\n    for p in processes:\n        p.join()\n\n    # print the output\n    while not tasks_that_are_done.empty():\n        print(tasks_that_are_done.get())\n\n    return True\n\n\nif __name__ == '__main__':\n    main()\n```\nDepending on the number of task, the code will take some time to show you the output. The output of the following code will vary from time to time. ![python multiprocessing example](https:\/\/journaldev.nyc3.cdn.digitaloceanspaces.com\/2017\/09\/python-multiprocessing-example-1.png)\n\n### [Python multiprocessing Pool](https:\/\/www.digitalocean.com\/community\/tutorials\/python-multiprocessing-example#python-multiprocessing-pool)\nPython multiprocessing Pool can be used for parallel execution of a function across multiple input values, distributing the input data across processes (data parallelism). Below is a simple Python multiprocessing Pool example.\n```\nfrom multiprocessing import Pool\n\nimport time\n\nwork = ([\"A\", 5], [\"B\", 2], [\"C\", 1], [\"D\", 3])\n\n\ndef work_log(work_data):\n    print(\" Process %s waiting %s seconds\" % (work_data[0], work_data[1]))\n    time.sleep(int(work_data[1]))\n    print(\" Process %s Finished.\" % work_data[0])\n\n\ndef pool_handler():\n    p = Pool(2)\n    p.map(work_log, work)\n\n\nif __name__ == '__main__':\n    pool_handler()\n```\nBelow image shows the output of the above program. Notice that pool size is 2, so two executions of `work_log` function is happening in parallel. When one of the function processing finishes, it picks the next argument and so on. ![python multiprocessing pool](https:\/\/journaldev.nyc3.cdn.digitaloceanspaces.com\/2017\/09\/python-multiprocessing-pool.png) So, that’s all for python multiprocessing module. Reference: [Official Documentation](https:\/\/docs.python.org\/3.6\/library\/multiprocessing.html)","meta_canonical":null}

3. Robots.txt Check

Query:

Response:

4. Spam/Ban Check

Query:

Response:

5. Seen Status Check

ℹ️ Skipped - page is already crawled

📄

INDEXABLE

✅

CRAWLED

12 days ago

🤖

ROBOTS ALLOWED

Page Info Filters

Filter	Status	Condition	Details
HTTP status	PASS	`download_http_code = 200`	HTTP 200
Age cutoff	PASS	`download_stamp > now() - 6 MONTH`	0.4 months ago
History drop	PASS	`isNull(history_drop_reason)`	No drop reason
Spam/ban	PASS	`fh_dont_index != 1 AND ml_spam_score = 0`	ml_spam_score=0
Canonical	PASS	`meta_canonical IS NULL OR = '' OR = src_unparsed`	Not set

Page Details

Property	Value
URL	https://www.digitalocean.com/community/tutorials/python-multiprocessing-example
Last Crawled	2026-03-25 22:29:27 (12 days ago)
First Indexed	2022-08-05 09:05:31 (3 years ago)
HTTP Status Code	200
Meta Title	Python Multiprocessing Example \| DigitalOcean
Meta Description	Speed up your Python code with multiprocessing. Learn how to parallelize tasks for faster results.
Meta Canonical	null
Boilerpipe Text	In our previous tutorial, we learned about Python CSV Example . In this tutorial we are going to learn Python Multiprocessing with examples. Python Multiprocessing Parallel processing is getting more attention nowadays. If you still don’t know about the parallel processing, learn from wikipedia . As CPU manufacturers start adding more and more cores to their processors, creating parallel code is a great way to improve performance. Python introduced multiprocessing module to let us write parallel code. To understand the main motivation of this module, we have to know some basics about parallel programming. After reading this article, we hope that, you would be able to gather some knowledge on this topic. Python Multiprocessing Process, Queue and Locks There are plenty of classes in python multiprocessing module for building a parallel program. Among them, three basic classes are Process , Queue and Lock . These classes will help you to build a parallel program. But before describing about those, let us initiate this topic with simple code. To make a parallel program useful, you have to know how many cores are there in you pc. Python Multiprocessing module enables you to know that. The following simple code will print the number of cores in your pc. import multiprocessing print ( "Number of cpu : " , multiprocessing . cpu_count ( ) ) The following output may vary for your pc. For me, number of cores is 8. Python multiprocessing Process class Python multiprocessing Process class is an abstraction that sets up another Python process, provides it to run code and a way for the parent application to control execution. There are two important functions that belongs to the Process class - start() and join() function. At first, we need to write a function, that will be run by the process. Then, we need to instantiate a process object. If we create a process object, nothing will happen until we tell it to start processing via start() function. Then, the process will run and return its result. After that we tell the process to complete via join() function. Without join() function call, process will remain idle and won’t terminate. So if you create many processes and don’t terminate them, you may face scarcity of resources. Then you may need to kill them manually. One important thing is, if you want to pass any argument through the process you need to use args keyword argument. The following code will be helpful to understand the usage of Process class. from multiprocessing import Process def print_func ( continent = 'Asia' ) : print ( 'The name of continent is : ' , continent ) if __name__ == "__main__" : # confirms that the code is under main function names = [ 'America' , 'Europe' , 'Africa' ] procs = [ ] proc = Process ( target = print_func ) # instantiating without any argument procs . append ( proc ) proc . start ( ) # instantiating process with arguments for name in names : # print(name) proc = Process ( target = print_func , args = ( name , ) ) procs . append ( proc ) proc . start ( ) # complete the processes for proc in procs : proc . join ( ) The output of the following code will be: Python multiprocessing Queue class You have basic knowledge about computer data-structure, you probably know about Queue. Python Multiprocessing modules provides Queue class that is exactly a First-In-First-Out data structure. They can store any pickle Python object (though simple ones are best) and are extremely useful for sharing data between processes. Queues are specially useful when passed as a parameter to a Process’ target function to enable the Process to consume data. By using put() function we can insert data to then queue and using get() we can get items from queues. See the following code for a quick example. from multiprocessing import Queue colors = [ 'red' , 'green' , 'blue' , 'black' ] cnt = 1 # instantiating a queue object queue = Queue ( ) print ( 'pushing items to queue:' ) for color in colors : print ( 'item no: ' , cnt , ' ' , color ) queue . put ( color ) cnt += 1 print ( '\npopping items from queue:' ) cnt = 0 while not queue . empty ( ) : print ( 'item no: ' , cnt , ' ' , queue . get ( ) ) cnt += 1 Python multiprocessing Lock Class The task of Lock class is quite simple. It allows code to claim lock so that no other process can execute the similar code until the lock has be released. So the task of Lock class is mainly two. One is to claim lock and other is to release the lock. To claim lock the, acquire() function is used and to release lock release() function is used. Python multiprocessing example In this Python multiprocessing example, we will merge all our knowledge together. Suppose we have some tasks to accomplish. To get that task done, we will use several processes. So, we will maintain two queue. One will contain the tasks and the other will contain the log of completed task. Then we instantiate the processes to complete the task. Note that the python Queue class is already synchronized. That means, we don’t need to use the Lock class to block multiple process to access the same queue object. That’s why, we don’t need to use Lock class in this case. Below is the implementation where we are adding tasks to the queue, then creating processes and starting them, then using join() to complete the processes. Finally we are printing the log from the second queue. from multiprocessing import Lock , Process , Queue , current_process import time import queue # imported for using queue.Empty exception def do_job ( tasks_to_accomplish , tasks_that_are_done ) : while True : try : ''' try to get task from the queue. get_nowait() function will raise queue.Empty exception if the queue is empty. queue(False) function would do the same task also. ''' task = tasks_to_accomplish . get_nowait ( ) except queue . Empty : break else : ''' if no exception has been raised, add the task completion message to task_that_are_done queue ''' print ( task ) tasks_that_are_done . put ( task + ' is done by ' + current_process ( ) . name ) time . sleep ( .5 ) return True def main ( ) : number_of_task = 10 number_of_processes = 4 tasks_to_accomplish = Queue ( ) tasks_that_are_done = Queue ( ) processes = [ ] for i in range ( number_of_task ) : tasks_to_accomplish . put ( "Task no " + str ( i ) ) # creating processes for w in range ( number_of_processes ) : p = Process ( target = do_job , args = ( tasks_to_accomplish , tasks_that_are_done ) ) processes . append ( p ) p . start ( ) # completing process for p in processes : p . join ( ) # print the output while not tasks_that_are_done . empty ( ) : print ( tasks_that_are_done . get ( ) ) return True if __name__ == '__main__' : main ( ) Depending on the number of task, the code will take some time to show you the output. The output of the following code will vary from time to time. Python multiprocessing Pool Python multiprocessing Pool can be used for parallel execution of a function across multiple input values, distributing the input data across processes (data parallelism). Below is a simple Python multiprocessing Pool example. from multiprocessing import Pool import time work = ( [ "A" , 5 ] , [ "B" , 2 ] , [ "C" , 1 ] , [ "D" , 3 ] ) def work_log ( work_data ) : print ( " Process %s waiting %s seconds" % ( work_data [ 0 ] , work_data [ 1 ] ) ) time . sleep ( int ( work_data [ 1 ] ) ) print ( " Process %s Finished." % work_data [ 0 ] ) def pool_handler ( ) : p = Pool ( 2 ) p . map ( work_log , work ) if __name__ == '__main__' : pool_handler ( ) Below image shows the output of the above program. Notice that pool size is 2, so two executions of work_log function is happening in parallel. When one of the function processing finishes, it picks the next argument and so on. So, that’s all for python multiprocessing module. Reference: Official Documentation
Markdown	[Now Available: OpenAI GPT-5.4 on the Gradient™ AI Platform](https://www.digitalocean.com/blog/whats-new-on-gradient-ai-platform#week-of-march-2) [Now Available: OpenAI GPT-5.3 Codex on the Gradient™ AI Platform](https://www.digitalocean.com/blog/whats-new-on-gradient-ai-platform#week-of-february-23rd) [Now Available: Claude Sonnet 4.6 on the Gradient™ AI Platform](https://www.digitalocean.com/blog/whats-new-on-gradient-ai-platform#week-of-february-17th) [Heroku is shifting to sustaining engineering. Migrate your workloads to DigitalOcean](https://www.digitalocean.com/blog/migrate-heroku-to-digitalocean) [Now Available: Claude Opus 4.6 on Gradient™ AI Platform](https://www.digitalocean.com/blog/claude-opus-4-6-gradient-ai-platform) [🔥 Run OpenClaw on DigitalOcean App Platform](https://www.digitalocean.com/blog/openclaw-digitalocean-app-platform) [Join us for Deploy in San Francisco, on April 28, 2026. Register now](https://www.digitalocean.com/deploy) - [Blog](https://www.digitalocean.com/blog) - [Docs](https://docs.digitalocean.com/products) - [Careers](https://www.digitalocean.com/careers) - [Get Support](https://www.digitalocean.com/support) - [Contact Sales](https://www.digitalocean.com/company/contact/sales?referrer=tophat) [DigitalOcean](https://www.digitalocean.com/) - Products - Featured Products - [Droplets Scalable virtual machines](https://www.digitalocean.com/products/droplets) - [Kubernetes Scale more effectively](https://www.digitalocean.com/products/kubernetes) - [Gradient™ AI Inference Cloud Build and scale with AI](https://www.digitalocean.com/products/gradient) - [Cloudways Managed cloud hosting](https://www.digitalocean.com/products/cloudways) - [App Platform Get apps to market faster](https://www.digitalocean.com/products/app-platform) - [Managed Databases Fully-managed database hosting](https://www.digitalocean.com/products/managed-databases) [See all products](https://www.digitalocean.com/products) - Solutions - - - [AI and Machine Learning](https://www.digitalocean.com/products/gradient) Develop, train, and deploy AI apps - [GPUs](https://www.digitalocean.com/products/gradient/gpu-droplets) - [Platform](https://www.digitalocean.com/products/gradient/platform) - [1-Click Models](https://www.digitalocean.com/products/gradient/1-click-models) - [HR Knowledge Assistant](https://www.digitalocean.com/solutions/ai-hr-knowledge-assistant) - [Code Copilot](https://www.digitalocean.com/solutions/ai-code-copilot) - [Support Ticket Triage](https://www.digitalocean.com/solutions/ai-support-ticket-triage) - [Recommendation Engine](https://www.digitalocean.com/solutions/ai-recommendation-engine) - [Blockchain](https://www.digitalocean.com/solutions/blockchain) Infrastructure for decentralized apps - [Blogs, Forums and Content Websites](https://www.digitalocean.com/solutions/content-hosting) Lightning-fast, reliable CMS hosting - [Wordpress](https://www.digitalocean.com/solutions/wordpress-hosting) - [Ghost](https://marketplace.digitalocean.com/apps/ghost) - [Mastodon](https://marketplace.digitalocean.com/apps/mastodon) - [Data Analytics](https://www.digitalocean.com/solutions/data-analytics) Real-time data processing at scale - [Data Streaming](https://www.digitalocean.com/solutions/data-streaming-cloud) - [AdTech & Martech](https://www.digitalocean.com/solutions/adtech-martech) - [Kafka](https://www.digitalocean.com/products/managed-databases-kafka?utm_source=top-nav-data-analytics) - - [Developer Tools](https://www.digitalocean.com/solutions/developer-tools) DevOps and CI/CD solutions - [CI/CD](https://www.digitalocean.com/solutions/cicd-pipelines) - [Prototyping](https://www.digitalocean.com/solutions/prototype-hosting) - [Digital Marketing Agencies](https://www.digitalocean.com/solutions/digital-marketing-agencies) Power your clients’ websites and campaigns - [Freelancer](https://www.digitalocean.com/solutions/freelancer-website-hosting) - [IT Consulting](https://www.digitalocean.com/solutions/it-consulting) - [Ecommerce](https://www.digitalocean.com/solutions/ecommerce-hosting) Build beautiful online storefronts - [Dropshipping](https://www.digitalocean.com/solutions/dropshipping-hosting) - [WooCommerce](https://marketplace.digitalocean.com/apps/wordpress-woocommerce) - [Magento](https://marketplace.digitalocean.com/apps/magento-2-open-source) - [Game Development](https://www.digitalocean.com/solutions/gaming-development) Low-latency multiplayer servers - [Minecraft Hosting](https://marketplace.digitalocean.com/apps/minecraft-java-edition-server) - - [IoT](https://www.digitalocean.com/solutions/iot-cloud) Connect to the power of the cloud - [Kafka](https://www.digitalocean.com/products/managed-databases-kafka?utm_source=top-nav-iot) - [ISVs](https://www.digitalocean.com/solutions/isv) Streamlined ISV application development - [Secure Web Hosting](https://www.digitalocean.com/solutions/secure-web-hosting) Powerful protection from DDoS and more - [Private VPN](https://www.digitalocean.com/solutions/vpn) - [Startup Cloud Hosting](https://www.digitalocean.com/solutions/startups) Scalable, cost-effective infrastructure - [Small Business](https://www.digitalocean.com/solutions/small-business-website-hosting) - [Video Streaming](https://www.digitalocean.com/solutions/streaming) High-bandwidth, low-latency delivery - [Kafka](https://www.digitalocean.com/products/managed-databases-kafka?utm_source=top-nav-video-streaming) - - [Web and Mobile Apps](https://www.digitalocean.com/solutions/web-mobile-apps) Simple cross-platform app hosting - [cPanel](https://www.digitalocean.com/solutions/cpanel-hosting) - [Docker](https://www.digitalocean.com/solutions/docker-hosting) - [Next.js](https://www.digitalocean.com/solutions/nextjs-hosting) - [Node.js](https://www.digitalocean.com/solutions/nodejs-hosting) - [Website Hosting](https://www.digitalocean.com/solutions/website-hosting) Fast page loads and reliable site uptime - [VPS Hosting](https://www.digitalocean.com/solutions/vps-hosting) - [Virtual Machines](https://www.digitalocean.com/solutions/virtual-machines) [See all solutions](https://www.digitalocean.com/business) - Developers - Our Community - [Community Home DevOps and development guides](https://www.digitalocean.com/community) - [CSS-Tricks All things web design](https://css-tricks.com/) - [The Wave Content to level up your business.](https://www.digitalocean.com/resources) - Partners - DigitalOcean Partner Programs - [Become a Partner](https://www.digitalocean.com/partners/pod) - [Partner Services Program](https://www.digitalocean.com/partners/services) - [DigitalOcean AI Partner Program](https://www.digitalocean.com/partners/ai-partner-program) - [Marketplace](https://marketplace.digitalocean.com/) - [DigitalOcean Startups](https://www.digitalocean.com/startups) - [Connect with a Partner](https://www.digitalocean.com/partners/directory) - [Pricing](https://www.digitalocean.com/pricing) - Log in - [Community](https://www.digitalocean.com/api/dynamic-content/v1/login?success_redirect=https%3A%2F%2Fwww.digitalocean.com%2Fcommunity%2Ftutorials%2Fpython-multiprocessing-example&error_redirect=https%3A%2F%2Fwww.digitalocean.com%2Fauth-error&type=login) - [DigitalOcean](https://cloud.digitalocean.com/login) - Sign up - [Community](https://www.digitalocean.com/api/dynamic-content/v1/login?success_redirect=https%3A%2F%2Fwww.digitalocean.com%2Fcommunity%2Ftutorials%2Fpython-multiprocessing-example&error_redirect=https%3A%2F%2Fwww.digitalocean.com%2Fauth-error&type=register) - [DigitalOcean](https://cloud.digitalocean.com/registrations/new) - [Blog](https://www.digitalocean.com/blog) - [Docs](https://docs.digitalocean.com/products) - [Careers](https://www.digitalocean.com/careers) - [Get Support](https://www.digitalocean.com/support) - [Contact Sales](https://www.digitalocean.com/company/contact/sales?referrer=tophat) - Log in - [Community](https://www.digitalocean.com/api/dynamic-content/v1/login?success_redirect=https%3A%2F%2Fwww.digitalocean.com%2Fcommunity%2Ftutorials%2Fpython-multiprocessing-example&error_redirect=https%3A%2F%2Fwww.digitalocean.com%2Fauth-error&type=login) - [DigitalOcean](https://cloud.digitalocean.com/login) - Sign up - [Community](https://www.digitalocean.com/api/dynamic-content/v1/login?success_redirect=https%3A%2F%2Fwww.digitalocean.com%2Fcommunity%2Ftutorials%2Fpython-multiprocessing-example&error_redirect=https%3A%2F%2Fwww.digitalocean.com%2Fauth-error&type=register) - [DigitalOcean](https://cloud.digitalocean.com/registrations/new) - [Tutorials](https://www.digitalocean.com/community/tutorials) - [Questions](https://www.digitalocean.com/community/questions) - [Product Docs](https://docs.digitalocean.com/) - Search Community ## Report this What is the reason for this report? This undefined is spam This undefined is offensive This undefined is off-topic This undefined is other Submit ## Table of contents 1. [Python Multiprocessing](https://www.digitalocean.com/community/tutorials/python-multiprocessing-example#python-multiprocessing) 1. [Tutorials](https://www.digitalocean.com/community/tutorials?subtype=tutorial) 2. [Python](https://www.digitalocean.com/community/tags/python) 3. Python Multiprocessing Example [Tutorial](https://www.digitalocean.com/community/tutorials?subtype=tutorial) # Python Multiprocessing Example Published on August 3, 2022 [Python](https://www.digitalocean.com/community/tags/python) ![Anish Singh Walia](https://www.gravatar.com/avatar/c91039d94d0ab6e82a10aaf08e3c1c341d5744c8533ff6d077ee4a765246d8d3?default=retro&size=256)![Pankaj Kumar](https://www.gravatar.com/avatar/dcb62410bd20fd354c21af393bec6e67b235a09758848745cfd4405e05bba1f4?default=retro&size=256) By [Anish Singh Walia](https://www.digitalocean.com/community/users/asinghwalia) and [Pankaj Kumar](https://www.digitalocean.com/community/users/pankajkumarjournaldev) ![Python Multiprocessing Example](https://www.digitalocean.com/api/static-content/v1/images?src=%2F_next%2Fstatic%2Fmedia%2Fintro-to-cloud.d49bc5f7.jpeg&width=1920) Table of contents Popular topics In our previous tutorial, we learned about [Python CSV Example](https://www.digitalocean.com/community/tutorials/parse-csv-files-in-python). In this tutorial we are going to learn Python Multiprocessing with examples. ## [Python Multiprocessing](https://www.digitalocean.com/community/tutorials/python-multiprocessing-example#python-multiprocessing) Parallel processing is getting more attention nowadays. If you still don’t know about the parallel processing, learn from [wikipedia](https://en.wikipedia.org/wiki/Parallel_processing). As CPU manufacturers start adding more and more cores to their processors, creating parallel code is a great way to improve performance. Python introduced multiprocessing module to let us write parallel code. To understand the main motivation of this module, we have to know some basics about parallel programming. After reading this article, we hope that, you would be able to gather some knowledge on this topic. ### [Python Multiprocessing Process, Queue and Locks](https://www.digitalocean.com/community/tutorials/python-multiprocessing-example#python-multiprocessing-process-queue-and-locks) There are plenty of classes in python multiprocessing module for building a parallel program. Among them, three basic classes are `Process`, `Queue` and `Lock`. These classes will help you to build a parallel program. But before describing about those, let us initiate this topic with simple code. To make a parallel program useful, you have to know how many cores are there in you pc. Python Multiprocessing module enables you to know that. The following simple code will print the number of cores in your pc. ``` import multiprocessing print("Number of cpu : ", multiprocessing.cpu_count()) ``` Copy The following output may vary for your pc. For me, number of cores is 8. ![python multiprocessing example cpu count](https://journaldev.nyc3.cdn.digitaloceanspaces.com/2017/09/python-multiprocessing-cpu-count.png) ### [Python multiprocessing Process class](https://www.digitalocean.com/community/tutorials/python-multiprocessing-example#python-multiprocessing-process-class) Python multiprocessing `Process` class is an abstraction that sets up another Python process, provides it to run code and a way for the parent application to control execution. There are two important functions that belongs to the Process class - `start()` and `join()` function. At first, we need to write a function, that will be run by the process. Then, we need to instantiate a process object. If we create a process object, nothing will happen until we tell it to start processing via `start()` function. Then, the process will run and return its result. After that we tell the process to complete via `join()` function. Without `join()` function call, process will remain idle and won’t terminate. So if you create many processes and don’t terminate them, you may face scarcity of resources. Then you may need to kill them manually. One important thing is, if you want to pass any argument through the process you need to use `args` keyword argument. The following code will be helpful to understand the usage of Process class. ``` from multiprocessing import Process def print_func(continent='Asia'): print('The name of continent is : ', continent) if __name__ == "__main__": # confirms that the code is under main function names = ['America', 'Europe', 'Africa'] procs = [] proc = Process(target=print_func) # instantiating without any argument procs.append(proc) proc.start() # instantiating process with arguments for name in names: # print(name) proc = Process(target=print_func, args=(name,)) procs.append(proc) proc.start() # complete the processes for proc in procs: proc.join() ``` Copy The output of the following code will be: ![python multiprocessing example process class](https://journaldev.nyc3.cdn.digitaloceanspaces.com/2017/09/python-multiprocessing-process.png) ### [Python multiprocessing Queue class](https://www.digitalocean.com/community/tutorials/python-multiprocessing-example#python-multiprocessing-queue-class) You have basic knowledge about computer data-structure, you probably know about Queue. Python Multiprocessing modules provides `Queue` class that is exactly a First-In-First-Out data structure. They can store any pickle Python object (though simple ones are best) and are extremely useful for sharing data between processes. Queues are specially useful when passed as a parameter to a Process’ target function to enable the Process to consume data. By using `put()` function we can insert data to then queue and using `get()` we can get items from queues. See the following code for a quick example. ``` from multiprocessing import Queue colors = ['red', 'green', 'blue', 'black'] cnt = 1 # instantiating a queue object queue = Queue() print('pushing items to queue:') for color in colors: print('item no: ', cnt, ' ', color) queue.put(color) cnt += 1 print('\npopping items from queue:') cnt = 0 while not queue.empty(): print('item no: ', cnt, ' ', queue.get()) cnt += 1 ``` Copy ![python multiprocessing queue](https://journaldev.nyc3.cdn.digitaloceanspaces.com/2017/09/python-multiprocessing-queue.png) ### [Python multiprocessing Lock Class](https://www.digitalocean.com/community/tutorials/python-multiprocessing-example#python-multiprocessing-lock-class) The task of Lock class is quite simple. It allows code to claim lock so that no other process can execute the similar code until the lock has be released. So the task of Lock class is mainly two. One is to claim lock and other is to release the lock. To claim lock the, `acquire()` function is used and to release lock `release()` function is used. ### [Python multiprocessing example](https://www.digitalocean.com/community/tutorials/python-multiprocessing-example#python-multiprocessing-example) In this Python multiprocessing example, we will merge all our knowledge together. Suppose we have some tasks to accomplish. To get that task done, we will use several processes. So, we will maintain two queue. One will contain the tasks and the other will contain the log of completed task. Then we instantiate the processes to complete the task. Note that the python Queue class is already synchronized. That means, we don’t need to use the Lock class to block multiple process to access the same queue object. That’s why, we don’t need to use Lock class in this case. Below is the implementation where we are adding tasks to the queue, then creating processes and starting them, then using `join()` to complete the processes. Finally we are printing the log from the second queue. ``` from multiprocessing import Lock, Process, Queue, current_process import time import queue # imported for using queue.Empty exception def do_job(tasks_to_accomplish, tasks_that_are_done): while True: try: ''' try to get task from the queue. get_nowait() function will raise queue.Empty exception if the queue is empty. queue(False) function would do the same task also. ''' task = tasks_to_accomplish.get_nowait() except queue.Empty: break else: ''' if no exception has been raised, add the task completion message to task_that_are_done queue ''' print(task) tasks_that_are_done.put(task + ' is done by ' + current_process().name) time.sleep(.5) return True def main(): number_of_task = 10 number_of_processes = 4 tasks_to_accomplish = Queue() tasks_that_are_done = Queue() processes = [] for i in range(number_of_task): tasks_to_accomplish.put("Task no " + str(i)) # creating processes for w in range(number_of_processes): p = Process(target=do_job, args=(tasks_to_accomplish, tasks_that_are_done)) processes.append(p) p.start() # completing process for p in processes: p.join() # print the output while not tasks_that_are_done.empty(): print(tasks_that_are_done.get()) return True if __name__ == '__main__': main() ``` Copy Depending on the number of task, the code will take some time to show you the output. The output of the following code will vary from time to time. ![python multiprocessing example](https://journaldev.nyc3.cdn.digitaloceanspaces.com/2017/09/python-multiprocessing-example-1.png) ### [Python multiprocessing Pool](https://www.digitalocean.com/community/tutorials/python-multiprocessing-example#python-multiprocessing-pool) Python multiprocessing Pool can be used for parallel execution of a function across multiple input values, distributing the input data across processes (data parallelism). Below is a simple Python multiprocessing Pool example. ``` from multiprocessing import Pool import time work = (["A", 5], ["B", 2], ["C", 1], ["D", 3]) def work_log(work_data): print(" Process %s waiting %s seconds" % (work_data[0], work_data[1])) time.sleep(int(work_data[1])) print(" Process %s Finished." % work_data[0]) def pool_handler(): p = Pool(2) p.map(work_log, work) if __name__ == '__main__': pool_handler() ``` Copy Below image shows the output of the above program. Notice that pool size is 2, so two executions of `work_log` function is happening in parallel. When one of the function processing finishes, it picks the next argument and so on. ![python multiprocessing pool](https://journaldev.nyc3.cdn.digitaloceanspaces.com/2017/09/python-multiprocessing-pool.png) So, that’s all for python multiprocessing module. Reference: [Official Documentation](https://docs.python.org/3.6/library/multiprocessing.html) Thanks for learning with the DigitalOcean Community. Check out our offerings for compute, storage, networking, and managed databases. [Learn more about our products](https://www.digitalocean.com/products "Learn more about our products") ### About the author(s) ![Anish Singh Walia](https://www.gravatar.com/avatar/c91039d94d0ab6e82a10aaf08e3c1c341d5744c8533ff6d077ee4a765246d8d3?default=retro&size=256) Anish Singh Walia Author Sr Technical Writer and Team Lead [See author profile](https://www.digitalocean.com/community/users/asinghwalia) I help Businesses scale with AI x SEO x (authentic) Content that revives traffic and keeps leads flowing \\| 3,000,000+ Average monthly readers on Medium \\| Sr Technical Writer(Team Lead) @ DigitalOcean \\| Ex-Cloud Consultant @ AMEX \\| Ex-Site Reliability Engineer(DevOps)@Nutanix [See author profile](https://www.digitalocean.com/community/users/asinghwalia) ![Pankaj Kumar](https://www.gravatar.com/avatar/dcb62410bd20fd354c21af393bec6e67b235a09758848745cfd4405e05bba1f4?default=retro&size=256) Pankaj Kumar Author [See author profile](https://www.digitalocean.com/community/users/pankajkumarjournaldev) Java and Python Developer for 20+ years, Open Source Enthusiast, Founder of https://www.askpython.com/, https://www.linuxfordevices.com/, and JournalDev.com (acquired by DigitalOcean). Passionate about writing technical articles and sharing knowledge with others. Love Java, Python, Unix and related technologies. Follow my X @PankajWebDev [See author profile](https://www.digitalocean.com/community/users/pankajkumarjournaldev) Category: [Tutorial](https://www.digitalocean.com/community/tutorials?subtype=tutorial) Tags: [Python](https://www.digitalocean.com/community/tags/python) While we believe that this content benefits our community, we have not yet thoroughly reviewed it. If you have any suggestions for improvements, please let us know by clicking the “report an issue“ button at the bottom of the tutorial. #### Still looking for an answer? [Ask a question](https://www.digitalocean.com/community/questions)[Search for more help](https://www.digitalocean.com/community) Was this helpful? Yes No Comments(10) Follow-up questions(0) ![JournalDev](https://www.gravatar.com/avatar/7c7a4546c8430a5a91a338a177bf8e7e8e153fe7eaee076acfa6d46029efef49?default=retro) [JournalDev](https://www.digitalocean.com/community/users/journaldev)![DigitalOcean Employee badge](https://www.digitalocean.com/api/static-content/v1/images?src=%2F_next%2Fstatic%2Fmedia%2Fdo-logo.15453cf1.svg&width=32) [July 15, 2018](https://www.digitalocean.com/community/tutorials/python-multiprocessing-example?comment=177730) Show less Hi You wrote “Without join() function call, process will remain idle and won’t terminate”. That is not correct., the process will terminate with or without join(). The difference is: -With join(): the program will wait for every process to finish -Without join(): each process will run without waiting for other processes to finish’ Thks \- namta (1) replies ![JournalDev](https://www.gravatar.com/avatar/7c7a4546c8430a5a91a338a177bf8e7e8e153fe7eaee076acfa6d46029efef49?default=retro) [JournalDev](https://www.digitalocean.com/community/users/journaldev)![DigitalOcean Employee badge](https://www.digitalocean.com/api/static-content/v1/images?src=%2F_next%2Fstatic%2Fmedia%2Fdo-logo.15453cf1.svg&width=32) [September 27, 2018](https://www.digitalocean.com/community/tutorials/python-multiprocessing-example?comment=177731) Show less Why does the start() must be called within main function ? \- xram ![JournalDev](https://www.gravatar.com/avatar/7c7a4546c8430a5a91a338a177bf8e7e8e153fe7eaee076acfa6d46029efef49?default=retro) [JournalDev](https://www.digitalocean.com/community/users/journaldev)![DigitalOcean Employee badge](https://www.digitalocean.com/api/static-content/v1/images?src=%2F_next%2Fstatic%2Fmedia%2Fdo-logo.15453cf1.svg&width=32) [February 7, 2019](https://www.digitalocean.com/community/tutorials/python-multiprocessing-example?comment=177732) Show less In the do\_job function ,if there other method to replace time.sleep? For some function ,you may need not to know the possible time to sleep. \- RUI ZHANG ![JournalDev](https://www.gravatar.com/avatar/7c7a4546c8430a5a91a338a177bf8e7e8e153fe7eaee076acfa6d46029efef49?default=retro) [JournalDev](https://www.digitalocean.com/community/users/journaldev)![DigitalOcean Employee badge](https://www.digitalocean.com/api/static-content/v1/images?src=%2F_next%2Fstatic%2Fmedia%2Fdo-logo.15453cf1.svg&width=32) [February 11, 2019](https://www.digitalocean.com/community/tutorials/python-multiprocessing-example?comment=177733) Show less Thank you but I think there isn’t enough explanation. \- Emre ATAKLI ![JournalDev](https://www.gravatar.com/avatar/7c7a4546c8430a5a91a338a177bf8e7e8e153fe7eaee076acfa6d46029efef49?default=retro) [JournalDev](https://www.digitalocean.com/community/users/journaldev)![DigitalOcean Employee badge](https://www.digitalocean.com/api/static-content/v1/images?src=%2F_next%2Fstatic%2Fmedia%2Fdo-logo.15453cf1.svg&width=32) [June 11, 2019](https://www.digitalocean.com/community/tutorials/python-multiprocessing-example?comment=177734) Show less How to Kill the Proc? is Proc.terminate() works best \- Senthilkumar Rajendran ![JournalDev](https://www.gravatar.com/avatar/7c7a4546c8430a5a91a338a177bf8e7e8e153fe7eaee076acfa6d46029efef49?default=retro) [JournalDev](https://www.digitalocean.com/community/users/journaldev)![DigitalOcean Employee badge](https://www.digitalocean.com/api/static-content/v1/images?src=%2F_next%2Fstatic%2Fmedia%2Fdo-logo.15453cf1.svg&width=32) [June 20, 2019](https://www.digitalocean.com/community/tutorials/python-multiprocessing-example?comment=177735) Show less Great article man ! Simple and very informative. Even official python documentation for multiprocessing is not this simple and easy to read. \- Ashish Dhiman ![JournalDev](https://www.gravatar.com/avatar/7c7a4546c8430a5a91a338a177bf8e7e8e153fe7eaee076acfa6d46029efef49?default=retro) [JournalDev](https://www.digitalocean.com/community/users/journaldev)![DigitalOcean Employee badge](https://www.digitalocean.com/api/static-content/v1/images?src=%2F_next%2Fstatic%2Fmedia%2Fdo-logo.15453cf1.svg&width=32) [July 12, 2019](https://www.digitalocean.com/community/tutorials/python-multiprocessing-example?comment=177736) Show less Very easy to understand and informative! Thanks a lot man\! \- Kenny ![JournalDev](https://www.gravatar.com/avatar/7c7a4546c8430a5a91a338a177bf8e7e8e153fe7eaee076acfa6d46029efef49?default=retro) [JournalDev](https://www.digitalocean.com/community/users/journaldev)![DigitalOcean Employee badge](https://www.digitalocean.com/api/static-content/v1/images?src=%2F_next%2Fstatic%2Fmedia%2Fdo-logo.15453cf1.svg&width=32) [February 5, 2020](https://www.digitalocean.com/community/tutorials/python-multiprocessing-example?comment=177737) Show less Thanks, your two way communication example helped me finally close in on my bug. One that your code will have as well if number or size of messages in the queue fills the buffer used by it. ‘.get()’ needs to be called on the Queue being used to send messages back from the sub-process (child) to the main process (parent) before the ‘.join()’ call that blocks main until the sub-process has completed. What is not seen from glancing at the Python code is that underneath, the Queue depends on a limited size buffer, even if no maximum count is set. Because of that, if the get() and join() are out of order a deadlock can happen where the sub-process cannot exit because it cannot finish putting things in the queue, and the main process cannot continue to where it removes things from the queue because it is waiting on the sub-process to exit. “…you need to make sure that all items which have been put on the queue will eventually be removed before the process is joined.” - <https://docs.python.org/3.8/library/multiprocessing.html#programming-guidelines> \- Kevin Whalen ![JournalDev](https://www.gravatar.com/avatar/7c7a4546c8430a5a91a338a177bf8e7e8e153fe7eaee076acfa6d46029efef49?default=retro) [JournalDev](https://www.digitalocean.com/community/users/journaldev)![DigitalOcean Employee badge](https://www.digitalocean.com/api/static-content/v1/images?src=%2F_next%2Fstatic%2Fmedia%2Fdo-logo.15453cf1.svg&width=32) [May 8, 2020](https://www.digitalocean.com/community/tutorials/python-multiprocessing-example?comment=177738) Show less Thanks was helpful as always. The simple explanation is the strong point. \- Sukesh Suvarna ![JournalDev](https://www.gravatar.com/avatar/7c7a4546c8430a5a91a338a177bf8e7e8e153fe7eaee076acfa6d46029efef49?default=retro) [JournalDev](https://www.digitalocean.com/community/users/journaldev)![DigitalOcean Employee badge](https://www.digitalocean.com/api/static-content/v1/images?src=%2F_next%2Fstatic%2Fmedia%2Fdo-logo.15453cf1.svg&width=32) [June 6, 2020](https://www.digitalocean.com/community/tutorials/python-multiprocessing-example?comment=177739) Show less Please help me to resolve this issue <https://stackoverflow.com/q/62237516/13193575> \- Lovely Load more comments [![Creative Commons](https://www.digitalocean.com/api/static-content/v1/images?src=%2F_next%2Fstatic%2Fmedia%2Fcreativecommons.c0a877f1.png&width=384)](https://creativecommons.org/licenses/by-nc-sa/4.0/)This work is licensed under a Creative Commons Attribution-NonCommercial- ShareAlike 4.0 International License. ## Deploy on DigitalOcean Click below to sign up for DigitalOcean's virtual machines, Databases, and AIML products. [Sign up](https://cloud.digitalocean.com/registrations/new?refcode=f6fcd01aaffb) ## Popular Topics 1. [AI/ML](https://www.digitalocean.com/community/tags/ai-ml) 2. [Ubuntu](https://www.digitalocean.com/community/tags/ubuntu) 3. [Linux Basics](https://www.digitalocean.com/community/tags/linux-basics) 4. [JavaScript](https://www.digitalocean.com/community/tags/javascript) 5. [Python](https://www.digitalocean.com/community/tags/python) 6. [MySQL](https://www.digitalocean.com/community/tags/mysql) 7. [Docker](https://www.digitalocean.com/community/tags/docker) 8. [Kubernetes](https://www.digitalocean.com/community/tags/kubernetes) 9. [All tutorials](https://www.digitalocean.com/community/tutorials) 10. [Talk to an expert](https://www.digitalocean.com/company/contact/sales?referrer=tutorials) ### Connect on Discord Join the conversation in our Discord to connect with fellow developers [Visit Discord](https://discord.gg/digitalocean) ## Featured tutorials 1. [SOLID Design Principles Explained: Building Better Software Architecture](https://www.digitalocean.com/community/tutorials/s-o-l-i-d-the-first-five-principles-of-object-oriented-design) 2. [How To Remove Docker Images, Containers, and Volumes](https://www.digitalocean.com/community/tutorials/how-to-remove-docker-images-containers-and-volumes) 3. [How to Create a MySQL User and Grant Privileges (Step-by-Step)](https://www.digitalocean.com/community/tutorials/how-to-create-a-new-user-and-grant-permissions-in-mysql) - [All tutorials](https://www.digitalocean.com/community/tutorials) - [All topic tags](https://www.digitalocean.com/community/tags) ##### Join the Tech Talk - [Python Multiprocessing](https://www.digitalocean.com/community/tutorials/python-multiprocessing-example#python-multiprocessing) - ## Deploy on DigitalOcean Click below to sign up for DigitalOcean's virtual machines, Databases, and AIML products. [Sign up](https://cloud.digitalocean.com/registrations/new?refcode=f6fcd01aaffb) ## Popular Topics 1. [AI/ML](https://www.digitalocean.com/community/tags/ai-ml) 2. [Ubuntu](https://www.digitalocean.com/community/tags/ubuntu) 3. [Linux Basics](https://www.digitalocean.com/community/tags/linux-basics) 4. [JavaScript](https://www.digitalocean.com/community/tags/javascript) 5. [Python](https://www.digitalocean.com/community/tags/python) 6. [MySQL](https://www.digitalocean.com/community/tags/mysql) 7. [Docker](https://www.digitalocean.com/community/tags/docker) 8. [Kubernetes](https://www.digitalocean.com/community/tags/kubernetes) 9. [All tutorials](https://www.digitalocean.com/community/tutorials) 10. [Talk to an expert](https://www.digitalocean.com/company/contact/sales?referrer=tutorials) ### Connect on Discord Join the conversation in our Discord to connect with fellow developers [Visit Discord](https://discord.gg/digitalocean) ## Featured tutorials 1. [SOLID Design Principles Explained: Building Better Software Architecture](https://www.digitalocean.com/community/tutorials/s-o-l-i-d-the-first-five-principles-of-object-oriented-design) 2. [How To Remove Docker Images, Containers, and Volumes](https://www.digitalocean.com/community/tutorials/how-to-remove-docker-images-containers-and-volumes) 3. [How to Create a MySQL User and Grant Privileges (Step-by-Step)](https://www.digitalocean.com/community/tutorials/how-to-create-a-new-user-and-grant-permissions-in-mysql) - [All tutorials](https://www.digitalocean.com/community/tutorials) - [All topic tags](https://www.digitalocean.com/community/tags) ![](https://www.digitalocean.com/api/static-content/v1/images?src=%2F_next%2Fstatic%2Fmedia%2Ftutorials-2-tulip.764b9f59.svg&width=1920) ## Become a contributor for community Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation. [Sign Up](https://www.digitalocean.com/community/pages/write-for-digitalocean) ![](https://www.digitalocean.com/api/static-content/v1/images?src=%2F_next%2Fstatic%2Fmedia%2Fdocs-2-kiwi.239a03ef.svg&width=1920) ## DigitalOcean Documentation Full documentation for every DigitalOcean product. [Learn more](https://docs.digitalocean.com/) ![](https://www.digitalocean.com/api/static-content/v1/images?src=%2F_next%2Fstatic%2Fmedia%2Fblogs-1-lavender.495d1f00.svg&width=1920) ## Resources for startups and AI-native businesses The Wave has everything you need to know about building a business, from raising funding to marketing your product. [Learn more](https://www.digitalocean.com/resources) ## Get our newsletter Stay up to date by signing up for DigitalOcean’s Infrastructure as a Newsletter. New accounts only. By submitting your email you agree to our [Privacy Policy](https://www.digitalocean.com/legal/privacy-policy) ## The developer cloud Scale up as you grow — whether you're running one virtual machine or ten thousand. [View all products](https://www.digitalocean.com/products) ![](https://www.digitalocean.com/api/static-content/v1/images?src=%2F_next%2Fstatic%2Fmedia%2Fclouds-mobile.5d14bead.svg&width=3840) ## Get started for free Sign up and get \$200 in credit for your first 60 days with DigitalOcean.\* [Get started](https://cloud.digitalocean.com/registrations/new?refcode=f6fcd01aaffb) ![](https://www.digitalocean.com/api/static-content/v1/images?src=%2F_next%2Fstatic%2Fmedia%2Fwaves-mobile.a054c63e.svg&width=3840) \*This promotional offer applies to new accounts only. ## Company - [About](https://www.digitalocean.com/about) - [Leadership](https://www.digitalocean.com/leadership/executive-management) - [Blog](https://www.digitalocean.com/blog) - [Careers](https://www.digitalocean.com/careers) - [Customers](https://www.digitalocean.com/customers) - [Partners](https://www.digitalocean.com/partners) - [Referral Program](https://www.digitalocean.com/referral-program) - [Affiliate Program](https://www.digitalocean.com/affiliates) - [Press](https://www.digitalocean.com/press) - [Legal](https://www.digitalocean.com/legal) - [Privacy Policy](https://www.digitalocean.com/legal/privacy-policy) - [Security](https://www.digitalocean.com/security) - [Investor Relations](https://investors.digitalocean.com/) ## Products - [Overview](https://www.digitalocean.com/products) - [Droplets](https://www.digitalocean.com/products/droplets) - [Kubernetes](https://www.digitalocean.com/products/kubernetes) - [Functions](https://www.digitalocean.com/products/functions) - [App Platform](https://www.digitalocean.com/products/app-platform) - [Gradient™ AI GPU Droplets](https://www.digitalocean.com/products/gradient/gpu-droplets) - [Gradient™ AI Bare Metal GPUs](https://www.digitalocean.com/products/gradient/bare-metal-gpus) - [Gradient™ AI 1-Click Models](https://www.digitalocean.com/products/gradient/1-click-models) - [Gradient™ AI Platform](https://www.digitalocean.com/products/gradient/platform) - [Load Balancers](https://www.digitalocean.com/products/load-balancers) - [Managed Databases](https://www.digitalocean.com/products/managed-databases) - [Spaces](https://www.digitalocean.com/products/spaces) - [Block Storage](https://www.digitalocean.com/products/block-storage) - [Network File Storage](https://www.digitalocean.com/products/storage/network-file-storage) - [API](https://docs.digitalocean.com/reference/api) - [Uptime](https://www.digitalocean.com/products/uptime-monitoring) - [Identity and Access Management](https://www.digitalocean.com/products/identity-access-management) - [Cloudways](https://www.digitalocean.com/products/cloudways) ## Resources - [Community Tutorials](https://www.digitalocean.com/community/tutorials) - [Community Q\&A](https://www.digitalocean.com/community/questions) - [CSS-Tricks](https://css-tricks.com/) - [Write for DOnations](https://www.digitalocean.com/community/pages/write-for-digitalocean) - [Currents Research](https://www.digitalocean.com/currents) - [DigitalOcean Startups](https://www.digitalocean.com/startups) - [Wavemakers Program](https://www.digitalocean.com/wavemakers) - [Compass Council](https://www.digitalocean.com/research) - [Open Source](https://www.digitalocean.com/open-source) - [Newsletter Signup](https://www.digitalocean.com/community#iaan) - [Marketplace](https://www.digitalocean.com/products/marketplace) - [Pricing](https://www.digitalocean.com/pricing) - [Pricing Calculator](https://www.digitalocean.com/pricing/calculator) - [Documentation](https://docs.digitalocean.com/) - [Release Notes](https://docs.digitalocean.com/release-notes) - [Code of Conduct](https://www.digitalocean.com/community/pages/code-of-conduct) - [Shop Swag](http://store.digitalocean.com/) ## Solutions - [Website Hosting](https://www.digitalocean.com/solutions/website-hosting) - [VPS Hosting](https://www.digitalocean.com/solutions/vps-hosting) - [Web & Mobile Apps](https://www.digitalocean.com/solutions/web-mobile-apps) - [Game Development](https://www.digitalocean.com/solutions/gaming-development) - [Streaming](https://www.digitalocean.com/solutions/streaming) - [VPN](https://www.digitalocean.com/solutions/vpn) - [SaaS Platforms](https://www.digitalocean.com/solutions/saas) - [Cloud Hosting for Blockchain](https://www.digitalocean.com/solutions/blockchain) - [Startup Resources](https://www.digitalocean.com/resources) - [Migration Assistance](https://www.digitalocean.com/migrate) ## Contact - [Support](https://www.digitalocean.com/support) - [Sales](https://www.digitalocean.com/company/contact/sales?referrer=footer) - [Report Abuse](https://www.digitalocean.com/company/contact/abuse) - [System Status](https://status.digitalocean.com/) - [Share your ideas](https://ideas.digitalocean.com/) ## Company - [About](https://www.digitalocean.com/about) - [Leadership](https://www.digitalocean.com/leadership/executive-management) - [Blog](https://www.digitalocean.com/blog) - [Careers](https://www.digitalocean.com/careers) - [Customers](https://www.digitalocean.com/customers) - [Partners](https://www.digitalocean.com/partners) - [Referral Program](https://www.digitalocean.com/referral-program) - [Affiliate Program](https://www.digitalocean.com/affiliates) - [Press](https://www.digitalocean.com/press) - [Legal](https://www.digitalocean.com/legal) - [Privacy Policy](https://www.digitalocean.com/legal/privacy-policy) - [Security](https://www.digitalocean.com/security) - [Investor Relations](https://investors.digitalocean.com/) ## Products - [Overview](https://www.digitalocean.com/products) - [Droplets](https://www.digitalocean.com/products/droplets) - [Kubernetes](https://www.digitalocean.com/products/kubernetes) - [Functions](https://www.digitalocean.com/products/functions) - [App Platform](https://www.digitalocean.com/products/app-platform) - [Gradient™ AI GPU Droplets](https://www.digitalocean.com/products/gradient/gpu-droplets) - [Gradient™ AI Bare Metal GPUs](https://www.digitalocean.com/products/gradient/bare-metal-gpus) - [Gradient™ AI 1-Click Models](https://www.digitalocean.com/products/gradient/1-click-models) - [Gradient™ AI Platform](https://www.digitalocean.com/products/gradient/platform) - [Load Balancers](https://www.digitalocean.com/products/load-balancers) - [Managed Databases](https://www.digitalocean.com/products/managed-databases) - [Spaces](https://www.digitalocean.com/products/spaces) - [Block Storage](https://www.digitalocean.com/products/block-storage) - [Network File Storage](https://www.digitalocean.com/products/storage/network-file-storage) - [API](https://docs.digitalocean.com/reference/api) - [Uptime](https://www.digitalocean.com/products/uptime-monitoring) - [Identity and Access Management](https://www.digitalocean.com/products/identity-access-management) - [Cloudways](https://www.digitalocean.com/products/cloudways) ## Resources - [Community Tutorials](https://www.digitalocean.com/community/tutorials) - [Community Q\&A](https://www.digitalocean.com/community/questions) - [CSS-Tricks](https://css-tricks.com/) - [Write for DOnations](https://www.digitalocean.com/community/pages/write-for-digitalocean) - [Currents Research](https://www.digitalocean.com/currents) - [DigitalOcean Startups](https://www.digitalocean.com/startups) - [Wavemakers Program](https://www.digitalocean.com/wavemakers) - [Compass Council](https://www.digitalocean.com/research) - [Open Source](https://www.digitalocean.com/open-source) - [Newsletter Signup](https://www.digitalocean.com/community#iaan) - [Marketplace](https://www.digitalocean.com/products/marketplace) - [Pricing](https://www.digitalocean.com/pricing) - [Pricing Calculator](https://www.digitalocean.com/pricing/calculator) - [Documentation](https://docs.digitalocean.com/) - [Release Notes](https://docs.digitalocean.com/release-notes) - [Code of Conduct](https://www.digitalocean.com/community/pages/code-of-conduct) - [Shop Swag](http://store.digitalocean.com/) ## Solutions - [Website Hosting](https://www.digitalocean.com/solutions/website-hosting) - [VPS Hosting](https://www.digitalocean.com/solutions/vps-hosting) - [Web & Mobile Apps](https://www.digitalocean.com/solutions/web-mobile-apps) - [Game Development](https://www.digitalocean.com/solutions/gaming-development) - [Streaming](https://www.digitalocean.com/solutions/streaming) - [VPN](https://www.digitalocean.com/solutions/vpn) - [SaaS Platforms](https://www.digitalocean.com/solutions/saas) - [Cloud Hosting for Blockchain](https://www.digitalocean.com/solutions/blockchain) - [Startup Resources](https://www.digitalocean.com/resources) - [Migration Assistance](https://www.digitalocean.com/migrate) ## Contact - [Support](https://www.digitalocean.com/support) - [Sales](https://www.digitalocean.com/company/contact/sales?referrer=footer) - [Report Abuse](https://www.digitalocean.com/company/contact/abuse) - [System Status](https://status.digitalocean.com/) - [Share your ideas](https://ideas.digitalocean.com/) © 2026 DigitalOcean, LLC.[Sitemap](https://www.digitalocean.com/sitemap).[Cookie Preferences]() This site uses cookies and related technologies, as described in our [privacy policy](https://www.digitalocean.com/legal/privacy-policy/), for purposes that may include site operation, analytics, enhanced user experience, or advertising. You may choose to consent to our use of these technologies, or manage your own preferences. Please visit our [cookie policy](https://www.digitalocean.com/legal/cookie-policy) for more information. Agree & Proceed Decline All Manage Choices Loading... ## Community ## Product Docs ## Marketplace ## DigitalOcean Blog navigate go exit
Readable Markdown	In our previous tutorial, we learned about [Python CSV Example](https://www.digitalocean.com/community/tutorials/parse-csv-files-in-python). In this tutorial we are going to learn Python Multiprocessing with examples. ## [Python Multiprocessing](https://www.digitalocean.com/community/tutorials/python-multiprocessing-example#python-multiprocessing) Parallel processing is getting more attention nowadays. If you still don’t know about the parallel processing, learn from [wikipedia](https://en.wikipedia.org/wiki/Parallel_processing). As CPU manufacturers start adding more and more cores to their processors, creating parallel code is a great way to improve performance. Python introduced multiprocessing module to let us write parallel code. To understand the main motivation of this module, we have to know some basics about parallel programming. After reading this article, we hope that, you would be able to gather some knowledge on this topic. ### [Python Multiprocessing Process, Queue and Locks](https://www.digitalocean.com/community/tutorials/python-multiprocessing-example#python-multiprocessing-process-queue-and-locks) There are plenty of classes in python multiprocessing module for building a parallel program. Among them, three basic classes are `Process`, `Queue` and `Lock`. These classes will help you to build a parallel program. But before describing about those, let us initiate this topic with simple code. To make a parallel program useful, you have to know how many cores are there in you pc. Python Multiprocessing module enables you to know that. The following simple code will print the number of cores in your pc. ``` import multiprocessing print("Number of cpu : ", multiprocessing.cpu_count()) ``` The following output may vary for your pc. For me, number of cores is 8. ![python multiprocessing example cpu count](https://journaldev.nyc3.cdn.digitaloceanspaces.com/2017/09/python-multiprocessing-cpu-count.png) ### [Python multiprocessing Process class](https://www.digitalocean.com/community/tutorials/python-multiprocessing-example#python-multiprocessing-process-class) Python multiprocessing `Process` class is an abstraction that sets up another Python process, provides it to run code and a way for the parent application to control execution. There are two important functions that belongs to the Process class - `start()` and `join()` function. At first, we need to write a function, that will be run by the process. Then, we need to instantiate a process object. If we create a process object, nothing will happen until we tell it to start processing via `start()` function. Then, the process will run and return its result. After that we tell the process to complete via `join()` function. Without `join()` function call, process will remain idle and won’t terminate. So if you create many processes and don’t terminate them, you may face scarcity of resources. Then you may need to kill them manually. One important thing is, if you want to pass any argument through the process you need to use `args` keyword argument. The following code will be helpful to understand the usage of Process class. ``` from multiprocessing import Process def print_func(continent='Asia'): print('The name of continent is : ', continent) if __name__ == "__main__": # confirms that the code is under main function names = ['America', 'Europe', 'Africa'] procs = [] proc = Process(target=print_func) # instantiating without any argument procs.append(proc) proc.start() # instantiating process with arguments for name in names: # print(name) proc = Process(target=print_func, args=(name,)) procs.append(proc) proc.start() # complete the processes for proc in procs: proc.join() ``` The output of the following code will be: ![python multiprocessing example process class](https://journaldev.nyc3.cdn.digitaloceanspaces.com/2017/09/python-multiprocessing-process.png) ### [Python multiprocessing Queue class](https://www.digitalocean.com/community/tutorials/python-multiprocessing-example#python-multiprocessing-queue-class) You have basic knowledge about computer data-structure, you probably know about Queue. Python Multiprocessing modules provides `Queue` class that is exactly a First-In-First-Out data structure. They can store any pickle Python object (though simple ones are best) and are extremely useful for sharing data between processes. Queues are specially useful when passed as a parameter to a Process’ target function to enable the Process to consume data. By using `put()` function we can insert data to then queue and using `get()` we can get items from queues. See the following code for a quick example. ``` from multiprocessing import Queue colors = ['red', 'green', 'blue', 'black'] cnt = 1 # instantiating a queue object queue = Queue() print('pushing items to queue:') for color in colors: print('item no: ', cnt, ' ', color) queue.put(color) cnt += 1 print('\npopping items from queue:') cnt = 0 while not queue.empty(): print('item no: ', cnt, ' ', queue.get()) cnt += 1 ``` ![python multiprocessing queue](https://journaldev.nyc3.cdn.digitaloceanspaces.com/2017/09/python-multiprocessing-queue.png) ### [Python multiprocessing Lock Class](https://www.digitalocean.com/community/tutorials/python-multiprocessing-example#python-multiprocessing-lock-class) The task of Lock class is quite simple. It allows code to claim lock so that no other process can execute the similar code until the lock has be released. So the task of Lock class is mainly two. One is to claim lock and other is to release the lock. To claim lock the, `acquire()` function is used and to release lock `release()` function is used. ### [Python multiprocessing example](https://www.digitalocean.com/community/tutorials/python-multiprocessing-example#python-multiprocessing-example) In this Python multiprocessing example, we will merge all our knowledge together. Suppose we have some tasks to accomplish. To get that task done, we will use several processes. So, we will maintain two queue. One will contain the tasks and the other will contain the log of completed task. Then we instantiate the processes to complete the task. Note that the python Queue class is already synchronized. That means, we don’t need to use the Lock class to block multiple process to access the same queue object. That’s why, we don’t need to use Lock class in this case. Below is the implementation where we are adding tasks to the queue, then creating processes and starting them, then using `join()` to complete the processes. Finally we are printing the log from the second queue. ``` from multiprocessing import Lock, Process, Queue, current_process import time import queue # imported for using queue.Empty exception def do_job(tasks_to_accomplish, tasks_that_are_done): while True: try: ''' try to get task from the queue. get_nowait() function will raise queue.Empty exception if the queue is empty. queue(False) function would do the same task also. ''' task = tasks_to_accomplish.get_nowait() except queue.Empty: break else: ''' if no exception has been raised, add the task completion message to task_that_are_done queue ''' print(task) tasks_that_are_done.put(task + ' is done by ' + current_process().name) time.sleep(.5) return True def main(): number_of_task = 10 number_of_processes = 4 tasks_to_accomplish = Queue() tasks_that_are_done = Queue() processes = [] for i in range(number_of_task): tasks_to_accomplish.put("Task no " + str(i)) # creating processes for w in range(number_of_processes): p = Process(target=do_job, args=(tasks_to_accomplish, tasks_that_are_done)) processes.append(p) p.start() # completing process for p in processes: p.join() # print the output while not tasks_that_are_done.empty(): print(tasks_that_are_done.get()) return True if __name__ == '__main__': main() ``` Depending on the number of task, the code will take some time to show you the output. The output of the following code will vary from time to time. ![python multiprocessing example](https://journaldev.nyc3.cdn.digitaloceanspaces.com/2017/09/python-multiprocessing-example-1.png) ### [Python multiprocessing Pool](https://www.digitalocean.com/community/tutorials/python-multiprocessing-example#python-multiprocessing-pool) Python multiprocessing Pool can be used for parallel execution of a function across multiple input values, distributing the input data across processes (data parallelism). Below is a simple Python multiprocessing Pool example. ``` from multiprocessing import Pool import time work = (["A", 5], ["B", 2], ["C", 1], ["D", 3]) def work_log(work_data): print(" Process %s waiting %s seconds" % (work_data[0], work_data[1])) time.sleep(int(work_data[1])) print(" Process %s Finished." % work_data[0]) def pool_handler(): p = Pool(2) p.map(work_log, work) if __name__ == '__main__': pool_handler() ``` Below image shows the output of the above program. Notice that pool size is 2, so two executions of `work_log` function is happening in parallel. When one of the function processing finishes, it picks the next argument and so on. ![python multiprocessing pool](https://journaldev.nyc3.cdn.digitaloceanspaces.com/2017/09/python-multiprocessing-pool.png) So, that’s all for python multiprocessing module. Reference: [Official Documentation](https://docs.python.org/3.6/library/multiprocessing.html)
Shard	83 (laksa)
Root Hash	13428457316079428483
Unparsed URL	com,digitalocean!www,/community/tutorials/python-multiprocessing-example s443