🕷️ Crawler Inspector

URL Lookup

Direct Parameter Lookup

Raw Queries and Responses

1. Shard Calculation

Query:

Response:

Calculated Shard: 73 (from laksa191)

2. Crawled Status Check

Query:

curl -X POST \
  'http://laksa073.int.ahrefs:8124/' \
  -H 'Content-Type: text/plain' \
  -H 'X-ClickHouse-Database: crawler3' \
  -H 'Authorization: Basic YXBpOg==' \
  -d 'SELECT getAhrefsURLFromUnparsed(src_unparsed) AS found_url, ifNull(toUnixTimestamp(download_stamp), 0) AS crawl_time, ifNull(toUnixTimestamp(props_url_first_seen), 0) AS first_indexed_time, download_http_code AS http_code, src_unparsed AS src_unparsed, src_root_hash AS src_root_hash, history_drop_reason AS history_drop_reason, meta_title AS meta_title, meta_descriptions AS meta_descriptions, attrs_boilerpipe_text AS attrs_boilerpipe_text, attrs_markdown AS attrs_markdown, attrs_readable_markdown AS attrs_readable_markdown, meta_canonical AS meta_canonical FROM crawler3.page_info_local FINAL PREWHERE (src_root_hash, src_unparsed) IN ((getAhrefsRootHashFromUnparsed(getAhrefsUnparsedNoserviceFromURL(\'https://devarea.com/python-multitasking-multithreading-multiprocessing/\')), getAhrefsUnparsedNoserviceFromURL(\'https://devarea.com/python-multitasking-multithreading-multiprocessing/\'))) FORMAT JSONEachRow'

Response:

{"found_url":"https:\/\/devarea.com\/python-multitasking-multithreading-multiprocessing\/","crawl_time":1775896305,"first_indexed_time":1554398768,"http_code":200,"src_unparsed":"com,devarea!\/python-multitasking-multithreading-multiprocessing\/ s443","src_root_hash":"2261861757304640073","history_drop_reason":null,"meta_title":"Python Multitasking – MultiThreading and MultiProcessing – Developers Area","meta_descriptions":[],"attrs_boilerpipe_text":"Modern operating systems allow a user to run several applications simultaneously, so that some tasks can be run in the background while the user continues with other work in the foreground. The user can also run multiple copies of the same program at the same time.\nTo create a task we can use process or thread. Process has its private resources including memory mapping, files and other os objects. Multiple threads can run on the same process and share all its resources but if one thread fail it will kill all other threads in its process.\nPython has many packages to handle multi tasking, in this post i will cover some\nos.fork\nUnix\/Linux\/OS X specific (i.e. all but windows). The function creates a child process that start running after the fork return. We call fork once but it returns twice on the parent and on the child. Consider the following code:\nimport os\r\n\r\nx=os.fork()\r\nif x:\r\n    print('hello parent')\r\nelse:\r\n    print('hello child')\nIf you run it , you will see both prints:\nhello parent\r\nhello child\nOn fork call, the OS creates a child process and fork returns twice – on the parent the return value is the child process id and on the child fork returns 0. The point behind this weird behaviour is the Join-Fork parallel pattern – you run a code and want to use another processor for a complex task – fork a child, divide the work and join again:\nimport os\r\n\r\nret=0\r\n# we need to do something complex so lets create a child\r\nx=os.fork()\r\nif x:\r\n    print('do half work by parent')\r\n    ret = os.wait()\r\nelse:\r\n    print('do half work by child')\r\n    os._exit(10)\r\n\r\nprint(\"only parent here res=\" + str(os.WEXITSTATUS(ret[1])))\nBefore the fork call and after the if statement only one process exists , The child does its task and finish returning a value to the parent, If the parent finish before the child it will wait for him\nIt is important to call wait otherwise the child remains zombie\nos.system\nRuns a command in the shell (bash, cmd, etc)\nsometimes you only need to run another program or command for example you want to do something that easily done with a command like format\nimport os\r\n\r\nstatus = os.system('script2.py')\r\nprint (\"exit status\",status)\nos.popen\nRun a process and read its output as a file for example:\nimport os\r\n\r\nfor line in os.popen('ps aux').readlines():\r\n    print(line)\nThe subprocess package\nAn attempt to replace os specific function with platform independent. You can run a process and open a pipe in any os for example:\nimport subprocess\r\n\r\nsubprocess.run('script2.py')\r\npr = subprocess.Popen('ls')\r\npr.wait();\nto communicate with the output use:\nimport subprocess\r\nimport sys\r\n\r\nproc = subprocess.Popen([sys.executable, 'myapp.py', 'data'],\r\n                        stdout = subprocess.PIPE,\r\n                        stderr = subprocess.PIPE)\r\n(output, error) = proc.communicate()\r\nif error != None:\r\n    print(\"error:\", error.decode())\r\nprint(\"output:\", output.decode())\nThe multiprocessing package\nAnother useful package for process creation and other methods.\nThe multiprocessing module is suitable for sharing data or tasks between processor cores. It does not use threading, but processes instead. Processes are inherently more “expensive” that threads, so they are not worth using for trivial data sets or tasks\nimport multiprocessing\r\ndef fn(a,b):\r\n    print (\"vals=\",a,b)\r\n\r\nproc1 = multiprocessing.Process(target=fn, args=(10,20))\r\nproc2 = multiprocessing.Process(target=fn, args=(10,20))\r\nproc1.start()\r\nproc2.start()\r\nproc1.join()\r\nproc2.join()\nWorking with Threads\nThreads run on the same process address space – it is easy to share data between them but if one thread fails all other threads in the same process killed. To create a thread we use the\nthreading\npackage. We write a class derived from Thread and declare the function run. To create a thread we need to create an object from the class and call start method:\nimport threading\r\nimport time\r\nclass MyThread (threading.Thread):\r\n    def run (self):\r\n        for i in range(5):\r\n            print (\"In thread\", self.name)\r\n            time.sleep(2)\r\n\r\nth1 = MyThread()\r\nth2 = MyThread()\r\nth1.setName(\"t1\")\r\nth2.setName(\"t2\")\r\nth1.start()\r\nth2.start()\r\nprint (\"From main\")\r\nth1.join()\r\nth2.join()\noutput:\nIn thread t1\r\nIn thread t2\r\nFrom main\r\nIn thread t1\r\nIn thread t2\r\nIn thread t1\r\nIn thread t2\r\n....\nYou can also define the thread function as global (without class):\nfrom threading import Thread\r\nimport time\r\n\r\ndef myfunc(*args):\r\n    print \"From thread\", args\r\n    time.sleep(5)\r\n   \r\ntid1 = Thread(target=myfunc, args='T1')\r\ntid2 = Thread(target=myfunc, args='T2')\r\ntid1.start()\r\ntid2.start()\r\nprint \"From main\"\r\ntid1.join()\r\ntid2.join()\nSynchronization Objects\nAccessing resources needs synchronization. We can find the following objects in threading package:\nCondition variables – similar to linux posix condition variables\nEvents – similar to windows events\nLock – similar to mutex\nSemaphore\nLock Example:\nimport threading\r\nimport time\r\n\r\nlc = threading.Lock()\r\n\r\n\r\ndef myfunc1(*args):\r\n    global lc\r\n    lc.acquire()\r\n    print(\"From thread\", args)\r\n    lc.release()\r\n\r\ndef myfunc2(*args):\r\n    global lc\r\n    lc.acquire()\r\n    print(\"From thread\", args)\r\n    lc.release()\r\n\r\n\r\ntid1 = threading.Thread(target=myfunc1, args='T1')\r\ntid2 = threading.Thread(target=myfunc2, args='T2')\r\ntid1.start()\r\ntid2.start()\r\nprint(\"From main\")\r\ntid1.join()\r\ntid2.join()\nInter Process Communication (IPC)\nThreads share the same address space so its easy to send data from one thread to another but processes live in a different address spaces and thats why we need an IPC object. You can use pipe, fifo, message queue and more\nMessage Queue Example:\nIn the following example we create 2 processes and use a queue for communication. One process send message to the queue and the other receive:\nimport multiprocessing\r\nimport time\r\n\r\ndef fn1(q):\r\n    while True:\r\n        x=q.get()\r\n        print (\"val=\",x)\r\n\r\ndef fn2(q):\r\n    while True:\r\n        time.sleep(2)\r\n        q.put(\"hello\")\r\n\r\nqueue = multiprocessing.Queue()\r\nproc1 = multiprocessing.Process(target=fn1, args=(queue,))\r\nproc2 = multiprocessing.Process(target=fn2, args=(queue,))\r\nproc1.start()\r\nproc2.start()\nYou can find more methods for creating threads and processes, using synchronization objects and IPC","attrs_markdown":"[Developers Area](https:\/\/devarea.com\/ \"Developers Area\")\n\n[Just another Developers site](https:\/\/devarea.com\/ \"Just another Developers site\")\n\nToggle navigation\n\n- [Home](https:\/\/devarea.com\/)\n- [Labs](https:\/\/devarea.com\/labs\/)\n- [Nice Projects links](https:\/\/devarea.com\/nice-projects-links\/)\n- [GitHub](https:\/\/github.com\/dev-area)\n- [Contact Me](https:\/\/devarea.com\/contact\/)\n\n## [Python Multitasking – MultiThreading and MultiProcessing](https:\/\/devarea.com\/python-multitasking-multithreading-multiprocessing\/)\nBy [Liran B.H](https:\/\/devarea.com\/author\/liran\/) \\|\n\n[November 30, 2017](https:\/\/devarea.com\/2017\/11\/30\/)\n\n\\| [3 Comments](https:\/\/devarea.com\/python-multitasking-multithreading-multiprocessing\/#comments) \\| [python](https:\/\/devarea.com\/category\/python\/ \"View all posts in python\")\n\nModern operating systems allow a user to run several applications simultaneously, so that some tasks can be run in the background while the user continues with other work in the foreground. The user can also run multiple copies of the same program at the same time.\n\nTo create a task we can use process or thread. Process has its private resources including memory mapping, files and other os objects. Multiple threads can run on the same process and share all its resources but if one thread fail it will kill all other threads in its process.\n\nPython has many packages to handle multi tasking, in this post i will cover some\n\n### os.fork\nUnix\/Linux\/OS X specific (i.e. all but windows). The function creates a child process that start running after the fork return. We call fork once but it returns twice on the parent and on the child. Consider the following code:\n```\nimport os\r\n\r\nx=os.fork()\r\nif x:\r\n    print('hello parent')\r\nelse:\r\n    print('hello child')\n```\nIf you run it , you will see both prints:\n```\nhello parent\r\nhello child\n```\nOn fork call, the OS creates a child process and fork returns twice – on the parent the return value is the child process id and on the child fork returns 0. The point behind this weird behaviour is the Join-Fork parallel pattern – you run a code and want to use another processor for a complex task – fork a child, divide the work and join again:\n```\nimport os\r\n\r\nret=0\r\n# we need to do something complex so lets create a child\r\nx=os.fork()\r\nif x:\r\n    print('do half work by parent')\r\n    ret = os.wait()\r\nelse:\r\n    print('do half work by child')\r\n    os._exit(10)\r\n\r\nprint(\"only parent here res=\" + str(os.WEXITSTATUS(ret[1])))\n```\nBefore the fork call and after the if statement only one process exists , The child does its task and finish returning a value to the parent, If the parent finish before the child it will wait for him\n\nIt is important to call wait otherwise the child remains zombie\n\n### os.system\nRuns a command in the shell (bash, cmd, etc)\n\nsometimes you only need to run another program or command for example you want to do something that easily done with a command like format\n```\nimport os\r\n\r\nstatus = os.system('script2.py')\r\nprint (\"exit status\",status)\n```\n### os.popen\nRun a process and read its output as a file for example:\n```\nimport os\r\n\r\nfor line in os.popen('ps aux').readlines():\r\n    print(line)\n```\n### The subprocess package\nAn attempt to replace os specific function with platform independent. You can run a process and open a pipe in any os for example:\n```\nimport subprocess\r\n\r\nsubprocess.run('script2.py')\r\npr = subprocess.Popen('ls')\r\npr.wait();\n```\nto communicate with the output use:\n```\nimport subprocess\r\nimport sys\r\n\r\nproc = subprocess.Popen([sys.executable, 'myapp.py', 'data'],\r\n                        stdout = subprocess.PIPE,\r\n                        stderr = subprocess.PIPE)\r\n(output, error) = proc.communicate()\r\nif error != None:\r\n    print(\"error:\", error.decode())\r\nprint(\"output:\", output.decode())\n```\n### The multiprocessing package\nAnother useful package for process creation and other methods.\n\nThe multiprocessing module is suitable for sharing data or tasks between processor cores. It does not use threading, but processes instead. Processes are inherently more “expensive” that threads, so they are not worth using for trivial data sets or tasks\n```\nimport multiprocessing\r\ndef fn(a,b):\r\n    print (\"vals=\",a,b)\r\n\r\nproc1 = multiprocessing.Process(target=fn, args=(10,20))\r\nproc2 = multiprocessing.Process(target=fn, args=(10,20))\r\nproc1.start()\r\nproc2.start()\r\nproc1.join()\r\nproc2.join()\n```\n### Working with Threads\nThreads run on the same process address space – it is easy to share data between them but if one thread fails all other threads in the same process killed. To create a thread we use the **threading** package. We write a class derived from Thread and declare the function run. To create a thread we need to create an object from the class and call start method:\n```\nimport threading\r\nimport time\r\nclass MyThread (threading.Thread):\r\n    def run (self):\r\n        for i in range(5):\r\n            print (\"In thread\", self.name)\r\n            time.sleep(2)\r\n\r\nth1 = MyThread()\r\nth2 = MyThread()\r\nth1.setName(\"t1\")\r\nth2.setName(\"t2\")\r\nth1.start()\r\nth2.start()\r\nprint (\"From main\")\r\nth1.join()\r\nth2.join()\n```\noutput:\n```\nIn thread t1\r\nIn thread t2\r\nFrom main\r\nIn thread t1\r\nIn thread t2\r\nIn thread t1\r\nIn thread t2\r\n....\n```\nYou can also define the thread function as global (without class):\n```\nfrom threading import Thread\r\nimport time\r\n\r\ndef myfunc(*args):\r\n    print \"From thread\", args\r\n    time.sleep(5)\r\n   \r\ntid1 = Thread(target=myfunc, args='T1')\r\ntid2 = Thread(target=myfunc, args='T2')\r\ntid1.start()\r\ntid2.start()\r\nprint \"From main\"\r\ntid1.join()\r\ntid2.join()\n```\n###\n### Synchronization Objects\nAccessing resources needs synchronization. We can find the following objects in threading package:\n\n- Condition variables – similar to linux posix condition variables\n- Events – similar to windows events\n- Lock – similar to mutex\n- Semaphore\n#### Lock Example:\n```\nimport threading\r\nimport time\r\n\r\nlc = threading.Lock()\r\n\r\n\r\ndef myfunc1(*args):\r\n    global lc\r\n    lc.acquire()\r\n    print(\"From thread\", args)\r\n    lc.release()\r\n\r\ndef myfunc2(*args):\r\n    global lc\r\n    lc.acquire()\r\n    print(\"From thread\", args)\r\n    lc.release()\r\n\r\n\r\ntid1 = threading.Thread(target=myfunc1, args='T1')\r\ntid2 = threading.Thread(target=myfunc2, args='T2')\r\ntid1.start()\r\ntid2.start()\r\nprint(\"From main\")\r\ntid1.join()\r\ntid2.join()\n```\n###\n### Inter Process Communication (IPC)\nThreads share the same address space so its easy to send data from one thread to another but processes live in a different address spaces and thats why we need an IPC object. You can use pipe, fifo, message queue and more\n\n#### Message Queue Example:\nIn the following example we create 2 processes and use a queue for communication. One process send message to the queue and the other receive:\n```\nimport multiprocessing\r\nimport time\r\n\r\ndef fn1(q):\r\n    while True:\r\n        x=q.get()\r\n        print (\"val=\",x)\r\n\r\ndef fn2(q):\r\n    while True:\r\n        time.sleep(2)\r\n        q.put(\"hello\")\r\n\r\nqueue = multiprocessing.Queue()\r\nproc1 = multiprocessing.Process(target=fn1, args=(queue,))\r\nproc2 = multiprocessing.Process(target=fn2, args=(queue,))\r\nproc1.start()\r\nproc2.start()\n```\nYou can find more methods for creating threads and processes, using synchronization objects and IPC\n\nTagged [python](https:\/\/devarea.com\/tag\/python\/)\n\n## Post navigation\n[← Python Machine Learning Example – Linear Regression](https:\/\/devarea.com\/python-machine-learning-example-linear-regression\/)\n\n[Linux – Handling Signals in a Multithreaded Application →](https:\/\/devarea.com\/linux-handling-signals-in-a-multithreaded-application\/)\n\n## 3 thoughts on “Python Multitasking – MultiThreading and MultiProcessing”\n1. ![](https:\/\/secure.gravatar.com\/avatar\/c941b0ff8558631eafac4df0d5f21514a3a74cdc4973a8646ad99f4808273a2f?s=128&d=mm&r=g)\n   \n   [anushri](https:\/\/www.besanttechnologies.com\/training-courses\/python-training-institute-in-bangalore)\n   \n   Multiple threading are useful create program small size its use full to workout.\n   \n   [March 1, 2018](https:\/\/devarea.com\/python-multitasking-multithreading-multiprocessing\/#comment-602)\n   \n   \\-\n2. ![](https:\/\/secure.gravatar.com\/avatar\/f3c8a19e7523417bf916d633e3f1abde7d123c580d0b9d238fbc4e903c474b22?s=128&d=mm&r=g)\n   \n   Sergio\n   \n   Many thanks, very useful post\\!\n   \n   [May 28, 2019](https:\/\/devarea.com\/python-multitasking-multithreading-multiprocessing\/#comment-7016)\n   \n   \\-\n3. ![](https:\/\/secure.gravatar.com\/avatar\/fde9409b241151326a7ce105e5cb1bc44ad4f1a9a948512f912312ed7fe8f0f4?s=128&d=mm&r=g)\n   \n   [mama bear t shirt](http:\/\/ow.ly\/pqtI50wN35r)\n   \n   Hi friends, its fantastic post on the topic of teachingand fully defined, keep  \n    it up all the time.\n   \n   [November 30, 2019](https:\/\/devarea.com\/python-multitasking-multithreading-multiprocessing\/#comment-8856)\n   \n   \\-\n\nComments are closed.\n\n### Subscribe to our Newsletter\n### Categories\n- [Android](https:\/\/devarea.com\/category\/android\/)\n- [Angular](https:\/\/devarea.com\/category\/angular\/)\n- [Django](https:\/\/devarea.com\/category\/django\/)\n- [General](https:\/\/devarea.com\/category\/general\/)\n- [Linux](https:\/\/devarea.com\/category\/linux\/)\n- [Machine Learning](https:\/\/devarea.com\/category\/machine-learning\/)\n- [python](https:\/\/devarea.com\/category\/python\/)\n\n### Recent Posts\n- [Building a Web App with Angular, Django and Django REST](https:\/\/devarea.com\/building-a-web-app-with-angular-django-and-django-rest\/)\n- [Introduction To Network Filters – Linux](https:\/\/devarea.com\/introduction-to-network-filters-linux\/)\n- [Linear Regression With Numpy](https:\/\/devarea.com\/linear-regression-with-numpy\/)\n- [Python – Working With Virtual Environments](https:\/\/devarea.com\/python-working-with-virtual-environments\/)\n- [Python – Text Processing Introduction](https:\/\/devarea.com\/python-text-processing-introduction\/)\n\n### Tags\n[Android](https:\/\/devarea.com\/tag\/android\/) [Android Internals](https:\/\/devarea.com\/tag\/android-internals\/) [angular](https:\/\/devarea.com\/tag\/angular\/) [AOSP](https:\/\/devarea.com\/tag\/aosp\/) [Assembly](https:\/\/devarea.com\/tag\/assembly\/) [C](https:\/\/devarea.com\/tag\/c\/) [Debug](https:\/\/devarea.com\/tag\/debug\/) [django](https:\/\/devarea.com\/tag\/django\/) [git](https:\/\/devarea.com\/tag\/git\/) [Java](https:\/\/devarea.com\/tag\/java\/) [Kernel](https:\/\/devarea.com\/tag\/kernel\/) [Linux](https:\/\/devarea.com\/tag\/linux\/) [Machine Learning](https:\/\/devarea.com\/tag\/machine-learning\/) [Networking](https:\/\/devarea.com\/tag\/networking\/) [numpy](https:\/\/devarea.com\/tag\/numpy\/) [pandas](https:\/\/devarea.com\/tag\/pandas\/) [python](https:\/\/devarea.com\/tag\/python\/) [rest](https:\/\/devarea.com\/tag\/rest\/) [scipy](https:\/\/devarea.com\/tag\/scipy\/) [security](https:\/\/devarea.com\/tag\/security\/)\n### Subscribe to our Newsletter\n### Archives\n- [April 2019](https:\/\/devarea.com\/2019\/04\/)\n- [March 2019](https:\/\/devarea.com\/2019\/03\/)\n- [May 2018](https:\/\/devarea.com\/2018\/05\/)\n- [March 2018](https:\/\/devarea.com\/2018\/03\/)\n- [February 2018](https:\/\/devarea.com\/2018\/02\/)\n- [January 2018](https:\/\/devarea.com\/2018\/01\/)\n- [December 2017](https:\/\/devarea.com\/2017\/12\/)\n- [November 2017](https:\/\/devarea.com\/2017\/11\/)\n- [October 2017](https:\/\/devarea.com\/2017\/10\/)\n- [September 2017](https:\/\/devarea.com\/2017\/09\/)\n- [August 2017](https:\/\/devarea.com\/2017\/08\/)\n- [May 2017](https:\/\/devarea.com\/2017\/05\/)\n- [March 2017](https:\/\/devarea.com\/2017\/03\/)\n\n### Categories\n- [Android](https:\/\/devarea.com\/category\/android\/)\n- [Angular](https:\/\/devarea.com\/category\/angular\/)\n- [Django](https:\/\/devarea.com\/category\/django\/)\n- [General](https:\/\/devarea.com\/category\/general\/)\n- [Linux](https:\/\/devarea.com\/category\/linux\/)\n- [Machine Learning](https:\/\/devarea.com\/category\/machine-learning\/)\n- [python](https:\/\/devarea.com\/category\/python\/)\n\n### Recent Posts\n- [Building a Web App with Angular, Django and Django REST](https:\/\/devarea.com\/building-a-web-app-with-angular-django-and-django-rest\/)\n- [Introduction To Network Filters – Linux](https:\/\/devarea.com\/introduction-to-network-filters-linux\/)\n- [Linear Regression With Numpy](https:\/\/devarea.com\/linear-regression-with-numpy\/)\n- [Python – Working With Virtual Environments](https:\/\/devarea.com\/python-working-with-virtual-environments\/)\n- [Python – Text Processing Introduction](https:\/\/devarea.com\/python-text-processing-introduction\/)\n\n### Tags\n[Android](https:\/\/devarea.com\/tag\/android\/) [Android Internals](https:\/\/devarea.com\/tag\/android-internals\/) [angular](https:\/\/devarea.com\/tag\/angular\/) [AOSP](https:\/\/devarea.com\/tag\/aosp\/) [Assembly](https:\/\/devarea.com\/tag\/assembly\/) [C](https:\/\/devarea.com\/tag\/c\/) [Debug](https:\/\/devarea.com\/tag\/debug\/) [django](https:\/\/devarea.com\/tag\/django\/) [git](https:\/\/devarea.com\/tag\/git\/) [Java](https:\/\/devarea.com\/tag\/java\/) [Kernel](https:\/\/devarea.com\/tag\/kernel\/) [Linux](https:\/\/devarea.com\/tag\/linux\/) [Machine Learning](https:\/\/devarea.com\/tag\/machine-learning\/) [Networking](https:\/\/devarea.com\/tag\/networking\/) [numpy](https:\/\/devarea.com\/tag\/numpy\/) [pandas](https:\/\/devarea.com\/tag\/pandas\/) [python](https:\/\/devarea.com\/tag\/python\/) [rest](https:\/\/devarea.com\/tag\/rest\/) [scipy](https:\/\/devarea.com\/tag\/scipy\/) [security](https:\/\/devarea.com\/tag\/security\/)\n\n### Subscribe to our mailing list\n### Follow Me\n\n***\n[ShopIsle](http:\/\/themeisle.com\/themes\/shop-isle\/) powered by [WordPress](http:\/\/wordpress.org\/)","attrs_readable_markdown":"Modern operating systems allow a user to run several applications simultaneously, so that some tasks can be run in the background while the user continues with other work in the foreground. The user can also run multiple copies of the same program at the same time.\n\nTo create a task we can use process or thread. Process has its private resources including memory mapping, files and other os objects. Multiple threads can run on the same process and share all its resources but if one thread fail it will kill all other threads in its process.\n\nPython has many packages to handle multi tasking, in this post i will cover some\n\n### os.fork\nUnix\/Linux\/OS X specific (i.e. all but windows). The function creates a child process that start running after the fork return. We call fork once but it returns twice on the parent and on the child. Consider the following code:\n```\nimport os\r\n\r\nx=os.fork()\r\nif x:\r\n    print('hello parent')\r\nelse:\r\n    print('hello child')\n```\nIf you run it , you will see both prints:\n```\nhello parent\r\nhello child\n```\nOn fork call, the OS creates a child process and fork returns twice – on the parent the return value is the child process id and on the child fork returns 0. The point behind this weird behaviour is the Join-Fork parallel pattern – you run a code and want to use another processor for a complex task – fork a child, divide the work and join again:\n```\nimport os\r\n\r\nret=0\r\n# we need to do something complex so lets create a child\r\nx=os.fork()\r\nif x:\r\n    print('do half work by parent')\r\n    ret = os.wait()\r\nelse:\r\n    print('do half work by child')\r\n    os._exit(10)\r\n\r\nprint(\"only parent here res=\" + str(os.WEXITSTATUS(ret[1])))\n```\nBefore the fork call and after the if statement only one process exists , The child does its task and finish returning a value to the parent, If the parent finish before the child it will wait for him\n\nIt is important to call wait otherwise the child remains zombie\n\n### os.system\nRuns a command in the shell (bash, cmd, etc)\n\nsometimes you only need to run another program or command for example you want to do something that easily done with a command like format\n```\nimport os\r\n\r\nstatus = os.system('script2.py')\r\nprint (\"exit status\",status)\n```\n### os.popen\nRun a process and read its output as a file for example:\n```\nimport os\r\n\r\nfor line in os.popen('ps aux').readlines():\r\n    print(line)\n```\n### The subprocess package\nAn attempt to replace os specific function with platform independent. You can run a process and open a pipe in any os for example:\n```\nimport subprocess\r\n\r\nsubprocess.run('script2.py')\r\npr = subprocess.Popen('ls')\r\npr.wait();\n```\nto communicate with the output use:\n```\nimport subprocess\r\nimport sys\r\n\r\nproc = subprocess.Popen([sys.executable, 'myapp.py', 'data'],\r\n                        stdout = subprocess.PIPE,\r\n                        stderr = subprocess.PIPE)\r\n(output, error) = proc.communicate()\r\nif error != None:\r\n    print(\"error:\", error.decode())\r\nprint(\"output:\", output.decode())\n```\n### The multiprocessing package\nAnother useful package for process creation and other methods.\n\nThe multiprocessing module is suitable for sharing data or tasks between processor cores. It does not use threading, but processes instead. Processes are inherently more “expensive” that threads, so they are not worth using for trivial data sets or tasks\n```\nimport multiprocessing\r\ndef fn(a,b):\r\n    print (\"vals=\",a,b)\r\n\r\nproc1 = multiprocessing.Process(target=fn, args=(10,20))\r\nproc2 = multiprocessing.Process(target=fn, args=(10,20))\r\nproc1.start()\r\nproc2.start()\r\nproc1.join()\r\nproc2.join()\n```\n### Working with Threads\nThreads run on the same process address space – it is easy to share data between them but if one thread fails all other threads in the same process killed. To create a thread we use the **threading** package. We write a class derived from Thread and declare the function run. To create a thread we need to create an object from the class and call start method:\n```\nimport threading\r\nimport time\r\nclass MyThread (threading.Thread):\r\n    def run (self):\r\n        for i in range(5):\r\n            print (\"In thread\", self.name)\r\n            time.sleep(2)\r\n\r\nth1 = MyThread()\r\nth2 = MyThread()\r\nth1.setName(\"t1\")\r\nth2.setName(\"t2\")\r\nth1.start()\r\nth2.start()\r\nprint (\"From main\")\r\nth1.join()\r\nth2.join()\n```\noutput:\n```\nIn thread t1\r\nIn thread t2\r\nFrom main\r\nIn thread t1\r\nIn thread t2\r\nIn thread t1\r\nIn thread t2\r\n....\n```\nYou can also define the thread function as global (without class):\n```\nfrom threading import Thread\r\nimport time\r\n\r\ndef myfunc(*args):\r\n    print \"From thread\", args\r\n    time.sleep(5)\r\n   \r\ntid1 = Thread(target=myfunc, args='T1')\r\ntid2 = Thread(target=myfunc, args='T2')\r\ntid1.start()\r\ntid2.start()\r\nprint \"From main\"\r\ntid1.join()\r\ntid2.join()\n```\n### Synchronization Objects\nAccessing resources needs synchronization. We can find the following objects in threading package:\n\n- Condition variables – similar to linux posix condition variables\n- Events – similar to windows events\n- Lock – similar to mutex\n- Semaphore\n#### Lock Example:\n```\nimport threading\r\nimport time\r\n\r\nlc = threading.Lock()\r\n\r\n\r\ndef myfunc1(*args):\r\n    global lc\r\n    lc.acquire()\r\n    print(\"From thread\", args)\r\n    lc.release()\r\n\r\ndef myfunc2(*args):\r\n    global lc\r\n    lc.acquire()\r\n    print(\"From thread\", args)\r\n    lc.release()\r\n\r\n\r\ntid1 = threading.Thread(target=myfunc1, args='T1')\r\ntid2 = threading.Thread(target=myfunc2, args='T2')\r\ntid1.start()\r\ntid2.start()\r\nprint(\"From main\")\r\ntid1.join()\r\ntid2.join()\n```\n### Inter Process Communication (IPC)\nThreads share the same address space so its easy to send data from one thread to another but processes live in a different address spaces and thats why we need an IPC object. You can use pipe, fifo, message queue and more\n\n#### Message Queue Example:\nIn the following example we create 2 processes and use a queue for communication. One process send message to the queue and the other receive:\n```\nimport multiprocessing\r\nimport time\r\n\r\ndef fn1(q):\r\n    while True:\r\n        x=q.get()\r\n        print (\"val=\",x)\r\n\r\ndef fn2(q):\r\n    while True:\r\n        time.sleep(2)\r\n        q.put(\"hello\")\r\n\r\nqueue = multiprocessing.Queue()\r\nproc1 = multiprocessing.Process(target=fn1, args=(queue,))\r\nproc2 = multiprocessing.Process(target=fn2, args=(queue,))\r\nproc1.start()\r\nproc2.start()\n```\nYou can find more methods for creating threads and processes, using synchronization objects and IPC","meta_canonical":null}

3. Robots.txt Check

Query:

Response:

4. Spam/Ban Check

Query:

Response:

5. Seen Status Check

ℹ️ Skipped - page is already crawled

📄

INDEXABLE

✅

CRAWLED

6 days ago

🤖

ROBOTS ALLOWED

Page Info Filters

Filter	Status	Condition	Details
HTTP status	PASS	`download_http_code = 200`	HTTP 200
Age cutoff	PASS	`download_stamp > now() - 6 MONTH`	0.2 months ago
History drop	PASS	`isNull(history_drop_reason)`	No drop reason
Spam/ban	PASS	`fh_dont_index != 1 AND ml_spam_score = 0`	ml_spam_score=0
Canonical	PASS	`meta_canonical IS NULL OR = '' OR = src_unparsed`	Not set

Page Details

Property	Value
URL	https://devarea.com/python-multitasking-multithreading-multiprocessing/
Last Crawled	2026-04-11 08:31:45 (6 days ago)
First Indexed	2019-04-04 17:26:08 (7 years ago)
HTTP Status Code	200
Meta Title	Python Multitasking – MultiThreading and MultiProcessing – Developers Area
Meta Description	null
Meta Canonical	null
Boilerpipe Text	Modern operating systems allow a user to run several applications simultaneously, so that some tasks can be run in the background while the user continues with other work in the foreground. The user can also run multiple copies of the same program at the same time. To create a task we can use process or thread. Process has its private resources including memory mapping, files and other os objects. Multiple threads can run on the same process and share all its resources but if one thread fail it will kill all other threads in its process. Python has many packages to handle multi tasking, in this post i will cover some os.fork Unix/Linux/OS X specific (i.e. all but windows). The function creates a child process that start running after the fork return. We call fork once but it returns twice on the parent and on the child. Consider the following code: import os x=os.fork() if x: print('hello parent') else: print('hello child') If you run it , you will see both prints: hello parent hello child On fork call, the OS creates a child process and fork returns twice – on the parent the return value is the child process id and on the child fork returns 0. The point behind this weird behaviour is the Join-Fork parallel pattern – you run a code and want to use another processor for a complex task – fork a child, divide the work and join again: import os ret=0 # we need to do something complex so lets create a child x=os.fork() if x: print('do half work by parent') ret = os.wait() else: print('do half work by child') os._exit(10) print("only parent here res=" + str(os.WEXITSTATUS(ret[1]))) Before the fork call and after the if statement only one process exists , The child does its task and finish returning a value to the parent, If the parent finish before the child it will wait for him It is important to call wait otherwise the child remains zombie os.system Runs a command in the shell (bash, cmd, etc) sometimes you only need to run another program or command for example you want to do something that easily done with a command like format import os status = os.system('script2.py') print ("exit status",status) os.popen Run a process and read its output as a file for example: import os for line in os.popen('ps aux').readlines(): print(line) The subprocess package An attempt to replace os specific function with platform independent. You can run a process and open a pipe in any os for example: import subprocess subprocess.run('script2.py') pr = subprocess.Popen('ls') pr.wait(); to communicate with the output use: import subprocess import sys proc = subprocess.Popen([sys.executable, 'myapp.py', 'data'], stdout = subprocess.PIPE, stderr = subprocess.PIPE) (output, error) = proc.communicate() if error != None: print("error:", error.decode()) print("output:", output.decode()) The multiprocessing package Another useful package for process creation and other methods. The multiprocessing module is suitable for sharing data or tasks between processor cores. It does not use threading, but processes instead. Processes are inherently more “expensive” that threads, so they are not worth using for trivial data sets or tasks import multiprocessing def fn(a,b): print ("vals=",a,b) proc1 = multiprocessing.Process(target=fn, args=(10,20)) proc2 = multiprocessing.Process(target=fn, args=(10,20)) proc1.start() proc2.start() proc1.join() proc2.join() Working with Threads Threads run on the same process address space – it is easy to share data between them but if one thread fails all other threads in the same process killed. To create a thread we use the threading package. We write a class derived from Thread and declare the function run. To create a thread we need to create an object from the class and call start method: import threading import time class MyThread (threading.Thread): def run (self): for i in range(5): print ("In thread", self.name) time.sleep(2) th1 = MyThread() th2 = MyThread() th1.setName("t1") th2.setName("t2") th1.start() th2.start() print ("From main") th1.join() th2.join() output: In thread t1 In thread t2 From main In thread t1 In thread t2 In thread t1 In thread t2 .... You can also define the thread function as global (without class): from threading import Thread import time def myfunc(args): print "From thread", args time.sleep(5) tid1 = Thread(target=myfunc, args='T1') tid2 = Thread(target=myfunc, args='T2') tid1.start() tid2.start() print "From main" tid1.join() tid2.join() Synchronization Objects Accessing resources needs synchronization. We can find the following objects in threading package: Condition variables – similar to linux posix condition variables Events – similar to windows events Lock – similar to mutex Semaphore Lock Example: import threading import time lc = threading.Lock() def myfunc1(args): global lc lc.acquire() print("From thread", args) lc.release() def myfunc2(*args): global lc lc.acquire() print("From thread", args) lc.release() tid1 = threading.Thread(target=myfunc1, args='T1') tid2 = threading.Thread(target=myfunc2, args='T2') tid1.start() tid2.start() print("From main") tid1.join() tid2.join() Inter Process Communication (IPC) Threads share the same address space so its easy to send data from one thread to another but processes live in a different address spaces and thats why we need an IPC object. You can use pipe, fifo, message queue and more Message Queue Example: In the following example we create 2 processes and use a queue for communication. One process send message to the queue and the other receive: import multiprocessing import time def fn1(q): while True: x=q.get() print ("val=",x) def fn2(q): while True: time.sleep(2) q.put("hello") queue = multiprocessing.Queue() proc1 = multiprocessing.Process(target=fn1, args=(queue,)) proc2 = multiprocessing.Process(target=fn2, args=(queue,)) proc1.start() proc2.start() You can find more methods for creating threads and processes, using synchronization objects and IPC
Markdown	[Developers Area](https://devarea.com/ "Developers Area") [Just another Developers site](https://devarea.com/ "Just another Developers site") Toggle navigation - [Home](https://devarea.com/) - [Labs](https://devarea.com/labs/) - [Nice Projects links](https://devarea.com/nice-projects-links/) - [GitHub](https://github.com/dev-area) - [Contact Me](https://devarea.com/contact/) ## [Python Multitasking – MultiThreading and MultiProcessing](https://devarea.com/python-multitasking-multithreading-multiprocessing/) By [Liran B.H](https://devarea.com/author/liran/) \\| [November 30, 2017](https://devarea.com/2017/11/30/) \\| [3 Comments](https://devarea.com/python-multitasking-multithreading-multiprocessing/#comments) \\| [python](https://devarea.com/category/python/ "View all posts in python") Modern operating systems allow a user to run several applications simultaneously, so that some tasks can be run in the background while the user continues with other work in the foreground. The user can also run multiple copies of the same program at the same time. To create a task we can use process or thread. Process has its private resources including memory mapping, files and other os objects. Multiple threads can run on the same process and share all its resources but if one thread fail it will kill all other threads in its process. Python has many packages to handle multi tasking, in this post i will cover some ### os.fork Unix/Linux/OS X specific (i.e. all but windows). The function creates a child process that start running after the fork return. We call fork once but it returns twice on the parent and on the child. Consider the following code: ``` import os x=os.fork() if x: print('hello parent') else: print('hello child') ``` If you run it , you will see both prints: ``` hello parent hello child ``` On fork call, the OS creates a child process and fork returns twice – on the parent the return value is the child process id and on the child fork returns 0. The point behind this weird behaviour is the Join-Fork parallel pattern – you run a code and want to use another processor for a complex task – fork a child, divide the work and join again: ``` import os ret=0 # we need to do something complex so lets create a child x=os.fork() if x: print('do half work by parent') ret = os.wait() else: print('do half work by child') os._exit(10) print("only parent here res=" + str(os.WEXITSTATUS(ret[1]))) ``` Before the fork call and after the if statement only one process exists , The child does its task and finish returning a value to the parent, If the parent finish before the child it will wait for him It is important to call wait otherwise the child remains zombie ### os.system Runs a command in the shell (bash, cmd, etc) sometimes you only need to run another program or command for example you want to do something that easily done with a command like format ``` import os status = os.system('script2.py') print ("exit status",status) ``` ### os.popen Run a process and read its output as a file for example: ``` import os for line in os.popen('ps aux').readlines(): print(line) ``` ### The subprocess package An attempt to replace os specific function with platform independent. You can run a process and open a pipe in any os for example: ``` import subprocess subprocess.run('script2.py') pr = subprocess.Popen('ls') pr.wait(); ``` to communicate with the output use: ``` import subprocess import sys proc = subprocess.Popen([sys.executable, 'myapp.py', 'data'], stdout = subprocess.PIPE, stderr = subprocess.PIPE) (output, error) = proc.communicate() if error != None: print("error:", error.decode()) print("output:", output.decode()) ``` ### The multiprocessing package Another useful package for process creation and other methods. The multiprocessing module is suitable for sharing data or tasks between processor cores. It does not use threading, but processes instead. Processes are inherently more “expensive” that threads, so they are not worth using for trivial data sets or tasks ``` import multiprocessing def fn(a,b): print ("vals=",a,b) proc1 = multiprocessing.Process(target=fn, args=(10,20)) proc2 = multiprocessing.Process(target=fn, args=(10,20)) proc1.start() proc2.start() proc1.join() proc2.join() ``` ### Working with Threads Threads run on the same process address space – it is easy to share data between them but if one thread fails all other threads in the same process killed. To create a thread we use the threading package. We write a class derived from Thread and declare the function run. To create a thread we need to create an object from the class and call start method: ``` import threading import time class MyThread (threading.Thread): def run (self): for i in range(5): print ("In thread", self.name) time.sleep(2) th1 = MyThread() th2 = MyThread() th1.setName("t1") th2.setName("t2") th1.start() th2.start() print ("From main") th1.join() th2.join() ``` output: ``` In thread t1 In thread t2 From main In thread t1 In thread t2 In thread t1 In thread t2 .... ``` You can also define the thread function as global (without class): ``` from threading import Thread import time def myfunc(args): print "From thread", args time.sleep(5) tid1 = Thread(target=myfunc, args='T1') tid2 = Thread(target=myfunc, args='T2') tid1.start() tid2.start() print "From main" tid1.join() tid2.join() ``` ### ### Synchronization Objects Accessing resources needs synchronization. We can find the following objects in threading package: - Condition variables – similar to linux posix condition variables - Events – similar to windows events - Lock – similar to mutex - Semaphore #### Lock Example: ``` import threading import time lc = threading.Lock() def myfunc1(args): global lc lc.acquire() print("From thread", args) lc.release() def myfunc2(args): global lc lc.acquire() print("From thread", args) lc.release() tid1 = threading.Thread(target=myfunc1, args='T1') tid2 = threading.Thread(target=myfunc2, args='T2') tid1.start() tid2.start() print("From main") tid1.join() tid2.join() ``` ### ### Inter Process Communication (IPC) Threads share the same address space so its easy to send data from one thread to another but processes live in a different address spaces and thats why we need an IPC object. You can use pipe, fifo, message queue and more #### Message Queue Example: In the following example we create 2 processes and use a queue for communication. One process send message to the queue and the other receive: ``` import multiprocessing import time def fn1(q): while True: x=q.get() print ("val=",x) def fn2(q): while True: time.sleep(2) q.put("hello") queue = multiprocessing.Queue() proc1 = multiprocessing.Process(target=fn1, args=(queue,)) proc2 = multiprocessing.Process(target=fn2, args=(queue,)) proc1.start() proc2.start() ``` You can find more methods for creating threads and processes, using synchronization objects and IPC Tagged [python](https://devarea.com/tag/python/) ## Post navigation [← Python Machine Learning Example – Linear Regression](https://devarea.com/python-machine-learning-example-linear-regression/) [Linux – Handling Signals in a Multithreaded Application →](https://devarea.com/linux-handling-signals-in-a-multithreaded-application/) ## 3 thoughts on “Python Multitasking – MultiThreading and MultiProcessing” 1. ![](https://secure.gravatar.com/avatar/c941b0ff8558631eafac4df0d5f21514a3a74cdc4973a8646ad99f4808273a2f?s=128&d=mm&r=g) [anushri](https://www.besanttechnologies.com/training-courses/python-training-institute-in-bangalore) Multiple threading are useful create program small size its use full to workout. [March 1, 2018](https://devarea.com/python-multitasking-multithreading-multiprocessing/#comment-602) \- 2. ![](https://secure.gravatar.com/avatar/f3c8a19e7523417bf916d633e3f1abde7d123c580d0b9d238fbc4e903c474b22?s=128&d=mm&r=g) Sergio Many thanks, very useful post\! [May 28, 2019](https://devarea.com/python-multitasking-multithreading-multiprocessing/#comment-7016) \- 3. ![](https://secure.gravatar.com/avatar/fde9409b241151326a7ce105e5cb1bc44ad4f1a9a948512f912312ed7fe8f0f4?s=128&d=mm&r=g) [mama bear t shirt](http://ow.ly/pqtI50wN35r) Hi friends, its fantastic post on the topic of teachingand fully defined, keep it up all the time. [November 30, 2019](https://devarea.com/python-multitasking-multithreading-multiprocessing/#comment-8856) \- Comments are closed. ### Subscribe to our Newsletter ### Categories - [Android](https://devarea.com/category/android/) - [Angular](https://devarea.com/category/angular/) - [Django](https://devarea.com/category/django/) - [General](https://devarea.com/category/general/) - [Linux](https://devarea.com/category/linux/) - [Machine Learning](https://devarea.com/category/machine-learning/) - [python](https://devarea.com/category/python/) ### Recent Posts - [Building a Web App with Angular, Django and Django REST](https://devarea.com/building-a-web-app-with-angular-django-and-django-rest/) - [Introduction To Network Filters – Linux](https://devarea.com/introduction-to-network-filters-linux/) - [Linear Regression With Numpy](https://devarea.com/linear-regression-with-numpy/) - [Python – Working With Virtual Environments](https://devarea.com/python-working-with-virtual-environments/) - [Python – Text Processing Introduction](https://devarea.com/python-text-processing-introduction/) ### Tags [Android](https://devarea.com/tag/android/) [Android Internals](https://devarea.com/tag/android-internals/) [angular](https://devarea.com/tag/angular/) [AOSP](https://devarea.com/tag/aosp/) [Assembly](https://devarea.com/tag/assembly/) [C](https://devarea.com/tag/c/) [Debug](https://devarea.com/tag/debug/) [django](https://devarea.com/tag/django/) [git](https://devarea.com/tag/git/) [Java](https://devarea.com/tag/java/) [Kernel](https://devarea.com/tag/kernel/) [Linux](https://devarea.com/tag/linux/) [Machine Learning](https://devarea.com/tag/machine-learning/) [Networking](https://devarea.com/tag/networking/) [numpy](https://devarea.com/tag/numpy/) [pandas](https://devarea.com/tag/pandas/) [python](https://devarea.com/tag/python/) [rest](https://devarea.com/tag/rest/) [scipy](https://devarea.com/tag/scipy/) [security](https://devarea.com/tag/security/) ### Subscribe to our Newsletter ### Archives - [April 2019](https://devarea.com/2019/04/) - [March 2019](https://devarea.com/2019/03/) - [May 2018](https://devarea.com/2018/05/) - [March 2018](https://devarea.com/2018/03/) - [February 2018](https://devarea.com/2018/02/) - [January 2018](https://devarea.com/2018/01/) - [December 2017](https://devarea.com/2017/12/) - [November 2017](https://devarea.com/2017/11/) - [October 2017](https://devarea.com/2017/10/) - [September 2017](https://devarea.com/2017/09/) - [August 2017](https://devarea.com/2017/08/) - [May 2017](https://devarea.com/2017/05/) - [March 2017](https://devarea.com/2017/03/) ### Categories - [Android](https://devarea.com/category/android/) - [Angular](https://devarea.com/category/angular/) - [Django](https://devarea.com/category/django/) - [General](https://devarea.com/category/general/) - [Linux](https://devarea.com/category/linux/) - [Machine Learning](https://devarea.com/category/machine-learning/) - [python](https://devarea.com/category/python/) ### Recent Posts - [Building a Web App with Angular, Django and Django REST](https://devarea.com/building-a-web-app-with-angular-django-and-django-rest/) - [Introduction To Network Filters – Linux](https://devarea.com/introduction-to-network-filters-linux/) - [Linear Regression With Numpy](https://devarea.com/linear-regression-with-numpy/) - [Python – Working With Virtual Environments](https://devarea.com/python-working-with-virtual-environments/) - [Python – Text Processing Introduction](https://devarea.com/python-text-processing-introduction/) ### Tags [Android](https://devarea.com/tag/android/) [Android Internals](https://devarea.com/tag/android-internals/) [angular](https://devarea.com/tag/angular/) [AOSP](https://devarea.com/tag/aosp/) [Assembly](https://devarea.com/tag/assembly/) [C](https://devarea.com/tag/c/) [Debug](https://devarea.com/tag/debug/) [django](https://devarea.com/tag/django/) [git](https://devarea.com/tag/git/) [Java](https://devarea.com/tag/java/) [Kernel](https://devarea.com/tag/kernel/) [Linux](https://devarea.com/tag/linux/) [Machine Learning](https://devarea.com/tag/machine-learning/) [Networking](https://devarea.com/tag/networking/) [numpy](https://devarea.com/tag/numpy/) [pandas](https://devarea.com/tag/pandas/) [python](https://devarea.com/tag/python/) [rest](https://devarea.com/tag/rest/) [scipy](https://devarea.com/tag/scipy/) [security](https://devarea.com/tag/security/) ### Subscribe to our mailing list ### Follow Me ** [ShopIsle](http://themeisle.com/themes/shop-isle/) powered by [WordPress](http://wordpress.org/)
Readable Markdown	Modern operating systems allow a user to run several applications simultaneously, so that some tasks can be run in the background while the user continues with other work in the foreground. The user can also run multiple copies of the same program at the same time. To create a task we can use process or thread. Process has its private resources including memory mapping, files and other os objects. Multiple threads can run on the same process and share all its resources but if one thread fail it will kill all other threads in its process. Python has many packages to handle multi tasking, in this post i will cover some ### os.fork Unix/Linux/OS X specific (i.e. all but windows). The function creates a child process that start running after the fork return. We call fork once but it returns twice on the parent and on the child. Consider the following code: ``` import os x=os.fork() if x: print('hello parent') else: print('hello child') ``` If you run it , you will see both prints: ``` hello parent hello child ``` On fork call, the OS creates a child process and fork returns twice – on the parent the return value is the child process id and on the child fork returns 0. The point behind this weird behaviour is the Join-Fork parallel pattern – you run a code and want to use another processor for a complex task – fork a child, divide the work and join again: ``` import os ret=0 # we need to do something complex so lets create a child x=os.fork() if x: print('do half work by parent') ret = os.wait() else: print('do half work by child') os._exit(10) print("only parent here res=" + str(os.WEXITSTATUS(ret[1]))) ``` Before the fork call and after the if statement only one process exists , The child does its task and finish returning a value to the parent, If the parent finish before the child it will wait for him It is important to call wait otherwise the child remains zombie ### os.system Runs a command in the shell (bash, cmd, etc) sometimes you only need to run another program or command for example you want to do something that easily done with a command like format ``` import os status = os.system('script2.py') print ("exit status",status) ``` ### os.popen Run a process and read its output as a file for example: ``` import os for line in os.popen('ps aux').readlines(): print(line) ``` ### The subprocess package An attempt to replace os specific function with platform independent. You can run a process and open a pipe in any os for example: ``` import subprocess subprocess.run('script2.py') pr = subprocess.Popen('ls') pr.wait(); ``` to communicate with the output use: ``` import subprocess import sys proc = subprocess.Popen([sys.executable, 'myapp.py', 'data'], stdout = subprocess.PIPE, stderr = subprocess.PIPE) (output, error) = proc.communicate() if error != None: print("error:", error.decode()) print("output:", output.decode()) ``` ### The multiprocessing package Another useful package for process creation and other methods. The multiprocessing module is suitable for sharing data or tasks between processor cores. It does not use threading, but processes instead. Processes are inherently more “expensive” that threads, so they are not worth using for trivial data sets or tasks ``` import multiprocessing def fn(a,b): print ("vals=",a,b) proc1 = multiprocessing.Process(target=fn, args=(10,20)) proc2 = multiprocessing.Process(target=fn, args=(10,20)) proc1.start() proc2.start() proc1.join() proc2.join() ``` ### Working with Threads Threads run on the same process address space – it is easy to share data between them but if one thread fails all other threads in the same process killed. To create a thread we use the threading package. We write a class derived from Thread and declare the function run. To create a thread we need to create an object from the class and call start method: ``` import threading import time class MyThread (threading.Thread): def run (self): for i in range(5): print ("In thread", self.name) time.sleep(2) th1 = MyThread() th2 = MyThread() th1.setName("t1") th2.setName("t2") th1.start() th2.start() print ("From main") th1.join() th2.join() ``` output: ``` In thread t1 In thread t2 From main In thread t1 In thread t2 In thread t1 In thread t2 .... ``` You can also define the thread function as global (without class): ``` from threading import Thread import time def myfunc(args): print "From thread", args time.sleep(5) tid1 = Thread(target=myfunc, args='T1') tid2 = Thread(target=myfunc, args='T2') tid1.start() tid2.start() print "From main" tid1.join() tid2.join() ``` ### Synchronization Objects Accessing resources needs synchronization. We can find the following objects in threading package: - Condition variables – similar to linux posix condition variables - Events – similar to windows events - Lock – similar to mutex - Semaphore #### Lock Example: ``` import threading import time lc = threading.Lock() def myfunc1(args): global lc lc.acquire() print("From thread", args) lc.release() def myfunc2(*args): global lc lc.acquire() print("From thread", args) lc.release() tid1 = threading.Thread(target=myfunc1, args='T1') tid2 = threading.Thread(target=myfunc2, args='T2') tid1.start() tid2.start() print("From main") tid1.join() tid2.join() ``` ### Inter Process Communication (IPC) Threads share the same address space so its easy to send data from one thread to another but processes live in a different address spaces and thats why we need an IPC object. You can use pipe, fifo, message queue and more #### Message Queue Example: In the following example we create 2 processes and use a queue for communication. One process send message to the queue and the other receive: ``` import multiprocessing import time def fn1(q): while True: x=q.get() print ("val=",x) def fn2(q): while True: time.sleep(2) q.put("hello") queue = multiprocessing.Queue() proc1 = multiprocessing.Process(target=fn1, args=(queue,)) proc2 = multiprocessing.Process(target=fn2, args=(queue,)) proc1.start() proc2.start() ``` You can find more methods for creating threads and processes, using synchronization objects and IPC
Shard	73 (laksa)
Root Hash	2261861757304640073
Unparsed URL	com,devarea!/python-multitasking-multithreading-multiprocessing/ s443