🕷️ Crawler Inspector

URL Lookup

Direct Parameter Lookup

Raw Queries and Responses

1. Shard Calculation

Query:

Response:

Calculated Shard: 160 (from laksa033)

2. Crawled Status Check

Query:

curl -X POST \
  'http://laksa160.int.ahrefs:8124/' \
  -H 'Content-Type: text/plain' \
  -H 'X-ClickHouse-Database: crawler3' \
  -H 'Authorization: Basic YXBpOg==' \
  -d 'SELECT getAhrefsURLFromUnparsed(src_unparsed) AS found_url, ifNull(toUnixTimestamp(download_stamp), 0) AS crawl_time, ifNull(toUnixTimestamp(props_url_first_seen), 0) AS first_indexed_time, download_http_code AS http_code, src_unparsed AS src_unparsed, src_root_hash AS src_root_hash, history_drop_reason AS history_drop_reason, meta_title AS meta_title, meta_descriptions AS meta_descriptions, attrs_boilerpipe_text AS attrs_boilerpipe_text, attrs_markdown AS attrs_markdown, attrs_readable_markdown AS attrs_readable_markdown, meta_canonical AS meta_canonical FROM crawler3.page_info_local FINAL PREWHERE (src_root_hash, src_unparsed) IN ((getAhrefsRootHashFromUnparsed(getAhrefsUnparsedNoserviceFromURL(\'https://www.gaohongnan.com/software_engineering/python/gil.html\')), getAhrefsUnparsedNoserviceFromURL(\'https://www.gaohongnan.com/software_engineering/python/gil.html\'))) FORMAT JSONEachRow'

Response:

{"found_url":"https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html","crawl_time":1771966477,"first_indexed_time":1730279528,"http_code":200,"src_unparsed":"com,gaohongnan!www,\/software_engineering\/python\/gil.html s443","src_root_hash":"5340751680682619160","history_drop_reason":null,"meta_title":"Global Interpreter Lock (GIL) — Omniverse","meta_descriptions":[],"attrs_boilerpipe_text":"Reference Counting in Python In Python,  reference counting  is a memory management technique used to keep\ntrack of how many references (or “pointers”) exist to an object in memory. When\nan object’s reference count drops to zero, Python automatically frees the memory\nallocated to that object. import sys a = [] b = a a_ref_count = sys . getrefcount ( a ) print ( a_ref_count ) # Output: 3 a = []  creates an empty list  []  and assigns it to variable  a . b = a  makes  b  reference the same list as  a . sys.getrefcount(a)  returns the number of references to the list  a  points\nto. The count is  3  because: a  references the list. b  references the list. The  sys.getrefcount(a)  function temporarily creates another reference\nwhen it’s called. As a beginner python programmer, we thank our lucky stars that Python manages\nmemory for us. But, as a good student, we ask, why is managing the reference\ncount important? First,  memory management  helps Python automatically manage memory by\nkeeping track of objects and freeing memory when objects are no longer needed.\nFor example, if we delete the reference  b , the reference count of  a  drops to\n2. del b a_ref_count = sys . getrefcount ( a ) print ( a_ref_count ) # Output: 2 Second,  avoiding memory leaks  ensures that memory is reused efficiently and\nprevents memory from being wasted on unused objects. Threads and Race Conditions A  race condition  occurs when two or more threads access shared data\nsimultaneously, and the final outcome depends on the sequence of execution. For example, if two threads try to increment a counter from 5 to 6: Thread A reads counter (value = 5) Thread B reads counter (value = 5) Thread A adds 1 (5 + 1 = 6) Thread B adds 1 (5 + 1 = 6) Thread A writes 6 Thread B writes 6 Even though two increment operations occurred, the counter only increased by 1\ninstead of 2. This happens because both threads read the original value before\neither could update it. The final result depends on which thread writes last,\nand the program “races” to an incorrect outcome. Show me the code to illustrate the race condition! \"\"\"With reference to effective python book chapter 54. Ref: https:\/\/github.com\/bslatkin\/effectivepython\/blob\/master\/example_code\/item_54.py \"\"\" import logging import threading from threading import Barrier from typing import List logging . basicConfig ( level = logging . DEBUG , format = \" %(asctime)s  -  %(levelname)s  -  %(message)s \" ) NUM_THREADS = 5 BARRIER = Barrier ( NUM_THREADS ) class Counter : def __init__ ( self ) -> None : self . count = 0 def increment ( self , offset : int ) -> None : self . count += offset def worker ( thread_index : int , total_iterations : int , counter : Counter ) -> None : \"\"\"The barrier is used to synchronize the threads so that they all start counting     at the same time. This makes it easier to get a race condition since we wait for     the other threads to start else in the loop we always have an order that the     first thread likely starts first and then the second and so on.     \"\"\" BARRIER . wait () logging . debug ( \"Thread  %s , starting\" , thread_index ) for _ in range ( total_iterations ): counter . increment ( 1 ) def thread_unsafe ( total_iterations : int ) -> None : counter = Counter () threads : List [ threading . Thread ] = [] for index in range ( NUM_THREADS ): thread = threading . Thread ( target = worker , args = ( index , total_iterations , counter )) threads . append ( thread ) for thread in threads : thread . start () for thread in threads : thread . join () expected = total_iterations * NUM_THREADS found = counter . count logging . info ( \"Counter should be  %s , got  %s \" , expected , found ) if __name__ == \"__main__\" : total_iterations = 10 ** 6 thread_unsafe ( total_iterations ) 2024-10-29 10:22:29,521 - DEBUG - Thread 4, starting\n2024-10-29 10:22:29,522 - DEBUG - Thread 0, starting\n2024-10-29 10:22:29,522 - DEBUG - Thread 3, starting\n2024-10-29 10:22:29,522 - DEBUG - Thread 2, starting\n2024-10-29 10:22:29,523 - DEBUG - Thread 1, starting\n2024-10-29 10:22:29,951 - INFO - Counter should be 5000000, got 4564365\n Protecting Reference Counts with Locks So we circle back a bit to the reference count thingy earlier. Since we saw how\nrace conditions can happen, the same thing can happen to the reference count of\nan object - which means that the  reference count  of an object can be\nmodified by multiple threads. If two threads try to increment or decrement the\nreference count simultaneously without protection, it can lead to\ninconsistencies, such as the case where reference counts might reach zero\nincorrectly, freeing memory while it’s still in use, causing crashes or bugs. How do we prevent this? We first understand the rough idea of a  lock . A  lock  is a synchronization mechanism used to control access to shared\nresources. When a thread acquires a lock, other threads must wait until the lock\nis released before accessing the resource. In the first example without a lock, both threads might read the same\n counter  value before incrementing it, causing some increments to be lost. In the second example, the lock ensures that only one thread can modify the\ncounter at a time. The  with lock:  statement automatically acquires the lock before entering\nthe block and releases it when exiting. This guarantees that the final counter value will always be correct. \"\"\"With reference to effective python book chapter 54. Ref: https:\/\/github.com\/bslatkin\/effectivepython\/blob\/master\/example_code\/item_54.py \"\"\" import logging import threading from threading import Barrier from typing import List logging . basicConfig ( level = logging . DEBUG , format = \" %(asctime)s  -  %(levelname)s  -  %(message)s \" ) NUM_THREADS = 5 BARRIER = Barrier ( NUM_THREADS ) class CounterLock : def __init__ ( self ) -> None : self . count = 0 self . lock = threading . Lock () def increment ( self , offset : int ) -> None : with self . lock : self . count += offset def worker ( thread_index : int , total_iterations : int , counter : Counter ) -> None : \"\"\"The barrier is used to synchronize the threads so that they all start counting     at the same time. This makes it easier to get a race condition since we wait for     the other threads to start else in the loop we always have an order that the     first thread likely starts first and then the second and so on.     \"\"\" BARRIER . wait () logging . debug ( \"Thread  %s , starting\" , thread_index ) for _ in range ( total_iterations ): counter . increment ( 1 ) def thread_safe ( total_iterations : int ) -> None : counter = CounterLock () threads : List [ threading . Thread ] = [] for index in range ( NUM_THREADS ): thread = threading . Thread ( target = worker , args = ( index , total_iterations , counter )) threads . append ( thread ) for thread in threads : thread . start () for thread in threads : thread . join () expected = total_iterations * NUM_THREADS found = counter . count logging . info ( \"Counter should be  %s , got  %s \" , expected , found ) if __name__ == \"__main__\" : total_iterations = 10 ** 6 thread_safe ( total_iterations ) 2024-10-29 10:22:29,959 - DEBUG - Thread 4, starting\n2024-10-29 10:22:29,959 - DEBUG - Thread 0, starting\n2024-10-29 10:22:29,959 - DEBUG - Thread 3, starting\n2024-10-29 10:22:29,960 - DEBUG - Thread 2, starting\n2024-10-29 10:22:29,960 - DEBUG - Thread 1, starting\n2024-10-29 10:22:30,821 - INFO - Counter should be 5000000, got 5000000\n Deadlocks A  deadlock  occurs when two or more threads are waiting indefinitely for\nlocks held by each other, preventing them from making progress. import threading import time class BankAccount : def __init__ ( self , id : int , balance : float ) -> None : self . id : int = id self . balance : float = balance self . lock : threading . Lock = threading . Lock () def __str__ ( self ) -> str : return f \"Account  { self . id } : $ { self . balance } \" def unsafe_transfer ( from_account : BankAccount , to_account : BankAccount , amount : float ) -> None : with from_account . lock : print ( f \"Locked account  { from_account . id } \" ) time . sleep ( 0.5 ) # Simulate some work and make deadlock more likely with to_account . lock : print ( f \"Locked account  { to_account . id } \" ) if from_account . balance >= amount : from_account . balance -= amount to_account . balance += amount print ( f \"Transferred $ { amount }  from Account  { from_account . id }  to Account  { to_account . id } \" ) def safe_transfer ( from_account : BankAccount , to_account : BankAccount , amount : float ) -> None : first : BankAccount = from_account if from_account . id < to_account . id else to_account second : BankAccount = to_account if from_account . id < to_account . id else from_account with first . lock : print ( f \"Locked account  { first . id } \" ) time . sleep ( 0.5 ) # Simulate some work with second . lock : print ( f \"Locked account  { second . id } \" ) if from_account . balance >= amount : from_account . balance -= amount to_account . balance += amount print ( f \"Transferred $ { amount }  from Account  { from_account . id }  to Account  { to_account . id } \" ) account1 : BankAccount = BankAccount ( 1 , 1000 ) account2 : BankAccount = BankAccount ( 2 , 1000 ) print ( \"Initial balances:\" ) print ( account1 ) print ( account2 ) print ( \" \\n Trying unsafe transfers (will likely deadlock):\" ) # Create threads with unsafe transfers (will deadlock) thread1 : threading . Thread = threading . Thread ( target = unsafe_transfer , args = ( account1 , account2 , 500 )) thread2 : threading . Thread = threading . Thread ( target = unsafe_transfer , args = ( account2 , account1 , 300 )) thread1 . start () thread2 . start () thread1 . join () thread2 . join () thread1  acquires the lock on  account1  and waits to acquire the lock on\n account2 . thread2  acquires the lock on  account2  and waits to acquire the lock on\n account1 . Both threads are waiting for each other to release the locks, causing a\ndeadlock. Why do we need to know this? Because as we saw earlier, locks are useful for\npreventing race conditions. But using locks not so carefully can lead to\ndeadlocks. GIL will aim to solve this as well. The Global Interpreter Lock (GIL) in Python The  Global Interpreter Lock (GIL)  is a  mutex  (lock) that protects access\nto Python objects, preventing multiple native threads from executing Python\nbytecodes simultaneously in the same process. To solve deadlocks, gil only has  one lock  for the entire process - thus\nthe scenario where two threads are waiting for each other to release a lock\nis avoided. In more intuition, the gil allows only  one thread  to execute\nbytecode at a time. With locks in place, race conditions are mostly resolved. With gil, it has its own trade-offs: Limits Multi-threaded Performance:  Because the GIL allows only one\nthread to execute Python code at a time, CPU-bound multi-threaded programs\ndon’t benefit from multiple cores. They run almost as if they were\nsingle-threaded. Inefficiency in Multi-core Systems:  In CPU-bound tasks, the GIL can\nbecome a bottleneck, preventing Python programs from fully utilizing\nmulti-core processors. Let’s see an example of how gil can limit multi-threaded performance. CPU-bound tasks We first compare the performance of single-threaded vs multi-threaded and note\nfor cpu-bound tasks, the time taken is almost the same. import threading import time import requests def cpu_bound_task (): \"\"\"CPU intensive task - calculating sum of numbers\"\"\" count = 0 for _ in range ( 20_000_000 ): count += 1 return count def run_tasks_single_thread ( task , num_iterations ): start_time = time . time () for _ in range ( num_iterations ): task () end_time = time . time () return end_time - start_time def run_tasks_multi_thread ( task , num_threads ): start_time = time . time () threads : List [ threading . Thread ] = [] for _ in range ( num_threads ): t = threading . Thread ( target = task ) threads . append ( t ) t . start () for t in threads : t . join () end_time = time . time () return end_time - start_time # NOTE: CPU-bound tasks num_tasks = 4 print ( \"CPU-bound task comparison:\" ) single_thread_cpu = run_tasks_single_thread ( cpu_bound_task , num_tasks ) print ( f \"Single-threaded time:  { single_thread_cpu : .2f }  seconds\" ) multi_thread_cpu = run_tasks_multi_thread ( cpu_bound_task , num_tasks ) print ( f \"Multi-threaded time:  { multi_thread_cpu : .2f }  seconds\" ) print ( f \"Speed difference:  { single_thread_cpu \/ multi_thread_cpu : .2f } x \\n \" ) CPU-bound task comparison:\nSingle-threaded time: 1.71 seconds\nMulti-threaded time: 1.62 seconds\nSpeed difference: 1.06x\n Next, we compare the performance of single-threaded vs multi-threaded vs\nmulti-process. We note that multi-process is the fastest and this is expected\nbecause there is no GIL in multi-process. import multiprocessing def run_tasks_multi_process ( task , num_processes ): \"\"\"Run tasks using multiple processes\"\"\" start_time = time . time () processes : List [ multiprocessing . Process ] = [] for _ in range ( num_processes ): p = multiprocessing . Process ( target = task ) processes . append ( p ) p . start () for p in processes : p . join () end_time = time . time () return end_time - start_time # Comparison of all three approaches num_tasks = 4 print ( \"CPU-bound task comparison:\" ) single_thread_cpu = run_tasks_single_thread ( cpu_bound_task , num_tasks ) print ( f \"Single-threaded time:  { single_thread_cpu : .2f }  seconds\" ) multi_thread_cpu = run_tasks_multi_thread ( cpu_bound_task , num_tasks ) print ( f \"Multi-threaded time:  { multi_thread_cpu : .2f }  seconds\" ) multi_process_cpu = run_tasks_multi_process ( cpu_bound_task , num_tasks ) print ( f \"Multi-process time:  { multi_process_cpu : .2f }  seconds\" ) print ( f \" \\n Speed comparison (relative to single-thread):\" ) print ( f \"Threading speedup:  { single_thread_cpu \/ multi_thread_cpu : .2f } x\" ) print ( f \"Multiprocessing speedup:  { single_thread_cpu \/ multi_process_cpu : .2f } x \\n \" ) CPU-bound task comparison:\nSingle-threaded time: 1.73 seconds\nMulti-threaded time: 1.58 seconds\nMulti-process time: 0.09 seconds\n\nSpeed comparison (relative to single-thread):\nThreading speedup: 1.10x\nMultiprocessing speedup: 18.59x\n Traceback (most recent call last):\n  File \"<string>\", line 1, in <module>\n  File \"\/opt\/homebrew\/Caskroom\/miniconda\/base\/envs\/omniverse\/lib\/python3.9\/multiprocessing\/spawn.py\", line 116, in spawn_main\nTraceback (most recent call last):\n  File \"<string>\", line 1, in <module>\n  File \"\/opt\/homebrew\/Caskroom\/miniconda\/base\/envs\/omniverse\/lib\/python3.9\/multiprocessing\/spawn.py\", line 116, in spawn_main\nTraceback (most recent call last):\n  File \"<string>\", line 1, in <module>\n  File \"\/opt\/homebrew\/Caskroom\/miniconda\/base\/envs\/omniverse\/lib\/python3.9\/multiprocessing\/spawn.py\", line 116, in spawn_main\n    exitcode = _main(fd, parent_sentinel)\n  File \"\/opt\/homebrew\/Caskroom\/miniconda\/base\/envs\/omniverse\/lib\/python3.9\/multiprocessing\/spawn.py\", line 126, in _main\n        exitcode = _main(fd, parent_sentinel)exitcode = _main(fd, parent_sentinel)\n\n  File \"\/opt\/homebrew\/Caskroom\/miniconda\/base\/envs\/omniverse\/lib\/python3.9\/multiprocessing\/spawn.py\", line 126, in _main\n  File \"\/opt\/homebrew\/Caskroom\/miniconda\/base\/envs\/omniverse\/lib\/python3.9\/multiprocessing\/spawn.py\", line 126, in _main\n    self = reduction.pickle.load(from_parent)\nAttributeError: Can't get attribute 'cpu_bound_task' on <module '__main__' (built-in)>\n    self = reduction.pickle.load(from_parent)\nAttributeError: Can't get attribute 'cpu_bound_task' on <module '__main__' (built-in)>\n    self = reduction.pickle.load(from_parent)\nAttributeError: Can't get attribute 'cpu_bound_task' on <module '__main__' (built-in)>\nTraceback (most recent call last):\n  File \"<string>\", line 1, in <module>\n  File \"\/opt\/homebrew\/Caskroom\/miniconda\/base\/envs\/omniverse\/lib\/python3.9\/multiprocessing\/spawn.py\", line 116, in spawn_main\n    exitcode = _main(fd, parent_sentinel)\n  File \"\/opt\/homebrew\/Caskroom\/miniconda\/base\/envs\/omniverse\/lib\/python3.9\/multiprocessing\/spawn.py\", line 126, in _main\n    self = reduction.pickle.load(from_parent)\nAttributeError: Can't get attribute 'cpu_bound_task' on <module '__main__' (built-in)>\n GIL Is Released During IO-bound Tasks As mentioned, the GIL is released during IO-bound tasks. As to why, please see\nthis  post . Let’s see an example of this. def io_bound_task (): \"\"\"IO intensive task - making HTTP requests\"\"\" url = \"https:\/\/api.github.com\" response = requests . get ( url ) return response . status_code # NOTE: IO-bound tasks print ( \"IO-bound task comparison:\" ) single_thread_io = run_tasks_single_thread ( io_bound_task , num_tasks ) print ( f \"Single-threaded time:  { single_thread_io : .2f }  seconds\" ) multi_thread_io = run_tasks_multi_thread ( io_bound_task , num_tasks ) print ( f \"Multi-threaded time:  { multi_thread_io : .2f }  seconds\" ) print ( f \"Speed difference:  { single_thread_io \/ multi_thread_io : .2f } x\" ) 2024-10-29 20:14:18,709 - DEBUG - Starting new HTTPS connection (1): api.github.com:443\n IO-bound task comparison:\n 2024-10-29 20:14:19,014 - DEBUG - https:\/\/api.github.com:443 \"GET \/ HTTP\/1.1\" 200 510\n2024-10-29 20:14:19,021 - DEBUG - Starting new HTTPS connection (1): api.github.com:443\n2024-10-29 20:14:19,072 - DEBUG - https:\/\/api.github.com:443 \"GET \/ HTTP\/1.1\" 200 510\n2024-10-29 20:14:19,075 - DEBUG - Starting new HTTPS connection (1): api.github.com:443\n2024-10-29 20:14:19,122 - DEBUG - https:\/\/api.github.com:443 \"GET \/ HTTP\/1.1\" 200 510\n2024-10-29 20:14:19,126 - DEBUG - Starting new HTTPS connection (1): api.github.com:443\n2024-10-29 20:14:19,167 - DEBUG - https:\/\/api.github.com:443 \"GET \/ HTTP\/1.1\" 200 510\n2024-10-29 20:14:19,174 - DEBUG - Starting new HTTPS connection (1): api.github.com:443\n2024-10-29 20:14:19,174 - DEBUG - Starting new HTTPS connection (1): api.github.com:443\n2024-10-29 20:14:19,174 - DEBUG - Starting new HTTPS connection (1): api.github.com:443\n2024-10-29 20:14:19,174 - DEBUG - Starting new HTTPS connection (1): api.github.com:443\n2024-10-29 20:14:19,256 - DEBUG - https:\/\/api.github.com:443 \"GET \/ HTTP\/1.1\" 200 510\n2024-10-29 20:14:19,259 - DEBUG - https:\/\/api.github.com:443 \"GET \/ HTTP\/1.1\" 200 510\n2024-10-29 20:14:19,261 - DEBUG - https:\/\/api.github.com:443 \"GET \/ HTTP\/1.1\" 200 510\n2024-10-29 20:14:19,261 - DEBUG - https:\/\/api.github.com:443 \"GET \/ HTTP\/1.1\" 200 510\n Single-threaded time: 0.50 seconds\nMulti-threaded time: 0.09 seconds\nSpeed difference: 5.24x\n Indeed, the multi-threaded version is faster than the single-threaded version. References and Further Readings","attrs_markdown":"[Skip to main content](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#main-content)\n\nBack to top\n\n[![Omniverse - Home](https:\/\/www.gaohongnan.com\/_static\/logo.png)](https:\/\/www.gaohongnan.com\/intro.html)\n\n- [Omniverse](https:\/\/www.gaohongnan.com\/intro.html)\n\nNotations\n\n- [Machine Learning Notations](https:\/\/www.gaohongnan.com\/notations\/machine_learning.html)\n\nInfluential Ideas and Papers\n\n- [Generative Pre-trained Transformers](https:\/\/www.gaohongnan.com\/influential\/generative_pretrained_transformer\/01_intro.html)\n  - [Notations](https:\/\/www.gaohongnan.com\/influential\/generative_pretrained_transformer\/02_notations.html)\n  - [The Concept of Generative Pre-trained Transformers (GPT)](https:\/\/www.gaohongnan.com\/influential\/generative_pretrained_transformer\/03_concept.html)\n  - [The Implementation of Generative Pre-trained Transformers (GPT)](https:\/\/www.gaohongnan.com\/influential\/generative_pretrained_transformer\/04_implementation.html)\n  - [Training a Mini-GPT to Learn Two-Digit Addition](https:\/\/www.gaohongnan.com\/influential\/generative_pretrained_transformer\/05_adder.html)\n- [Low-Rank Adaptation Of Large Language Models](https:\/\/www.gaohongnan.com\/influential\/low_rank_adaptation\/01_intro.html)\n  - [Concept](https:\/\/www.gaohongnan.com\/influential\/low_rank_adaptation\/02_concept.html)\n  - [Implementation](https:\/\/www.gaohongnan.com\/influential\/low_rank_adaptation\/03_implementation.html)\n- [Chain-of-Verification Reduces Hallucination in Large Language Models](https:\/\/www.gaohongnan.com\/influential\/cove\/cove.html)\n- [Empirical Risk Minimization](https:\/\/www.gaohongnan.com\/influential\/empirical_risk_minimization\/01_intro.html)\n  - [Concept: Empirical Risk Minimization](https:\/\/www.gaohongnan.com\/influential\/empirical_risk_minimization\/02_concept.html)\n  - [Bayes Optimal Classifier](https:\/\/www.gaohongnan.com\/influential\/empirical_risk_minimization\/03_bayes_optimal_classifier.html)\n- [Is The Learning Problem Solvable?](https:\/\/www.gaohongnan.com\/influential\/learning_theory\/01_intro.html)\n  - [Concept: Learning Theory](https:\/\/www.gaohongnan.com\/influential\/learning_theory\/02_concept.html)\n- [Lloyd’s K-Means Clustering Algorithm](https:\/\/www.gaohongnan.com\/influential\/kmeans_clustering\/01_intro.html)\n  - [Concept: K-Means Clustering](https:\/\/www.gaohongnan.com\/influential\/kmeans_clustering\/02_concept.html)\n  - [Implementation: K-Means (Lloyd)](https:\/\/www.gaohongnan.com\/influential\/kmeans_clustering\/03_implementation.html)\n  - [Application: Image Compression and Segmentation](https:\/\/www.gaohongnan.com\/influential\/kmeans_clustering\/04_image_segmentation.html)\n  - [Conceptual Questions](https:\/\/www.gaohongnan.com\/influential\/kmeans_clustering\/05_conceptual_questions.html)\n- [Naive Bayes](https:\/\/www.gaohongnan.com\/influential\/naive_bayes\/01_intro.html)\n  - [Concept](https:\/\/www.gaohongnan.com\/influential\/naive_bayes\/02_concept.html)\n  - [Naives Bayes Implementation](https:\/\/www.gaohongnan.com\/influential\/naive_bayes\/03_implementation.html)\n  - [Naive Bayes Application: Penguins](https:\/\/www.gaohongnan.com\/influential\/naive_bayes\/04_example_penguins.html)\n  - [Naive Bayes Application (MNIST)](https:\/\/www.gaohongnan.com\/influential\/naive_bayes\/05_application_mnist.html)\n- [Mixture Models](https:\/\/www.gaohongnan.com\/influential\/gaussian_mixture_models\/01_intro.html)\n  - [Concept](https:\/\/www.gaohongnan.com\/influential\/gaussian_mixture_models\/02_concept.html)\n  - [Gaussian Mixture Models Implementation](https:\/\/www.gaohongnan.com\/influential\/gaussian_mixture_models\/03_implementation.html)\n- [Linear Regression](https:\/\/www.gaohongnan.com\/influential\/linear_regression\/01_intro.html)\n  - [Concept](https:\/\/www.gaohongnan.com\/influential\/linear_regression\/02_concept.html)\n  - [Implementation](https:\/\/www.gaohongnan.com\/influential\/linear_regression\/03_implementation.html)\n\nPlaybook\n\n- [Training Dynamics And Tricks](https:\/\/www.gaohongnan.com\/playbook\/training\/intro.html)\n  - [How to Calculate the Number of FLOPs in Transformer Based Models?](https:\/\/www.gaohongnan.com\/playbook\/training\/how_to_calculate_flops_in_transformer_based_models.html)\n  - [Why Does Cosine Annealing With Warmup Stabilize Training?](https:\/\/www.gaohongnan.com\/playbook\/training\/why_cosine_annealing_warmup_stabilize_training.html)\n  - [How To Fine-Tune Decoder-Only Models For Sequence Classification Using Last Token Pooling?](https:\/\/www.gaohongnan.com\/playbook\/training\/how_to_finetune_decoder_with_last_token_pooling.html)\n  - [How To Fine-Tune Decoder-Only Models For Sequence Classification With Cross-Attention?](https:\/\/www.gaohongnan.com\/playbook\/training\/how_to_finetune_decoder_with_cross_attention.html)\n  - [How To Do Teacher-Student Knowledge Distillation?](https:\/\/www.gaohongnan.com\/playbook\/training\/how_to_teacher_student_knowledge_distillation.html)\n- [Softmax Preserves Order, Is Translation Invariant But Not Invariant Under Scaling.](https:\/\/www.gaohongnan.com\/playbook\/why_softmax_preserves_order_translation_invariant_not_invariant_scaling.html)\n- [How to Inspect Function and Class Signatures in Python?](https:\/\/www.gaohongnan.com\/playbook\/how_to_inspect_function_and_class_signatures.html)\n\nProbability Theory\n\n- [Chapter 1. Mathematical Preliminaries](https:\/\/www.gaohongnan.com\/probability_theory\/01_mathematical_preliminaries\/intro.html)\n  - [Permutations and Combinations](https:\/\/www.gaohongnan.com\/probability_theory\/01_mathematical_preliminaries\/01_combinatorics.html)\n  - [Calculus](https:\/\/www.gaohongnan.com\/probability_theory\/01_mathematical_preliminaries\/02_calculus.html)\n  - [Contour Maps](https:\/\/www.gaohongnan.com\/probability_theory\/01_mathematical_preliminaries\/03_contours.html)\n  - [Exercises](https:\/\/www.gaohongnan.com\/probability_theory\/01_mathematical_preliminaries\/exercises.html)\n- [Chapter 2. Probability](https:\/\/www.gaohongnan.com\/probability_theory\/02_probability\/intro.html)\n  - [Probability Space](https:\/\/www.gaohongnan.com\/probability_theory\/02_probability\/0202_probability_space.html)\n  - [Probability Axioms](https:\/\/www.gaohongnan.com\/probability_theory\/02_probability\/0203_probability_axioms.html)\n  - [Conditional Probability](https:\/\/www.gaohongnan.com\/probability_theory\/02_probability\/0204_conditional_probability.html)\n  - [Independence](https:\/\/www.gaohongnan.com\/probability_theory\/02_probability\/0205_independence.html)\n  - [Baye’s Theorem and the Law of Total Probability](https:\/\/www.gaohongnan.com\/probability_theory\/02_probability\/0206_bayes_theorem.html)\n  - [Summary](https:\/\/www.gaohongnan.com\/probability_theory\/02_probability\/summary.html)\n- [Chapter 3. Discrete Random Variables](https:\/\/www.gaohongnan.com\/probability_theory\/03_discrete_random_variables\/intro.html)\n  - [Random Variables](https:\/\/www.gaohongnan.com\/probability_theory\/03_discrete_random_variables\/0301_random_variables.html)\n  - [Discrete Random Variables](https:\/\/www.gaohongnan.com\/probability_theory\/03_discrete_random_variables\/0302_discrete_random_variables.html)\n  - [Probability Mass Function](https:\/\/www.gaohongnan.com\/probability_theory\/03_discrete_random_variables\/0303_probability_mass_function.html)\n  - [Cumulative Distribution Function](https:\/\/www.gaohongnan.com\/probability_theory\/03_discrete_random_variables\/0304_cumulative_distribution_function.html)\n  - [Expectation](https:\/\/www.gaohongnan.com\/probability_theory\/03_discrete_random_variables\/0305_expectation.html)\n  - [Moments and Variance](https:\/\/www.gaohongnan.com\/probability_theory\/03_discrete_random_variables\/0306_moments_and_variance.html)\n  - [Discrete Uniform Distribution](https:\/\/www.gaohongnan.com\/probability_theory\/03_discrete_random_variables\/uniform\/intro.html)\n    - [Concept](https:\/\/www.gaohongnan.com\/probability_theory\/03_discrete_random_variables\/uniform\/0307_discrete_uniform_distribution_concept.html)\n    - [Application](https:\/\/www.gaohongnan.com\/probability_theory\/03_discrete_random_variables\/uniform\/0307_discrete_uniform_distribution_application.html)\n  - [Bernoulli Distribution](https:\/\/www.gaohongnan.com\/probability_theory\/03_discrete_random_variables\/bernoulli\/intro.html)\n    - [Concept](https:\/\/www.gaohongnan.com\/probability_theory\/03_discrete_random_variables\/bernoulli\/0308_bernoulli_distribution_concept.html)\n    - [Application](https:\/\/www.gaohongnan.com\/probability_theory\/03_discrete_random_variables\/bernoulli\/0308_bernoulli_distribution_application.html)\n  - [Independent and Identically Distributed (IID)](https:\/\/www.gaohongnan.com\/probability_theory\/03_discrete_random_variables\/iid.html)\n  - [Binomial Distribution](https:\/\/www.gaohongnan.com\/probability_theory\/03_discrete_random_variables\/binomial\/intro.html)\n    - [Concept](https:\/\/www.gaohongnan.com\/probability_theory\/03_discrete_random_variables\/binomial\/0309_binomial_distribution_concept.html)\n    - [Implementation](https:\/\/www.gaohongnan.com\/probability_theory\/03_discrete_random_variables\/binomial\/0309_binomial_distribution_implementation.html)\n    - [Real World Examples](https:\/\/www.gaohongnan.com\/probability_theory\/03_discrete_random_variables\/binomial\/0309_binomial_distribution_application.html)\n  - [Geometric Distribution](https:\/\/www.gaohongnan.com\/probability_theory\/03_discrete_random_variables\/geometric\/intro.html)\n    - [Concept](https:\/\/www.gaohongnan.com\/probability_theory\/03_discrete_random_variables\/geometric\/0310_geometric_distribution_concept.html)\n  - [Poisson Distribution](https:\/\/www.gaohongnan.com\/probability_theory\/03_discrete_random_variables\/poisson\/intro.html)\n    - [Concept](https:\/\/www.gaohongnan.com\/probability_theory\/03_discrete_random_variables\/poisson\/0311_poisson_distribution_concept.html)\n    - [Implementation](https:\/\/www.gaohongnan.com\/probability_theory\/03_discrete_random_variables\/poisson\/0311_poisson_distribution_implementation.html)\n  - [Important](https:\/\/www.gaohongnan.com\/probability_theory\/03_discrete_random_variables\/summary.html)\n  - [Exercises](https:\/\/www.gaohongnan.com\/probability_theory\/03_discrete_random_variables\/exercises.html)\n- [Chapter 4. Continuous Random Variables](https:\/\/www.gaohongnan.com\/probability_theory\/04_continuous_random_variables\/intro.html)\n  - [From Discrete to Continuous](https:\/\/www.gaohongnan.com\/probability_theory\/04_continuous_random_variables\/from_discrete_to_continuous.html)\n  - [Continuous Random Variables](https:\/\/www.gaohongnan.com\/probability_theory\/04_continuous_random_variables\/0401_continuous_random_variables.html)\n  - [Probability Density Function](https:\/\/www.gaohongnan.com\/probability_theory\/04_continuous_random_variables\/0402_probability_density_function.html)\n  - [Expectation](https:\/\/www.gaohongnan.com\/probability_theory\/04_continuous_random_variables\/0403_expectation.html)\n  - [Moments and Variance](https:\/\/www.gaohongnan.com\/probability_theory\/04_continuous_random_variables\/0404_moments_and_variance.html)\n  - [Cumulative Distribution Function](https:\/\/www.gaohongnan.com\/probability_theory\/04_continuous_random_variables\/0405_cumulative_distribution_function.html)\n  - [Mean, Median and Mode](https:\/\/www.gaohongnan.com\/probability_theory\/04_continuous_random_variables\/0406_mean_median_mode.html)\n  - [Continuous Uniform Distribution](https:\/\/www.gaohongnan.com\/probability_theory\/04_continuous_random_variables\/0407_continuous_uniform_distribution.html)\n  - [Exponential Distribution](https:\/\/www.gaohongnan.com\/probability_theory\/04_continuous_random_variables\/0408_exponential_distribution.html)\n  - [Gaussian Distribution](https:\/\/www.gaohongnan.com\/probability_theory\/04_continuous_random_variables\/0409_gaussian_distribution.html)\n  - [Skewness and Kurtosis](https:\/\/www.gaohongnan.com\/probability_theory\/04_continuous_random_variables\/0410_skewness_and_kurtosis.html)\n  - [Convolution and Sum of Random Variables](https:\/\/www.gaohongnan.com\/probability_theory\/04_continuous_random_variables\/0411_convolve_and_sum_of_random_variables.html)\n  - [Functions of Random Variables](https:\/\/www.gaohongnan.com\/probability_theory\/04_continuous_random_variables\/0412_functions_of_random_variables.html)\n- [Chapter 5. Joint Distributions](https:\/\/www.gaohongnan.com\/probability_theory\/05_joint_distributions\/intro.html)\n  - [From Single Variable to Joint Distributions](https:\/\/www.gaohongnan.com\/probability_theory\/05_joint_distributions\/from_single_variable_to_joint_distributions.html)\n  - [Joint PMF and PDF](https:\/\/www.gaohongnan.com\/probability_theory\/05_joint_distributions\/0501_joint_pmf_pdf\/intro.html)\n    - [Concept](https:\/\/www.gaohongnan.com\/probability_theory\/05_joint_distributions\/0501_joint_pmf_pdf\/concept.html)\n  - [Joint Expectation and Correlation](https:\/\/www.gaohongnan.com\/probability_theory\/05_joint_distributions\/0502_joint_expectation_and_correlation\/intro.html)\n    - [Concept](https:\/\/www.gaohongnan.com\/probability_theory\/05_joint_distributions\/0502_joint_expectation_and_correlation\/concept.html)\n  - [Conditional PMF and PDF](https:\/\/www.gaohongnan.com\/probability_theory\/05_joint_distributions\/0503_conditional_pmf_pdf\/intro.html)\n    - [Concept](https:\/\/www.gaohongnan.com\/probability_theory\/05_joint_distributions\/0503_conditional_pmf_pdf\/concept.html)\n    - [Application](https:\/\/www.gaohongnan.com\/probability_theory\/05_joint_distributions\/0503_conditional_pmf_pdf\/application.html)\n  - [Conditional Expectation and Variance](https:\/\/www.gaohongnan.com\/probability_theory\/05_joint_distributions\/0504_conditional_expectation_variance\/intro.html)\n    - [Concept](https:\/\/www.gaohongnan.com\/probability_theory\/05_joint_distributions\/0504_conditional_expectation_variance\/concept.html)\n    - [Exercises](https:\/\/www.gaohongnan.com\/probability_theory\/05_joint_distributions\/0504_conditional_expectation_variance\/exercises.html)\n  - [Sum of Random Variables](https:\/\/www.gaohongnan.com\/probability_theory\/05_joint_distributions\/0505_sum_of_random_variables\/intro.html)\n    - [Concept](https:\/\/www.gaohongnan.com\/probability_theory\/05_joint_distributions\/0505_sum_of_random_variables\/concept.html)\n  - [Random Vectors](https:\/\/www.gaohongnan.com\/probability_theory\/05_joint_distributions\/0506_random_vectors\/intro.html)\n    - [Concept](https:\/\/www.gaohongnan.com\/probability_theory\/05_joint_distributions\/0506_random_vectors\/concept.html)\n  - [Multivariate Gaussian Distribution](https:\/\/www.gaohongnan.com\/probability_theory\/05_joint_distributions\/0507_multivariate_gaussian\/intro.html)\n    - [Concept](https:\/\/www.gaohongnan.com\/probability_theory\/05_joint_distributions\/0507_multivariate_gaussian\/concept.html)\n    - [Application: Plots and Transformations](https:\/\/www.gaohongnan.com\/probability_theory\/05_joint_distributions\/0507_multivariate_gaussian\/application_transformation.html)\n    - [Covariance Matrix is Positive Semi-Definite](https:\/\/www.gaohongnan.com\/probability_theory\/05_joint_distributions\/0507_multivariate_gaussian\/psd.html)\n    - [Eigendecomposition and Covariance Matrix](https:\/\/www.gaohongnan.com\/probability_theory\/05_joint_distributions\/0507_multivariate_gaussian\/eigendecomposition.html)\n    - [The Geometry of Multivariate Gaussians](https:\/\/www.gaohongnan.com\/probability_theory\/05_joint_distributions\/0507_multivariate_gaussian\/geometry_of_multivariate_gaussian.html)\n- [Chapter 6. Sample Statistics](https:\/\/www.gaohongnan.com\/probability_theory\/06_sample_statistics\/intro.html)\n  - [Moment Generating and Characteristic Functions](https:\/\/www.gaohongnan.com\/probability_theory\/06_sample_statistics\/0601_moment_generating_and_characteristic_functions\/intro.html)\n    - [Moment Generating Function](https:\/\/www.gaohongnan.com\/probability_theory\/06_sample_statistics\/0601_moment_generating_and_characteristic_functions\/moment_generating_function.html)\n    - [Application: Moment Generating Function and the Sum of Random Variables](https:\/\/www.gaohongnan.com\/probability_theory\/06_sample_statistics\/0601_moment_generating_and_characteristic_functions\/moment_generating_function_application_sum_of_rv.html)\n    - [Characteristic Function](https:\/\/www.gaohongnan.com\/probability_theory\/06_sample_statistics\/0601_moment_generating_and_characteristic_functions\/characteristic_function.html)\n  - [Probability Inequalities](https:\/\/www.gaohongnan.com\/probability_theory\/06_sample_statistics\/0602_probability_inequalities\/intro.html)\n    - [Probability Inequalities](https:\/\/www.gaohongnan.com\/probability_theory\/06_sample_statistics\/0602_probability_inequalities\/concept.html)\n    - [Application: Learning Theory](https:\/\/www.gaohongnan.com\/probability_theory\/06_sample_statistics\/0602_probability_inequalities\/application.html)\n  - [Law of Large Numbers](https:\/\/www.gaohongnan.com\/probability_theory\/06_sample_statistics\/0603_law_of_large_numbers\/intro.html)\n    - [Concept](https:\/\/www.gaohongnan.com\/probability_theory\/06_sample_statistics\/0603_law_of_large_numbers\/concept.html)\n    - [Convergence of Sample Average](https:\/\/www.gaohongnan.com\/probability_theory\/06_sample_statistics\/0603_law_of_large_numbers\/convergence.html)\n    - [Application: Learning Theory](https:\/\/www.gaohongnan.com\/probability_theory\/06_sample_statistics\/0603_law_of_large_numbers\/application.html)\n- [Chapter 8. Estimation Theory](https:\/\/www.gaohongnan.com\/probability_theory\/08_estimation_theory\/intro.html)\n  - [Maximum Likelihood Estimation](https:\/\/www.gaohongnan.com\/probability_theory\/08_estimation_theory\/maximum_likelihood_estimation\/intro.html)\n    - [Concept](https:\/\/www.gaohongnan.com\/probability_theory\/08_estimation_theory\/maximum_likelihood_estimation\/concept.html)\n\nOperations\n\n- [Distributed Systems](https:\/\/www.gaohongnan.com\/operations\/distributed\/intro.html)\n  - [Notations](https:\/\/www.gaohongnan.com\/operations\/distributed\/01_notations.html)\n  - [Basics Of Distributed Data Parallelism](https:\/\/www.gaohongnan.com\/operations\/distributed\/02_basics.html)\n  - [How to Setup SLURM and ParallelCluster in AWS](https:\/\/www.gaohongnan.com\/operations\/distributed\/03_how_to_setup_slurm_in_aws.html)\n  - [Ablations](https:\/\/www.gaohongnan.com\/operations\/distributed\/04_ablation.html)\n- [Profiling](https:\/\/www.gaohongnan.com\/operations\/profiling\/intro.html)\n  - [Synchronize CUDA To Time CUDA Operations](https:\/\/www.gaohongnan.com\/operations\/profiling\/01_synchronize.html)\n  - [Profiling Code With Timeit](https:\/\/www.gaohongnan.com\/operations\/profiling\/02_timeit.html)\n  - [PyTorch’s Event And Profiler](https:\/\/www.gaohongnan.com\/operations\/profiling\/03_time_profiler.html)\n  - [Profile GPT Small Time And Memory](https:\/\/www.gaohongnan.com\/operations\/profiling\/04_small_gpt_profile.html)\n  - [CUDA Memory Allocations](https:\/\/www.gaohongnan.com\/operations\/profiling\/05_memory_leak.html)\n- [The Lifecycle of an AIOps System](https:\/\/www.gaohongnan.com\/operations\/machine_learning_lifecycle\/00_intro.html)\n  - [Stage 1. Problem Formulation](https:\/\/www.gaohongnan.com\/operations\/machine_learning_lifecycle\/01_problem_formulation.html)\n  - [Stage 2. Project Scoping And Framing The Problem](https:\/\/www.gaohongnan.com\/operations\/machine_learning_lifecycle\/02_project_scoping.html)\n  - [Stage 3. Data Pipeline (Data Engineering and DataOps)](https:\/\/www.gaohongnan.com\/operations\/machine_learning_lifecycle\/03_dataops_pipeline\/03_dataops_pipeline.html)\n    - [Stage 3.1. Data Source and Formats](https:\/\/www.gaohongnan.com\/operations\/machine_learning_lifecycle\/03_dataops_pipeline\/031_data_source_and_format.html)\n    - [Stage 3.2. Data Model and Storage](https:\/\/www.gaohongnan.com\/operations\/machine_learning_lifecycle\/03_dataops_pipeline\/032_data_model_and_storage.html)\n    - [Stage 3.3. Extract, Transform, Load (ETL)](https:\/\/www.gaohongnan.com\/operations\/machine_learning_lifecycle\/03_dataops_pipeline\/033_etl.html)\n  - [Stage 4. Data Extraction (MLOps), Data Analysis (Data Science), Data Preparation (Data Science)](https:\/\/www.gaohongnan.com\/operations\/machine_learning_lifecycle\/04_mlops_data_pipeline.html)\n  - [Stage 5. Model Development and Training (MLOps)](https:\/\/www.gaohongnan.com\/operations\/machine_learning_lifecycle\/05_model_development_selection_and_training\/05_ml_training_pipeline.html)\n    - [Stage 5.1. Model Selection](https:\/\/www.gaohongnan.com\/operations\/machine_learning_lifecycle\/05_model_development_selection_and_training\/051_model_selection.html)\n    - [Stage 5.2. Metric Selection](https:\/\/www.gaohongnan.com\/operations\/machine_learning_lifecycle\/05_model_development_selection_and_training\/052_metric_selection.html)\n    - [Stage 5.3. Experiment Tracking And Versioning](https:\/\/www.gaohongnan.com\/operations\/machine_learning_lifecycle\/05_model_development_selection_and_training\/053_experiment_tracking.html)\n    - [Stage 5.4. Model Testing](https:\/\/www.gaohongnan.com\/operations\/machine_learning_lifecycle\/05_model_development_selection_and_training\/054_model_testing.html)\n  - [Stage 6. Model Evaluation (MLOps)](https:\/\/www.gaohongnan.com\/operations\/machine_learning_lifecycle\/06_model_evaluation.html)\n  - [Stage 7. Model Validation, Registry and Pushing Model to Production (MLOps)](https:\/\/www.gaohongnan.com\/operations\/machine_learning_lifecycle\/07_model_validation_registry_and_pushing_model_to_production.html)\n  - [Stage 8. Model Serving (MLOps)](https:\/\/www.gaohongnan.com\/operations\/machine_learning_lifecycle\/08_model_deployment_and_serving.html)\n  - [Stage 9. Model Monitoring (MLOps)](https:\/\/www.gaohongnan.com\/operations\/machine_learning_lifecycle\/09_model_monitoring.html)\n  - [Stage 10. Continuous Integration, Deployment, Learning and Training (DevOps, DataOps, MLOps)](https:\/\/www.gaohongnan.com\/operations\/machine_learning_lifecycle\/010_continuous_integration_deployment_learning_and_training.html)\n\nSoftware Engineering\n\n- [Config, State, Metadata Management](https:\/\/www.gaohongnan.com\/software_engineering\/config_management\/intro.html)\n  - [Configuration Management](https:\/\/www.gaohongnan.com\/software_engineering\/config_management\/concept.html)\n  - [Pydantic And Hydra](https:\/\/www.gaohongnan.com\/software_engineering\/config_management\/01-pydra.html)\n  - [State And Metadata Management](https:\/\/www.gaohongnan.com\/software_engineering\/config_management\/02-state.html)\n- [Design Patterns](https:\/\/www.gaohongnan.com\/software_engineering\/design_patterns\/intro.html)\n  - [Dependency Inversion Principle](https:\/\/www.gaohongnan.com\/software_engineering\/design_patterns\/dependency_inversion_principle.html)\n  - [Named Constructor](https:\/\/www.gaohongnan.com\/software_engineering\/design_patterns\/named_constructor.html)\n  - [Strategy](https:\/\/www.gaohongnan.com\/software_engineering\/design_patterns\/strategy.html)\n  - [Registry](https:\/\/www.gaohongnan.com\/software_engineering\/design_patterns\/registry.html)\n  - [Context Object Pattern (God Object)](https:\/\/www.gaohongnan.com\/software_engineering\/design_patterns\/god_object_pattern.html)\n  - [Factory Method](https:\/\/www.gaohongnan.com\/software_engineering\/design_patterns\/factory_method.html)\n  - [Singleton](https:\/\/www.gaohongnan.com\/software_engineering\/design_patterns\/singleton.html)\n- [Python](https:\/\/www.gaohongnan.com\/software_engineering\/python\/intro.html)\n  - [Init vs New](https:\/\/www.gaohongnan.com\/software_engineering\/python\/new_vs_init.html)\n  - [Global Interpreter Lock (GIL)](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html)\n  - [The Iterator Protocol](https:\/\/www.gaohongnan.com\/software_engineering\/python\/iterator_protocol.html)\n  - [Decorator](https:\/\/www.gaohongnan.com\/software_engineering\/python\/decorator.html)\n  - [Generators Over Lists For Memory Efficiency](https:\/\/www.gaohongnan.com\/software_engineering\/python\/generators_over_lists.html)\n  - [Pydantic Is All You Need - Jason Liu](https:\/\/www.gaohongnan.com\/software_engineering\/python\/pydantic.html)\n  - [Do Not Use Mutable Default Arguments](https:\/\/www.gaohongnan.com\/software_engineering\/python\/mutable_default.html)\n  - [Set Over List For Frequent Membership Tests](https:\/\/www.gaohongnan.com\/software_engineering\/python\/set_vs_list.html)\n  - [Late Binding Closures](https:\/\/www.gaohongnan.com\/software_engineering\/python\/late_binding_closures.html)\n  - [Is vs Equality](https:\/\/www.gaohongnan.com\/software_engineering\/python\/is_vs_equality.html)\n- [Concurrency, Parallelism and Asynchronous Programming](https:\/\/www.gaohongnan.com\/software_engineering\/concurrency_parallelism_asynchronous\/intro.html)\n  - [Overview Of Concurrency, Parallelism, and Asynchronous Execution](https:\/\/www.gaohongnan.com\/software_engineering\/concurrency_parallelism_asynchronous\/overview.html)\n  - [Thread Safety](https:\/\/www.gaohongnan.com\/software_engineering\/concurrency_parallelism_asynchronous\/insights\/locks_for_thread_safety.html)\n  - [A Rudimentary Introduction to Generator and Yield in Python](https:\/\/www.gaohongnan.com\/software_engineering\/concurrency_parallelism_asynchronous\/generator_yield.html)\n\nComputer Science\n\n- [Type Theory, A Very Rudimentary Introduction](https:\/\/www.gaohongnan.com\/computer_science\/type_theory\/intro.html)\n  - [Subtypes](https:\/\/www.gaohongnan.com\/computer_science\/type_theory\/01-subtypes.html)\n  - [Type Safety](https:\/\/www.gaohongnan.com\/computer_science\/type_theory\/02-type-safety.html)\n  - [Subsumption](https:\/\/www.gaohongnan.com\/computer_science\/type_theory\/03-subsumption.html)\n  - [Generics and Type Variables](https:\/\/www.gaohongnan.com\/computer_science\/type_theory\/04-generics.html)\n  - [Bound and Constraint in Generics and Type Variables](https:\/\/www.gaohongnan.com\/computer_science\/type_theory\/05-typevar-bound-constraints.html)\n  - [Invariance, Covariance and Contravariance](https:\/\/www.gaohongnan.com\/computer_science\/type_theory\/06-invariance-covariance-contravariance.html)\n  - [Function Overloading](https:\/\/www.gaohongnan.com\/computer_science\/type_theory\/07-pep-3124-overloading.html)\n  - [Sentinel Types](https:\/\/www.gaohongnan.com\/computer_science\/type_theory\/08-pep-661-sentinel-values.html)\n\nData Structures and Algorithms\n\n- [Complexity Analysis](https:\/\/www.gaohongnan.com\/dsa\/complexity_analysis\/intro.html)\n  - [Master Theorem](https:\/\/www.gaohongnan.com\/dsa\/complexity_analysis\/master_theorem.html)\n- [List\/Array](https:\/\/www.gaohongnan.com\/dsa\/array\/intro.html)\n  - [Concept](https:\/\/www.gaohongnan.com\/dsa\/array\/concept.html)\n  - [Questions](https:\/\/www.gaohongnan.com\/dsa\/array\/questions\/intro.html)\n    - [Two Sum](https:\/\/www.gaohongnan.com\/dsa\/array\/questions\/01-two-sum.html)\n- [Hash Map](https:\/\/www.gaohongnan.com\/dsa\/hash_map\/intro.html)\n  - [Concept](https:\/\/www.gaohongnan.com\/dsa\/hash_map\/concept.html)\n  - [Questions](https:\/\/www.gaohongnan.com\/dsa\/hash_map\/questions\/intro.html)\n    - [Two Sum](https:\/\/www.gaohongnan.com\/dsa\/hash_map\/questions\/01-two-sum.html)\n    - [Group Anagrams](https:\/\/www.gaohongnan.com\/dsa\/hash_map\/questions\/49-group-anagrams.html)\n- [Two Pointers And Sliding Window](https:\/\/www.gaohongnan.com\/dsa\/two_pointers\/intro.html)\n  - [Two Pointers](https:\/\/www.gaohongnan.com\/dsa\/two_pointers\/two_pointers.html)\n  - [Sliding Window](https:\/\/www.gaohongnan.com\/dsa\/two_pointers\/sliding_window.html)\n  - [Questions](https:\/\/www.gaohongnan.com\/dsa\/two_pointers\/questions\/intro.html)\n    - [Two Pointers](https:\/\/www.gaohongnan.com\/dsa\/two_pointers\/questions\/two_pointers\/intro.html)\n      - [Remove Duplicates from Sorted Array](https:\/\/www.gaohongnan.com\/dsa\/two_pointers\/questions\/two_pointers\/26-remove-duplicates-from-sorted-array.html)\n      - [Two Sum II - Input Array Is Sorted](https:\/\/www.gaohongnan.com\/dsa\/two_pointers\/questions\/two_pointers\/167-two-sum-ii-input-array-is-sorted.html)\n    - [Sliding Window](https:\/\/www.gaohongnan.com\/dsa\/two_pointers\/questions\/sliding_window\/intro.html)\n      - [Find All Anagrams in a String](https:\/\/www.gaohongnan.com\/dsa\/two_pointers\/questions\/sliding_window\/438-find-all-anagrams-in-a-string.html)\n- [Stack](https:\/\/www.gaohongnan.com\/dsa\/stack\/intro.html)\n  - [Concept](https:\/\/www.gaohongnan.com\/dsa\/stack\/concept.html)\n  - [Questions](https:\/\/www.gaohongnan.com\/dsa\/stack\/questions\/intro.html)\n    - [Valid Parentheses](https:\/\/www.gaohongnan.com\/dsa\/stack\/questions\/20-valid-parentheses.html)\n    - [Min Stack](https:\/\/www.gaohongnan.com\/dsa\/stack\/questions\/155-min-stack.html)\n    - [Implement Queue using Stacks](https:\/\/www.gaohongnan.com\/dsa\/stack\/questions\/232-implement-queue-using-stacks.html)\n    - [Reverse String](https:\/\/www.gaohongnan.com\/dsa\/stack\/questions\/344-reverse-string.html)\n- [Queue](https:\/\/www.gaohongnan.com\/dsa\/queue\/intro.html)\n  - [Concept](https:\/\/www.gaohongnan.com\/dsa\/queue\/concept.html)\n  - [Double Ended Queue](https:\/\/www.gaohongnan.com\/dsa\/queue\/dequeue.html)\n    - [Easy - Hot Potatoes](https:\/\/www.gaohongnan.com\/dsa\/queue\/questions\/hot-potatoes.html)\n    - [Palindrome Checker](https:\/\/www.gaohongnan.com\/dsa\/queue\/questions\/125-valid-palindrome.html)\n- [Linear Search](https:\/\/www.gaohongnan.com\/dsa\/searching_algorithms\/linear_search\/intro.html)\n  - [Concept](https:\/\/www.gaohongnan.com\/dsa\/searching_algorithms\/linear_search\/concept.html)\n- [Binary Search](https:\/\/www.gaohongnan.com\/dsa\/searching_algorithms\/binary_search\/intro.html)\n  - [Concept](https:\/\/www.gaohongnan.com\/dsa\/searching_algorithms\/binary_search\/concept.html)\n  - [Koko Eating Bananas](https:\/\/www.gaohongnan.com\/dsa\/searching_algorithms\/binary_search\/problems\/875-koko-eating-bananas.html)\n\nLinear Algebra\n\n- [Preliminaries](https:\/\/www.gaohongnan.com\/linear_algebra\/01_preliminaries\/intro.html)\n  - [Fields](https:\/\/www.gaohongnan.com\/linear_algebra\/01_preliminaries\/01-fields.html)\n  - [Systems of Linear Equations](https:\/\/www.gaohongnan.com\/linear_algebra\/01_preliminaries\/02-systems-of-linear-equations.html)\n- [Vectors](https:\/\/www.gaohongnan.com\/linear_algebra\/02_vectors\/intro.html)\n  - [Vector and Its Definition](https:\/\/www.gaohongnan.com\/linear_algebra\/02_vectors\/01-vector-definition.html)\n  - [Vector and Its Operations](https:\/\/www.gaohongnan.com\/linear_algebra\/02_vectors\/02-vector-operation.html)\n  - [Vector Norm and Distance](https:\/\/www.gaohongnan.com\/linear_algebra\/02_vectors\/03-vector-norm.html)\n  - [A First Look at Vector Products](https:\/\/www.gaohongnan.com\/linear_algebra\/02_vectors\/04-vector-products.html)\n\nReferences, Resources and Roadmap\n\n- [Bibliography](https:\/\/www.gaohongnan.com\/bibliography.html)\n- [IEEE (Style) Citations](https:\/\/www.gaohongnan.com\/citations.html)\n\n- [![Colab logo](https:\/\/www.gaohongnan.com\/_static\/images\/logo_colab.png) Colab](https:\/\/colab.research.google.com\/github\/gao-hongnan\/omniverse\/blob\/main\/omniverse\/software_engineering\/python\/gil.ipynb \"Launch on Colab\")\n\n- [Repository](https:\/\/github.com\/gao-hongnan\/omniverse \"Source repository\")\n- [Open issue](https:\/\/github.com\/gao-hongnan\/omniverse\/issues\/new?title=Issue%20on%20page%20%2Fsoftware_engineering\/python\/gil.html&body=Your%20issue%20content%20here. \"Open an issue\")\n\n- [.ipynb](https:\/\/www.gaohongnan.com\/_sources\/software_engineering\/python\/gil.ipynb \"Download source file\")\n- .pdf\n\n# Global Interpreter Lock (GIL)\n## Contents\n- [Reference Counting in Python](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#reference-counting-in-python)\n- [Threads and Race Conditions](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#threads-and-race-conditions)\n- [Protecting Reference Counts with Locks](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#protecting-reference-counts-with-locks)\n- [Deadlocks](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#deadlocks)\n- [The Global Interpreter Lock (GIL) in Python](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#the-global-interpreter-lock-gil-in-python)\n- [CPU-bound tasks](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#cpu-bound-tasks)\n- [GIL Is Released During IO-bound Tasks](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#gil-is-released-during-io-bound-tasks)\n- [References and Further Readings](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#references-and-further-readings)\n\n# Global Interpreter Lock (GIL)[\\#](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#global-interpreter-lock-gil \"Link to this heading\")\n[![Twitter Handle](https:\/\/img.shields.io\/badge\/Twitter-@gaohongnan-blue?style=social&logo=twitter)](https:\/\/twitter.com\/gaohongnan) [![LinkedIn Profile](https:\/\/img.shields.io\/badge\/@gaohongnan-blue?style=social&logo=linkedin)](https:\/\/linkedin.com\/in\/gao-hongnan) [![GitHub Profile](https:\/\/img.shields.io\/badge\/GitHub-gao--hongnan-lightgrey?style=social&logo=github)](https:\/\/github.com\/gao-hongnan) ![Tag](https:\/\/img.shields.io\/badge\/Tag-Organized_Chaos-orange)\n\n- [Reference Counting in Python](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#reference-counting-in-python)\n- [Threads and Race Conditions](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#threads-and-race-conditions)\n- [Protecting Reference Counts with Locks](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#protecting-reference-counts-with-locks)\n- [Deadlocks](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#deadlocks)\n- [The Global Interpreter Lock (GIL) in Python](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#the-global-interpreter-lock-gil-in-python)\n- [CPU-bound tasks](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#cpu-bound-tasks)\n- [GIL Is Released During IO-bound Tasks](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#gil-is-released-during-io-bound-tasks)\n- [References and Further Readings](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#references-and-further-readings)\n\n## [Reference Counting in Python](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#id1)[\\#](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#reference-counting-in-python \"Link to this heading\")\nIn Python, **reference counting** is a memory management technique used to keep track of how many references (or “pointers”) exist to an object in memory. When an object’s reference count drops to zero, Python automatically frees the memory allocated to that object.\n\n```\nimport sys\n\na = []\nb = a\na_ref_count = sys.getrefcount(a)\nprint(a_ref_count)  # Output: 3\n```\n```\n3\n```\n\n1. `a = []` creates an empty list `[]` and assigns it to variable `a`.\n2. `b = a` makes `b` reference the same list as `a`.\n3. `sys.getrefcount(a)` returns the number of references to the list `a` points to. The count is `3` because:\n   - `a` references the list.\n   - `b` references the list.\n   - The `sys.getrefcount(a)` function temporarily creates another reference when it’s called.\n\nAs a beginner python programmer, we thank our lucky stars that Python manages memory for us. But, as a good student, we ask, why is managing the reference count important?\n\nFirst, **memory management** helps Python automatically manage memory by keeping track of objects and freeing memory when objects are no longer needed. For example, if we delete the reference `b`, the reference count of `a` drops to 2.\n\n```\ndel b\na_ref_count = sys.getrefcount(a)\nprint(a_ref_count)  # Output: 2\n```\n```\n2\n```\n\nSecond, **avoiding memory leaks** ensures that memory is reused efficiently and prevents memory from being wasted on unused objects.\n\n## [Threads and Race Conditions](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#id2)[\\#](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#threads-and-race-conditions \"Link to this heading\")\nA **race condition** occurs when two or more threads access shared data simultaneously, and the final outcome depends on the sequence of execution.\n\nFor example, if two threads try to increment a counter from 5 to 6:\n\n1. Thread A reads counter (value = 5)\n2. Thread B reads counter (value = 5)\n3. Thread A adds 1 (5 + 1 = 6)\n4. Thread B adds 1 (5 + 1 = 6)\n5. Thread A writes 6\n6. Thread B writes 6\n\nEven though two increment operations occurred, the counter only increased by 1 instead of 2. This happens because both threads read the original value before either could update it. The final result depends on which thread writes last, and the program “races” to an incorrect outcome.\n\nShow me the code to illustrate the race condition\\!\n\n```\n\"\"\"With reference to effective python book chapter 54.\nRef: https:\/\/github.com\/bslatkin\/effectivepython\/blob\/master\/example_code\/item_54.py\n\"\"\"\nimport logging\nimport threading\nfrom threading import Barrier\nfrom typing import List\n\nlogging.basicConfig(level=logging.DEBUG, format=\"%(asctime)s - %(levelname)s - %(message)s\")\n\nNUM_THREADS = 5\nBARRIER = Barrier(NUM_THREADS)\n\n\nclass Counter:\n    def __init__(self) -> None:\n        self.count = 0\n\n    def increment(self, offset: int) -> None:\n        self.count += offset\n\n\ndef worker(thread_index: int, total_iterations: int, counter: Counter) -> None:\n    \"\"\"The barrier is used to synchronize the threads so that they all start counting\n    at the same time. This makes it easier to get a race condition since we wait for\n    the other threads to start else in the loop we always have an order that the\n    first thread likely starts first and then the second and so on.\n    \"\"\"\n    BARRIER.wait()\n    logging.debug(\"Thread %s, starting\", thread_index)\n    for _ in range(total_iterations):\n        counter.increment(1)\n\n\ndef thread_unsafe(total_iterations: int) -> None:\n    counter = Counter()\n\n    threads: List[threading.Thread] = []\n    for index in range(NUM_THREADS):\n        thread = threading.Thread(target=worker, args=(index, total_iterations, counter))\n        threads.append(thread)\n\n    for thread in threads:\n        thread.start()\n\n    for thread in threads:\n        thread.join()\n\n    expected = total_iterations * NUM_THREADS\n    found = counter.count\n    logging.info(\"Counter should be %s, got %s\", expected, found)\n\n\nif __name__ == \"__main__\":\n    total_iterations = 10**6\n\n    thread_unsafe(total_iterations)\n```\n```\n2024-10-29 10:22:29,521 - DEBUG - Thread 4, starting\n2024-10-29 10:22:29,522 - DEBUG - Thread 0, starting\n2024-10-29 10:22:29,522 - DEBUG - Thread 3, starting\n2024-10-29 10:22:29,522 - DEBUG - Thread 2, starting\n2024-10-29 10:22:29,523 - DEBUG - Thread 1, starting\n2024-10-29 10:22:29,951 - INFO - Counter should be 5000000, got 4564365\n```\n\n## [Protecting Reference Counts with Locks](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#id3)[\\#](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#protecting-reference-counts-with-locks \"Link to this heading\")\nSo we circle back a bit to the reference count thingy earlier. Since we saw how race conditions can happen, the same thing can happen to the reference count of an object - which means that the **reference count** of an object can be modified by multiple threads. If two threads try to increment or decrement the reference count simultaneously without protection, it can lead to inconsistencies, such as the case where reference counts might reach zero incorrectly, freeing memory while it’s still in use, causing crashes or bugs.\n\nHow do we prevent this? We first understand the rough idea of a **lock**.\n\nA **lock** is a synchronization mechanism used to control access to shared resources. When a thread acquires a lock, other threads must wait until the lock is released before accessing the resource.\n\n- In the first example without a lock, both threads might read the same `counter` value before incrementing it, causing some increments to be lost.\n- In the second example, the lock ensures that only one thread can modify the counter at a time.\n- The `with lock:` statement automatically acquires the lock before entering the block and releases it when exiting.\n- This guarantees that the final counter value will always be correct.\n\n```\n\"\"\"With reference to effective python book chapter 54.\nRef: https:\/\/github.com\/bslatkin\/effectivepython\/blob\/master\/example_code\/item_54.py\n\"\"\"\nimport logging\nimport threading\nfrom threading import Barrier\nfrom typing import List\n\nlogging.basicConfig(level=logging.DEBUG, format=\"%(asctime)s - %(levelname)s - %(message)s\")\n\nNUM_THREADS = 5\nBARRIER = Barrier(NUM_THREADS)\n\n\nclass CounterLock:\n    def __init__(self) -> None:\n        self.count = 0\n        self.lock = threading.Lock()\n\n    def increment(self, offset: int) -> None:\n        with self.lock:\n            self.count += offset\n\n\ndef worker(thread_index: int, total_iterations: int, counter: Counter) -> None:\n    \"\"\"The barrier is used to synchronize the threads so that they all start counting\n    at the same time. This makes it easier to get a race condition since we wait for\n    the other threads to start else in the loop we always have an order that the\n    first thread likely starts first and then the second and so on.\n    \"\"\"\n    BARRIER.wait()\n    logging.debug(\"Thread %s, starting\", thread_index)\n    for _ in range(total_iterations):\n        counter.increment(1)\n\n\ndef thread_safe(total_iterations: int) -> None:\n    counter = CounterLock()\n\n    threads: List[threading.Thread] = []\n    for index in range(NUM_THREADS):\n        thread = threading.Thread(target=worker, args=(index, total_iterations, counter))\n        threads.append(thread)\n\n    for thread in threads:\n        thread.start()\n\n    for thread in threads:\n        thread.join()\n\n    expected = total_iterations * NUM_THREADS\n    found = counter.count\n\n    logging.info(\"Counter should be %s, got %s\", expected, found)\n\n\nif __name__ == \"__main__\":\n    total_iterations = 10**6\n\n    thread_safe(total_iterations)\n```\n```\n2024-10-29 10:22:29,959 - DEBUG - Thread 4, starting\n2024-10-29 10:22:29,959 - DEBUG - Thread 0, starting\n2024-10-29 10:22:29,959 - DEBUG - Thread 3, starting\n2024-10-29 10:22:29,960 - DEBUG - Thread 2, starting\n2024-10-29 10:22:29,960 - DEBUG - Thread 1, starting\n2024-10-29 10:22:30,821 - INFO - Counter should be 5000000, got 5000000\n```\n\n## [Deadlocks](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#id4)[\\#](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#deadlocks \"Link to this heading\")\nA **deadlock** occurs when two or more threads are waiting indefinitely for locks held by each other, preventing them from making progress.\n```\nimport threading\nimport time\n\n\nclass BankAccount:\n    def __init__(self, id: int, balance: float) -> None:\n        self.id: int = id\n        self.balance: float = balance\n        self.lock: threading.Lock = threading.Lock()\n\n    def __str__(self) -> str:\n        return f\"Account {self.id}: ${self.balance}\"\n\n\ndef unsafe_transfer(from_account: BankAccount, to_account: BankAccount, amount: float) -> None:\n    with from_account.lock:\n        print(f\"Locked account {from_account.id}\")\n        time.sleep(0.5)  # Simulate some work and make deadlock more likely\n        with to_account.lock:\n            print(f\"Locked account {to_account.id}\")\n            if from_account.balance >= amount:\n                from_account.balance -= amount\n                to_account.balance += amount\n                print(f\"Transferred ${amount} from Account {from_account.id} to Account {to_account.id}\")\n\n\ndef safe_transfer(from_account: BankAccount, to_account: BankAccount, amount: float) -> None:\n    first: BankAccount = from_account if from_account.id < to_account.id else to_account\n    second: BankAccount = to_account if from_account.id < to_account.id else from_account\n\n    with first.lock:\n        print(f\"Locked account {first.id}\")\n        time.sleep(0.5)  # Simulate some work\n        with second.lock:\n            print(f\"Locked account {second.id}\")\n            if from_account.balance >= amount:\n                from_account.balance -= amount\n                to_account.balance += amount\n                print(f\"Transferred ${amount} from Account {from_account.id} to Account {to_account.id}\")\n\n\naccount1: BankAccount = BankAccount(1, 1000)\naccount2: BankAccount = BankAccount(2, 1000)\n\nprint(\"Initial balances:\")\nprint(account1)\nprint(account2)\nprint(\"\\nTrying unsafe transfers (will likely deadlock):\")\n\n# Create threads with unsafe transfers (will deadlock)\nthread1: threading.Thread = threading.Thread(target=unsafe_transfer, args=(account1, account2, 500))\nthread2: threading.Thread = threading.Thread(target=unsafe_transfer, args=(account2, account1, 300))\n\nthread1.start()\nthread2.start()\nthread1.join()\nthread2.join()\n```\n1. `thread1` acquires the lock on `account1` and waits to acquire the lock on `account2`.\n2. `thread2` acquires the lock on `account2` and waits to acquire the lock on `account1`.\n3. Both threads are waiting for each other to release the locks, causing a deadlock.\n\nWhy do we need to know this? Because as we saw earlier, locks are useful for preventing race conditions. But using locks not so carefully can lead to deadlocks. GIL will aim to solve this as well.\n\n## [The Global Interpreter Lock (GIL) in Python](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#id5)[\\#](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#the-global-interpreter-lock-gil-in-python \"Link to this heading\")\nThe **Global Interpreter Lock (GIL)** is a **mutex** (lock) that protects access to Python objects, preventing multiple native threads from executing Python bytecodes simultaneously in the same process.\n\n- To solve deadlocks, gil only has **one lock** for the entire process - thus the scenario where two threads are waiting for each other to release a lock is avoided. In more intuition, the gil allows only **one thread** to execute bytecode at a time.\n- With locks in place, race conditions are mostly resolved.\n\nWith gil, it has its own trade-offs:\n\n- **Limits Multi-threaded Performance:** Because the GIL allows only one thread to execute Python code at a time, CPU-bound multi-threaded programs don’t benefit from multiple cores. They run almost as if they were single-threaded.\n- **Inefficiency in Multi-core Systems:** In CPU-bound tasks, the GIL can become a bottleneck, preventing Python programs from fully utilizing multi-core processors.\n\nLet’s see an example of how gil can limit multi-threaded performance.\n\n## [CPU-bound tasks](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#id6)[\\#](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#cpu-bound-tasks \"Link to this heading\")\nWe first compare the performance of single-threaded vs multi-threaded and note for cpu-bound tasks, the time taken is almost the same.\n\n```\nimport threading\nimport time\nimport requests\n\n\ndef cpu_bound_task():\n    \"\"\"CPU intensive task - calculating sum of numbers\"\"\"\n    count = 0\n    for _ in range(20_000_000):\n        count += 1\n    return count\n\n\ndef run_tasks_single_thread(task, num_iterations):\n    start_time = time.time()\n\n    for _ in range(num_iterations):\n        task()\n\n    end_time = time.time()\n    return end_time - start_time\n\n\ndef run_tasks_multi_thread(task, num_threads):\n    start_time = time.time()\n\n    threads: List[threading.Thread] = []\n    for _ in range(num_threads):\n        t = threading.Thread(target=task)\n        threads.append(t)\n        t.start()\n\n    for t in threads:\n        t.join()\n\n    end_time = time.time()\n    return end_time - start_time\n\n\n# NOTE: CPU-bound tasks\nnum_tasks = 4\nprint(\"CPU-bound task comparison:\")\nsingle_thread_cpu = run_tasks_single_thread(cpu_bound_task, num_tasks)\nprint(f\"Single-threaded time: {single_thread_cpu:.2f} seconds\")\n\nmulti_thread_cpu = run_tasks_multi_thread(cpu_bound_task, num_tasks)\nprint(f\"Multi-threaded time: {multi_thread_cpu:.2f} seconds\")\nprint(f\"Speed difference: {single_thread_cpu\/multi_thread_cpu:.2f}x\\n\")\n```\n```\nCPU-bound task comparison:\nSingle-threaded time: 1.71 seconds\nMulti-threaded time: 1.62 seconds\nSpeed difference: 1.06x\n```\n\nNext, we compare the performance of single-threaded vs multi-threaded vs multi-process. We note that multi-process is the fastest and this is expected because there is no GIL in multi-process.\n\n```\nimport multiprocessing\n\n\ndef run_tasks_multi_process(task, num_processes):\n    \"\"\"Run tasks using multiple processes\"\"\"\n    start_time = time.time()\n\n    processes: List[multiprocessing.Process] = []\n    for _ in range(num_processes):\n        p = multiprocessing.Process(target=task)\n        processes.append(p)\n        p.start()\n\n    for p in processes:\n        p.join()\n\n    end_time = time.time()\n    return end_time - start_time\n\n\n# Comparison of all three approaches\nnum_tasks = 4\n\nprint(\"CPU-bound task comparison:\")\nsingle_thread_cpu = run_tasks_single_thread(cpu_bound_task, num_tasks)\nprint(f\"Single-threaded time: {single_thread_cpu:.2f} seconds\")\n\nmulti_thread_cpu = run_tasks_multi_thread(cpu_bound_task, num_tasks)\nprint(f\"Multi-threaded time: {multi_thread_cpu:.2f} seconds\")\n\nmulti_process_cpu = run_tasks_multi_process(cpu_bound_task, num_tasks)\nprint(f\"Multi-process time: {multi_process_cpu:.2f} seconds\")\n\nprint(f\"\\nSpeed comparison (relative to single-thread):\")\nprint(f\"Threading speedup: {single_thread_cpu\/multi_thread_cpu:.2f}x\")\nprint(f\"Multiprocessing speedup: {single_thread_cpu\/multi_process_cpu:.2f}x\\n\")\n```\n```\nCPU-bound task comparison:\nSingle-threaded time: 1.73 seconds\nMulti-threaded time: 1.58 seconds\nMulti-process time: 0.09 seconds\n\nSpeed comparison (relative to single-thread):\nThreading speedup: 1.10x\nMultiprocessing speedup: 18.59x\n```\n```\nTraceback (most recent call last):\n  File \"<string>\", line 1, in <module>\n  File \"\/opt\/homebrew\/Caskroom\/miniconda\/base\/envs\/omniverse\/lib\/python3.9\/multiprocessing\/spawn.py\", line 116, in spawn_main\nTraceback (most recent call last):\n  File \"<string>\", line 1, in <module>\n  File \"\/opt\/homebrew\/Caskroom\/miniconda\/base\/envs\/omniverse\/lib\/python3.9\/multiprocessing\/spawn.py\", line 116, in spawn_main\nTraceback (most recent call last):\n  File \"<string>\", line 1, in <module>\n  File \"\/opt\/homebrew\/Caskroom\/miniconda\/base\/envs\/omniverse\/lib\/python3.9\/multiprocessing\/spawn.py\", line 116, in spawn_main\n    exitcode = _main(fd, parent_sentinel)\n  File \"\/opt\/homebrew\/Caskroom\/miniconda\/base\/envs\/omniverse\/lib\/python3.9\/multiprocessing\/spawn.py\", line 126, in _main\n        exitcode = _main(fd, parent_sentinel)exitcode = _main(fd, parent_sentinel)\n\n  File \"\/opt\/homebrew\/Caskroom\/miniconda\/base\/envs\/omniverse\/lib\/python3.9\/multiprocessing\/spawn.py\", line 126, in _main\n  File \"\/opt\/homebrew\/Caskroom\/miniconda\/base\/envs\/omniverse\/lib\/python3.9\/multiprocessing\/spawn.py\", line 126, in _main\n    self = reduction.pickle.load(from_parent)\nAttributeError: Can't get attribute 'cpu_bound_task' on <module '__main__' (built-in)>\n    self = reduction.pickle.load(from_parent)\nAttributeError: Can't get attribute 'cpu_bound_task' on <module '__main__' (built-in)>\n    self = reduction.pickle.load(from_parent)\nAttributeError: Can't get attribute 'cpu_bound_task' on <module '__main__' (built-in)>\nTraceback (most recent call last):\n  File \"<string>\", line 1, in <module>\n  File \"\/opt\/homebrew\/Caskroom\/miniconda\/base\/envs\/omniverse\/lib\/python3.9\/multiprocessing\/spawn.py\", line 116, in spawn_main\n    exitcode = _main(fd, parent_sentinel)\n  File \"\/opt\/homebrew\/Caskroom\/miniconda\/base\/envs\/omniverse\/lib\/python3.9\/multiprocessing\/spawn.py\", line 126, in _main\n    self = reduction.pickle.load(from_parent)\nAttributeError: Can't get attribute 'cpu_bound_task' on <module '__main__' (built-in)>\n```\n\n## [GIL Is Released During IO-bound Tasks](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#id7)[\\#](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#gil-is-released-during-io-bound-tasks \"Link to this heading\")\nAs mentioned, the GIL is released during IO-bound tasks. As to why, please see this [post](https:\/\/stackoverflow.com\/questions\/1294382\/what-is-the-global-interpreter-lock-gil-in-cpython\/55309364#55309364).\n\nLet’s see an example of this.\n\n```\ndef io_bound_task():\n    \"\"\"IO intensive task - making HTTP requests\"\"\"\n    url = \"https:\/\/api.github.com\"\n    response = requests.get(url)\n    return response.status_code\n\n\n# NOTE: IO-bound tasks\nprint(\"IO-bound task comparison:\")\nsingle_thread_io = run_tasks_single_thread(io_bound_task, num_tasks)\nprint(f\"Single-threaded time: {single_thread_io:.2f} seconds\")\n\nmulti_thread_io = run_tasks_multi_thread(io_bound_task, num_tasks)\nprint(f\"Multi-threaded time: {multi_thread_io:.2f} seconds\")\nprint(f\"Speed difference: {single_thread_io\/multi_thread_io:.2f}x\")\n```\n```\n2024-10-29 20:14:18,709 - DEBUG - Starting new HTTPS connection (1): api.github.com:443\n```\n```\nIO-bound task comparison:\n```\n```\n2024-10-29 20:14:19,014 - DEBUG - https:\/\/api.github.com:443 \"GET \/ HTTP\/1.1\" 200 510\n2024-10-29 20:14:19,021 - DEBUG - Starting new HTTPS connection (1): api.github.com:443\n2024-10-29 20:14:19,072 - DEBUG - https:\/\/api.github.com:443 \"GET \/ HTTP\/1.1\" 200 510\n2024-10-29 20:14:19,075 - DEBUG - Starting new HTTPS connection (1): api.github.com:443\n2024-10-29 20:14:19,122 - DEBUG - https:\/\/api.github.com:443 \"GET \/ HTTP\/1.1\" 200 510\n2024-10-29 20:14:19,126 - DEBUG - Starting new HTTPS connection (1): api.github.com:443\n2024-10-29 20:14:19,167 - DEBUG - https:\/\/api.github.com:443 \"GET \/ HTTP\/1.1\" 200 510\n2024-10-29 20:14:19,174 - DEBUG - Starting new HTTPS connection (1): api.github.com:443\n2024-10-29 20:14:19,174 - DEBUG - Starting new HTTPS connection (1): api.github.com:443\n2024-10-29 20:14:19,174 - DEBUG - Starting new HTTPS connection (1): api.github.com:443\n2024-10-29 20:14:19,174 - DEBUG - Starting new HTTPS connection (1): api.github.com:443\n2024-10-29 20:14:19,256 - DEBUG - https:\/\/api.github.com:443 \"GET \/ HTTP\/1.1\" 200 510\n2024-10-29 20:14:19,259 - DEBUG - https:\/\/api.github.com:443 \"GET \/ HTTP\/1.1\" 200 510\n2024-10-29 20:14:19,261 - DEBUG - https:\/\/api.github.com:443 \"GET \/ HTTP\/1.1\" 200 510\n2024-10-29 20:14:19,261 - DEBUG - https:\/\/api.github.com:443 \"GET \/ HTTP\/1.1\" 200 510\n```\n```\nSingle-threaded time: 0.50 seconds\nMulti-threaded time: 0.09 seconds\nSpeed difference: 5.24x\n```\n\nIndeed, the multi-threaded version is faster than the single-threaded version.\n\n## [References and Further Readings](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#id8)[\\#](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#references-and-further-readings \"Link to this heading\")\n- [The Global Interpreter Lock (GIL) in Python](https:\/\/realpython.com\/python-gil\/)\n- <https:\/\/stackoverflow.com\/questions\/1294382\/what-is-the-global-interpreter-lock-gil-in-cpython\/55309364#55309364>\n\n[previous Init vs New](https:\/\/www.gaohongnan.com\/software_engineering\/python\/new_vs_init.html \"previous page\")\n\n[next The Iterator Protocol](https:\/\/www.gaohongnan.com\/software_engineering\/python\/iterator_protocol.html \"next page\")\n\nContents\n\n- [Reference Counting in Python](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#reference-counting-in-python)\n- [Threads and Race Conditions](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#threads-and-race-conditions)\n- [Protecting Reference Counts with Locks](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#protecting-reference-counts-with-locks)\n- [Deadlocks](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#deadlocks)\n- [The Global Interpreter Lock (GIL) in Python](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#the-global-interpreter-lock-gil-in-python)\n- [CPU-bound tasks](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#cpu-bound-tasks)\n- [GIL Is Released During IO-bound Tasks](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#gil-is-released-during-io-bound-tasks)\n- [References and Further Readings](https:\/\/www.gaohongnan.com\/software_engineering\/python\/gil.html#references-and-further-readings)\n\nBy Gao Hongnan\n\n© Copyright 2024.\n\n### Connect with me\\!\n[Follow on LinkedIn](https:\/\/www.linkedin.com\/in\/gao-hongnan\/)\n\n[Follow on X](https:\/\/x.com\/gaohongnan)\n### Share this page\\!\n[Share on LinkedIn]()\n\n[Share on X]()","attrs_readable_markdown":null,"meta_canonical":null}

3. Robots.txt Check

Query:

Response:

4. Spam/Ban Check

Query:

Response:

5. Seen Status Check

ℹ️ Skipped - page is already crawled

📄

INDEXABLE

✅

CRAWLED

1 month ago

🤖

ROBOTS ALLOWED

Page Info Filters

Filter	Status	Condition	Details
HTTP status	PASS	`download_http_code = 200`	HTTP 200
Age cutoff	PASS	`download_stamp > now() - 6 MONTH`	1.4 months ago
History drop	PASS	`isNull(history_drop_reason)`	No drop reason
Spam/ban	PASS	`fh_dont_index != 1 AND ml_spam_score = 0`	ml_spam_score=0
Canonical	PASS	`meta_canonical IS NULL OR = '' OR = src_unparsed`	Not set

Page Details

Property	Value
URL	https://www.gaohongnan.com/software_engineering/python/gil.html
Last Crawled	2026-02-24 20:54:37 (1 month ago)
First Indexed	2024-10-30 09:12:08 (1 year ago)
HTTP Status Code	200
Meta Title	Global Interpreter Lock (GIL) — Omniverse
Meta Description	null
Meta Canonical	null
Boilerpipe Text	Reference Counting in Python In Python, reference counting is a memory management technique used to keep track of how many references (or “pointers”) exist to an object in memory. When an object’s reference count drops to zero, Python automatically frees the memory allocated to that object. import sys a = [] b = a a_ref_count = sys . getrefcount ( a ) print ( a_ref_count ) # Output: 3 a = [] creates an empty list [] and assigns it to variable a . b = a makes b reference the same list as a . sys.getrefcount(a) returns the number of references to the list a points to. The count is 3 because: a references the list. b references the list. The sys.getrefcount(a) function temporarily creates another reference when it’s called. As a beginner python programmer, we thank our lucky stars that Python manages memory for us. But, as a good student, we ask, why is managing the reference count important? First, memory management helps Python automatically manage memory by keeping track of objects and freeing memory when objects are no longer needed. For example, if we delete the reference b , the reference count of a drops to 2. del b a_ref_count = sys . getrefcount ( a ) print ( a_ref_count ) # Output: 2 Second, avoiding memory leaks ensures that memory is reused efficiently and prevents memory from being wasted on unused objects. Threads and Race Conditions A race condition occurs when two or more threads access shared data simultaneously, and the final outcome depends on the sequence of execution. For example, if two threads try to increment a counter from 5 to 6: Thread A reads counter (value = 5) Thread B reads counter (value = 5) Thread A adds 1 (5 + 1 = 6) Thread B adds 1 (5 + 1 = 6) Thread A writes 6 Thread B writes 6 Even though two increment operations occurred, the counter only increased by 1 instead of 2. This happens because both threads read the original value before either could update it. The final result depends on which thread writes last, and the program “races” to an incorrect outcome. Show me the code to illustrate the race condition! """With reference to effective python book chapter 54. Ref: https://github.com/bslatkin/effectivepython/blob/master/example_code/item_54.py """ import logging import threading from threading import Barrier from typing import List logging . basicConfig ( level = logging . DEBUG , format = " %(asctime)s - %(levelname)s - %(message)s " ) NUM_THREADS = 5 BARRIER = Barrier ( NUM_THREADS ) class Counter : def __init__ ( self ) -> None : self . count = 0 def increment ( self , offset : int ) -> None : self . count += offset def worker ( thread_index : int , total_iterations : int , counter : Counter ) -> None : """The barrier is used to synchronize the threads so that they all start counting at the same time. This makes it easier to get a race condition since we wait for the other threads to start else in the loop we always have an order that the first thread likely starts first and then the second and so on. """ BARRIER . wait () logging . debug ( "Thread %s , starting" , thread_index ) for _ in range ( total_iterations ): counter . increment ( 1 ) def thread_unsafe ( total_iterations : int ) -> None : counter = Counter () threads : List [ threading . Thread ] = [] for index in range ( NUM_THREADS ): thread = threading . Thread ( target = worker , args = ( index , total_iterations , counter )) threads . append ( thread ) for thread in threads : thread . start () for thread in threads : thread . join () expected = total_iterations * NUM_THREADS found = counter . count logging . info ( "Counter should be %s , got %s " , expected , found ) if __name__ == "__main__" : total_iterations = 10 ** 6 thread_unsafe ( total_iterations ) 2024-10-29 10:22:29,521 - DEBUG - Thread 4, starting 2024-10-29 10:22:29,522 - DEBUG - Thread 0, starting 2024-10-29 10:22:29,522 - DEBUG - Thread 3, starting 2024-10-29 10:22:29,522 - DEBUG - Thread 2, starting 2024-10-29 10:22:29,523 - DEBUG - Thread 1, starting 2024-10-29 10:22:29,951 - INFO - Counter should be 5000000, got 4564365 Protecting Reference Counts with Locks So we circle back a bit to the reference count thingy earlier. Since we saw how race conditions can happen, the same thing can happen to the reference count of an object - which means that the reference count of an object can be modified by multiple threads. If two threads try to increment or decrement the reference count simultaneously without protection, it can lead to inconsistencies, such as the case where reference counts might reach zero incorrectly, freeing memory while it’s still in use, causing crashes or bugs. How do we prevent this? We first understand the rough idea of a lock . A lock is a synchronization mechanism used to control access to shared resources. When a thread acquires a lock, other threads must wait until the lock is released before accessing the resource. In the first example without a lock, both threads might read the same counter value before incrementing it, causing some increments to be lost. In the second example, the lock ensures that only one thread can modify the counter at a time. The with lock: statement automatically acquires the lock before entering the block and releases it when exiting. This guarantees that the final counter value will always be correct. """With reference to effective python book chapter 54. Ref: https://github.com/bslatkin/effectivepython/blob/master/example_code/item_54.py """ import logging import threading from threading import Barrier from typing import List logging . basicConfig ( level = logging . DEBUG , format = " %(asctime)s - %(levelname)s - %(message)s " ) NUM_THREADS = 5 BARRIER = Barrier ( NUM_THREADS ) class CounterLock : def __init__ ( self ) -> None : self . count = 0 self . lock = threading . Lock () def increment ( self , offset : int ) -> None : with self . lock : self . count += offset def worker ( thread_index : int , total_iterations : int , counter : Counter ) -> None : """The barrier is used to synchronize the threads so that they all start counting at the same time. This makes it easier to get a race condition since we wait for the other threads to start else in the loop we always have an order that the first thread likely starts first and then the second and so on. """ BARRIER . wait () logging . debug ( "Thread %s , starting" , thread_index ) for _ in range ( total_iterations ): counter . increment ( 1 ) def thread_safe ( total_iterations : int ) -> None : counter = CounterLock () threads : List [ threading . Thread ] = [] for index in range ( NUM_THREADS ): thread = threading . Thread ( target = worker , args = ( index , total_iterations , counter )) threads . append ( thread ) for thread in threads : thread . start () for thread in threads : thread . join () expected = total_iterations * NUM_THREADS found = counter . count logging . info ( "Counter should be %s , got %s " , expected , found ) if __name__ == "__main__" : total_iterations = 10 ** 6 thread_safe ( total_iterations ) 2024-10-29 10:22:29,959 - DEBUG - Thread 4, starting 2024-10-29 10:22:29,959 - DEBUG - Thread 0, starting 2024-10-29 10:22:29,959 - DEBUG - Thread 3, starting 2024-10-29 10:22:29,960 - DEBUG - Thread 2, starting 2024-10-29 10:22:29,960 - DEBUG - Thread 1, starting 2024-10-29 10:22:30,821 - INFO - Counter should be 5000000, got 5000000 Deadlocks A deadlock occurs when two or more threads are waiting indefinitely for locks held by each other, preventing them from making progress. import threading import time class BankAccount : def __init__ ( self , id : int , balance : float ) -> None : self . id : int = id self . balance : float = balance self . lock : threading . Lock = threading . Lock () def __str__ ( self ) -> str : return f "Account { self . id } : $ { self . balance } " def unsafe_transfer ( from_account : BankAccount , to_account : BankAccount , amount : float ) -> None : with from_account . lock : print ( f "Locked account { from_account . id } " ) time . sleep ( 0.5 ) # Simulate some work and make deadlock more likely with to_account . lock : print ( f "Locked account { to_account . id } " ) if from_account . balance >= amount : from_account . balance -= amount to_account . balance += amount print ( f "Transferred $ { amount } from Account { from_account . id } to Account { to_account . id } " ) def safe_transfer ( from_account : BankAccount , to_account : BankAccount , amount : float ) -> None : first : BankAccount = from_account if from_account . id < to_account . id else to_account second : BankAccount = to_account if from_account . id < to_account . id else from_account with first . lock : print ( f "Locked account { first . id } " ) time . sleep ( 0.5 ) # Simulate some work with second . lock : print ( f "Locked account { second . id } " ) if from_account . balance >= amount : from_account . balance -= amount to_account . balance += amount print ( f "Transferred $ { amount } from Account { from_account . id } to Account { to_account . id } " ) account1 : BankAccount = BankAccount ( 1 , 1000 ) account2 : BankAccount = BankAccount ( 2 , 1000 ) print ( "Initial balances:" ) print ( account1 ) print ( account2 ) print ( " \n Trying unsafe transfers (will likely deadlock):" ) # Create threads with unsafe transfers (will deadlock) thread1 : threading . Thread = threading . Thread ( target = unsafe_transfer , args = ( account1 , account2 , 500 )) thread2 : threading . Thread = threading . Thread ( target = unsafe_transfer , args = ( account2 , account1 , 300 )) thread1 . start () thread2 . start () thread1 . join () thread2 . join () thread1 acquires the lock on account1 and waits to acquire the lock on account2 . thread2 acquires the lock on account2 and waits to acquire the lock on account1 . Both threads are waiting for each other to release the locks, causing a deadlock. Why do we need to know this? Because as we saw earlier, locks are useful for preventing race conditions. But using locks not so carefully can lead to deadlocks. GIL will aim to solve this as well. The Global Interpreter Lock (GIL) in Python The Global Interpreter Lock (GIL) is a mutex (lock) that protects access to Python objects, preventing multiple native threads from executing Python bytecodes simultaneously in the same process. To solve deadlocks, gil only has one lock for the entire process - thus the scenario where two threads are waiting for each other to release a lock is avoided. In more intuition, the gil allows only one thread to execute bytecode at a time. With locks in place, race conditions are mostly resolved. With gil, it has its own trade-offs: Limits Multi-threaded Performance: Because the GIL allows only one thread to execute Python code at a time, CPU-bound multi-threaded programs don’t benefit from multiple cores. They run almost as if they were single-threaded. Inefficiency in Multi-core Systems: In CPU-bound tasks, the GIL can become a bottleneck, preventing Python programs from fully utilizing multi-core processors. Let’s see an example of how gil can limit multi-threaded performance. CPU-bound tasks We first compare the performance of single-threaded vs multi-threaded and note for cpu-bound tasks, the time taken is almost the same. import threading import time import requests def cpu_bound_task (): """CPU intensive task - calculating sum of numbers""" count = 0 for _ in range ( 20_000_000 ): count += 1 return count def run_tasks_single_thread ( task , num_iterations ): start_time = time . time () for _ in range ( num_iterations ): task () end_time = time . time () return end_time - start_time def run_tasks_multi_thread ( task , num_threads ): start_time = time . time () threads : List [ threading . Thread ] = [] for _ in range ( num_threads ): t = threading . Thread ( target = task ) threads . append ( t ) t . start () for t in threads : t . join () end_time = time . time () return end_time - start_time # NOTE: CPU-bound tasks num_tasks = 4 print ( "CPU-bound task comparison:" ) single_thread_cpu = run_tasks_single_thread ( cpu_bound_task , num_tasks ) print ( f "Single-threaded time: { single_thread_cpu : .2f } seconds" ) multi_thread_cpu = run_tasks_multi_thread ( cpu_bound_task , num_tasks ) print ( f "Multi-threaded time: { multi_thread_cpu : .2f } seconds" ) print ( f "Speed difference: { single_thread_cpu / multi_thread_cpu : .2f } x \n " ) CPU-bound task comparison: Single-threaded time: 1.71 seconds Multi-threaded time: 1.62 seconds Speed difference: 1.06x Next, we compare the performance of single-threaded vs multi-threaded vs multi-process. We note that multi-process is the fastest and this is expected because there is no GIL in multi-process. import multiprocessing def run_tasks_multi_process ( task , num_processes ): """Run tasks using multiple processes""" start_time = time . time () processes : List [ multiprocessing . Process ] = [] for _ in range ( num_processes ): p = multiprocessing . Process ( target = task ) processes . append ( p ) p . start () for p in processes : p . join () end_time = time . time () return end_time - start_time # Comparison of all three approaches num_tasks = 4 print ( "CPU-bound task comparison:" ) single_thread_cpu = run_tasks_single_thread ( cpu_bound_task , num_tasks ) print ( f "Single-threaded time: { single_thread_cpu : .2f } seconds" ) multi_thread_cpu = run_tasks_multi_thread ( cpu_bound_task , num_tasks ) print ( f "Multi-threaded time: { multi_thread_cpu : .2f } seconds" ) multi_process_cpu = run_tasks_multi_process ( cpu_bound_task , num_tasks ) print ( f "Multi-process time: { multi_process_cpu : .2f } seconds" ) print ( f " \n Speed comparison (relative to single-thread):" ) print ( f "Threading speedup: { single_thread_cpu / multi_thread_cpu : .2f } x" ) print ( f "Multiprocessing speedup: { single_thread_cpu / multi_process_cpu : .2f } x \n " ) CPU-bound task comparison: Single-threaded time: 1.73 seconds Multi-threaded time: 1.58 seconds Multi-process time: 0.09 seconds Speed comparison (relative to single-thread): Threading speedup: 1.10x Multiprocessing speedup: 18.59x Traceback (most recent call last): File "<string>", line 1, in <module> File "/opt/homebrew/Caskroom/miniconda/base/envs/omniverse/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main Traceback (most recent call last): File "<string>", line 1, in <module> File "/opt/homebrew/Caskroom/miniconda/base/envs/omniverse/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main Traceback (most recent call last): File "<string>", line 1, in <module> File "/opt/homebrew/Caskroom/miniconda/base/envs/omniverse/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main exitcode = _main(fd, parent_sentinel) File "/opt/homebrew/Caskroom/miniconda/base/envs/omniverse/lib/python3.9/multiprocessing/spawn.py", line 126, in _main exitcode = _main(fd, parent_sentinel)exitcode = _main(fd, parent_sentinel) File "/opt/homebrew/Caskroom/miniconda/base/envs/omniverse/lib/python3.9/multiprocessing/spawn.py", line 126, in _main File "/opt/homebrew/Caskroom/miniconda/base/envs/omniverse/lib/python3.9/multiprocessing/spawn.py", line 126, in _main self = reduction.pickle.load(from_parent) AttributeError: Can't get attribute 'cpu_bound_task' on <module '__main__' (built-in)> self = reduction.pickle.load(from_parent) AttributeError: Can't get attribute 'cpu_bound_task' on <module '__main__' (built-in)> self = reduction.pickle.load(from_parent) AttributeError: Can't get attribute 'cpu_bound_task' on <module '__main__' (built-in)> Traceback (most recent call last): File "<string>", line 1, in <module> File "/opt/homebrew/Caskroom/miniconda/base/envs/omniverse/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main exitcode = _main(fd, parent_sentinel) File "/opt/homebrew/Caskroom/miniconda/base/envs/omniverse/lib/python3.9/multiprocessing/spawn.py", line 126, in _main self = reduction.pickle.load(from_parent) AttributeError: Can't get attribute 'cpu_bound_task' on <module '__main__' (built-in)> GIL Is Released During IO-bound Tasks As mentioned, the GIL is released during IO-bound tasks. As to why, please see this post . Let’s see an example of this. def io_bound_task (): """IO intensive task - making HTTP requests""" url = "https://api.github.com" response = requests . get ( url ) return response . status_code # NOTE: IO-bound tasks print ( "IO-bound task comparison:" ) single_thread_io = run_tasks_single_thread ( io_bound_task , num_tasks ) print ( f "Single-threaded time: { single_thread_io : .2f } seconds" ) multi_thread_io = run_tasks_multi_thread ( io_bound_task , num_tasks ) print ( f "Multi-threaded time: { multi_thread_io : .2f } seconds" ) print ( f "Speed difference: { single_thread_io / multi_thread_io : .2f } x" ) 2024-10-29 20:14:18,709 - DEBUG - Starting new HTTPS connection (1): api.github.com:443 IO-bound task comparison: 2024-10-29 20:14:19,014 - DEBUG - https://api.github.com:443 "GET / HTTP/1.1" 200 510 2024-10-29 20:14:19,021 - DEBUG - Starting new HTTPS connection (1): api.github.com:443 2024-10-29 20:14:19,072 - DEBUG - https://api.github.com:443 "GET / HTTP/1.1" 200 510 2024-10-29 20:14:19,075 - DEBUG - Starting new HTTPS connection (1): api.github.com:443 2024-10-29 20:14:19,122 - DEBUG - https://api.github.com:443 "GET / HTTP/1.1" 200 510 2024-10-29 20:14:19,126 - DEBUG - Starting new HTTPS connection (1): api.github.com:443 2024-10-29 20:14:19,167 - DEBUG - https://api.github.com:443 "GET / HTTP/1.1" 200 510 2024-10-29 20:14:19,174 - DEBUG - Starting new HTTPS connection (1): api.github.com:443 2024-10-29 20:14:19,174 - DEBUG - Starting new HTTPS connection (1): api.github.com:443 2024-10-29 20:14:19,174 - DEBUG - Starting new HTTPS connection (1): api.github.com:443 2024-10-29 20:14:19,174 - DEBUG - Starting new HTTPS connection (1): api.github.com:443 2024-10-29 20:14:19,256 - DEBUG - https://api.github.com:443 "GET / HTTP/1.1" 200 510 2024-10-29 20:14:19,259 - DEBUG - https://api.github.com:443 "GET / HTTP/1.1" 200 510 2024-10-29 20:14:19,261 - DEBUG - https://api.github.com:443 "GET / HTTP/1.1" 200 510 2024-10-29 20:14:19,261 - DEBUG - https://api.github.com:443 "GET / HTTP/1.1" 200 510 Single-threaded time: 0.50 seconds Multi-threaded time: 0.09 seconds Speed difference: 5.24x Indeed, the multi-threaded version is faster than the single-threaded version. References and Further Readings
Markdown	[Skip to main content](https://www.gaohongnan.com/software_engineering/python/gil.html#main-content) Back to top [![Omniverse - Home](https://www.gaohongnan.com/_static/logo.png)](https://www.gaohongnan.com/intro.html) - [Omniverse](https://www.gaohongnan.com/intro.html) Notations - [Machine Learning Notations](https://www.gaohongnan.com/notations/machine_learning.html) Influential Ideas and Papers - [Generative Pre-trained Transformers](https://www.gaohongnan.com/influential/generative_pretrained_transformer/01_intro.html) - [Notations](https://www.gaohongnan.com/influential/generative_pretrained_transformer/02_notations.html) - [The Concept of Generative Pre-trained Transformers (GPT)](https://www.gaohongnan.com/influential/generative_pretrained_transformer/03_concept.html) - [The Implementation of Generative Pre-trained Transformers (GPT)](https://www.gaohongnan.com/influential/generative_pretrained_transformer/04_implementation.html) - [Training a Mini-GPT to Learn Two-Digit Addition](https://www.gaohongnan.com/influential/generative_pretrained_transformer/05_adder.html) - [Low-Rank Adaptation Of Large Language Models](https://www.gaohongnan.com/influential/low_rank_adaptation/01_intro.html) - [Concept](https://www.gaohongnan.com/influential/low_rank_adaptation/02_concept.html) - [Implementation](https://www.gaohongnan.com/influential/low_rank_adaptation/03_implementation.html) - [Chain-of-Verification Reduces Hallucination in Large Language Models](https://www.gaohongnan.com/influential/cove/cove.html) - [Empirical Risk Minimization](https://www.gaohongnan.com/influential/empirical_risk_minimization/01_intro.html) - [Concept: Empirical Risk Minimization](https://www.gaohongnan.com/influential/empirical_risk_minimization/02_concept.html) - [Bayes Optimal Classifier](https://www.gaohongnan.com/influential/empirical_risk_minimization/03_bayes_optimal_classifier.html) - [Is The Learning Problem Solvable?](https://www.gaohongnan.com/influential/learning_theory/01_intro.html) - [Concept: Learning Theory](https://www.gaohongnan.com/influential/learning_theory/02_concept.html) - [Lloyd’s K-Means Clustering Algorithm](https://www.gaohongnan.com/influential/kmeans_clustering/01_intro.html) - [Concept: K-Means Clustering](https://www.gaohongnan.com/influential/kmeans_clustering/02_concept.html) - [Implementation: K-Means (Lloyd)](https://www.gaohongnan.com/influential/kmeans_clustering/03_implementation.html) - [Application: Image Compression and Segmentation](https://www.gaohongnan.com/influential/kmeans_clustering/04_image_segmentation.html) - [Conceptual Questions](https://www.gaohongnan.com/influential/kmeans_clustering/05_conceptual_questions.html) - [Naive Bayes](https://www.gaohongnan.com/influential/naive_bayes/01_intro.html) - [Concept](https://www.gaohongnan.com/influential/naive_bayes/02_concept.html) - [Naives Bayes Implementation](https://www.gaohongnan.com/influential/naive_bayes/03_implementation.html) - [Naive Bayes Application: Penguins](https://www.gaohongnan.com/influential/naive_bayes/04_example_penguins.html) - [Naive Bayes Application (MNIST)](https://www.gaohongnan.com/influential/naive_bayes/05_application_mnist.html) - [Mixture Models](https://www.gaohongnan.com/influential/gaussian_mixture_models/01_intro.html) - [Concept](https://www.gaohongnan.com/influential/gaussian_mixture_models/02_concept.html) - [Gaussian Mixture Models Implementation](https://www.gaohongnan.com/influential/gaussian_mixture_models/03_implementation.html) - [Linear Regression](https://www.gaohongnan.com/influential/linear_regression/01_intro.html) - [Concept](https://www.gaohongnan.com/influential/linear_regression/02_concept.html) - [Implementation](https://www.gaohongnan.com/influential/linear_regression/03_implementation.html) Playbook - [Training Dynamics And Tricks](https://www.gaohongnan.com/playbook/training/intro.html) - [How to Calculate the Number of FLOPs in Transformer Based Models?](https://www.gaohongnan.com/playbook/training/how_to_calculate_flops_in_transformer_based_models.html) - [Why Does Cosine Annealing With Warmup Stabilize Training?](https://www.gaohongnan.com/playbook/training/why_cosine_annealing_warmup_stabilize_training.html) - [How To Fine-Tune Decoder-Only Models For Sequence Classification Using Last Token Pooling?](https://www.gaohongnan.com/playbook/training/how_to_finetune_decoder_with_last_token_pooling.html) - [How To Fine-Tune Decoder-Only Models For Sequence Classification With Cross-Attention?](https://www.gaohongnan.com/playbook/training/how_to_finetune_decoder_with_cross_attention.html) - [How To Do Teacher-Student Knowledge Distillation?](https://www.gaohongnan.com/playbook/training/how_to_teacher_student_knowledge_distillation.html) - [Softmax Preserves Order, Is Translation Invariant But Not Invariant Under Scaling.](https://www.gaohongnan.com/playbook/why_softmax_preserves_order_translation_invariant_not_invariant_scaling.html) - [How to Inspect Function and Class Signatures in Python?](https://www.gaohongnan.com/playbook/how_to_inspect_function_and_class_signatures.html) Probability Theory - [Chapter 1. Mathematical Preliminaries](https://www.gaohongnan.com/probability_theory/01_mathematical_preliminaries/intro.html) - [Permutations and Combinations](https://www.gaohongnan.com/probability_theory/01_mathematical_preliminaries/01_combinatorics.html) - [Calculus](https://www.gaohongnan.com/probability_theory/01_mathematical_preliminaries/02_calculus.html) - [Contour Maps](https://www.gaohongnan.com/probability_theory/01_mathematical_preliminaries/03_contours.html) - [Exercises](https://www.gaohongnan.com/probability_theory/01_mathematical_preliminaries/exercises.html) - [Chapter 2. Probability](https://www.gaohongnan.com/probability_theory/02_probability/intro.html) - [Probability Space](https://www.gaohongnan.com/probability_theory/02_probability/0202_probability_space.html) - [Probability Axioms](https://www.gaohongnan.com/probability_theory/02_probability/0203_probability_axioms.html) - [Conditional Probability](https://www.gaohongnan.com/probability_theory/02_probability/0204_conditional_probability.html) - [Independence](https://www.gaohongnan.com/probability_theory/02_probability/0205_independence.html) - [Baye’s Theorem and the Law of Total Probability](https://www.gaohongnan.com/probability_theory/02_probability/0206_bayes_theorem.html) - [Summary](https://www.gaohongnan.com/probability_theory/02_probability/summary.html) - [Chapter 3. Discrete Random Variables](https://www.gaohongnan.com/probability_theory/03_discrete_random_variables/intro.html) - [Random Variables](https://www.gaohongnan.com/probability_theory/03_discrete_random_variables/0301_random_variables.html) - [Discrete Random Variables](https://www.gaohongnan.com/probability_theory/03_discrete_random_variables/0302_discrete_random_variables.html) - [Probability Mass Function](https://www.gaohongnan.com/probability_theory/03_discrete_random_variables/0303_probability_mass_function.html) - [Cumulative Distribution Function](https://www.gaohongnan.com/probability_theory/03_discrete_random_variables/0304_cumulative_distribution_function.html) - [Expectation](https://www.gaohongnan.com/probability_theory/03_discrete_random_variables/0305_expectation.html) - [Moments and Variance](https://www.gaohongnan.com/probability_theory/03_discrete_random_variables/0306_moments_and_variance.html) - [Discrete Uniform Distribution](https://www.gaohongnan.com/probability_theory/03_discrete_random_variables/uniform/intro.html) - [Concept](https://www.gaohongnan.com/probability_theory/03_discrete_random_variables/uniform/0307_discrete_uniform_distribution_concept.html) - [Application](https://www.gaohongnan.com/probability_theory/03_discrete_random_variables/uniform/0307_discrete_uniform_distribution_application.html) - [Bernoulli Distribution](https://www.gaohongnan.com/probability_theory/03_discrete_random_variables/bernoulli/intro.html) - [Concept](https://www.gaohongnan.com/probability_theory/03_discrete_random_variables/bernoulli/0308_bernoulli_distribution_concept.html) - [Application](https://www.gaohongnan.com/probability_theory/03_discrete_random_variables/bernoulli/0308_bernoulli_distribution_application.html) - [Independent and Identically Distributed (IID)](https://www.gaohongnan.com/probability_theory/03_discrete_random_variables/iid.html) - [Binomial Distribution](https://www.gaohongnan.com/probability_theory/03_discrete_random_variables/binomial/intro.html) - [Concept](https://www.gaohongnan.com/probability_theory/03_discrete_random_variables/binomial/0309_binomial_distribution_concept.html) - [Implementation](https://www.gaohongnan.com/probability_theory/03_discrete_random_variables/binomial/0309_binomial_distribution_implementation.html) - [Real World Examples](https://www.gaohongnan.com/probability_theory/03_discrete_random_variables/binomial/0309_binomial_distribution_application.html) - [Geometric Distribution](https://www.gaohongnan.com/probability_theory/03_discrete_random_variables/geometric/intro.html) - [Concept](https://www.gaohongnan.com/probability_theory/03_discrete_random_variables/geometric/0310_geometric_distribution_concept.html) - [Poisson Distribution](https://www.gaohongnan.com/probability_theory/03_discrete_random_variables/poisson/intro.html) - [Concept](https://www.gaohongnan.com/probability_theory/03_discrete_random_variables/poisson/0311_poisson_distribution_concept.html) - [Implementation](https://www.gaohongnan.com/probability_theory/03_discrete_random_variables/poisson/0311_poisson_distribution_implementation.html) - [Important](https://www.gaohongnan.com/probability_theory/03_discrete_random_variables/summary.html) - [Exercises](https://www.gaohongnan.com/probability_theory/03_discrete_random_variables/exercises.html) - [Chapter 4. Continuous Random Variables](https://www.gaohongnan.com/probability_theory/04_continuous_random_variables/intro.html) - [From Discrete to Continuous](https://www.gaohongnan.com/probability_theory/04_continuous_random_variables/from_discrete_to_continuous.html) - [Continuous Random Variables](https://www.gaohongnan.com/probability_theory/04_continuous_random_variables/0401_continuous_random_variables.html) - [Probability Density Function](https://www.gaohongnan.com/probability_theory/04_continuous_random_variables/0402_probability_density_function.html) - [Expectation](https://www.gaohongnan.com/probability_theory/04_continuous_random_variables/0403_expectation.html) - [Moments and Variance](https://www.gaohongnan.com/probability_theory/04_continuous_random_variables/0404_moments_and_variance.html) - [Cumulative Distribution Function](https://www.gaohongnan.com/probability_theory/04_continuous_random_variables/0405_cumulative_distribution_function.html) - [Mean, Median and Mode](https://www.gaohongnan.com/probability_theory/04_continuous_random_variables/0406_mean_median_mode.html) - [Continuous Uniform Distribution](https://www.gaohongnan.com/probability_theory/04_continuous_random_variables/0407_continuous_uniform_distribution.html) - [Exponential Distribution](https://www.gaohongnan.com/probability_theory/04_continuous_random_variables/0408_exponential_distribution.html) - [Gaussian Distribution](https://www.gaohongnan.com/probability_theory/04_continuous_random_variables/0409_gaussian_distribution.html) - [Skewness and Kurtosis](https://www.gaohongnan.com/probability_theory/04_continuous_random_variables/0410_skewness_and_kurtosis.html) - [Convolution and Sum of Random Variables](https://www.gaohongnan.com/probability_theory/04_continuous_random_variables/0411_convolve_and_sum_of_random_variables.html) - [Functions of Random Variables](https://www.gaohongnan.com/probability_theory/04_continuous_random_variables/0412_functions_of_random_variables.html) - [Chapter 5. Joint Distributions](https://www.gaohongnan.com/probability_theory/05_joint_distributions/intro.html) - [From Single Variable to Joint Distributions](https://www.gaohongnan.com/probability_theory/05_joint_distributions/from_single_variable_to_joint_distributions.html) - [Joint PMF and PDF](https://www.gaohongnan.com/probability_theory/05_joint_distributions/0501_joint_pmf_pdf/intro.html) - [Concept](https://www.gaohongnan.com/probability_theory/05_joint_distributions/0501_joint_pmf_pdf/concept.html) - [Joint Expectation and Correlation](https://www.gaohongnan.com/probability_theory/05_joint_distributions/0502_joint_expectation_and_correlation/intro.html) - [Concept](https://www.gaohongnan.com/probability_theory/05_joint_distributions/0502_joint_expectation_and_correlation/concept.html) - [Conditional PMF and PDF](https://www.gaohongnan.com/probability_theory/05_joint_distributions/0503_conditional_pmf_pdf/intro.html) - [Concept](https://www.gaohongnan.com/probability_theory/05_joint_distributions/0503_conditional_pmf_pdf/concept.html) - [Application](https://www.gaohongnan.com/probability_theory/05_joint_distributions/0503_conditional_pmf_pdf/application.html) - [Conditional Expectation and Variance](https://www.gaohongnan.com/probability_theory/05_joint_distributions/0504_conditional_expectation_variance/intro.html) - [Concept](https://www.gaohongnan.com/probability_theory/05_joint_distributions/0504_conditional_expectation_variance/concept.html) - [Exercises](https://www.gaohongnan.com/probability_theory/05_joint_distributions/0504_conditional_expectation_variance/exercises.html) - [Sum of Random Variables](https://www.gaohongnan.com/probability_theory/05_joint_distributions/0505_sum_of_random_variables/intro.html) - [Concept](https://www.gaohongnan.com/probability_theory/05_joint_distributions/0505_sum_of_random_variables/concept.html) - [Random Vectors](https://www.gaohongnan.com/probability_theory/05_joint_distributions/0506_random_vectors/intro.html) - [Concept](https://www.gaohongnan.com/probability_theory/05_joint_distributions/0506_random_vectors/concept.html) - [Multivariate Gaussian Distribution](https://www.gaohongnan.com/probability_theory/05_joint_distributions/0507_multivariate_gaussian/intro.html) - [Concept](https://www.gaohongnan.com/probability_theory/05_joint_distributions/0507_multivariate_gaussian/concept.html) - [Application: Plots and Transformations](https://www.gaohongnan.com/probability_theory/05_joint_distributions/0507_multivariate_gaussian/application_transformation.html) - [Covariance Matrix is Positive Semi-Definite](https://www.gaohongnan.com/probability_theory/05_joint_distributions/0507_multivariate_gaussian/psd.html) - [Eigendecomposition and Covariance Matrix](https://www.gaohongnan.com/probability_theory/05_joint_distributions/0507_multivariate_gaussian/eigendecomposition.html) - [The Geometry of Multivariate Gaussians](https://www.gaohongnan.com/probability_theory/05_joint_distributions/0507_multivariate_gaussian/geometry_of_multivariate_gaussian.html) - [Chapter 6. Sample Statistics](https://www.gaohongnan.com/probability_theory/06_sample_statistics/intro.html) - [Moment Generating and Characteristic Functions](https://www.gaohongnan.com/probability_theory/06_sample_statistics/0601_moment_generating_and_characteristic_functions/intro.html) - [Moment Generating Function](https://www.gaohongnan.com/probability_theory/06_sample_statistics/0601_moment_generating_and_characteristic_functions/moment_generating_function.html) - [Application: Moment Generating Function and the Sum of Random Variables](https://www.gaohongnan.com/probability_theory/06_sample_statistics/0601_moment_generating_and_characteristic_functions/moment_generating_function_application_sum_of_rv.html) - [Characteristic Function](https://www.gaohongnan.com/probability_theory/06_sample_statistics/0601_moment_generating_and_characteristic_functions/characteristic_function.html) - [Probability Inequalities](https://www.gaohongnan.com/probability_theory/06_sample_statistics/0602_probability_inequalities/intro.html) - [Probability Inequalities](https://www.gaohongnan.com/probability_theory/06_sample_statistics/0602_probability_inequalities/concept.html) - [Application: Learning Theory](https://www.gaohongnan.com/probability_theory/06_sample_statistics/0602_probability_inequalities/application.html) - [Law of Large Numbers](https://www.gaohongnan.com/probability_theory/06_sample_statistics/0603_law_of_large_numbers/intro.html) - [Concept](https://www.gaohongnan.com/probability_theory/06_sample_statistics/0603_law_of_large_numbers/concept.html) - [Convergence of Sample Average](https://www.gaohongnan.com/probability_theory/06_sample_statistics/0603_law_of_large_numbers/convergence.html) - [Application: Learning Theory](https://www.gaohongnan.com/probability_theory/06_sample_statistics/0603_law_of_large_numbers/application.html) - [Chapter 8. Estimation Theory](https://www.gaohongnan.com/probability_theory/08_estimation_theory/intro.html) - [Maximum Likelihood Estimation](https://www.gaohongnan.com/probability_theory/08_estimation_theory/maximum_likelihood_estimation/intro.html) - [Concept](https://www.gaohongnan.com/probability_theory/08_estimation_theory/maximum_likelihood_estimation/concept.html) Operations - [Distributed Systems](https://www.gaohongnan.com/operations/distributed/intro.html) - [Notations](https://www.gaohongnan.com/operations/distributed/01_notations.html) - [Basics Of Distributed Data Parallelism](https://www.gaohongnan.com/operations/distributed/02_basics.html) - [How to Setup SLURM and ParallelCluster in AWS](https://www.gaohongnan.com/operations/distributed/03_how_to_setup_slurm_in_aws.html) - [Ablations](https://www.gaohongnan.com/operations/distributed/04_ablation.html) - [Profiling](https://www.gaohongnan.com/operations/profiling/intro.html) - [Synchronize CUDA To Time CUDA Operations](https://www.gaohongnan.com/operations/profiling/01_synchronize.html) - [Profiling Code With Timeit](https://www.gaohongnan.com/operations/profiling/02_timeit.html) - [PyTorch’s Event And Profiler](https://www.gaohongnan.com/operations/profiling/03_time_profiler.html) - [Profile GPT Small Time And Memory](https://www.gaohongnan.com/operations/profiling/04_small_gpt_profile.html) - [CUDA Memory Allocations](https://www.gaohongnan.com/operations/profiling/05_memory_leak.html) - [The Lifecycle of an AIOps System](https://www.gaohongnan.com/operations/machine_learning_lifecycle/00_intro.html) - [Stage 1. Problem Formulation](https://www.gaohongnan.com/operations/machine_learning_lifecycle/01_problem_formulation.html) - [Stage 2. Project Scoping And Framing The Problem](https://www.gaohongnan.com/operations/machine_learning_lifecycle/02_project_scoping.html) - [Stage 3. Data Pipeline (Data Engineering and DataOps)](https://www.gaohongnan.com/operations/machine_learning_lifecycle/03_dataops_pipeline/03_dataops_pipeline.html) - [Stage 3.1. Data Source and Formats](https://www.gaohongnan.com/operations/machine_learning_lifecycle/03_dataops_pipeline/031_data_source_and_format.html) - [Stage 3.2. Data Model and Storage](https://www.gaohongnan.com/operations/machine_learning_lifecycle/03_dataops_pipeline/032_data_model_and_storage.html) - [Stage 3.3. Extract, Transform, Load (ETL)](https://www.gaohongnan.com/operations/machine_learning_lifecycle/03_dataops_pipeline/033_etl.html) - [Stage 4. Data Extraction (MLOps), Data Analysis (Data Science), Data Preparation (Data Science)](https://www.gaohongnan.com/operations/machine_learning_lifecycle/04_mlops_data_pipeline.html) - [Stage 5. Model Development and Training (MLOps)](https://www.gaohongnan.com/operations/machine_learning_lifecycle/05_model_development_selection_and_training/05_ml_training_pipeline.html) - [Stage 5.1. Model Selection](https://www.gaohongnan.com/operations/machine_learning_lifecycle/05_model_development_selection_and_training/051_model_selection.html) - [Stage 5.2. Metric Selection](https://www.gaohongnan.com/operations/machine_learning_lifecycle/05_model_development_selection_and_training/052_metric_selection.html) - [Stage 5.3. Experiment Tracking And Versioning](https://www.gaohongnan.com/operations/machine_learning_lifecycle/05_model_development_selection_and_training/053_experiment_tracking.html) - [Stage 5.4. Model Testing](https://www.gaohongnan.com/operations/machine_learning_lifecycle/05_model_development_selection_and_training/054_model_testing.html) - [Stage 6. Model Evaluation (MLOps)](https://www.gaohongnan.com/operations/machine_learning_lifecycle/06_model_evaluation.html) - [Stage 7. Model Validation, Registry and Pushing Model to Production (MLOps)](https://www.gaohongnan.com/operations/machine_learning_lifecycle/07_model_validation_registry_and_pushing_model_to_production.html) - [Stage 8. Model Serving (MLOps)](https://www.gaohongnan.com/operations/machine_learning_lifecycle/08_model_deployment_and_serving.html) - [Stage 9. Model Monitoring (MLOps)](https://www.gaohongnan.com/operations/machine_learning_lifecycle/09_model_monitoring.html) - [Stage 10. Continuous Integration, Deployment, Learning and Training (DevOps, DataOps, MLOps)](https://www.gaohongnan.com/operations/machine_learning_lifecycle/010_continuous_integration_deployment_learning_and_training.html) Software Engineering - [Config, State, Metadata Management](https://www.gaohongnan.com/software_engineering/config_management/intro.html) - [Configuration Management](https://www.gaohongnan.com/software_engineering/config_management/concept.html) - [Pydantic And Hydra](https://www.gaohongnan.com/software_engineering/config_management/01-pydra.html) - [State And Metadata Management](https://www.gaohongnan.com/software_engineering/config_management/02-state.html) - [Design Patterns](https://www.gaohongnan.com/software_engineering/design_patterns/intro.html) - [Dependency Inversion Principle](https://www.gaohongnan.com/software_engineering/design_patterns/dependency_inversion_principle.html) - [Named Constructor](https://www.gaohongnan.com/software_engineering/design_patterns/named_constructor.html) - [Strategy](https://www.gaohongnan.com/software_engineering/design_patterns/strategy.html) - [Registry](https://www.gaohongnan.com/software_engineering/design_patterns/registry.html) - [Context Object Pattern (God Object)](https://www.gaohongnan.com/software_engineering/design_patterns/god_object_pattern.html) - [Factory Method](https://www.gaohongnan.com/software_engineering/design_patterns/factory_method.html) - [Singleton](https://www.gaohongnan.com/software_engineering/design_patterns/singleton.html) - [Python](https://www.gaohongnan.com/software_engineering/python/intro.html) - [Init vs New](https://www.gaohongnan.com/software_engineering/python/new_vs_init.html) - [Global Interpreter Lock (GIL)](https://www.gaohongnan.com/software_engineering/python/gil.html) - [The Iterator Protocol](https://www.gaohongnan.com/software_engineering/python/iterator_protocol.html) - [Decorator](https://www.gaohongnan.com/software_engineering/python/decorator.html) - [Generators Over Lists For Memory Efficiency](https://www.gaohongnan.com/software_engineering/python/generators_over_lists.html) - [Pydantic Is All You Need - Jason Liu](https://www.gaohongnan.com/software_engineering/python/pydantic.html) - [Do Not Use Mutable Default Arguments](https://www.gaohongnan.com/software_engineering/python/mutable_default.html) - [Set Over List For Frequent Membership Tests](https://www.gaohongnan.com/software_engineering/python/set_vs_list.html) - [Late Binding Closures](https://www.gaohongnan.com/software_engineering/python/late_binding_closures.html) - [Is vs Equality](https://www.gaohongnan.com/software_engineering/python/is_vs_equality.html) - [Concurrency, Parallelism and Asynchronous Programming](https://www.gaohongnan.com/software_engineering/concurrency_parallelism_asynchronous/intro.html) - [Overview Of Concurrency, Parallelism, and Asynchronous Execution](https://www.gaohongnan.com/software_engineering/concurrency_parallelism_asynchronous/overview.html) - [Thread Safety](https://www.gaohongnan.com/software_engineering/concurrency_parallelism_asynchronous/insights/locks_for_thread_safety.html) - [A Rudimentary Introduction to Generator and Yield in Python](https://www.gaohongnan.com/software_engineering/concurrency_parallelism_asynchronous/generator_yield.html) Computer Science - [Type Theory, A Very Rudimentary Introduction](https://www.gaohongnan.com/computer_science/type_theory/intro.html) - [Subtypes](https://www.gaohongnan.com/computer_science/type_theory/01-subtypes.html) - [Type Safety](https://www.gaohongnan.com/computer_science/type_theory/02-type-safety.html) - [Subsumption](https://www.gaohongnan.com/computer_science/type_theory/03-subsumption.html) - [Generics and Type Variables](https://www.gaohongnan.com/computer_science/type_theory/04-generics.html) - [Bound and Constraint in Generics and Type Variables](https://www.gaohongnan.com/computer_science/type_theory/05-typevar-bound-constraints.html) - [Invariance, Covariance and Contravariance](https://www.gaohongnan.com/computer_science/type_theory/06-invariance-covariance-contravariance.html) - [Function Overloading](https://www.gaohongnan.com/computer_science/type_theory/07-pep-3124-overloading.html) - [Sentinel Types](https://www.gaohongnan.com/computer_science/type_theory/08-pep-661-sentinel-values.html) Data Structures and Algorithms - [Complexity Analysis](https://www.gaohongnan.com/dsa/complexity_analysis/intro.html) - [Master Theorem](https://www.gaohongnan.com/dsa/complexity_analysis/master_theorem.html) - [List/Array](https://www.gaohongnan.com/dsa/array/intro.html) - [Concept](https://www.gaohongnan.com/dsa/array/concept.html) - [Questions](https://www.gaohongnan.com/dsa/array/questions/intro.html) - [Two Sum](https://www.gaohongnan.com/dsa/array/questions/01-two-sum.html) - [Hash Map](https://www.gaohongnan.com/dsa/hash_map/intro.html) - [Concept](https://www.gaohongnan.com/dsa/hash_map/concept.html) - [Questions](https://www.gaohongnan.com/dsa/hash_map/questions/intro.html) - [Two Sum](https://www.gaohongnan.com/dsa/hash_map/questions/01-two-sum.html) - [Group Anagrams](https://www.gaohongnan.com/dsa/hash_map/questions/49-group-anagrams.html) - [Two Pointers And Sliding Window](https://www.gaohongnan.com/dsa/two_pointers/intro.html) - [Two Pointers](https://www.gaohongnan.com/dsa/two_pointers/two_pointers.html) - [Sliding Window](https://www.gaohongnan.com/dsa/two_pointers/sliding_window.html) - [Questions](https://www.gaohongnan.com/dsa/two_pointers/questions/intro.html) - [Two Pointers](https://www.gaohongnan.com/dsa/two_pointers/questions/two_pointers/intro.html) - [Remove Duplicates from Sorted Array](https://www.gaohongnan.com/dsa/two_pointers/questions/two_pointers/26-remove-duplicates-from-sorted-array.html) - [Two Sum II - Input Array Is Sorted](https://www.gaohongnan.com/dsa/two_pointers/questions/two_pointers/167-two-sum-ii-input-array-is-sorted.html) - [Sliding Window](https://www.gaohongnan.com/dsa/two_pointers/questions/sliding_window/intro.html) - [Find All Anagrams in a String](https://www.gaohongnan.com/dsa/two_pointers/questions/sliding_window/438-find-all-anagrams-in-a-string.html) - [Stack](https://www.gaohongnan.com/dsa/stack/intro.html) - [Concept](https://www.gaohongnan.com/dsa/stack/concept.html) - [Questions](https://www.gaohongnan.com/dsa/stack/questions/intro.html) - [Valid Parentheses](https://www.gaohongnan.com/dsa/stack/questions/20-valid-parentheses.html) - [Min Stack](https://www.gaohongnan.com/dsa/stack/questions/155-min-stack.html) - [Implement Queue using Stacks](https://www.gaohongnan.com/dsa/stack/questions/232-implement-queue-using-stacks.html) - [Reverse String](https://www.gaohongnan.com/dsa/stack/questions/344-reverse-string.html) - [Queue](https://www.gaohongnan.com/dsa/queue/intro.html) - [Concept](https://www.gaohongnan.com/dsa/queue/concept.html) - [Double Ended Queue](https://www.gaohongnan.com/dsa/queue/dequeue.html) - [Easy - Hot Potatoes](https://www.gaohongnan.com/dsa/queue/questions/hot-potatoes.html) - [Palindrome Checker](https://www.gaohongnan.com/dsa/queue/questions/125-valid-palindrome.html) - [Linear Search](https://www.gaohongnan.com/dsa/searching_algorithms/linear_search/intro.html) - [Concept](https://www.gaohongnan.com/dsa/searching_algorithms/linear_search/concept.html) - [Binary Search](https://www.gaohongnan.com/dsa/searching_algorithms/binary_search/intro.html) - [Concept](https://www.gaohongnan.com/dsa/searching_algorithms/binary_search/concept.html) - [Koko Eating Bananas](https://www.gaohongnan.com/dsa/searching_algorithms/binary_search/problems/875-koko-eating-bananas.html) Linear Algebra - [Preliminaries](https://www.gaohongnan.com/linear_algebra/01_preliminaries/intro.html) - [Fields](https://www.gaohongnan.com/linear_algebra/01_preliminaries/01-fields.html) - [Systems of Linear Equations](https://www.gaohongnan.com/linear_algebra/01_preliminaries/02-systems-of-linear-equations.html) - [Vectors](https://www.gaohongnan.com/linear_algebra/02_vectors/intro.html) - [Vector and Its Definition](https://www.gaohongnan.com/linear_algebra/02_vectors/01-vector-definition.html) - [Vector and Its Operations](https://www.gaohongnan.com/linear_algebra/02_vectors/02-vector-operation.html) - [Vector Norm and Distance](https://www.gaohongnan.com/linear_algebra/02_vectors/03-vector-norm.html) - [A First Look at Vector Products](https://www.gaohongnan.com/linear_algebra/02_vectors/04-vector-products.html) References, Resources and Roadmap - [Bibliography](https://www.gaohongnan.com/bibliography.html) - [IEEE (Style) Citations](https://www.gaohongnan.com/citations.html) - [![Colab logo](https://www.gaohongnan.com/_static/images/logo_colab.png) Colab](https://colab.research.google.com/github/gao-hongnan/omniverse/blob/main/omniverse/software_engineering/python/gil.ipynb "Launch on Colab") - [Repository](https://github.com/gao-hongnan/omniverse "Source repository") - [Open issue](https://github.com/gao-hongnan/omniverse/issues/new?title=Issue%20on%20page%20%2Fsoftware_engineering/python/gil.html&body=Your%20issue%20content%20here. "Open an issue") - [.ipynb](https://www.gaohongnan.com/_sources/software_engineering/python/gil.ipynb "Download source file") - .pdf # Global Interpreter Lock (GIL) ## Contents - [Reference Counting in Python](https://www.gaohongnan.com/software_engineering/python/gil.html#reference-counting-in-python) - [Threads and Race Conditions](https://www.gaohongnan.com/software_engineering/python/gil.html#threads-and-race-conditions) - [Protecting Reference Counts with Locks](https://www.gaohongnan.com/software_engineering/python/gil.html#protecting-reference-counts-with-locks) - [Deadlocks](https://www.gaohongnan.com/software_engineering/python/gil.html#deadlocks) - [The Global Interpreter Lock (GIL) in Python](https://www.gaohongnan.com/software_engineering/python/gil.html#the-global-interpreter-lock-gil-in-python) - [CPU-bound tasks](https://www.gaohongnan.com/software_engineering/python/gil.html#cpu-bound-tasks) - [GIL Is Released During IO-bound Tasks](https://www.gaohongnan.com/software_engineering/python/gil.html#gil-is-released-during-io-bound-tasks) - [References and Further Readings](https://www.gaohongnan.com/software_engineering/python/gil.html#references-and-further-readings) # Global Interpreter Lock (GIL)[\#](https://www.gaohongnan.com/software_engineering/python/gil.html#global-interpreter-lock-gil "Link to this heading") [![Twitter Handle](https://img.shields.io/badge/Twitter-@gaohongnan-blue?style=social&logo=twitter)](https://twitter.com/gaohongnan) [![LinkedIn Profile](https://img.shields.io/badge/@gaohongnan-blue?style=social&logo=linkedin)](https://linkedin.com/in/gao-hongnan) [![GitHub Profile](https://img.shields.io/badge/GitHub-gao--hongnan-lightgrey?style=social&logo=github)](https://github.com/gao-hongnan) ![Tag](https://img.shields.io/badge/Tag-Organized_Chaos-orange) - [Reference Counting in Python](https://www.gaohongnan.com/software_engineering/python/gil.html#reference-counting-in-python) - [Threads and Race Conditions](https://www.gaohongnan.com/software_engineering/python/gil.html#threads-and-race-conditions) - [Protecting Reference Counts with Locks](https://www.gaohongnan.com/software_engineering/python/gil.html#protecting-reference-counts-with-locks) - [Deadlocks](https://www.gaohongnan.com/software_engineering/python/gil.html#deadlocks) - [The Global Interpreter Lock (GIL) in Python](https://www.gaohongnan.com/software_engineering/python/gil.html#the-global-interpreter-lock-gil-in-python) - [CPU-bound tasks](https://www.gaohongnan.com/software_engineering/python/gil.html#cpu-bound-tasks) - [GIL Is Released During IO-bound Tasks](https://www.gaohongnan.com/software_engineering/python/gil.html#gil-is-released-during-io-bound-tasks) - [References and Further Readings](https://www.gaohongnan.com/software_engineering/python/gil.html#references-and-further-readings) ## [Reference Counting in Python](https://www.gaohongnan.com/software_engineering/python/gil.html#id1)[\#](https://www.gaohongnan.com/software_engineering/python/gil.html#reference-counting-in-python "Link to this heading") In Python, reference counting is a memory management technique used to keep track of how many references (or “pointers”) exist to an object in memory. When an object’s reference count drops to zero, Python automatically frees the memory allocated to that object. ``` import sys a = [] b = a a_ref_count = sys.getrefcount(a) print(a_ref_count) # Output: 3 ``` ``` 3 ``` 1. `a = []` creates an empty list `[]` and assigns it to variable `a`. 2. `b = a` makes `b` reference the same list as `a`. 3. `sys.getrefcount(a)` returns the number of references to the list `a` points to. The count is `3` because: - `a` references the list. - `b` references the list. - The `sys.getrefcount(a)` function temporarily creates another reference when it’s called. As a beginner python programmer, we thank our lucky stars that Python manages memory for us. But, as a good student, we ask, why is managing the reference count important? First, memory management helps Python automatically manage memory by keeping track of objects and freeing memory when objects are no longer needed. For example, if we delete the reference `b`, the reference count of `a` drops to 2. ``` del b a_ref_count = sys.getrefcount(a) print(a_ref_count) # Output: 2 ``` ``` 2 ``` Second, avoiding memory leaks ensures that memory is reused efficiently and prevents memory from being wasted on unused objects. ## [Threads and Race Conditions](https://www.gaohongnan.com/software_engineering/python/gil.html#id2)[\#](https://www.gaohongnan.com/software_engineering/python/gil.html#threads-and-race-conditions "Link to this heading") A race condition occurs when two or more threads access shared data simultaneously, and the final outcome depends on the sequence of execution. For example, if two threads try to increment a counter from 5 to 6: 1. Thread A reads counter (value = 5) 2. Thread B reads counter (value = 5) 3. Thread A adds 1 (5 + 1 = 6) 4. Thread B adds 1 (5 + 1 = 6) 5. Thread A writes 6 6. Thread B writes 6 Even though two increment operations occurred, the counter only increased by 1 instead of 2. This happens because both threads read the original value before either could update it. The final result depends on which thread writes last, and the program “races” to an incorrect outcome. Show me the code to illustrate the race condition\! ``` """With reference to effective python book chapter 54. Ref: https://github.com/bslatkin/effectivepython/blob/master/example_code/item_54.py """ import logging import threading from threading import Barrier from typing import List logging.basicConfig(level=logging.DEBUG, format="%(asctime)s - %(levelname)s - %(message)s") NUM_THREADS = 5 BARRIER = Barrier(NUM_THREADS) class Counter: def __init__(self) -> None: self.count = 0 def increment(self, offset: int) -> None: self.count += offset def worker(thread_index: int, total_iterations: int, counter: Counter) -> None: """The barrier is used to synchronize the threads so that they all start counting at the same time. This makes it easier to get a race condition since we wait for the other threads to start else in the loop we always have an order that the first thread likely starts first and then the second and so on. """ BARRIER.wait() logging.debug("Thread %s, starting", thread_index) for _ in range(total_iterations): counter.increment(1) def thread_unsafe(total_iterations: int) -> None: counter = Counter() threads: List[threading.Thread] = [] for index in range(NUM_THREADS): thread = threading.Thread(target=worker, args=(index, total_iterations, counter)) threads.append(thread) for thread in threads: thread.start() for thread in threads: thread.join() expected = total_iterations * NUM_THREADS found = counter.count logging.info("Counter should be %s, got %s", expected, found) if __name__ == "__main__": total_iterations = 106 thread_unsafe(total_iterations) ``` ``` 2024-10-29 10:22:29,521 - DEBUG - Thread 4, starting 2024-10-29 10:22:29,522 - DEBUG - Thread 0, starting 2024-10-29 10:22:29,522 - DEBUG - Thread 3, starting 2024-10-29 10:22:29,522 - DEBUG - Thread 2, starting 2024-10-29 10:22:29,523 - DEBUG - Thread 1, starting 2024-10-29 10:22:29,951 - INFO - Counter should be 5000000, got 4564365 ``` ## [Protecting Reference Counts with Locks](https://www.gaohongnan.com/software_engineering/python/gil.html#id3)[\#](https://www.gaohongnan.com/software_engineering/python/gil.html#protecting-reference-counts-with-locks "Link to this heading") So we circle back a bit to the reference count thingy earlier. Since we saw how race conditions can happen, the same thing can happen to the reference count of an object - which means that the reference count of an object can be modified by multiple threads. If two threads try to increment or decrement the reference count simultaneously without protection, it can lead to inconsistencies, such as the case where reference counts might reach zero incorrectly, freeing memory while it’s still in use, causing crashes or bugs. How do we prevent this? We first understand the rough idea of a lock. A lock** is a synchronization mechanism used to control access to shared resources. When a thread acquires a lock, other threads must wait until the lock is released before accessing the resource. - In the first example without a lock, both threads might read the same `counter` value before incrementing it, causing some increments to be lost. - In the second example, the lock ensures that only one thread can modify the counter at a time. - The `with lock:` statement automatically acquires the lock before entering the block and releases it when exiting. - This guarantees that the final counter value will always be correct. ``` """With reference to effective python book chapter 54. Ref: https://github.com/bslatkin/effectivepython/blob/master/example_code/item_54.py """ import logging import threading from threading import Barrier from typing import List logging.basicConfig(level=logging.DEBUG, format="%(asctime)s - %(levelname)s - %(message)s") NUM_THREADS = 5 BARRIER = Barrier(NUM_THREADS) class CounterLock: def __init__(self) -> None: self.count = 0 self.lock = threading.Lock() def increment(self, offset: int) -> None: with self.lock: self.count += offset def worker(thread_index: int, total_iterations: int, counter: Counter) -> None: """The barrier is used to synchronize the threads so that they all start counting at the same time. This makes it easier to get a race condition since we wait for the other threads to start else in the loop we always have an order that the first thread likely starts first and then the second and so on. """ BARRIER.wait() logging.debug("Thread %s, starting", thread_index) for _ in range(total_iterations): counter.increment(1) def thread_safe(total_iterations: int) -> None: counter = CounterLock() threads: List[threading.Thread] = [] for index in range(NUM_THREADS): thread = threading.Thread(target=worker, args=(index, total_iterations, counter)) threads.append(thread) for thread in threads: thread.start() for thread in threads: thread.join() expected = total_iterations * NUM_THREADS found = counter.count logging.info("Counter should be %s, got %s", expected, found) if __name__ == "__main__": total_iterations = 106 thread_safe(total_iterations) ``` ``` 2024-10-29 10:22:29,959 - DEBUG - Thread 4, starting 2024-10-29 10:22:29,959 - DEBUG - Thread 0, starting 2024-10-29 10:22:29,959 - DEBUG - Thread 3, starting 2024-10-29 10:22:29,960 - DEBUG - Thread 2, starting 2024-10-29 10:22:29,960 - DEBUG - Thread 1, starting 2024-10-29 10:22:30,821 - INFO - Counter should be 5000000, got 5000000 ``` ## [Deadlocks](https://www.gaohongnan.com/software_engineering/python/gil.html#id4)[\#](https://www.gaohongnan.com/software_engineering/python/gil.html#deadlocks "Link to this heading") A deadlock occurs when two or more threads are waiting indefinitely for locks held by each other, preventing them from making progress. ``` import threading import time class BankAccount: def __init__(self, id: int, balance: float) -> None: self.id: int = id self.balance: float = balance self.lock: threading.Lock = threading.Lock() def __str__(self) -> str: return f"Account {self.id}: ${self.balance}" def unsafe_transfer(from_account: BankAccount, to_account: BankAccount, amount: float) -> None: with from_account.lock: print(f"Locked account {from_account.id}") time.sleep(0.5) # Simulate some work and make deadlock more likely with to_account.lock: print(f"Locked account {to_account.id}") if from_account.balance >= amount: from_account.balance -= amount to_account.balance += amount print(f"Transferred ${amount} from Account {from_account.id} to Account {to_account.id}") def safe_transfer(from_account: BankAccount, to_account: BankAccount, amount: float) -> None: first: BankAccount = from_account if from_account.id < to_account.id else to_account second: BankAccount = to_account if from_account.id < to_account.id else from_account with first.lock: print(f"Locked account {first.id}") time.sleep(0.5) # Simulate some work with second.lock: print(f"Locked account {second.id}") if from_account.balance >= amount: from_account.balance -= amount to_account.balance += amount print(f"Transferred ${amount} from Account {from_account.id} to Account {to_account.id}") account1: BankAccount = BankAccount(1, 1000) account2: BankAccount = BankAccount(2, 1000) print("Initial balances:") print(account1) print(account2) print("\nTrying unsafe transfers (will likely deadlock):") # Create threads with unsafe transfers (will deadlock) thread1: threading.Thread = threading.Thread(target=unsafe_transfer, args=(account1, account2, 500)) thread2: threading.Thread = threading.Thread(target=unsafe_transfer, args=(account2, account1, 300)) thread1.start() thread2.start() thread1.join() thread2.join() ``` 1. `thread1` acquires the lock on `account1` and waits to acquire the lock on `account2`. 2. `thread2` acquires the lock on `account2` and waits to acquire the lock on `account1`. 3. Both threads are waiting for each other to release the locks, causing a deadlock. Why do we need to know this? Because as we saw earlier, locks are useful for preventing race conditions. But using locks not so carefully can lead to deadlocks. GIL will aim to solve this as well. ## [The Global Interpreter Lock (GIL) in Python](https://www.gaohongnan.com/software_engineering/python/gil.html#id5)[\#](https://www.gaohongnan.com/software_engineering/python/gil.html#the-global-interpreter-lock-gil-in-python "Link to this heading") The Global Interpreter Lock (GIL) is a mutex (lock) that protects access to Python objects, preventing multiple native threads from executing Python bytecodes simultaneously in the same process. - To solve deadlocks, gil only has one lock for the entire process - thus the scenario where two threads are waiting for each other to release a lock is avoided. In more intuition, the gil allows only one thread to execute bytecode at a time. - With locks in place, race conditions are mostly resolved. With gil, it has its own trade-offs: - Limits Multi-threaded Performance: Because the GIL allows only one thread to execute Python code at a time, CPU-bound multi-threaded programs don’t benefit from multiple cores. They run almost as if they were single-threaded. - Inefficiency in Multi-core Systems:** In CPU-bound tasks, the GIL can become a bottleneck, preventing Python programs from fully utilizing multi-core processors. Let’s see an example of how gil can limit multi-threaded performance. ## [CPU-bound tasks](https://www.gaohongnan.com/software_engineering/python/gil.html#id6)[\#](https://www.gaohongnan.com/software_engineering/python/gil.html#cpu-bound-tasks "Link to this heading") We first compare the performance of single-threaded vs multi-threaded and note for cpu-bound tasks, the time taken is almost the same. ``` import threading import time import requests def cpu_bound_task(): """CPU intensive task - calculating sum of numbers""" count = 0 for _ in range(20_000_000): count += 1 return count def run_tasks_single_thread(task, num_iterations): start_time = time.time() for _ in range(num_iterations): task() end_time = time.time() return end_time - start_time def run_tasks_multi_thread(task, num_threads): start_time = time.time() threads: List[threading.Thread] = [] for _ in range(num_threads): t = threading.Thread(target=task) threads.append(t) t.start() for t in threads: t.join() end_time = time.time() return end_time - start_time # NOTE: CPU-bound tasks num_tasks = 4 print("CPU-bound task comparison:") single_thread_cpu = run_tasks_single_thread(cpu_bound_task, num_tasks) print(f"Single-threaded time: {single_thread_cpu:.2f} seconds") multi_thread_cpu = run_tasks_multi_thread(cpu_bound_task, num_tasks) print(f"Multi-threaded time: {multi_thread_cpu:.2f} seconds") print(f"Speed difference: {single_thread_cpu/multi_thread_cpu:.2f}x\n") ``` ``` CPU-bound task comparison: Single-threaded time: 1.71 seconds Multi-threaded time: 1.62 seconds Speed difference: 1.06x ``` Next, we compare the performance of single-threaded vs multi-threaded vs multi-process. We note that multi-process is the fastest and this is expected because there is no GIL in multi-process. ``` import multiprocessing def run_tasks_multi_process(task, num_processes): """Run tasks using multiple processes""" start_time = time.time() processes: List[multiprocessing.Process] = [] for _ in range(num_processes): p = multiprocessing.Process(target=task) processes.append(p) p.start() for p in processes: p.join() end_time = time.time() return end_time - start_time # Comparison of all three approaches num_tasks = 4 print("CPU-bound task comparison:") single_thread_cpu = run_tasks_single_thread(cpu_bound_task, num_tasks) print(f"Single-threaded time: {single_thread_cpu:.2f} seconds") multi_thread_cpu = run_tasks_multi_thread(cpu_bound_task, num_tasks) print(f"Multi-threaded time: {multi_thread_cpu:.2f} seconds") multi_process_cpu = run_tasks_multi_process(cpu_bound_task, num_tasks) print(f"Multi-process time: {multi_process_cpu:.2f} seconds") print(f"\nSpeed comparison (relative to single-thread):") print(f"Threading speedup: {single_thread_cpu/multi_thread_cpu:.2f}x") print(f"Multiprocessing speedup: {single_thread_cpu/multi_process_cpu:.2f}x\n") ``` ``` CPU-bound task comparison: Single-threaded time: 1.73 seconds Multi-threaded time: 1.58 seconds Multi-process time: 0.09 seconds Speed comparison (relative to single-thread): Threading speedup: 1.10x Multiprocessing speedup: 18.59x ``` ``` Traceback (most recent call last): File "<string>", line 1, in <module> File "/opt/homebrew/Caskroom/miniconda/base/envs/omniverse/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main Traceback (most recent call last): File "<string>", line 1, in <module> File "/opt/homebrew/Caskroom/miniconda/base/envs/omniverse/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main Traceback (most recent call last): File "<string>", line 1, in <module> File "/opt/homebrew/Caskroom/miniconda/base/envs/omniverse/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main exitcode = _main(fd, parent_sentinel) File "/opt/homebrew/Caskroom/miniconda/base/envs/omniverse/lib/python3.9/multiprocessing/spawn.py", line 126, in _main exitcode = _main(fd, parent_sentinel)exitcode = _main(fd, parent_sentinel) File "/opt/homebrew/Caskroom/miniconda/base/envs/omniverse/lib/python3.9/multiprocessing/spawn.py", line 126, in _main File "/opt/homebrew/Caskroom/miniconda/base/envs/omniverse/lib/python3.9/multiprocessing/spawn.py", line 126, in _main self = reduction.pickle.load(from_parent) AttributeError: Can't get attribute 'cpu_bound_task' on <module '__main__' (built-in)> self = reduction.pickle.load(from_parent) AttributeError: Can't get attribute 'cpu_bound_task' on <module '__main__' (built-in)> self = reduction.pickle.load(from_parent) AttributeError: Can't get attribute 'cpu_bound_task' on <module '__main__' (built-in)> Traceback (most recent call last): File "<string>", line 1, in <module> File "/opt/homebrew/Caskroom/miniconda/base/envs/omniverse/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main exitcode = _main(fd, parent_sentinel) File "/opt/homebrew/Caskroom/miniconda/base/envs/omniverse/lib/python3.9/multiprocessing/spawn.py", line 126, in _main self = reduction.pickle.load(from_parent) AttributeError: Can't get attribute 'cpu_bound_task' on <module '__main__' (built-in)> ``` ## [GIL Is Released During IO-bound Tasks](https://www.gaohongnan.com/software_engineering/python/gil.html#id7)[\#](https://www.gaohongnan.com/software_engineering/python/gil.html#gil-is-released-during-io-bound-tasks "Link to this heading") As mentioned, the GIL is released during IO-bound tasks. As to why, please see this [post](https://stackoverflow.com/questions/1294382/what-is-the-global-interpreter-lock-gil-in-cpython/55309364#55309364). Let’s see an example of this. ``` def io_bound_task(): """IO intensive task - making HTTP requests""" url = "https://api.github.com" response = requests.get(url) return response.status_code # NOTE: IO-bound tasks print("IO-bound task comparison:") single_thread_io = run_tasks_single_thread(io_bound_task, num_tasks) print(f"Single-threaded time: {single_thread_io:.2f} seconds") multi_thread_io = run_tasks_multi_thread(io_bound_task, num_tasks) print(f"Multi-threaded time: {multi_thread_io:.2f} seconds") print(f"Speed difference: {single_thread_io/multi_thread_io:.2f}x") ``` ``` 2024-10-29 20:14:18,709 - DEBUG - Starting new HTTPS connection (1): api.github.com:443 ``` ``` IO-bound task comparison: ``` ``` 2024-10-29 20:14:19,014 - DEBUG - https://api.github.com:443 "GET / HTTP/1.1" 200 510 2024-10-29 20:14:19,021 - DEBUG - Starting new HTTPS connection (1): api.github.com:443 2024-10-29 20:14:19,072 - DEBUG - https://api.github.com:443 "GET / HTTP/1.1" 200 510 2024-10-29 20:14:19,075 - DEBUG - Starting new HTTPS connection (1): api.github.com:443 2024-10-29 20:14:19,122 - DEBUG - https://api.github.com:443 "GET / HTTP/1.1" 200 510 2024-10-29 20:14:19,126 - DEBUG - Starting new HTTPS connection (1): api.github.com:443 2024-10-29 20:14:19,167 - DEBUG - https://api.github.com:443 "GET / HTTP/1.1" 200 510 2024-10-29 20:14:19,174 - DEBUG - Starting new HTTPS connection (1): api.github.com:443 2024-10-29 20:14:19,174 - DEBUG - Starting new HTTPS connection (1): api.github.com:443 2024-10-29 20:14:19,174 - DEBUG - Starting new HTTPS connection (1): api.github.com:443 2024-10-29 20:14:19,174 - DEBUG - Starting new HTTPS connection (1): api.github.com:443 2024-10-29 20:14:19,256 - DEBUG - https://api.github.com:443 "GET / HTTP/1.1" 200 510 2024-10-29 20:14:19,259 - DEBUG - https://api.github.com:443 "GET / HTTP/1.1" 200 510 2024-10-29 20:14:19,261 - DEBUG - https://api.github.com:443 "GET / HTTP/1.1" 200 510 2024-10-29 20:14:19,261 - DEBUG - https://api.github.com:443 "GET / HTTP/1.1" 200 510 ``` ``` Single-threaded time: 0.50 seconds Multi-threaded time: 0.09 seconds Speed difference: 5.24x ``` Indeed, the multi-threaded version is faster than the single-threaded version. ## [References and Further Readings](https://www.gaohongnan.com/software_engineering/python/gil.html#id8)[\#](https://www.gaohongnan.com/software_engineering/python/gil.html#references-and-further-readings "Link to this heading") - [The Global Interpreter Lock (GIL) in Python](https://realpython.com/python-gil/) - <https://stackoverflow.com/questions/1294382/what-is-the-global-interpreter-lock-gil-in-cpython/55309364#55309364> [previous Init vs New](https://www.gaohongnan.com/software_engineering/python/new_vs_init.html "previous page") [next The Iterator Protocol](https://www.gaohongnan.com/software_engineering/python/iterator_protocol.html "next page") Contents - [Reference Counting in Python](https://www.gaohongnan.com/software_engineering/python/gil.html#reference-counting-in-python) - [Threads and Race Conditions](https://www.gaohongnan.com/software_engineering/python/gil.html#threads-and-race-conditions) - [Protecting Reference Counts with Locks](https://www.gaohongnan.com/software_engineering/python/gil.html#protecting-reference-counts-with-locks) - [Deadlocks](https://www.gaohongnan.com/software_engineering/python/gil.html#deadlocks) - [The Global Interpreter Lock (GIL) in Python](https://www.gaohongnan.com/software_engineering/python/gil.html#the-global-interpreter-lock-gil-in-python) - [CPU-bound tasks](https://www.gaohongnan.com/software_engineering/python/gil.html#cpu-bound-tasks) - [GIL Is Released During IO-bound Tasks](https://www.gaohongnan.com/software_engineering/python/gil.html#gil-is-released-during-io-bound-tasks) - [References and Further Readings](https://www.gaohongnan.com/software_engineering/python/gil.html#references-and-further-readings) By Gao Hongnan © Copyright 2024. ### Connect with me\! [Follow on LinkedIn](https://www.linkedin.com/in/gao-hongnan/) [Follow on X](https://x.com/gaohongnan) ### Share this page\! [Share on LinkedIn]() [Share on X]()
Readable Markdown	null
Shard	160 (laksa)
Root Hash	5340751680682619160
Unparsed URL	com,gaohongnan!www,/software_engineering/python/gil.html s443