🕷️ Crawler Inspector

URL Lookup

Direct Parameter Lookup

Raw Queries and Responses

1. Shard Calculation

Query:

Response:

Calculated Shard: 194 (from laksa177)

2. Crawled Status Check

Query:

curl -X POST \
  'http://laksa194.int.ahrefs:8124/' \
  -H 'Content-Type: text/plain' \
  -H 'X-ClickHouse-Database: crawler3' \
  -H 'Authorization: Basic YXBpOg==' \
  -d 'SELECT getAhrefsURLFromUnparsed(src_unparsed) AS found_url, ifNull(toUnixTimestamp(download_stamp), 0) AS crawl_time, ifNull(toUnixTimestamp(props_url_first_seen), 0) AS first_indexed_time, download_http_code AS http_code, src_unparsed AS src_unparsed, src_root_hash AS src_root_hash, history_drop_reason AS history_drop_reason, meta_title AS meta_title, meta_descriptions AS meta_descriptions, attrs_boilerpipe_text AS attrs_boilerpipe_text, attrs_markdown AS attrs_markdown, attrs_readable_markdown AS attrs_readable_markdown, meta_canonical AS meta_canonical FROM crawler3.page_info_local FINAL PREWHERE (src_root_hash, src_unparsed) IN ((getAhrefsRootHashFromUnparsed(getAhrefsUnparsedNoserviceFromURL(\'https://docs.pola.rs/user-guide/misc/multiprocessing/\')), getAhrefsUnparsedNoserviceFromURL(\'https://docs.pola.rs/user-guide/misc/multiprocessing/\'))) FORMAT JSONEachRow'

Response:

{"found_url":"https:\/\/docs.pola.rs\/user-guide\/misc\/multiprocessing\/","crawl_time":1774927645,"first_indexed_time":1703778585,"http_code":200,"src_unparsed":"rs,pola!docs,\/user-guide\/misc\/multiprocessing\/ s443","src_root_hash":"15785171083524017994","history_drop_reason":null,"meta_title":"Multiprocessing - Polars user guide","meta_descriptions":[],"attrs_boilerpipe_text":"TLDR: if you find that using Python's built-in\nmultiprocessing\nmodule together with Polars results\nin a Polars error about multiprocessing methods, you should make sure you are using\nspawn\n, not\nfork\n, as the starting method:\nfrom\nmultiprocessing\nimport\nget_context\ndef\nmy_fun\n(\ns\n):\nprint\n(\ns\n)\nwith\nget_context\n(\n\"spawn\"\n)\n.\nPool\n()\nas\npool\n:\npool\n.\nmap\n(\nmy_fun\n,\n[\n\"input1\"\n,\n\"input2\"\n,\n...\n])\nWhen not to use multiprocessing\nBefore we dive into the details, it is important to emphasize that Polars has been built from the\nstart to use all your CPU cores. It does this by executing computations which can be done in\nparallel in separate threads. For example, requesting two expressions in a\nselect\nstatement can be\ndone in parallel, with the results only being combined at the end. Another example is aggregating a\nvalue within groups using\ngroup_by().agg(<expr>)\n, each group can be evaluated separately. It is\nvery unlikely that the\nmultiprocessing\nmodule can improve your code performance in these cases. If\nyou're using the GPU Engine with Polars you should also avoid manual multiprocessing. When used\nsimultaneously, they can compete for system memory and processing power, leading to reduced\nperformance.\nSee\nthe optimizations section\nfor more optimizations.\nWhen to use multiprocessing\nAlthough Polars is multithreaded, other libraries may be single-threaded. When the other library is\nthe bottleneck, and the problem at hand is parallelizable, it makes sense to use multiprocessing to\ngain a speed up.\nThe problem with the default multiprocessing config\nSummary\nThe\nPython multiprocessing documentation\nlists the three methods to create a process pool:\nspawn\nfork\nforkserver\nThe description of fork is (as of 2022-10-15):\nThe parent process uses os.fork() to fork the Python interpreter. The child process, when it\nbegins, is effectively identical to the parent process. All resources of the parent are inherited\nby the child process. Note that safely forking a multithreaded process is problematic.\nAvailable on Unix only. The default on Unix.\nThe short summary is: Polars is multithreaded as to provide strong performance out-of-the-box. Thus,\nit cannot be combined with\nfork\n. If you are on Unix (Linux, BSD, etc), you are using\nfork\n,\nunless you explicitly override it.\nThe reason you may not have encountered this before is that pure Python code, and most Python\nlibraries, are (mostly) single threaded. Alternatively, you are on Windows or MacOS, on which\nfork\nis not even available as a method (for MacOS it was up to Python 3.7).\nThus one should use\nspawn\n, or\nforkserver\n, instead.\nspawn\nis available on all platforms and the\nsafest choice, and hence the recommended method.\nExample\nThe problem with\nfork\nis in the copying of the parent's process. Consider the example below, which\nis a slightly modified example posted on the\nPolars issue tracker\n:\nimport\nmultiprocessing\nimport\npolars\nas\npl\ndef\ntest_sub_process\n(\ndf\n:\npl\n.\nDataFrame\n,\njob_id\n):\ndf_filtered\n=\ndf\n.\nfilter\n(\npl\n.\ncol\n(\n\"a\"\n)\n>\n0\n)\nprint\n(\nf\n\"Filtered (job_id:\n{\njob_id\n}\n)\"\n,\ndf_filtered\n,\nsep\n=\n\"\n\\n\n\"\n)\ndef\ncreate_dataset\n():\nreturn\npl\n.\nDataFrame\n({\n\"a\"\n:\n[\n0\n,\n2\n,\n3\n,\n4\n,\n5\n],\n\"b\"\n:\n[\n0\n,\n4\n,\n5\n,\n56\n,\n4\n]})\ndef\nsetup\n():\n# some setup work\ndf\n=\ncreate_dataset\n()\ndf\n.\nwrite_parquet\n(\n\"\/tmp\/test.parquet\"\n)\ndef\nmain\n():\ntest_df\n=\npl\n.\nread_parquet\n(\n\"\/tmp\/test.parquet\"\n)\nfor\ni\nin\nrange\n(\n0\n,\n5\n):\nproc\n=\nmultiprocessing\n.\nget_context\n(\n\"spawn\"\n)\n.\nProcess\n(\ntarget\n=\ntest_sub_process\n,\nargs\n=\n(\ntest_df\n,\ni\n)\n)\nproc\n.\nstart\n()\nproc\n.\njoin\n()\nprint\n(\nf\n\"Executed sub process\n{\ni\n}\n\"\n)\nif\n__name__\n==\n\"__main__\"\n:\nsetup\n()\nmain\n()\nUsing\nfork\nas the method, instead of\nspawn\n, will cause a dead lock.\nThe fork method is equivalent to calling\nos.fork()\n, which is a system call as defined in\nthe POSIX standard\n:\nA process shall be created with a single thread. If a multi-threaded process calls fork(), the new\nprocess shall contain a replica of the calling thread and its entire address space, possibly\nincluding the states of mutexes and other resources. Consequently, to avoid errors, the child\nprocess may only execute async-signal-safe operations until such time as one of the exec functions\nis called.\nIn contrast,\nspawn\nwill create a completely new fresh Python interpreter, and not inherit the\nstate of mutexes.\nSo what happens in the code example? For reading the file with\npl.read_parquet\nthe file has to be\nlocked. Then\nos.fork()\nis called, copying the state of the parent process, including mutexes. Thus\nall child processes will copy the file lock in an acquired state, leaving them hanging indefinitely\nwaiting for the file lock to be released, which never happens.\nWhat makes debugging these issues tricky is that\nfork\ncan work. Change the example to not having\nthe call to\npl.read_parquet\n:\nimport\nmultiprocessing\nimport\npolars\nas\npl\ndef\ntest_sub_process\n(\ndf\n:\npl\n.\nDataFrame\n,\njob_id\n):\ndf_filtered\n=\ndf\n.\nfilter\n(\npl\n.\ncol\n(\n\"a\"\n)\n>\n0\n)\nprint\n(\nf\n\"Filtered (job_id:\n{\njob_id\n}\n)\"\n,\ndf_filtered\n,\nsep\n=\n\"\n\\n\n\"\n)\ndef\ncreate_dataset\n():\nreturn\npl\n.\nDataFrame\n({\n\"a\"\n:\n[\n0\n,\n2\n,\n3\n,\n4\n,\n5\n],\n\"b\"\n:\n[\n0\n,\n4\n,\n5\n,\n56\n,\n4\n]})\ndef\nmain\n():\ntest_df\n=\ncreate_dataset\n()\nfor\ni\nin\nrange\n(\n0\n,\n5\n):\nproc\n=\nmultiprocessing\n.\nget_context\n(\n\"fork\"\n)\n.\nProcess\n(\ntarget\n=\ntest_sub_process\n,\nargs\n=\n(\ntest_df\n,\ni\n)\n)\nproc\n.\nstart\n()\nproc\n.\njoin\n()\nprint\n(\nf\n\"Executed sub process\n{\ni\n}\n\"\n)\nif\n__name__\n==\n\"__main__\"\n:\nmain\n()\nThis works fine. Therefore debugging these issues in larger code bases, i.e. not the small toy\nexamples here, can be a real pain, as a seemingly unrelated change can break your multiprocessing\ncode. In general, one should therefore never use the\nfork\nstart method with multithreaded\nlibraries unless there are very specific requirements that cannot be met otherwise.\nPro's and cons of fork\nBased on the example, you may think, why is\nfork\navailable in Python to start with?\nFirst, probably because of historical reasons:\nspawn\nwas added to Python in version 3.4, whilst\nfork\nhas been part of Python from the 2.x series.\nSecond, there are several limitations for\nspawn\nand\nforkserver\nthat do not apply to\nfork\n, in\nparticular all arguments should be pickleable. See the\nPython multiprocessing docs\nfor more information.\nThird, because it is faster to create new processes compared to\nspawn\n, as\nspawn\nis effectively\nfork\n+ creating a brand new Python process without the locks by calling\nexecv\n. Hence the warning in\nthe Python docs that it is slower: there is more overhead to\nspawn\n. However, in almost all cases,\none would like to use multiple processes to speed up computations that take multiple minutes or even\nhours, meaning the overhead is negligible in the grand scheme of things. And more importantly, it\nactually works in combination with multithreaded libraries.\nFourth,\nspawn\nstarts a new process, and therefore it requires code to be importable, in contrast\nto\nfork\n. In particular, this means that when using\nspawn\nthe relevant code should not be in the\nglobal scope, such as in Jupyter notebooks or in plain scripts. Hence in the examples above, we\ndefine functions where we spawn within, and run those functions from a\n__main__\nclause. This is\nnot an issue for typical projects, but during quick experimentation in notebooks it could fail.\nReferences\nhttps:\/\/docs.python.org\/3\/library\/multiprocessing.html\nhttps:\/\/pythonspeed.com\/articles\/python-multiprocessing\/\nhttps:\/\/pubs.opengroup.org\/onlinepubs\/9699919799\/functions\/fork.html\nhttps:\/\/bnikolic.co.uk\/blog\/python\/parallelism\/2019\/11\/13\/python-forkserver-preload.html","attrs_markdown":"[Skip to content](https:\/\/docs.pola.rs\/user-guide\/misc\/multiprocessing\/#multiprocessing)\n\n[![logo](https:\/\/docs.pola.rs\/_build\/assets\/logo.png)](https:\/\/docs.pola.rs\/ \"Polars user guide\")\n\nPolars user guide\n\nMultiprocessing\n\nType to start searching\n\n[pola-rs\/polars py-1.39.3 37.9k 2.7k](https:\/\/github.com\/pola-rs\/polars \"Go to repository\")\n\n- [Polars](https:\/\/docs.pola.rs\/)\n- [Polars Cloud](https:\/\/docs.pola.rs\/polars-cloud\/)\n- [Polars on-premises](https:\/\/docs.pola.rs\/polars-on-premises\/)\n\n[![logo](https:\/\/docs.pola.rs\/_build\/assets\/logo.png)](https:\/\/docs.pola.rs\/ \"Polars user guide\")  Polars user guide\n\n[pola-rs\/polars py-1.39.3 37.9k 2.7k](https:\/\/github.com\/pola-rs\/polars \"Go to repository\")\n\n- Polars\n  Polars\n  \n  - [User guide](https:\/\/docs.pola.rs\/)\n    User guide\n    \n    - [Getting started](https:\/\/docs.pola.rs\/user-guide\/getting-started\/)\n    - [Installation](https:\/\/docs.pola.rs\/user-guide\/installation\/)\n    - [Concepts](https:\/\/docs.pola.rs\/user-guide\/concepts\/)\n      Concepts\n      \n      - [Data types and structures](https:\/\/docs.pola.rs\/user-guide\/concepts\/data-types-and-structures\/)\n      - [Expressions and contexts](https:\/\/docs.pola.rs\/user-guide\/concepts\/expressions-and-contexts\/)\n      - [Lazy API](https:\/\/docs.pola.rs\/user-guide\/concepts\/lazy-api\/)\n      - [Streaming](https:\/\/docs.pola.rs\/user-guide\/concepts\/streaming\/)\n    - [Expressions](https:\/\/docs.pola.rs\/user-guide\/expressions\/)\n      Expressions\n      \n      - [Basic operations](https:\/\/docs.pola.rs\/user-guide\/expressions\/basic-operations\/)\n      - [Expression expansion](https:\/\/docs.pola.rs\/user-guide\/expressions\/expression-expansion\/)\n      - [Casting](https:\/\/docs.pola.rs\/user-guide\/expressions\/casting\/)\n      - [Strings](https:\/\/docs.pola.rs\/user-guide\/expressions\/strings\/)\n      - [Lists and arrays](https:\/\/docs.pola.rs\/user-guide\/expressions\/lists-and-arrays\/)\n      - [Categorical data and enums](https:\/\/docs.pola.rs\/user-guide\/expressions\/categorical-data-and-enums\/)\n      - [Structs](https:\/\/docs.pola.rs\/user-guide\/expressions\/structs\/)\n      - [Missing data](https:\/\/docs.pola.rs\/user-guide\/expressions\/missing-data\/)\n      - [Aggregation](https:\/\/docs.pola.rs\/user-guide\/expressions\/aggregation\/)\n      - [Window functions](https:\/\/docs.pola.rs\/user-guide\/expressions\/window-functions\/)\n      - [Folds](https:\/\/docs.pola.rs\/user-guide\/expressions\/folds\/)\n      - [User-defined Python functions](https:\/\/docs.pola.rs\/user-guide\/expressions\/user-defined-python-functions\/)\n      - [Numpy functions](https:\/\/docs.pola.rs\/user-guide\/expressions\/numpy-functions\/)\n    - [Transformations](https:\/\/docs.pola.rs\/user-guide\/transformations\/)\n      Transformations\n      \n      - [Joins](https:\/\/docs.pola.rs\/user-guide\/transformations\/joins\/)\n      - [Concatenation](https:\/\/docs.pola.rs\/user-guide\/transformations\/concatenation\/)\n      - [Pivots](https:\/\/docs.pola.rs\/user-guide\/transformations\/pivot\/)\n      - [Unpivots](https:\/\/docs.pola.rs\/user-guide\/transformations\/unpivot\/)\n      - Time series\n        Time series\n        \n        - [Parsing](https:\/\/docs.pola.rs\/user-guide\/transformations\/time-series\/parsing\/)\n        - [Filtering](https:\/\/docs.pola.rs\/user-guide\/transformations\/time-series\/filter\/)\n        - [Grouping](https:\/\/docs.pola.rs\/user-guide\/transformations\/time-series\/rolling\/)\n        - [Resampling](https:\/\/docs.pola.rs\/user-guide\/transformations\/time-series\/resampling\/)\n        - [Time zones](https:\/\/docs.pola.rs\/user-guide\/transformations\/time-series\/timezones\/)\n    - [Lazy API](https:\/\/docs.pola.rs\/user-guide\/lazy\/)\n      Lazy API\n      \n      - [Usage](https:\/\/docs.pola.rs\/user-guide\/lazy\/using\/)\n      - [Optimizations](https:\/\/docs.pola.rs\/user-guide\/lazy\/optimizations\/)\n      - [Schema](https:\/\/docs.pola.rs\/user-guide\/lazy\/schemas\/)\n      - [DataType Expressions](https:\/\/docs.pola.rs\/user-guide\/lazy\/datatype_exprs\/)\n      - [Query plan](https:\/\/docs.pola.rs\/user-guide\/lazy\/query-plan\/)\n      - [Query execution](https:\/\/docs.pola.rs\/user-guide\/lazy\/execution\/)\n      - [Sources and sinks](https:\/\/docs.pola.rs\/user-guide\/lazy\/sources_sinks\/)\n      - [Multiplexing queries](https:\/\/docs.pola.rs\/user-guide\/lazy\/multiplexing\/)\n      - [GPU Support](https:\/\/docs.pola.rs\/user-guide\/lazy\/gpu\/)\n    - [IO](https:\/\/docs.pola.rs\/user-guide\/io\/)\n      IO\n      \n      - [CSV](https:\/\/docs.pola.rs\/user-guide\/io\/csv\/)\n      - [Excel](https:\/\/docs.pola.rs\/user-guide\/io\/excel\/)\n      - [Parquet](https:\/\/docs.pola.rs\/user-guide\/io\/parquet\/)\n      - [JSON files](https:\/\/docs.pola.rs\/user-guide\/io\/json\/)\n      - [Multiple](https:\/\/docs.pola.rs\/user-guide\/io\/multiple\/)\n      - [Hive](https:\/\/docs.pola.rs\/user-guide\/io\/hive\/)\n      - [Databases](https:\/\/docs.pola.rs\/user-guide\/io\/database\/)\n      - [Cloud storage](https:\/\/docs.pola.rs\/user-guide\/io\/cloud-storage\/)\n      - [Google BigQuery](https:\/\/docs.pola.rs\/user-guide\/io\/bigquery\/)\n      - [Hugging Face](https:\/\/docs.pola.rs\/user-guide\/io\/hugging-face\/)\n      - [Google Sheets (via Colab)](https:\/\/docs.pola.rs\/user-guide\/io\/sheets_colab\/)\n    - [Plugins](https:\/\/docs.pola.rs\/user-guide\/plugins\/)\n      Plugins\n      \n      - [Expression Plugins](https:\/\/docs.pola.rs\/user-guide\/plugins\/expr_plugins\/)\n      - [IO Plugins](https:\/\/docs.pola.rs\/user-guide\/plugins\/io_plugins\/)\n    - SQL\n      SQL\n      \n      - [Introduction](https:\/\/docs.pola.rs\/user-guide\/sql\/intro\/)\n      - [SHOW TABLES](https:\/\/docs.pola.rs\/user-guide\/sql\/show\/)\n      - [SELECT](https:\/\/docs.pola.rs\/user-guide\/sql\/select\/)\n      - [CREATE](https:\/\/docs.pola.rs\/user-guide\/sql\/create\/)\n      - [Common Table Expressions](https:\/\/docs.pola.rs\/user-guide\/sql\/cte\/)\n    - Migrating\n      Migrating\n      \n      - [Coming from Pandas](https:\/\/docs.pola.rs\/user-guide\/migration\/pandas\/)\n      - [Coming from Apache Spark](https:\/\/docs.pola.rs\/user-guide\/migration\/spark\/)\n    - Misc\n      Misc\n      \n      - [Ecosystem](https:\/\/docs.pola.rs\/user-guide\/ecosystem\/)\n      - Multiprocessing\n        [Multiprocessing](https:\/\/docs.pola.rs\/user-guide\/misc\/multiprocessing\/)\n        Table of contents\n        \n        - [When not to use multiprocessing](https:\/\/docs.pola.rs\/user-guide\/misc\/multiprocessing\/#when-not-to-use-multiprocessing)\n        - [When to use multiprocessing](https:\/\/docs.pola.rs\/user-guide\/misc\/multiprocessing\/#when-to-use-multiprocessing)\n        - [The problem with the default multiprocessing config](https:\/\/docs.pola.rs\/user-guide\/misc\/multiprocessing\/#the-problem-with-the-default-multiprocessing-config)\n          - [Summary](https:\/\/docs.pola.rs\/user-guide\/misc\/multiprocessing\/#summary)\n          - [Example](https:\/\/docs.pola.rs\/user-guide\/misc\/multiprocessing\/#example)\n          - [Pro's and cons of fork](https:\/\/docs.pola.rs\/user-guide\/misc\/multiprocessing\/#pros-and-cons-of-fork)\n        - [References](https:\/\/docs.pola.rs\/user-guide\/misc\/multiprocessing\/#references)\n      - [Visualization](https:\/\/docs.pola.rs\/user-guide\/misc\/visualization\/)\n      - [Styling](https:\/\/docs.pola.rs\/user-guide\/misc\/styling\/)\n      - [Comparison with other tools](https:\/\/docs.pola.rs\/user-guide\/misc\/comparison\/)\n      - [Arrow producer\/consumer](https:\/\/docs.pola.rs\/user-guide\/misc\/arrow\/)\n      - [Generating Polars code with LLMs](https:\/\/docs.pola.rs\/user-guide\/misc\/polars_llms\/)\n    - [GPU Support \\[Open Beta\\]](https:\/\/docs.pola.rs\/user-guide\/gpu-support\/)\n  - API\n    API\n    \n    - [Reference guide](https:\/\/docs.pola.rs\/api\/reference\/)\n  - Development\n    Development\n    \n    - [Contributing](https:\/\/docs.pola.rs\/development\/contributing\/)\n      Contributing\n      \n      - [IDE configuration](https:\/\/docs.pola.rs\/development\/contributing\/ide\/)\n      - [Test suite](https:\/\/docs.pola.rs\/development\/contributing\/test\/)\n      - [Continuous integration](https:\/\/docs.pola.rs\/development\/contributing\/ci\/)\n      - [Code style](https:\/\/docs.pola.rs\/development\/contributing\/code-style\/)\n    - [Versioning](https:\/\/docs.pola.rs\/development\/versioning\/)\n  - Releases\n    Releases\n    \n    - [Upgrade guides](https:\/\/docs.pola.rs\/releases\/upgrade\/)\n      Upgrade guides\n      \n      - [Version 1](https:\/\/docs.pola.rs\/releases\/upgrade\/1\/)\n      - [Version 0.20](https:\/\/docs.pola.rs\/releases\/upgrade\/0.20\/)\n      - [Version 0.19](https:\/\/docs.pola.rs\/releases\/upgrade\/0.19\/)\n    - [Changelog](https:\/\/docs.pola.rs\/releases\/changelog\/)\n- [Polars Cloud](https:\/\/docs.pola.rs\/polars-cloud\/)\n  Polars Cloud\n  \n  - [Getting started](https:\/\/docs.pola.rs\/polars-cloud\/quickstart\/)\n  - [Connect to your cloud](https:\/\/docs.pola.rs\/polars-cloud\/connect-cloud\/)\n  - Queries\n    Queries\n    \n    - [Execute remote query](https:\/\/docs.pola.rs\/polars-cloud\/run\/remote-query\/)\n    - [Distributed queries](https:\/\/docs.pola.rs\/polars-cloud\/run\/distributed-engine\/)\n    - [Query profiling](https:\/\/docs.pola.rs\/polars-cloud\/run\/query-profile\/)\n    - [Glossary](https:\/\/docs.pola.rs\/polars-cloud\/run\/glossary\/)\n  - Integrations\n    Integrations\n    \n    - [Orchestration](https:\/\/docs.pola.rs\/polars-cloud\/integrations\/)\n      Orchestration\n      \n      - [Airflow](https:\/\/docs.pola.rs\/polars-cloud\/integrations\/airflow\/)\n      - [Dagster](https:\/\/docs.pola.rs\/polars-cloud\/integrations\/dagster\/)\n      - [Prefect](https:\/\/docs.pola.rs\/polars-cloud\/integrations\/prefect\/)\n      - [AWS Lambda](https:\/\/docs.pola.rs\/polars-cloud\/integrations\/lambda\/)\n  - Concepts\n    Concepts\n    \n    - Context\n      Context\n      \n      - [Compute context introduction](https:\/\/docs.pola.rs\/polars-cloud\/context\/compute-context\/)\n      - [Reconnect to compute cluster](https:\/\/docs.pola.rs\/polars-cloud\/context\/reconnect\/)\n      - [Plugins and custom libraries](https:\/\/docs.pola.rs\/polars-cloud\/context\/plugins\/)\n      - [Proxy mode](https:\/\/docs.pola.rs\/polars-cloud\/context\/proxy-mode\/)\n    - Organizations\n      Organizations\n      \n      - [Set up organization](https:\/\/docs.pola.rs\/polars-cloud\/organization\/organizations\/)\n      - [Start trial period](https:\/\/docs.pola.rs\/polars-cloud\/organization\/start-trial\/)\n      - [Payment and billing](https:\/\/docs.pola.rs\/polars-cloud\/organization\/billing\/)\n      - [Manage members](https:\/\/docs.pola.rs\/polars-cloud\/organization\/members\/)\n    - Workspaces\n      Workspaces\n      \n      - [Workspace configuration](https:\/\/docs.pola.rs\/polars-cloud\/workspace\/settings\/)\n      - [Manage team](https:\/\/docs.pola.rs\/polars-cloud\/workspace\/team\/)\n    - Authentication\n      Authentication\n      \n      - [Logging in](https:\/\/docs.pola.rs\/polars-cloud\/explain\/authentication\/)\n      - [Using service accounts](https:\/\/docs.pola.rs\/polars-cloud\/explain\/service-accounts\/)\n    - Providers\n      Providers\n      \n      - AWS\n        AWS\n        \n        - [Infrastructure](https:\/\/docs.pola.rs\/polars-cloud\/providers\/aws\/infra\/)\n        - [Permissions](https:\/\/docs.pola.rs\/polars-cloud\/providers\/aws\/permissions\/)\n    - Misc\n      Misc\n      \n      - [CLI](https:\/\/docs.pola.rs\/polars-cloud\/cli\/)\n      - [Public datasets](https:\/\/docs.pola.rs\/polars-cloud\/public-datasets\/)\n      - [FAQ](https:\/\/docs.pola.rs\/polars-cloud\/faq\/)\n      - [API Reference](https:\/\/docs.cloud.pola.rs\/)\n  - API\n    API\n    \n    - [Reference guide](https:\/\/docs.cloud.pola.rs\/)\n- [Polars on-premises](https:\/\/docs.pola.rs\/polars-on-premises\/)\n  Polars on-premises\n  \n  - Kubernetes\n    Kubernetes\n    \n    - [Getting started](https:\/\/docs.pola.rs\/polars-on-premises\/kubernetes\/getting-started\/)\n  - Bare-metal\n    Bare-metal\n    \n    - [Getting started](https:\/\/docs.pola.rs\/polars-on-premises\/bare-metal\/getting-started\/)\n    - Configuration\n      Configuration\n      \n      - [Config file reference](https:\/\/docs.pola.rs\/polars-on-premises\/bare-metal\/configuration\/reference\/)\n      - [License](https:\/\/docs.pola.rs\/polars-on-premises\/bare-metal\/configuration\/license\/)\n      - [Profiling and host metrics](https:\/\/docs.pola.rs\/polars-on-premises\/bare-metal\/configuration\/monitoring\/)\n      - [Resource limits](https:\/\/docs.pola.rs\/polars-on-premises\/bare-metal\/configuration\/resource-limits\/)\n      - [Shuffle data](https:\/\/docs.pola.rs\/polars-on-premises\/bare-metal\/configuration\/shuffle-data\/)\n      - [Anonymous results](https:\/\/docs.pola.rs\/polars-on-premises\/bare-metal\/configuration\/anonymous-results\/)\n      - [Network addresses](https:\/\/docs.pola.rs\/polars-on-premises\/bare-metal\/configuration\/network-addresses\/)\n      - [Static leader configuration](https:\/\/docs.pola.rs\/polars-on-premises\/bare-metal\/configuration\/static-leader\/)\n      - [Example configurations](https:\/\/docs.pola.rs\/polars-on-premises\/bare-metal\/configuration\/example-configurations\/)\n    - [Environment variables](https:\/\/docs.pola.rs\/polars-on-premises\/bare-metal\/environment-variables\/)\n    - [Python environment](https:\/\/docs.pola.rs\/polars-on-premises\/bare-metal\/python-environment\/)\n\nTable of contents\n\n- [When not to use multiprocessing](https:\/\/docs.pola.rs\/user-guide\/misc\/multiprocessing\/#when-not-to-use-multiprocessing)\n- [When to use multiprocessing](https:\/\/docs.pola.rs\/user-guide\/misc\/multiprocessing\/#when-to-use-multiprocessing)\n- [The problem with the default multiprocessing config](https:\/\/docs.pola.rs\/user-guide\/misc\/multiprocessing\/#the-problem-with-the-default-multiprocessing-config)\n  - [Summary](https:\/\/docs.pola.rs\/user-guide\/misc\/multiprocessing\/#summary)\n  - [Example](https:\/\/docs.pola.rs\/user-guide\/misc\/multiprocessing\/#example)\n  - [Pro's and cons of fork](https:\/\/docs.pola.rs\/user-guide\/misc\/multiprocessing\/#pros-and-cons-of-fork)\n- [References](https:\/\/docs.pola.rs\/user-guide\/misc\/multiprocessing\/#references)\n\n# Multiprocessing\nTLDR: if you find that using Python's built-in `multiprocessing` module together with Polars results in a Polars error about multiprocessing methods, you should make sure you are using `spawn`, not `fork`, as the starting method:\n\n[Python](https:\/\/docs.pola.rs\/user-guide\/misc\/multiprocessing\/#__tabbed_1_1)\n```\n\n```\n## When not to use multiprocessing\nBefore we dive into the details, it is important to emphasize that Polars has been built from the start to use all your CPU cores. It does this by executing computations which can be done in parallel in separate threads. For example, requesting two expressions in a `select` statement can be done in parallel, with the results only being combined at the end. Another example is aggregating a value within groups using `group_by().agg(<expr>)`, each group can be evaluated separately. It is very unlikely that the `multiprocessing` module can improve your code performance in these cases. If you're using the GPU Engine with Polars you should also avoid manual multiprocessing. When used simultaneously, they can compete for system memory and processing power, leading to reduced performance.\n\nSee [the optimizations section](https:\/\/docs.pola.rs\/user-guide\/lazy\/optimizations\/) for more optimizations.\n\n## When to use multiprocessing\nAlthough Polars is multithreaded, other libraries may be single-threaded. When the other library is the bottleneck, and the problem at hand is parallelizable, it makes sense to use multiprocessing to gain a speed up.\n\n## The problem with the default multiprocessing config\n### Summary\nThe [Python multiprocessing documentation](https:\/\/docs.python.org\/3\/library\/multiprocessing.html) lists the three methods to create a process pool:\n\n1. spawn\n2. fork\n3. forkserver\n\nThe description of fork is (as of 2022-10-15):\n\n> The parent process uses os.fork() to fork the Python interpreter. The child process, when it begins, is effectively identical to the parent process. All resources of the parent are inherited by the child process. Note that safely forking a multithreaded process is problematic.\n> \n> Available on Unix only. The default on Unix.\n\nThe short summary is: Polars is multithreaded as to provide strong performance out-of-the-box. Thus, it cannot be combined with `fork`. If you are on Unix (Linux, BSD, etc), you are using `fork`, unless you explicitly override it.\n\nThe reason you may not have encountered this before is that pure Python code, and most Python libraries, are (mostly) single threaded. Alternatively, you are on Windows or MacOS, on which `fork` is not even available as a method (for MacOS it was up to Python 3.7).\n\nThus one should use `spawn`, or `forkserver`, instead. `spawn` is available on all platforms and the safest choice, and hence the recommended method.\n\n### Example\nThe problem with `fork` is in the copying of the parent's process. Consider the example below, which is a slightly modified example posted on the [Polars issue tracker](https:\/\/github.com\/pola-rs\/polars\/issues\/3144):\n\n[Python](https:\/\/docs.pola.rs\/user-guide\/misc\/multiprocessing\/#__tabbed_2_1)\n```\n\n```\n\nUsing `fork` as the method, instead of `spawn`, will cause a dead lock.\n\nThe fork method is equivalent to calling `os.fork()`, which is a system call as defined in [the POSIX standard](https:\/\/pubs.opengroup.org\/onlinepubs\/9699919799\/functions\/fork.html):\n\n> A process shall be created with a single thread. If a multi-threaded process calls fork(), the new process shall contain a replica of the calling thread and its entire address space, possibly including the states of mutexes and other resources. Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of the exec functions is called.\n\nIn contrast, `spawn` will create a completely new fresh Python interpreter, and not inherit the state of mutexes.\n\nSo what happens in the code example? For reading the file with `pl.read_parquet` the file has to be locked. Then `os.fork()` is called, copying the state of the parent process, including mutexes. Thus all child processes will copy the file lock in an acquired state, leaving them hanging indefinitely waiting for the file lock to be released, which never happens.\n\nWhat makes debugging these issues tricky is that `fork` can work. Change the example to not having the call to `pl.read_parquet`:\n\n[Python](https:\/\/docs.pola.rs\/user-guide\/misc\/multiprocessing\/#__tabbed_3_1)\n```\n\n```\n\nThis works fine. Therefore debugging these issues in larger code bases, i.e. not the small toy examples here, can be a real pain, as a seemingly unrelated change can break your multiprocessing code. In general, one should therefore never use the `fork` start method with multithreaded libraries unless there are very specific requirements that cannot be met otherwise.\n\n### Pro's and cons of fork\nBased on the example, you may think, why is `fork` available in Python to start with?\n\nFirst, probably because of historical reasons: `spawn` was added to Python in version 3.4, whilst `fork` has been part of Python from the 2.x series.\n\nSecond, there are several limitations for `spawn` and `forkserver` that do not apply to `fork`, in particular all arguments should be pickleable. See the [Python multiprocessing docs](https:\/\/docs.python.org\/3\/library\/multiprocessing.html#the-spawn-and-forkserver-start-methods) for more information.\n\nThird, because it is faster to create new processes compared to `spawn`, as `spawn` is effectively `fork` + creating a brand new Python process without the locks by calling [execv](https:\/\/pubs.opengroup.org\/onlinepubs\/9699919799\/functions\/exec.html). Hence the warning in the Python docs that it is slower: there is more overhead to `spawn`. However, in almost all cases, one would like to use multiple processes to speed up computations that take multiple minutes or even hours, meaning the overhead is negligible in the grand scheme of things. And more importantly, it actually works in combination with multithreaded libraries.\n\nFourth, `spawn` starts a new process, and therefore it requires code to be importable, in contrast to `fork`. In particular, this means that when using `spawn` the relevant code should not be in the global scope, such as in Jupyter notebooks or in plain scripts. Hence in the examples above, we define functions where we spawn within, and run those functions from a `__main__` clause. This is not an issue for typical projects, but during quick experimentation in notebooks it could fail.\n\n## References\n1. https:\/\/docs.python.org\/3\/library\/multiprocessing.html\n2. https:\/\/pythonspeed.com\/articles\/python-multiprocessing\/\n3. https:\/\/pubs.opengroup.org\/onlinepubs\/9699919799\/functions\/fork.html\n4. https:\/\/bnikolic.co.uk\/blog\/python\/parallelism\/2019\/11\/13\/python-forkserver-preload.html\n\n[Previous Ecosystem](https:\/\/docs.pola.rs\/user-guide\/ecosystem\/)\n\n[Next Visualization](https:\/\/docs.pola.rs\/user-guide\/misc\/visualization\/)\n\nMade with  [Material for MkDocs](https:\/\/squidfunk.github.io\/mkdocs-material\/)","attrs_readable_markdown":"TLDR: if you find that using Python's built-in `multiprocessing` module together with Polars results in a Polars error about multiprocessing methods, you should make sure you are using `spawn`, not `fork`, as the starting method:\n```\n\n```\n## When not to use multiprocessing\nBefore we dive into the details, it is important to emphasize that Polars has been built from the start to use all your CPU cores. It does this by executing computations which can be done in parallel in separate threads. For example, requesting two expressions in a `select` statement can be done in parallel, with the results only being combined at the end. Another example is aggregating a value within groups using `group_by().agg(<expr>)`, each group can be evaluated separately. It is very unlikely that the `multiprocessing` module can improve your code performance in these cases. If you're using the GPU Engine with Polars you should also avoid manual multiprocessing. When used simultaneously, they can compete for system memory and processing power, leading to reduced performance.\n\nSee [the optimizations section](https:\/\/docs.pola.rs\/user-guide\/lazy\/optimizations\/) for more optimizations.\n\n## When to use multiprocessing\nAlthough Polars is multithreaded, other libraries may be single-threaded. When the other library is the bottleneck, and the problem at hand is parallelizable, it makes sense to use multiprocessing to gain a speed up.\n\n## The problem with the default multiprocessing config\n### Summary\nThe [Python multiprocessing documentation](https:\/\/docs.python.org\/3\/library\/multiprocessing.html) lists the three methods to create a process pool:\n\n1. spawn\n2. fork\n3. forkserver\n\nThe description of fork is (as of 2022-10-15):\n\n> The parent process uses os.fork() to fork the Python interpreter. The child process, when it begins, is effectively identical to the parent process. All resources of the parent are inherited by the child process. Note that safely forking a multithreaded process is problematic.\n> \n> Available on Unix only. The default on Unix.\n\nThe short summary is: Polars is multithreaded as to provide strong performance out-of-the-box. Thus, it cannot be combined with `fork`. If you are on Unix (Linux, BSD, etc), you are using `fork`, unless you explicitly override it.\n\nThe reason you may not have encountered this before is that pure Python code, and most Python libraries, are (mostly) single threaded. Alternatively, you are on Windows or MacOS, on which `fork` is not even available as a method (for MacOS it was up to Python 3.7).\n\nThus one should use `spawn`, or `forkserver`, instead. `spawn` is available on all platforms and the safest choice, and hence the recommended method.\n\n### Example\nThe problem with `fork` is in the copying of the parent's process. Consider the example below, which is a slightly modified example posted on the [Polars issue tracker](https:\/\/github.com\/pola-rs\/polars\/issues\/3144):\n```\n\n```\nUsing `fork` as the method, instead of `spawn`, will cause a dead lock.\n\nThe fork method is equivalent to calling `os.fork()`, which is a system call as defined in [the POSIX standard](https:\/\/pubs.opengroup.org\/onlinepubs\/9699919799\/functions\/fork.html):\n\n> A process shall be created with a single thread. If a multi-threaded process calls fork(), the new process shall contain a replica of the calling thread and its entire address space, possibly including the states of mutexes and other resources. Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of the exec functions is called.\n\nIn contrast, `spawn` will create a completely new fresh Python interpreter, and not inherit the state of mutexes.\n\nSo what happens in the code example? For reading the file with `pl.read_parquet` the file has to be locked. Then `os.fork()` is called, copying the state of the parent process, including mutexes. Thus all child processes will copy the file lock in an acquired state, leaving them hanging indefinitely waiting for the file lock to be released, which never happens.\n\nWhat makes debugging these issues tricky is that `fork` can work. Change the example to not having the call to `pl.read_parquet`:\n```\n\n```\nThis works fine. Therefore debugging these issues in larger code bases, i.e. not the small toy examples here, can be a real pain, as a seemingly unrelated change can break your multiprocessing code. In general, one should therefore never use the `fork` start method with multithreaded libraries unless there are very specific requirements that cannot be met otherwise.\n\n### Pro's and cons of fork\nBased on the example, you may think, why is `fork` available in Python to start with?\n\nFirst, probably because of historical reasons: `spawn` was added to Python in version 3.4, whilst `fork` has been part of Python from the 2.x series.\n\nSecond, there are several limitations for `spawn` and `forkserver` that do not apply to `fork`, in particular all arguments should be pickleable. See the [Python multiprocessing docs](https:\/\/docs.python.org\/3\/library\/multiprocessing.html#the-spawn-and-forkserver-start-methods) for more information.\n\nThird, because it is faster to create new processes compared to `spawn`, as `spawn` is effectively `fork` + creating a brand new Python process without the locks by calling [execv](https:\/\/pubs.opengroup.org\/onlinepubs\/9699919799\/functions\/exec.html). Hence the warning in the Python docs that it is slower: there is more overhead to `spawn`. However, in almost all cases, one would like to use multiple processes to speed up computations that take multiple minutes or even hours, meaning the overhead is negligible in the grand scheme of things. And more importantly, it actually works in combination with multithreaded libraries.\n\nFourth, `spawn` starts a new process, and therefore it requires code to be importable, in contrast to `fork`. In particular, this means that when using `spawn` the relevant code should not be in the global scope, such as in Jupyter notebooks or in plain scripts. Hence in the examples above, we define functions where we spawn within, and run those functions from a `__main__` clause. This is not an issue for typical projects, but during quick experimentation in notebooks it could fail.\n\n## References\n1. https:\/\/docs.python.org\/3\/library\/multiprocessing.html\n2. https:\/\/pythonspeed.com\/articles\/python-multiprocessing\/\n3. https:\/\/pubs.opengroup.org\/onlinepubs\/9699919799\/functions\/fork.html\n4. https:\/\/bnikolic.co.uk\/blog\/python\/parallelism\/2019\/11\/13\/python-forkserver-preload.html","meta_canonical":null}

3. Robots.txt Check

Query:

Response:

4. Spam/Ban Check

Query:

Response:

5. Seen Status Check

ℹ️ Skipped - page is already crawled

📄

INDEXABLE

✅

CRAWLED

6 days ago

🤖

ROBOTS ALLOWED

Page Info Filters

Filter	Status	Condition	Details
HTTP status	PASS	`download_http_code = 200`	HTTP 200
Age cutoff	PASS	`download_stamp > now() - 6 MONTH`	0.2 months ago
History drop	PASS	`isNull(history_drop_reason)`	No drop reason
Spam/ban	PASS	`fh_dont_index != 1 AND ml_spam_score = 0`	ml_spam_score=0
Canonical	PASS	`meta_canonical IS NULL OR = '' OR = src_unparsed`	Not set

Page Details

Property	Value
URL	https://docs.pola.rs/user-guide/misc/multiprocessing/
Last Crawled	2026-03-31 03:27:25 (6 days ago)
First Indexed	2023-12-28 15:49:45 (2 years ago)
HTTP Status Code	200
Meta Title	Multiprocessing - Polars user guide
Meta Description	null
Meta Canonical	null
Boilerpipe Text	TLDR: if you find that using Python's built-in multiprocessing module together with Polars results in a Polars error about multiprocessing methods, you should make sure you are using spawn , not fork , as the starting method: from multiprocessing import get_context def my_fun ( s ): print ( s ) with get_context ( "spawn" ) . Pool () as pool : pool . map ( my_fun , [ "input1" , "input2" , ... ]) When not to use multiprocessing Before we dive into the details, it is important to emphasize that Polars has been built from the start to use all your CPU cores. It does this by executing computations which can be done in parallel in separate threads. For example, requesting two expressions in a select statement can be done in parallel, with the results only being combined at the end. Another example is aggregating a value within groups using group_by().agg(<expr>) , each group can be evaluated separately. It is very unlikely that the multiprocessing module can improve your code performance in these cases. If you're using the GPU Engine with Polars you should also avoid manual multiprocessing. When used simultaneously, they can compete for system memory and processing power, leading to reduced performance. See the optimizations section for more optimizations. When to use multiprocessing Although Polars is multithreaded, other libraries may be single-threaded. When the other library is the bottleneck, and the problem at hand is parallelizable, it makes sense to use multiprocessing to gain a speed up. The problem with the default multiprocessing config Summary The Python multiprocessing documentation lists the three methods to create a process pool: spawn fork forkserver The description of fork is (as of 2022-10-15): The parent process uses os.fork() to fork the Python interpreter. The child process, when it begins, is effectively identical to the parent process. All resources of the parent are inherited by the child process. Note that safely forking a multithreaded process is problematic. Available on Unix only. The default on Unix. The short summary is: Polars is multithreaded as to provide strong performance out-of-the-box. Thus, it cannot be combined with fork . If you are on Unix (Linux, BSD, etc), you are using fork , unless you explicitly override it. The reason you may not have encountered this before is that pure Python code, and most Python libraries, are (mostly) single threaded. Alternatively, you are on Windows or MacOS, on which fork is not even available as a method (for MacOS it was up to Python 3.7). Thus one should use spawn , or forkserver , instead. spawn is available on all platforms and the safest choice, and hence the recommended method. Example The problem with fork is in the copying of the parent's process. Consider the example below, which is a slightly modified example posted on the Polars issue tracker : import multiprocessing import polars as pl def test_sub_process ( df : pl . DataFrame , job_id ): df_filtered = df . filter ( pl . col ( "a" ) > 0 ) print ( f "Filtered (job_id: { job_id } )" , df_filtered , sep = " \n " ) def create_dataset (): return pl . DataFrame ({ "a" : [ 0 , 2 , 3 , 4 , 5 ], "b" : [ 0 , 4 , 5 , 56 , 4 ]}) def setup (): # some setup work df = create_dataset () df . write_parquet ( "/tmp/test.parquet" ) def main (): test_df = pl . read_parquet ( "/tmp/test.parquet" ) for i in range ( 0 , 5 ): proc = multiprocessing . get_context ( "spawn" ) . Process ( target = test_sub_process , args = ( test_df , i ) ) proc . start () proc . join () print ( f "Executed sub process { i } " ) if __name__ == "__main__" : setup () main () Using fork as the method, instead of spawn , will cause a dead lock. The fork method is equivalent to calling os.fork() , which is a system call as defined in the POSIX standard : A process shall be created with a single thread. If a multi-threaded process calls fork(), the new process shall contain a replica of the calling thread and its entire address space, possibly including the states of mutexes and other resources. Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of the exec functions is called. In contrast, spawn will create a completely new fresh Python interpreter, and not inherit the state of mutexes. So what happens in the code example? For reading the file with pl.read_parquet the file has to be locked. Then os.fork() is called, copying the state of the parent process, including mutexes. Thus all child processes will copy the file lock in an acquired state, leaving them hanging indefinitely waiting for the file lock to be released, which never happens. What makes debugging these issues tricky is that fork can work. Change the example to not having the call to pl.read_parquet : import multiprocessing import polars as pl def test_sub_process ( df : pl . DataFrame , job_id ): df_filtered = df . filter ( pl . col ( "a" ) > 0 ) print ( f "Filtered (job_id: { job_id } )" , df_filtered , sep = " \n " ) def create_dataset (): return pl . DataFrame ({ "a" : [ 0 , 2 , 3 , 4 , 5 ], "b" : [ 0 , 4 , 5 , 56 , 4 ]}) def main (): test_df = create_dataset () for i in range ( 0 , 5 ): proc = multiprocessing . get_context ( "fork" ) . Process ( target = test_sub_process , args = ( test_df , i ) ) proc . start () proc . join () print ( f "Executed sub process { i } " ) if __name__ == "__main__" : main () This works fine. Therefore debugging these issues in larger code bases, i.e. not the small toy examples here, can be a real pain, as a seemingly unrelated change can break your multiprocessing code. In general, one should therefore never use the fork start method with multithreaded libraries unless there are very specific requirements that cannot be met otherwise. Pro's and cons of fork Based on the example, you may think, why is fork available in Python to start with? First, probably because of historical reasons: spawn was added to Python in version 3.4, whilst fork has been part of Python from the 2.x series. Second, there are several limitations for spawn and forkserver that do not apply to fork , in particular all arguments should be pickleable. See the Python multiprocessing docs for more information. Third, because it is faster to create new processes compared to spawn , as spawn is effectively fork + creating a brand new Python process without the locks by calling execv . Hence the warning in the Python docs that it is slower: there is more overhead to spawn . However, in almost all cases, one would like to use multiple processes to speed up computations that take multiple minutes or even hours, meaning the overhead is negligible in the grand scheme of things. And more importantly, it actually works in combination with multithreaded libraries. Fourth, spawn starts a new process, and therefore it requires code to be importable, in contrast to fork . In particular, this means that when using spawn the relevant code should not be in the global scope, such as in Jupyter notebooks or in plain scripts. Hence in the examples above, we define functions where we spawn within, and run those functions from a __main__ clause. This is not an issue for typical projects, but during quick experimentation in notebooks it could fail. References https://docs.python.org/3/library/multiprocessing.html https://pythonspeed.com/articles/python-multiprocessing/ https://pubs.opengroup.org/onlinepubs/9699919799/functions/fork.html https://bnikolic.co.uk/blog/python/parallelism/2019/11/13/python-forkserver-preload.html
Markdown	[Skip to content](https://docs.pola.rs/user-guide/misc/multiprocessing/#multiprocessing) [![logo](https://docs.pola.rs/_build/assets/logo.png)](https://docs.pola.rs/ "Polars user guide") Polars user guide Multiprocessing Type to start searching [pola-rs/polars py-1.39.3 37.9k 2.7k](https://github.com/pola-rs/polars "Go to repository") - [Polars](https://docs.pola.rs/) - [Polars Cloud](https://docs.pola.rs/polars-cloud/) - [Polars on-premises](https://docs.pola.rs/polars-on-premises/) [![logo](https://docs.pola.rs/_build/assets/logo.png)](https://docs.pola.rs/ "Polars user guide") Polars user guide [pola-rs/polars py-1.39.3 37.9k 2.7k](https://github.com/pola-rs/polars "Go to repository") - Polars Polars - [User guide](https://docs.pola.rs/) User guide - [Getting started](https://docs.pola.rs/user-guide/getting-started/) - [Installation](https://docs.pola.rs/user-guide/installation/) - [Concepts](https://docs.pola.rs/user-guide/concepts/) Concepts - [Data types and structures](https://docs.pola.rs/user-guide/concepts/data-types-and-structures/) - [Expressions and contexts](https://docs.pola.rs/user-guide/concepts/expressions-and-contexts/) - [Lazy API](https://docs.pola.rs/user-guide/concepts/lazy-api/) - [Streaming](https://docs.pola.rs/user-guide/concepts/streaming/) - [Expressions](https://docs.pola.rs/user-guide/expressions/) Expressions - [Basic operations](https://docs.pola.rs/user-guide/expressions/basic-operations/) - [Expression expansion](https://docs.pola.rs/user-guide/expressions/expression-expansion/) - [Casting](https://docs.pola.rs/user-guide/expressions/casting/) - [Strings](https://docs.pola.rs/user-guide/expressions/strings/) - [Lists and arrays](https://docs.pola.rs/user-guide/expressions/lists-and-arrays/) - [Categorical data and enums](https://docs.pola.rs/user-guide/expressions/categorical-data-and-enums/) - [Structs](https://docs.pola.rs/user-guide/expressions/structs/) - [Missing data](https://docs.pola.rs/user-guide/expressions/missing-data/) - [Aggregation](https://docs.pola.rs/user-guide/expressions/aggregation/) - [Window functions](https://docs.pola.rs/user-guide/expressions/window-functions/) - [Folds](https://docs.pola.rs/user-guide/expressions/folds/) - [User-defined Python functions](https://docs.pola.rs/user-guide/expressions/user-defined-python-functions/) - [Numpy functions](https://docs.pola.rs/user-guide/expressions/numpy-functions/) - [Transformations](https://docs.pola.rs/user-guide/transformations/) Transformations - [Joins](https://docs.pola.rs/user-guide/transformations/joins/) - [Concatenation](https://docs.pola.rs/user-guide/transformations/concatenation/) - [Pivots](https://docs.pola.rs/user-guide/transformations/pivot/) - [Unpivots](https://docs.pola.rs/user-guide/transformations/unpivot/) - Time series Time series - [Parsing](https://docs.pola.rs/user-guide/transformations/time-series/parsing/) - [Filtering](https://docs.pola.rs/user-guide/transformations/time-series/filter/) - [Grouping](https://docs.pola.rs/user-guide/transformations/time-series/rolling/) - [Resampling](https://docs.pola.rs/user-guide/transformations/time-series/resampling/) - [Time zones](https://docs.pola.rs/user-guide/transformations/time-series/timezones/) - [Lazy API](https://docs.pola.rs/user-guide/lazy/) Lazy API - [Usage](https://docs.pola.rs/user-guide/lazy/using/) - [Optimizations](https://docs.pola.rs/user-guide/lazy/optimizations/) - [Schema](https://docs.pola.rs/user-guide/lazy/schemas/) - [DataType Expressions](https://docs.pola.rs/user-guide/lazy/datatype_exprs/) - [Query plan](https://docs.pola.rs/user-guide/lazy/query-plan/) - [Query execution](https://docs.pola.rs/user-guide/lazy/execution/) - [Sources and sinks](https://docs.pola.rs/user-guide/lazy/sources_sinks/) - [Multiplexing queries](https://docs.pola.rs/user-guide/lazy/multiplexing/) - [GPU Support](https://docs.pola.rs/user-guide/lazy/gpu/) - [IO](https://docs.pola.rs/user-guide/io/) IO - [CSV](https://docs.pola.rs/user-guide/io/csv/) - [Excel](https://docs.pola.rs/user-guide/io/excel/) - [Parquet](https://docs.pola.rs/user-guide/io/parquet/) - [JSON files](https://docs.pola.rs/user-guide/io/json/) - [Multiple](https://docs.pola.rs/user-guide/io/multiple/) - [Hive](https://docs.pola.rs/user-guide/io/hive/) - [Databases](https://docs.pola.rs/user-guide/io/database/) - [Cloud storage](https://docs.pola.rs/user-guide/io/cloud-storage/) - [Google BigQuery](https://docs.pola.rs/user-guide/io/bigquery/) - [Hugging Face](https://docs.pola.rs/user-guide/io/hugging-face/) - [Google Sheets (via Colab)](https://docs.pola.rs/user-guide/io/sheets_colab/) - [Plugins](https://docs.pola.rs/user-guide/plugins/) Plugins - [Expression Plugins](https://docs.pola.rs/user-guide/plugins/expr_plugins/) - [IO Plugins](https://docs.pola.rs/user-guide/plugins/io_plugins/) - SQL SQL - [Introduction](https://docs.pola.rs/user-guide/sql/intro/) - [SHOW TABLES](https://docs.pola.rs/user-guide/sql/show/) - [SELECT](https://docs.pola.rs/user-guide/sql/select/) - [CREATE](https://docs.pola.rs/user-guide/sql/create/) - [Common Table Expressions](https://docs.pola.rs/user-guide/sql/cte/) - Migrating Migrating - [Coming from Pandas](https://docs.pola.rs/user-guide/migration/pandas/) - [Coming from Apache Spark](https://docs.pola.rs/user-guide/migration/spark/) - Misc Misc - [Ecosystem](https://docs.pola.rs/user-guide/ecosystem/) - Multiprocessing [Multiprocessing](https://docs.pola.rs/user-guide/misc/multiprocessing/) Table of contents - [When not to use multiprocessing](https://docs.pola.rs/user-guide/misc/multiprocessing/#when-not-to-use-multiprocessing) - [When to use multiprocessing](https://docs.pola.rs/user-guide/misc/multiprocessing/#when-to-use-multiprocessing) - [The problem with the default multiprocessing config](https://docs.pola.rs/user-guide/misc/multiprocessing/#the-problem-with-the-default-multiprocessing-config) - [Summary](https://docs.pola.rs/user-guide/misc/multiprocessing/#summary) - [Example](https://docs.pola.rs/user-guide/misc/multiprocessing/#example) - [Pro's and cons of fork](https://docs.pola.rs/user-guide/misc/multiprocessing/#pros-and-cons-of-fork) - [References](https://docs.pola.rs/user-guide/misc/multiprocessing/#references) - [Visualization](https://docs.pola.rs/user-guide/misc/visualization/) - [Styling](https://docs.pola.rs/user-guide/misc/styling/) - [Comparison with other tools](https://docs.pola.rs/user-guide/misc/comparison/) - [Arrow producer/consumer](https://docs.pola.rs/user-guide/misc/arrow/) - [Generating Polars code with LLMs](https://docs.pola.rs/user-guide/misc/polars_llms/) - [GPU Support \[Open Beta\]](https://docs.pola.rs/user-guide/gpu-support/) - API API - [Reference guide](https://docs.pola.rs/api/reference/) - Development Development - [Contributing](https://docs.pola.rs/development/contributing/) Contributing - [IDE configuration](https://docs.pola.rs/development/contributing/ide/) - [Test suite](https://docs.pola.rs/development/contributing/test/) - [Continuous integration](https://docs.pola.rs/development/contributing/ci/) - [Code style](https://docs.pola.rs/development/contributing/code-style/) - [Versioning](https://docs.pola.rs/development/versioning/) - Releases Releases - [Upgrade guides](https://docs.pola.rs/releases/upgrade/) Upgrade guides - [Version 1](https://docs.pola.rs/releases/upgrade/1/) - [Version 0.20](https://docs.pola.rs/releases/upgrade/0.20/) - [Version 0.19](https://docs.pola.rs/releases/upgrade/0.19/) - [Changelog](https://docs.pola.rs/releases/changelog/) - [Polars Cloud](https://docs.pola.rs/polars-cloud/) Polars Cloud - [Getting started](https://docs.pola.rs/polars-cloud/quickstart/) - [Connect to your cloud](https://docs.pola.rs/polars-cloud/connect-cloud/) - Queries Queries - [Execute remote query](https://docs.pola.rs/polars-cloud/run/remote-query/) - [Distributed queries](https://docs.pola.rs/polars-cloud/run/distributed-engine/) - [Query profiling](https://docs.pola.rs/polars-cloud/run/query-profile/) - [Glossary](https://docs.pola.rs/polars-cloud/run/glossary/) - Integrations Integrations - [Orchestration](https://docs.pola.rs/polars-cloud/integrations/) Orchestration - [Airflow](https://docs.pola.rs/polars-cloud/integrations/airflow/) - [Dagster](https://docs.pola.rs/polars-cloud/integrations/dagster/) - [Prefect](https://docs.pola.rs/polars-cloud/integrations/prefect/) - [AWS Lambda](https://docs.pola.rs/polars-cloud/integrations/lambda/) - Concepts Concepts - Context Context - [Compute context introduction](https://docs.pola.rs/polars-cloud/context/compute-context/) - [Reconnect to compute cluster](https://docs.pola.rs/polars-cloud/context/reconnect/) - [Plugins and custom libraries](https://docs.pola.rs/polars-cloud/context/plugins/) - [Proxy mode](https://docs.pola.rs/polars-cloud/context/proxy-mode/) - Organizations Organizations - [Set up organization](https://docs.pola.rs/polars-cloud/organization/organizations/) - [Start trial period](https://docs.pola.rs/polars-cloud/organization/start-trial/) - [Payment and billing](https://docs.pola.rs/polars-cloud/organization/billing/) - [Manage members](https://docs.pola.rs/polars-cloud/organization/members/) - Workspaces Workspaces - [Workspace configuration](https://docs.pola.rs/polars-cloud/workspace/settings/) - [Manage team](https://docs.pola.rs/polars-cloud/workspace/team/) - Authentication Authentication - [Logging in](https://docs.pola.rs/polars-cloud/explain/authentication/) - [Using service accounts](https://docs.pola.rs/polars-cloud/explain/service-accounts/) - Providers Providers - AWS AWS - [Infrastructure](https://docs.pola.rs/polars-cloud/providers/aws/infra/) - [Permissions](https://docs.pola.rs/polars-cloud/providers/aws/permissions/) - Misc Misc - [CLI](https://docs.pola.rs/polars-cloud/cli/) - [Public datasets](https://docs.pola.rs/polars-cloud/public-datasets/) - [FAQ](https://docs.pola.rs/polars-cloud/faq/) - [API Reference](https://docs.cloud.pola.rs/) - API API - [Reference guide](https://docs.cloud.pola.rs/) - [Polars on-premises](https://docs.pola.rs/polars-on-premises/) Polars on-premises - Kubernetes Kubernetes - [Getting started](https://docs.pola.rs/polars-on-premises/kubernetes/getting-started/) - Bare-metal Bare-metal - [Getting started](https://docs.pola.rs/polars-on-premises/bare-metal/getting-started/) - Configuration Configuration - [Config file reference](https://docs.pola.rs/polars-on-premises/bare-metal/configuration/reference/) - [License](https://docs.pola.rs/polars-on-premises/bare-metal/configuration/license/) - [Profiling and host metrics](https://docs.pola.rs/polars-on-premises/bare-metal/configuration/monitoring/) - [Resource limits](https://docs.pola.rs/polars-on-premises/bare-metal/configuration/resource-limits/) - [Shuffle data](https://docs.pola.rs/polars-on-premises/bare-metal/configuration/shuffle-data/) - [Anonymous results](https://docs.pola.rs/polars-on-premises/bare-metal/configuration/anonymous-results/) - [Network addresses](https://docs.pola.rs/polars-on-premises/bare-metal/configuration/network-addresses/) - [Static leader configuration](https://docs.pola.rs/polars-on-premises/bare-metal/configuration/static-leader/) - [Example configurations](https://docs.pola.rs/polars-on-premises/bare-metal/configuration/example-configurations/) - [Environment variables](https://docs.pola.rs/polars-on-premises/bare-metal/environment-variables/) - [Python environment](https://docs.pola.rs/polars-on-premises/bare-metal/python-environment/) Table of contents - [When not to use multiprocessing](https://docs.pola.rs/user-guide/misc/multiprocessing/#when-not-to-use-multiprocessing) - [When to use multiprocessing](https://docs.pola.rs/user-guide/misc/multiprocessing/#when-to-use-multiprocessing) - [The problem with the default multiprocessing config](https://docs.pola.rs/user-guide/misc/multiprocessing/#the-problem-with-the-default-multiprocessing-config) - [Summary](https://docs.pola.rs/user-guide/misc/multiprocessing/#summary) - [Example](https://docs.pola.rs/user-guide/misc/multiprocessing/#example) - [Pro's and cons of fork](https://docs.pola.rs/user-guide/misc/multiprocessing/#pros-and-cons-of-fork) - [References](https://docs.pola.rs/user-guide/misc/multiprocessing/#references) # Multiprocessing TLDR: if you find that using Python's built-in `multiprocessing` module together with Polars results in a Polars error about multiprocessing methods, you should make sure you are using `spawn`, not `fork`, as the starting method: [Python](https://docs.pola.rs/user-guide/misc/multiprocessing/#__tabbed_1_1) ``` ``` ## When not to use multiprocessing Before we dive into the details, it is important to emphasize that Polars has been built from the start to use all your CPU cores. It does this by executing computations which can be done in parallel in separate threads. For example, requesting two expressions in a `select` statement can be done in parallel, with the results only being combined at the end. Another example is aggregating a value within groups using `group_by().agg(<expr>)`, each group can be evaluated separately. It is very unlikely that the `multiprocessing` module can improve your code performance in these cases. If you're using the GPU Engine with Polars you should also avoid manual multiprocessing. When used simultaneously, they can compete for system memory and processing power, leading to reduced performance. See [the optimizations section](https://docs.pola.rs/user-guide/lazy/optimizations/) for more optimizations. ## When to use multiprocessing Although Polars is multithreaded, other libraries may be single-threaded. When the other library is the bottleneck, and the problem at hand is parallelizable, it makes sense to use multiprocessing to gain a speed up. ## The problem with the default multiprocessing config ### Summary The [Python multiprocessing documentation](https://docs.python.org/3/library/multiprocessing.html) lists the three methods to create a process pool: 1. spawn 2. fork 3. forkserver The description of fork is (as of 2022-10-15): > The parent process uses os.fork() to fork the Python interpreter. The child process, when it begins, is effectively identical to the parent process. All resources of the parent are inherited by the child process. Note that safely forking a multithreaded process is problematic. > > Available on Unix only. The default on Unix. The short summary is: Polars is multithreaded as to provide strong performance out-of-the-box. Thus, it cannot be combined with `fork`. If you are on Unix (Linux, BSD, etc), you are using `fork`, unless you explicitly override it. The reason you may not have encountered this before is that pure Python code, and most Python libraries, are (mostly) single threaded. Alternatively, you are on Windows or MacOS, on which `fork` is not even available as a method (for MacOS it was up to Python 3.7). Thus one should use `spawn`, or `forkserver`, instead. `spawn` is available on all platforms and the safest choice, and hence the recommended method. ### Example The problem with `fork` is in the copying of the parent's process. Consider the example below, which is a slightly modified example posted on the [Polars issue tracker](https://github.com/pola-rs/polars/issues/3144): [Python](https://docs.pola.rs/user-guide/misc/multiprocessing/#__tabbed_2_1) ``` ``` Using `fork` as the method, instead of `spawn`, will cause a dead lock. The fork method is equivalent to calling `os.fork()`, which is a system call as defined in [the POSIX standard](https://pubs.opengroup.org/onlinepubs/9699919799/functions/fork.html): > A process shall be created with a single thread. If a multi-threaded process calls fork(), the new process shall contain a replica of the calling thread and its entire address space, possibly including the states of mutexes and other resources. Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of the exec functions is called. In contrast, `spawn` will create a completely new fresh Python interpreter, and not inherit the state of mutexes. So what happens in the code example? For reading the file with `pl.read_parquet` the file has to be locked. Then `os.fork()` is called, copying the state of the parent process, including mutexes. Thus all child processes will copy the file lock in an acquired state, leaving them hanging indefinitely waiting for the file lock to be released, which never happens. What makes debugging these issues tricky is that `fork` can work. Change the example to not having the call to `pl.read_parquet`: [Python](https://docs.pola.rs/user-guide/misc/multiprocessing/#__tabbed_3_1) ``` ``` This works fine. Therefore debugging these issues in larger code bases, i.e. not the small toy examples here, can be a real pain, as a seemingly unrelated change can break your multiprocessing code. In general, one should therefore never use the `fork` start method with multithreaded libraries unless there are very specific requirements that cannot be met otherwise. ### Pro's and cons of fork Based on the example, you may think, why is `fork` available in Python to start with? First, probably because of historical reasons: `spawn` was added to Python in version 3.4, whilst `fork` has been part of Python from the 2.x series. Second, there are several limitations for `spawn` and `forkserver` that do not apply to `fork`, in particular all arguments should be pickleable. See the [Python multiprocessing docs](https://docs.python.org/3/library/multiprocessing.html#the-spawn-and-forkserver-start-methods) for more information. Third, because it is faster to create new processes compared to `spawn`, as `spawn` is effectively `fork` + creating a brand new Python process without the locks by calling [execv](https://pubs.opengroup.org/onlinepubs/9699919799/functions/exec.html). Hence the warning in the Python docs that it is slower: there is more overhead to `spawn`. However, in almost all cases, one would like to use multiple processes to speed up computations that take multiple minutes or even hours, meaning the overhead is negligible in the grand scheme of things. And more importantly, it actually works in combination with multithreaded libraries. Fourth, `spawn` starts a new process, and therefore it requires code to be importable, in contrast to `fork`. In particular, this means that when using `spawn` the relevant code should not be in the global scope, such as in Jupyter notebooks or in plain scripts. Hence in the examples above, we define functions where we spawn within, and run those functions from a `__main__` clause. This is not an issue for typical projects, but during quick experimentation in notebooks it could fail. ## References 1. https://docs.python.org/3/library/multiprocessing.html 2. https://pythonspeed.com/articles/python-multiprocessing/ 3. https://pubs.opengroup.org/onlinepubs/9699919799/functions/fork.html 4. https://bnikolic.co.uk/blog/python/parallelism/2019/11/13/python-forkserver-preload.html [Previous Ecosystem](https://docs.pola.rs/user-guide/ecosystem/) [Next Visualization](https://docs.pola.rs/user-guide/misc/visualization/) Made with [Material for MkDocs](https://squidfunk.github.io/mkdocs-material/)
Readable Markdown	TLDR: if you find that using Python's built-in `multiprocessing` module together with Polars results in a Polars error about multiprocessing methods, you should make sure you are using `spawn`, not `fork`, as the starting method: ``` ``` ## When not to use multiprocessing Before we dive into the details, it is important to emphasize that Polars has been built from the start to use all your CPU cores. It does this by executing computations which can be done in parallel in separate threads. For example, requesting two expressions in a `select` statement can be done in parallel, with the results only being combined at the end. Another example is aggregating a value within groups using `group_by().agg(<expr>)`, each group can be evaluated separately. It is very unlikely that the `multiprocessing` module can improve your code performance in these cases. If you're using the GPU Engine with Polars you should also avoid manual multiprocessing. When used simultaneously, they can compete for system memory and processing power, leading to reduced performance. See [the optimizations section](https://docs.pola.rs/user-guide/lazy/optimizations/) for more optimizations. ## When to use multiprocessing Although Polars is multithreaded, other libraries may be single-threaded. When the other library is the bottleneck, and the problem at hand is parallelizable, it makes sense to use multiprocessing to gain a speed up. ## The problem with the default multiprocessing config ### Summary The [Python multiprocessing documentation](https://docs.python.org/3/library/multiprocessing.html) lists the three methods to create a process pool: 1. spawn 2. fork 3. forkserver The description of fork is (as of 2022-10-15): > The parent process uses os.fork() to fork the Python interpreter. The child process, when it begins, is effectively identical to the parent process. All resources of the parent are inherited by the child process. Note that safely forking a multithreaded process is problematic. > > Available on Unix only. The default on Unix. The short summary is: Polars is multithreaded as to provide strong performance out-of-the-box. Thus, it cannot be combined with `fork`. If you are on Unix (Linux, BSD, etc), you are using `fork`, unless you explicitly override it. The reason you may not have encountered this before is that pure Python code, and most Python libraries, are (mostly) single threaded. Alternatively, you are on Windows or MacOS, on which `fork` is not even available as a method (for MacOS it was up to Python 3.7). Thus one should use `spawn`, or `forkserver`, instead. `spawn` is available on all platforms and the safest choice, and hence the recommended method. ### Example The problem with `fork` is in the copying of the parent's process. Consider the example below, which is a slightly modified example posted on the [Polars issue tracker](https://github.com/pola-rs/polars/issues/3144): ``` ``` Using `fork` as the method, instead of `spawn`, will cause a dead lock. The fork method is equivalent to calling `os.fork()`, which is a system call as defined in [the POSIX standard](https://pubs.opengroup.org/onlinepubs/9699919799/functions/fork.html): > A process shall be created with a single thread. If a multi-threaded process calls fork(), the new process shall contain a replica of the calling thread and its entire address space, possibly including the states of mutexes and other resources. Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of the exec functions is called. In contrast, `spawn` will create a completely new fresh Python interpreter, and not inherit the state of mutexes. So what happens in the code example? For reading the file with `pl.read_parquet` the file has to be locked. Then `os.fork()` is called, copying the state of the parent process, including mutexes. Thus all child processes will copy the file lock in an acquired state, leaving them hanging indefinitely waiting for the file lock to be released, which never happens. What makes debugging these issues tricky is that `fork` can work. Change the example to not having the call to `pl.read_parquet`: ``` ``` This works fine. Therefore debugging these issues in larger code bases, i.e. not the small toy examples here, can be a real pain, as a seemingly unrelated change can break your multiprocessing code. In general, one should therefore never use the `fork` start method with multithreaded libraries unless there are very specific requirements that cannot be met otherwise. ### Pro's and cons of fork Based on the example, you may think, why is `fork` available in Python to start with? First, probably because of historical reasons: `spawn` was added to Python in version 3.4, whilst `fork` has been part of Python from the 2.x series. Second, there are several limitations for `spawn` and `forkserver` that do not apply to `fork`, in particular all arguments should be pickleable. See the [Python multiprocessing docs](https://docs.python.org/3/library/multiprocessing.html#the-spawn-and-forkserver-start-methods) for more information. Third, because it is faster to create new processes compared to `spawn`, as `spawn` is effectively `fork` + creating a brand new Python process without the locks by calling [execv](https://pubs.opengroup.org/onlinepubs/9699919799/functions/exec.html). Hence the warning in the Python docs that it is slower: there is more overhead to `spawn`. However, in almost all cases, one would like to use multiple processes to speed up computations that take multiple minutes or even hours, meaning the overhead is negligible in the grand scheme of things. And more importantly, it actually works in combination with multithreaded libraries. Fourth, `spawn` starts a new process, and therefore it requires code to be importable, in contrast to `fork`. In particular, this means that when using `spawn` the relevant code should not be in the global scope, such as in Jupyter notebooks or in plain scripts. Hence in the examples above, we define functions where we spawn within, and run those functions from a `__main__` clause. This is not an issue for typical projects, but during quick experimentation in notebooks it could fail. ## References 1. https://docs.python.org/3/library/multiprocessing.html 2. https://pythonspeed.com/articles/python-multiprocessing/ 3. https://pubs.opengroup.org/onlinepubs/9699919799/functions/fork.html 4. https://bnikolic.co.uk/blog/python/parallelism/2019/11/13/python-forkserver-preload.html
Shard	194 (laksa)
Root Hash	15785171083524017994
Unparsed URL	rs,pola!docs,/user-guide/misc/multiprocessing/ s443