🕷️ Crawler Inspector

 - [Learn]() [Get Started](https://pytorch.org/get-started/locally) [Tutorials](https://pytorch.org/tutorials) [Learn the Basics](https://pytorch.org/tutorials/beginner/basics/intro.html) [PyTorch Recipes](https://pytorch.org/tutorials/recipes/recipes_index.html) [Intro to PyTorch - YouTube Series](https://pytorch.org/tutorials/beginner/introyt.html) [Webinars](https://pytorch.org/webinars/) - [Community]() [Landscape](https://landscape.pytorch.org/) [Join the Ecosystem](https://pytorch.org/join-ecosystem) [Community Hub](https://pytorch.org/community-hub/) [Forums](https://discuss.pytorch.org/) [Developer Resources](https://pytorch.org/resources) [Contributor Awards](https://pytorch.org/contributor-awards/) [Community Events](https://pytorch.org/community-events/) [PyTorch Ambassadors](https://pytorch.org/programs/ambassadors/) - [Projects]() [PyTorch](https://pytorch.org/projects/pytorch/) [vLLM](https://pytorch.org/projects/vllm/) [DeepSpeed](https://pytorch.org/projects/deepspeed/) [Host Your Project](https://pytorch.org/projects/host-your-project/) - [Docs]() [PyTorch](https://pytorch.org/docs/stable/index.html) [Domains](https://pytorch.org/domains) - [Blogs & News]() [Blog](https://pytorch.org/blog/) [Announcements](https://pytorch.org/announcements) [Case Studies](https://pytorch.org/case-studies/) [Events](https://pytorch.org/events) [Newsletter](https://pytorch.org/newsletter) - [About]() [PyTorch Foundation](https://pytorch.org/foundation) [Members](https://pytorch.org/members) [Governing Board](https://pytorch.org/governing-board) [Technical Advisory Council](https://pytorch.org/tac) [Cloud Credit Program](https://pytorch.org/credits) [Staff](https://pytorch.org/staff) [Contact](https://pytorch.org/contact) - [JOIN](https://pytorch.org/join) - [Learn]() - [Get Started](https://pytorch.org/get-started/locally) - [Tutorials](https://pytorch.org/tutorials) - [Learn the Basics](https://pytorch.org/tutorials/beginner/basics/intro.html) - [PyTorch Recipes](https://pytorch.org/tutorials/recipes/recipes_index.html) - [Introduction to PyTorch - YouTube Series](https://pytorch.org/tutorials/beginner/introyt.html) - [Webinars](https://pytorch.org/webinars/) - [Community]() - [Landscape](https://landscape.pytorch.org/) - [Join the Ecosystem](https://pytorch.org/join-ecosystem) - [Community Hub](https://pytorch.org/community-hub/) - [Forums](https://discuss.pytorch.org/) - [Developer Resources](https://pytorch.org/resources) - [Contributor Awards](https://pytorch.org/contributor-awards/) - [Community Events](https://pytorch.org/community-events/) - [PyTorch Ambassadors](https://pytorch.org/programs/ambassadors/) - [Projects]() - [PyTorch](https://pytorch.org/projects/pytorch/) - [vLLM](https://pytorch.org/projects/vllm/) - [DeepSpeed](https://pytorch.org/projects/deepspeed/) - [Host Your Project](https://pytorch.org/projects/host-your-project/) - [Docs]() - [PyTorch](https://pytorch.org/docs/stable/index.html) - [Domains](https://pytorch.org/domains) - [Blog & News]() - [Blog](https://pytorch.org/blog/) - [Announcements](https://pytorch.org/announcements) - [Case Studies](https://pytorch.org/case-studies/) - [Events](https://pytorch.org/events) - [Newsletter](https://pytorch.org/newsletter) - [About]() - [PyTorch Foundation](https://pytorch.org/foundation) - [Members](https://pytorch.org/members) - [Governing Board](https://pytorch.org/governing-board) - [Technical Advisory Council](https://pytorch.org/tac) - [Cloud Credit Program](https://pytorch.org/credits) - [Staff](https://pytorch.org/staff) - [Contact](https://pytorch.org/contact) [Skip to main content](https://docs.pytorch.org/docs/2.8/multiprocessing.html#main-content) Back to top [Home](https://docs.pytorch.org/docs/2.8/index.html) - [Python API](https://docs.pytorch.org/docs/2.8/pytorch-api.html) - [Developer Notes](https://docs.pytorch.org/docs/2.8/notes.html) - [Community](https://docs.pytorch.org/docs/2.8/community/index.html) - [C++](https://docs.pytorch.org/cppdocs/) - [Tutorials](https://pytorch.org/tutorials/) - [X](https://x.com/PyTorch "X") - [GitHub](https://github.com/pytorch/pytorch "GitHub") - [PyTorch Forum](https://discuss.pytorch.org/ "PyTorch Forum") - [PyPi](https://pypi.org/project/torch/ "PyPi") - [Python API](https://docs.pytorch.org/docs/2.8/pytorch-api.html) - [Developer Notes](https://docs.pytorch.org/docs/2.8/notes.html) - [Community](https://docs.pytorch.org/docs/2.8/community/index.html) - [C++](https://docs.pytorch.org/cppdocs/) - [Tutorials](https://pytorch.org/tutorials/) - [X](https://x.com/PyTorch "X") - [GitHub](https://github.com/pytorch/pytorch "GitHub") - [PyTorch Forum](https://discuss.pytorch.org/ "PyTorch Forum") - [PyPi](https://pypi.org/project/torch/ "PyPi") - Multiprocess... Rate this Page ★ ★ ★ ★ ★ # Multiprocessing package - torch.multiprocessing[\#](https://docs.pytorch.org/docs/2.8/multiprocessing.html#module-torch.multiprocessing "Permalink to this heading") Created On: Dec 23, 2016 \| Last Updated On: Jun 08, 2025 torch.multiprocessing is a wrapper around the native [`multiprocessing`](https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing "(in Python v3.13)") module. It registers custom reducers, that use shared memory to provide shared views on the same data in different processes. Once the tensor/storage is moved to shared\_memory (see [`share_memory_()`](https://docs.pytorch.org/docs/2.8/generated/torch.Tensor.share_memory_.html#torch.Tensor.share_memory_ "torch.Tensor.share_memory_")), it will be possible to send it to other processes without making any copies. The API is 100% compatible with the original module - it’s enough to change `import multiprocessing` to `import torch.multiprocessing` to have all the tensors sent through the queues or shared via other mechanisms, moved to shared memory. Because of the similarity of APIs we do not document most of this package contents, and we recommend referring to very good docs of the original module. Warning If the main process exits abruptly (e.g. because of an incoming signal), Python’s `multiprocessing` sometimes fails to clean up its children. It’s a known caveat, so if you’re seeing any resource leaks after interrupting the interpreter, it probably means that this has just happened to you. ## Strategy management[\#](https://docs.pytorch.org/docs/2.8/multiprocessing.html#strategy-management "Permalink to this heading") torch.multiprocessing.get\_all\_sharing\_strategies()[\[source\]](https://github.com/pytorch/pytorch/blob/v2.8.0/torch/multiprocessing/__init__.py#L78)[\#](https://docs.pytorch.org/docs/2.8/multiprocessing.html#torch.multiprocessing.get_all_sharing_strategies "Permalink to this definition") Return a set of sharing strategies supported on a current system. torch.multiprocessing.get\_sharing\_strategy()[\[source\]](https://github.com/pytorch/pytorch/blob/v2.8.0/torch/multiprocessing/__init__.py#L73)[\#](https://docs.pytorch.org/docs/2.8/multiprocessing.html#torch.multiprocessing.get_sharing_strategy "Permalink to this definition") Return the current strategy for sharing CPU tensors. torch.multiprocessing.set\_sharing\_strategy(*new\_strategy*)[\[source\]](https://github.com/pytorch/pytorch/blob/v2.8.0/torch/multiprocessing/__init__.py#L61)[\#](https://docs.pytorch.org/docs/2.8/multiprocessing.html#torch.multiprocessing.set_sharing_strategy "Permalink to this definition") Set the strategy for sharing CPU tensors. Parameters **new\_strategy** ([*str*](https://docs.python.org/3/library/stdtypes.html#str "(in Python v3.13)")) – Name of the selected strategy. Should be one of the values returned by [`get_all_sharing_strategies()`](https://docs.pytorch.org/docs/2.8/multiprocessing.html#torch.multiprocessing.get_all_sharing_strategies "torch.multiprocessing.get_all_sharing_strategies"). ## Sharing CUDA tensors[\#](https://docs.pytorch.org/docs/2.8/multiprocessing.html#sharing-cuda-tensors "Permalink to this heading") Sharing CUDA tensors between processes is supported only in Python 3, using a `spawn` or `forkserver` start methods. Unlike CPU tensors, the sending process is required to keep the original tensor as long as the receiving process retains a copy of the tensor. The refcounting is implemented under the hood but requires users to follow the next best practices. Warning If the consumer process dies abnormally to a fatal signal, the shared tensor could be forever kept in memory as long as the sending process is running. 1. Release memory ASAP in the consumer. ``` ## Good x = queue.get() # do somethings with x del x ``` ``` ## Bad x = queue.get() # do somethings with x # do everything else (producer have to keep x in memory) ``` 1. Keep producer process running until all consumers exits. This will prevent the situation when the producer process releasing memory which is still in use by the consumer. ``` ## producer # send tensors, do something event.wait() ``` ``` ## consumer # receive tensors and use them event.set() ``` 1. Don’t pass received tensors. ``` # not going to work x = queue.get() queue_2.put(x) ``` ``` # you need to create a process-local copy x = queue.get() x_clone = x.clone() queue_2.put(x_clone) ``` ``` # putting and getting from the same queue in the same process will likely end up with segfault queue.put(tensor) x = queue.get() ``` ## Sharing strategies[\#](https://docs.pytorch.org/docs/2.8/multiprocessing.html#sharing-strategies "Permalink to this heading") This section provides a brief overview into how different sharing strategies work. Note that it applies only to CPU tensor - CUDA tensors will always use the CUDA API, as that’s the only way they can be shared. ### File descriptor - `file_descriptor`[\#](https://docs.pytorch.org/docs/2.8/multiprocessing.html#file-descriptor-file-descriptor "Permalink to this heading") Note This is the default strategy (except for macOS and OS X where it’s not supported). This strategy will use file descriptors as shared memory handles. Whenever a storage is moved to shared memory, a file descriptor obtained from `shm_open` is cached with the object, and when it’s going to be sent to other processes, the file descriptor will be transferred (e.g. via UNIX sockets) to it. The receiver will also cache the file descriptor and `mmap` it, to obtain a shared view onto the storage data. Note that if there will be a lot of tensors shared, this strategy will keep a large number of file descriptors open most of the time. If your system has low limits for the number of open file descriptors, and you can’t raise them, you should use the `file_system` strategy. ### File system - `file_system`[\#](https://docs.pytorch.org/docs/2.8/multiprocessing.html#file-system-file-system "Permalink to this heading") This strategy will use file names given to `shm_open` to identify the shared memory regions. This has a benefit of not requiring the implementation to cache the file descriptors obtained from it, but at the same time is prone to shared memory leaks. The file can’t be deleted right after its creation, because other processes need to access it to open their views. If the processes fatally crash, or are killed, and don’t call the storage destructors, the files will remain in the system. This is very serious, because they keep using up the memory until the system is restarted, or they’re freed manually. To counter the problem of shared memory file leaks, [`torch.multiprocessing`](https://docs.pytorch.org/docs/2.8/multiprocessing.html#module-torch.multiprocessing "torch.multiprocessing") will spawn a daemon named `torch_shm_manager` that will isolate itself from the current process group, and will keep track of all shared memory allocations. Once all processes connected to it exit, it will wait a moment to ensure there will be no new connections, and will iterate over all shared memory files allocated by the group. If it finds that any of them still exist, they will be deallocated. We’ve tested this method and it proved to be robust to various failures. Still, if your system has high enough limits, and `file_descriptor` is a supported strategy, we do not recommend switching to this one. ## Spawning subprocesses[\#](https://docs.pytorch.org/docs/2.8/multiprocessing.html#spawning-subprocesses "Permalink to this heading") Note Available for Python \>= 3.4. This depends on the `spawn` start method in Python’s `multiprocessing` package. Spawning a number of subprocesses to perform some function can be done by creating `Process` instances and calling `join` to wait for their completion. This approach works fine when dealing with a single subprocess but presents potential issues when dealing with multiple processes. Namely, joining processes sequentially implies they will terminate sequentially. If they don’t, and the first process does not terminate, the process termination will go unnoticed. Also, there are no native facilities for error propagation. The `spawn` function below addresses these concerns and takes care of error propagation, out of order termination, and will actively terminate processes upon detecting an error in one of them. torch.multiprocessing.spawn.spawn(*fn*, *args\=()*, *nprocs\=1*, *join\=True*, *daemon\=False*, *start\_method\='spawn'*)[\[source\]](https://github.com/pytorch/pytorch/blob/v2.8.0/torch/multiprocessing/spawn.py#L300)[\#](https://docs.pytorch.org/docs/2.8/multiprocessing.html#torch.multiprocessing.spawn.spawn "Permalink to this definition") Spawns `nprocs` processes that run `fn` with `args`. If one of the processes exits with a non-zero exit status, the remaining processes are killed and an exception is raised with the cause of termination. In the case an exception was caught in the child process, it is forwarded and its traceback is included in the exception raised in the parent process. Parameters - **fn** (*function*) – Function is called as the entrypoint of the spawned process. This function must be defined at the top level of a module so it can be pickled and spawned. This is a requirement imposed by multiprocessing. The function is called as `fn(i, *args)`, where `i` is the process index and `args` is the passed through tuple of arguments. - **args** ([*tuple*](https://docs.python.org/3/library/stdtypes.html#tuple "(in Python v3.13)")) – Arguments passed to `fn`. - **nprocs** ([*int*](https://docs.python.org/3/library/functions.html#int "(in Python v3.13)")) – Number of processes to spawn. - **join** ([*bool*](https://docs.python.org/3/library/functions.html#bool "(in Python v3.13)")) – Perform a blocking join on all processes. - **daemon** ([*bool*](https://docs.python.org/3/library/functions.html#bool "(in Python v3.13)")) – The spawned processes’ daemon flag. If set to True, daemonic processes will be created. - **start\_method** ([*str*](https://docs.python.org/3/library/stdtypes.html#str "(in Python v3.13)")) – (deprecated) this method will always use `spawn` as the start method. To use a different start method use `start_processes()`. Returns None if `join` is `True`, `ProcessContext` if `join` is `False` *class* torch.multiprocessing.SpawnContext[\[source\]](https://github.com/pytorch/pytorch/blob/v2.8.0/torch/multiprocessing/spawn.py#L218)[\#](https://docs.pytorch.org/docs/2.8/multiprocessing.html#torch.multiprocessing.SpawnContext "Permalink to this definition") Returned by [`spawn()`](https://docs.pytorch.org/docs/2.8/multiprocessing.html#module-torch.multiprocessing.spawn "torch.multiprocessing.spawn") when called with `join=False`. join(*timeout\=None*, *grace\_period\=None*)[\[source\]](https://github.com/pytorch/pytorch/blob/v2.8.0/torch/multiprocessing/spawn.py#L120)[\#](https://docs.pytorch.org/docs/2.8/multiprocessing.html#torch.multiprocessing.SpawnContext.join "Permalink to this definition") Join one or more processes within spawn context. Attempt to join one or more processes in this spawn context. If one of them exited with a non-zero exit status, this function kills the remaining processes (optionally with a grace period) and raises an exception with the cause of the first process exiting. Returns `True` if all processes have been joined successfully, `False` if there are more processes that need to be joined. Parameters - **timeout** ([*float*](https://docs.python.org/3/library/functions.html#float "(in Python v3.13)")) – Wait this long (in seconds) before giving up on waiting. - **grace\_period** ([*float*](https://docs.python.org/3/library/functions.html#float "(in Python v3.13)")) – When any processes fail, wait this long (in seconds) for others to shutdown gracefully before terminating them. If they still don’t exit, wait another grace period before killing them. Rate this Page ★ ★ ★ ★ ★ Send Feedback © Copyright PyTorch Contributors. Built with the [PyData Sphinx Theme](https://pydata-sphinx-theme.readthedocs.io/en/stable/index.html) 0.15.4. On this page - [Strategy management](https://docs.pytorch.org/docs/2.8/multiprocessing.html#strategy-management) - [`get_all_sharing_strategies()`](https://docs.pytorch.org/docs/2.8/multiprocessing.html#torch.multiprocessing.get_all_sharing_strategies) - [`get_sharing_strategy()`](https://docs.pytorch.org/docs/2.8/multiprocessing.html#torch.multiprocessing.get_sharing_strategy) - [`set_sharing_strategy()`](https://docs.pytorch.org/docs/2.8/multiprocessing.html#torch.multiprocessing.set_sharing_strategy) - [Sharing CUDA tensors](https://docs.pytorch.org/docs/2.8/multiprocessing.html#sharing-cuda-tensors) - [Sharing strategies](https://docs.pytorch.org/docs/2.8/multiprocessing.html#sharing-strategies) - [File descriptor - `file_descriptor`](https://docs.pytorch.org/docs/2.8/multiprocessing.html#file-descriptor-file-descriptor) - [File system - `file_system`](https://docs.pytorch.org/docs/2.8/multiprocessing.html#file-system-file-system) - [Spawning subprocesses](https://docs.pytorch.org/docs/2.8/multiprocessing.html#spawning-subprocesses) - [`spawn()`](https://docs.pytorch.org/docs/2.8/multiprocessing.html#torch.multiprocessing.spawn.spawn) - [`SpawnContext`](https://docs.pytorch.org/docs/2.8/multiprocessing.html#torch.multiprocessing.SpawnContext) - [`SpawnContext.join()`](https://docs.pytorch.org/docs/2.8/multiprocessing.html#torch.multiprocessing.SpawnContext.join) [Edit on GitHub](https://github.com/pytorch/pytorch/edit/main/docs/source/multiprocessing.md) [Show Source](https://docs.pytorch.org/docs/2.8/_sources/multiprocessing.md.txt) PyTorch Libraries - [torchao](https://docs.pytorch.org/ao) - [torchrec](https://docs.pytorch.org/torchrec) - [torchft](https://docs.pytorch.org/torchft) - [TorchCodec](https://docs.pytorch.org/torchcodec) - [torchvision](https://docs.pytorch.org/vision) - [ExecuTorch](https://docs.pytorch.org/executorch) - [PyTorch on XLA Devices](https://docs.pytorch.org/xla) ## Docs Access comprehensive developer documentation for PyTorch [View Docs](https://pytorch.org/docs/stable/index.html) ## Tutorials Get in-depth tutorials for beginners and advanced developers [View Tutorials](https://pytorch.org/tutorials) ## Resources Find development resources and get your questions answered [View Resources](https://pytorch.org/resources) **Stay in touch** for updates, event info, and the latest news By submitting this form, I consent to receive marketing emails from the LF and its projects regarding their events, training, research, developments, and related announcements. I understand that I can unsubscribe at any time using the links in the footers of the emails I receive. [Privacy Policy](https://www.linuxfoundation.org/privacy/). © PyTorch. Copyright © The Linux Foundation®. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For more information, including terms of use, privacy policy, and trademark usage, please see our [Policies](https://www.linuxfoundation.org/legal/policies) page. [Trademark Usage](https://www.linuxfoundation.org/trademark-usage). [Privacy Policy](http://www.linuxfoundation.org/privacy). To analyze traffic and optimize your experience, we serve cookies on this site. By clicking or navigating, you agree to allow our usage of cookies. As the current maintainers of this site, Facebook’s Cookies Policy applies. Learn more, including about available controls: [Cookies Policy](https://www.facebook.com/policies/cookies/).  To analyze traffic and optimize your experience, we serve cookies on this site. By clicking or navigating, you agree to allow our usage of cookies. As the current maintainers of this site, Facebook’s Cookies Policy applies. Learn more, including about available controls: [Cookies Policy](https://www.facebook.com/policies/cookies/).  © Copyright PyTorch Contributors. Created using [Sphinx](https://www.sphinx-doc.org/) 5.3.0. Built with the [PyData Sphinx Theme](https://pydata-sphinx-theme.readthedocs.io/en/stable/index.html) 0.15.4.

Created On: Dec 23, 2016 \| Last Updated On: Jun 08, 2025 torch.multiprocessing is a wrapper around the native [`multiprocessing`](https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing "(in Python v3.13)") module. It registers custom reducers, that use shared memory to provide shared views on the same data in different processes. Once the tensor/storage is moved to shared\_memory (see [`share_memory_()`](https://docs.pytorch.org/docs/2.8/generated/torch.Tensor.share_memory_.html#torch.Tensor.share_memory_ "torch.Tensor.share_memory_")), it will be possible to send it to other processes without making any copies. The API is 100% compatible with the original module - it’s enough to change `import multiprocessing` to `import torch.multiprocessing` to have all the tensors sent through the queues or shared via other mechanisms, moved to shared memory. Because of the similarity of APIs we do not document most of this package contents, and we recommend referring to very good docs of the original module. Warning If the main process exits abruptly (e.g. because of an incoming signal), Python’s `multiprocessing` sometimes fails to clean up its children. It’s a known caveat, so if you’re seeing any resource leaks after interrupting the interpreter, it probably means that this has just happened to you. ## Strategy management[\#](https://docs.pytorch.org/docs/2.8/multiprocessing.html#strategy-management "Permalink to this heading") torch.multiprocessing.get\_all\_sharing\_strategies()[\[source\]](https://github.com/pytorch/pytorch/blob/v2.8.0/torch/multiprocessing/__init__.py#L78)[\#](https://docs.pytorch.org/docs/2.8/multiprocessing.html#torch.multiprocessing.get_all_sharing_strategies "Permalink to this definition") Return a set of sharing strategies supported on a current system. torch.multiprocessing.get\_sharing\_strategy()[\[source\]](https://github.com/pytorch/pytorch/blob/v2.8.0/torch/multiprocessing/__init__.py#L73)[\#](https://docs.pytorch.org/docs/2.8/multiprocessing.html#torch.multiprocessing.get_sharing_strategy "Permalink to this definition") Return the current strategy for sharing CPU tensors. torch.multiprocessing.set\_sharing\_strategy(*new\_strategy*)[\[source\]](https://github.com/pytorch/pytorch/blob/v2.8.0/torch/multiprocessing/__init__.py#L61)[\#](https://docs.pytorch.org/docs/2.8/multiprocessing.html#torch.multiprocessing.set_sharing_strategy "Permalink to this definition") Set the strategy for sharing CPU tensors. Parameters **new\_strategy** ([*str*](https://docs.python.org/3/library/stdtypes.html#str "(in Python v3.13)")) – Name of the selected strategy. Should be one of the values returned by [`get_all_sharing_strategies()`](https://docs.pytorch.org/docs/2.8/multiprocessing.html#torch.multiprocessing.get_all_sharing_strategies "torch.multiprocessing.get_all_sharing_strategies"). ## Sharing CUDA tensors[\#](https://docs.pytorch.org/docs/2.8/multiprocessing.html#sharing-cuda-tensors "Permalink to this heading") Sharing CUDA tensors between processes is supported only in Python 3, using a `spawn` or `forkserver` start methods. Unlike CPU tensors, the sending process is required to keep the original tensor as long as the receiving process retains a copy of the tensor. The refcounting is implemented under the hood but requires users to follow the next best practices. Warning If the consumer process dies abnormally to a fatal signal, the shared tensor could be forever kept in memory as long as the sending process is running. 1. Release memory ASAP in the consumer. ``` ## Good x = queue.get() # do somethings with x del x ``` ``` ## Bad x = queue.get() # do somethings with x # do everything else (producer have to keep x in memory) ``` 1. Keep producer process running until all consumers exits. This will prevent the situation when the producer process releasing memory which is still in use by the consumer. ``` ## producer # send tensors, do something event.wait() ``` ``` ## consumer # receive tensors and use them event.set() ``` 1. Don’t pass received tensors. ``` # not going to work x = queue.get() queue_2.put(x) ``` ``` # you need to create a process-local copy x = queue.get() x_clone = x.clone() queue_2.put(x_clone) ``` ``` # putting and getting from the same queue in the same process will likely end up with segfault queue.put(tensor) x = queue.get() ``` ## Sharing strategies[\#](https://docs.pytorch.org/docs/2.8/multiprocessing.html#sharing-strategies "Permalink to this heading") This section provides a brief overview into how different sharing strategies work. Note that it applies only to CPU tensor - CUDA tensors will always use the CUDA API, as that’s the only way they can be shared. ### File descriptor - `file_descriptor`[\#](https://docs.pytorch.org/docs/2.8/multiprocessing.html#file-descriptor-file-descriptor "Permalink to this heading") Note This is the default strategy (except for macOS and OS X where it’s not supported). This strategy will use file descriptors as shared memory handles. Whenever a storage is moved to shared memory, a file descriptor obtained from `shm_open` is cached with the object, and when it’s going to be sent to other processes, the file descriptor will be transferred (e.g. via UNIX sockets) to it. The receiver will also cache the file descriptor and `mmap` it, to obtain a shared view onto the storage data. Note that if there will be a lot of tensors shared, this strategy will keep a large number of file descriptors open most of the time. If your system has low limits for the number of open file descriptors, and you can’t raise them, you should use the `file_system` strategy. ### File system - `file_system`[\#](https://docs.pytorch.org/docs/2.8/multiprocessing.html#file-system-file-system "Permalink to this heading") This strategy will use file names given to `shm_open` to identify the shared memory regions. This has a benefit of not requiring the implementation to cache the file descriptors obtained from it, but at the same time is prone to shared memory leaks. The file can’t be deleted right after its creation, because other processes need to access it to open their views. If the processes fatally crash, or are killed, and don’t call the storage destructors, the files will remain in the system. This is very serious, because they keep using up the memory until the system is restarted, or they’re freed manually. To counter the problem of shared memory file leaks, [`torch.multiprocessing`](https://docs.pytorch.org/docs/2.8/multiprocessing.html#module-torch.multiprocessing "torch.multiprocessing") will spawn a daemon named `torch_shm_manager` that will isolate itself from the current process group, and will keep track of all shared memory allocations. Once all processes connected to it exit, it will wait a moment to ensure there will be no new connections, and will iterate over all shared memory files allocated by the group. If it finds that any of them still exist, they will be deallocated. We’ve tested this method and it proved to be robust to various failures. Still, if your system has high enough limits, and `file_descriptor` is a supported strategy, we do not recommend switching to this one. ## Spawning subprocesses[\#](https://docs.pytorch.org/docs/2.8/multiprocessing.html#spawning-subprocesses "Permalink to this heading") Note Available for Python \>= 3.4. This depends on the `spawn` start method in Python’s `multiprocessing` package. Spawning a number of subprocesses to perform some function can be done by creating `Process` instances and calling `join` to wait for their completion. This approach works fine when dealing with a single subprocess but presents potential issues when dealing with multiple processes. Namely, joining processes sequentially implies they will terminate sequentially. If they don’t, and the first process does not terminate, the process termination will go unnoticed. Also, there are no native facilities for error propagation. The `spawn` function below addresses these concerns and takes care of error propagation, out of order termination, and will actively terminate processes upon detecting an error in one of them. torch.multiprocessing.spawn.spawn(*fn*, *args\=()*, *nprocs\=1*, *join\=True*, *daemon\=False*, *start\_method\='spawn'*)[\[source\]](https://github.com/pytorch/pytorch/blob/v2.8.0/torch/multiprocessing/spawn.py#L300)[\#](https://docs.pytorch.org/docs/2.8/multiprocessing.html#torch.multiprocessing.spawn.spawn "Permalink to this definition") Spawns `nprocs` processes that run `fn` with `args`. If one of the processes exits with a non-zero exit status, the remaining processes are killed and an exception is raised with the cause of termination. In the case an exception was caught in the child process, it is forwarded and its traceback is included in the exception raised in the parent process. Parameters - **fn** (*function*) – Function is called as the entrypoint of the spawned process. This function must be defined at the top level of a module so it can be pickled and spawned. This is a requirement imposed by multiprocessing. The function is called as `fn(i, *args)`, where `i` is the process index and `args` is the passed through tuple of arguments. - **args** ([*tuple*](https://docs.python.org/3/library/stdtypes.html#tuple "(in Python v3.13)")) – Arguments passed to `fn`. - **nprocs** ([*int*](https://docs.python.org/3/library/functions.html#int "(in Python v3.13)")) – Number of processes to spawn. - **join** ([*bool*](https://docs.python.org/3/library/functions.html#bool "(in Python v3.13)")) – Perform a blocking join on all processes. - **daemon** ([*bool*](https://docs.python.org/3/library/functions.html#bool "(in Python v3.13)")) – The spawned processes’ daemon flag. If set to True, daemonic processes will be created. - **start\_method** ([*str*](https://docs.python.org/3/library/stdtypes.html#str "(in Python v3.13)")) – (deprecated) this method will always use `spawn` as the start method. To use a different start method use `start_processes()`. Returns None if `join` is `True`, `ProcessContext` if `join` is `False` *class* torch.multiprocessing.SpawnContext[\[source\]](https://github.com/pytorch/pytorch/blob/v2.8.0/torch/multiprocessing/spawn.py#L218)[\#](https://docs.pytorch.org/docs/2.8/multiprocessing.html#torch.multiprocessing.SpawnContext "Permalink to this definition") Returned by [`spawn()`](https://docs.pytorch.org/docs/2.8/multiprocessing.html#module-torch.multiprocessing.spawn "torch.multiprocessing.spawn") when called with `join=False`. join(*timeout\=None*, *grace\_period\=None*)[\[source\]](https://github.com/pytorch/pytorch/blob/v2.8.0/torch/multiprocessing/spawn.py#L120)[\#](https://docs.pytorch.org/docs/2.8/multiprocessing.html#torch.multiprocessing.SpawnContext.join "Permalink to this definition") Join one or more processes within spawn context. Attempt to join one or more processes in this spawn context. If one of them exited with a non-zero exit status, this function kills the remaining processes (optionally with a grace period) and raises an exception with the cause of the first process exiting. Returns `True` if all processes have been joined successfully, `False` if there are more processes that need to be joined. Parameters - **timeout** ([*float*](https://docs.python.org/3/library/functions.html#float "(in Python v3.13)")) – Wait this long (in seconds) before giving up on waiting. - **grace\_period** ([*float*](https://docs.python.org/3/library/functions.html#float "(in Python v3.13)")) – When any processes fail, wait this long (in seconds) for others to shutdown gracefully before terminating them. If they still don’t exit, wait another grace period before killing them.