🕷️ Crawler Inspector

[](https://towardsdatascience.com/) Publish AI, ML & data-science insights to a global community of data professionals. [Sign in]() [Submit an Article](https://contributor.insightmediagroup.io/) - [Latest](https://towardsdatascience.com/latest/) - [Editor’s Picks](https://towardsdatascience.com/tag/editors-pick/) - [Deep Dives](https://towardsdatascience.com/tag/deep-dives/) - [Newsletter](https://towardsdatascience.com/tag/the-variable/) - [Write For TDS](https://towardsdatascience.com/submissions/) Toggle Mobile Navigation - [LinkedIn](https://www.linkedin.com/company/towards-data-science/?originalSubdomain=ca) - [X](https://x.com/TDataScience) Toggle Search # Taming the Python Import System Do it wrong so you can do it right [Han Qi](https://towardsdatascience.com/author/hanqi01/) Apr 25, 2021 4 min read Share  Photo by [Glen Carrie](https://unsplash.com/@glencarrie?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com/?utm_source=medium&utm_medium=referral) Through the years I have come across quite a few import issues and only got them to work based on trial and error. Imports are difficult because there are so many factors interacting with each other to make things work when they shouldn’t, and make things fail/get a warning when they should not. I’m finally tired of that and this article (mainly the loom video) aims to show you the factors affecting the success and failure of imports through a series of experiments that demonstrate many situations you may come across. It may not cover all cases, but lays out some of the possible interactions and hopefully gets you knowledgeable enough to know what to look out for so you can debug during yet unforeseen circumstances. This article is a summary of the video for a refresher, also to help you decide whether to watch the video, first-timers will do better beginning with the video. ### The video <https://www.loom.com/share/efea0a982d1d4f73a4d8cd5837b484c7> ### What you will learn 1. 2 modes of running python files a. As a module `python -m main` b. As a script `python main.py` 2. Absolute vs Relative Imports 3. How you write the import statements determine success/failure of code 4. The current directory in terminal you run from, and which folder you open VSCode in, determines success/failure of code 5. What’s **init**.py for? ### 2 modes of running Running as a module with -m places the current directory in the 1st index in `sys.path` , running as a script places the immediate parent directory of the script in the 1st index in `sys.path` . These 2 paths may be the same when the project hierarchy is simple and not nested, or when you run the program from a directory right above the module (python file), so nothing to worry here. However, when these 2 paths start differing, you can print `sys.path` to debug. ### Absolute vs Relative Imports Absolute imports use the full path of the module from the project root, relative imports use dot notation. Absolute is more explicit on where the module exists in a project hierarchy, relative could be used if you don’t want to type long imports and don’t need to know the position of the module relative to the modules it is importing. When doing relative imports, running as a script will cause `__package__` to be None, thus having no point of reference for the relativity concept so it will fail. Running as a module may still fail if the module was at the top level, which causes `__package__` to be `''` empty string. For other cases where `__package__` correctly contains the package name (eg. folder), relative imports work fine. Absolute import failures can be fixed with quick hacks (usually when importing modules into jupyter notebooks) like `sys.path.insert(0,path_to_package)` but that will not help relative import failures which depend on `__package__` having the right value. Pandas uses absolute while Sklearn uses relative. **Pandas**: <https://github.com/pandas-dev/pandas/blob/master/pandas/core/groupby/groupby.py> **Sklearn**: <https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/metrics/pairwise.py> ### How you write the imports determine success/failure `from folder.utils import add` vs `from utils import add` are two different ways of writing imports. The former searches for `folder` to import while the latter searches for `utils` . When things break down, look at `sys.path` to see whether what you are importing can be found in any of the paths. ### The current directory in terminal you run from, and which folder you open VSCode in, determines success/failure The current directory in terminal from which you run as module or script affects which path gets prepended to `sys.path` , which affects whether the imports in code (also depends on how you wrote them) can be found. The folder you open VSCode in affects the red underline warnings saying "xxx can’t be resolved". They are an eyesore and can be fixed by opening VSCode in the correct directory.(Assuming you are in the correct environment with imported packages installed) ### What is **init**.py Helps useful modules/functions bubble up from deep within a project hierarchy to the top level, so users or other modules can call them with a simpler `from folder import add` rather than the deeper `from folder.utils import add` . A similar concept can be seen from my other article (<https://towardsdatascience.com/whats-the-difference-between-pd-merge-and-df-merge-ab387bc20a2e>), where the `merge` function is being bubbled up from the `pandas.core.reshape.merge` module into the `pandas.core.reshape.api` namespace through the imports in that file, then further bubbled up to the top level of `pandas.merge` through the top level `__init__.py` in the pandas package at [https://github.com/pandas-dev/pandas/blob/v0.25.1/pandas/**init**.py\#L129-L143](https://github.com/pandas-dev/pandas/blob/v0.25.1/pandas/__init__.py#L129-L143). I welcome any feedback in the comments on corrections or unclear explanations, which I’ll use to update this article. References: 1. <https://stackoverflow.com/questions/16981921/relative-imports-in-python-3> 2. <https://stackoverflow.com/questions/14132789/relative-imports-for-the-billionth-time> 3. <https://stackoverflow.com/questions/22241420/execution-of-python-code-with-m-option-or-not> 4. <https://stackoverflow.com/questions/21233229/whats-the-purpose-of-the-package-attribute-in-python/> 5. <https://stackoverflow.com/questions/448271/what-is-init-py-for> *** Written By Han Qi [See all from Han Qi](https://towardsdatascience.com/author/hanqi01/) [Pandas](https://towardsdatascience.com/tag/pandas/), [Programming Tips](https://towardsdatascience.com/tag/programming-tips/), [Python](https://towardsdatascience.com/tag/python/), [Scikit Learn](https://towardsdatascience.com/tag/scikit-learn/) Share This Article - [Share on Facebook](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Ftaming-the-python-import-system-fbee2bf0a1e4%2F&title=Taming%20the%20Python%20Import%20System) - [Share on LinkedIn](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Ftaming-the-python-import-system-fbee2bf0a1e4%2F&title=Taming%20the%20Python%20Import%20System) - [Share on X](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Ftaming-the-python-import-system-fbee2bf0a1e4%2F&text=Taming%20the%20Python%20Import%20System) Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. [Write for TDS](https://towardsdatascience.com/questions-96667b06af5/) ## Related Articles -  ## [Must-Know in Statistics: The Bivariate Normal Projection Explained](https://towardsdatascience.com/must-know-in-statistics-the-bivariate-normal-projection-explained-ace7b2f70b5b/) [Data Science](https://towardsdatascience.com/category/data-science/) Derivation and practical examples of this powerful concept [Luigi Battistoni](https://towardsdatascience.com/author/lu-battistoni/) August 14, 2024 7 min read -  ## [Squashing the Average: A Dive into Penalized Quantile Regression for Python](https://towardsdatascience.com/squashing-the-average-a-dive-into-penalized-quantile-regression-for-python-8f3a996768b6/) [Data Science](https://towardsdatascience.com/category/data-science/) How to build penalized quantile regression models (with code!) [Álvaro Méndez Civieta](https://towardsdatascience.com/author/alvaromc317/) August 16, 2024 5 min read -  ## [The Math Behind Keras 3 Optimizers: Deep Understanding and Application](https://towardsdatascience.com/the-math-behind-keras-3-optimizers-deep-understanding-and-application-2e5ff95eb342/) [Data Science](https://towardsdatascience.com/category/data-science/) This is a bit different from what the books say. [Peng Qian](https://towardsdatascience.com/author/qtalen/) August 17, 2024 9 min read -  ## [Python Meets Pawn 2: Clustering Chess Grandmasters based on their Openings](https://towardsdatascience.com/python-meets-pawn-2-clustering-chess-grandmasters-based-on-their-openings-68440fc9f9b1/) In this blog, I will guide you through the process of analyzing Chess Grandmasters’ openings… [Mikayil Ahadli](https://towardsdatascience.com/author/mikayilahad/) December 22, 2023 7 min read -  ## [Spotting Spatiotemporal Patterns in Earthquake Data](https://towardsdatascience.com/spotting-spatiotemporal-patterns-in-earthquake-data-b07068b84314/) Use density-based clustering and survival analysis to estimate when earthquakes occur [Elliot Humphrey](https://towardsdatascience.com/author/elz1582/) January 10, 2024 10 min read -  ## [Stacked Ensembles for Advanced Predictive Modeling With H2O.ai and Optuna](https://towardsdatascience.com/stacked-ensembles-for-advanced-predictive-modeling-with-h2o-ai-and-optuna-8c339f8fb602/) [Data Science](https://towardsdatascience.com/category/data-science/) And how I placed top 10% in Europe’s largest machine learning competition with them\! [Sheila Teo](https://towardsdatascience.com/author/sheilateozy/) December 18, 2023 15 min read -  ## [How to Read and Analyze GDAT Files Using Python](https://towardsdatascience.com/how-to-read-and-analyze-gdat-files-using-python-5c8dece157d4/) [Data Science](https://towardsdatascience.com/category/data-science/) A quick tutorial on how to work with these computer-modelled binary files. [Madison Hunter](https://towardsdatascience.com/author/madison13/) April 18, 2024 11 min read - [YouTube](https://www.youtube.com/c/TowardsDataScience) - [X](https://x.com/TDataScience) - [LinkedIn](https://www.linkedin.com/company/towards-data-science/?originalSubdomain=ca) - [Threads](https://www.threads.net/@towardsdatascience) - [Bluesky](https://bsky.app/profile/towardsdatascience.com) [](https://towardsdatascience.com/) Your home for data science and Al. The world’s leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. © Insight Media Group, LLC 2026 Subscribe to Our Newsletter - [Write For TDS](https://towardsdatascience.com/questions-96667b06af5/) - [About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/) - [Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science) - [Privacy Policy](https://towardsdatascience.com/privacy-policy/) - [Terms of Use](https://towardsdatascience.com/website-terms-of-use/) 

Through the years I have come across quite a few import issues and only got them to work based on trial and error. Imports are difficult because there are so many factors interacting with each other to make things work when they shouldn’t, and make things fail/get a warning when they should not. I’m finally tired of that and this article (mainly the loom video) aims to show you the factors affecting the success and failure of imports through a series of experiments that demonstrate many situations you may come across. It may not cover all cases, but lays out some of the possible interactions and hopefully gets you knowledgeable enough to know what to look out for so you can debug during yet unforeseen circumstances. This article is a summary of the video for a refresher, also to help you decide whether to watch the video, first-timers will do better beginning with the video. ### The video <https://www.loom.com/share/efea0a982d1d4f73a4d8cd5837b484c7> ### What you will learn 1. 2 modes of running python files a. As a module `python -m main` b. As a script `python main.py` 2. Absolute vs Relative Imports 3. How you write the import statements determine success/failure of code 4. The current directory in terminal you run from, and which folder you open VSCode in, determines success/failure of code 5. What’s **init**.py for? ### 2 modes of running Running as a module with -m places the current directory in the 1st index in `sys.path` , running as a script places the immediate parent directory of the script in the 1st index in `sys.path` . These 2 paths may be the same when the project hierarchy is simple and not nested, or when you run the program from a directory right above the module (python file), so nothing to worry here. However, when these 2 paths start differing, you can print `sys.path` to debug. ### Absolute vs Relative Imports Absolute imports use the full path of the module from the project root, relative imports use dot notation. Absolute is more explicit on where the module exists in a project hierarchy, relative could be used if you don’t want to type long imports and don’t need to know the position of the module relative to the modules it is importing. When doing relative imports, running as a script will cause `__package__` to be None, thus having no point of reference for the relativity concept so it will fail. Running as a module may still fail if the module was at the top level, which causes `__package__` to be `''` empty string. For other cases where `__package__` correctly contains the package name (eg. folder), relative imports work fine. Absolute import failures can be fixed with quick hacks (usually when importing modules into jupyter notebooks) like `sys.path.insert(0,path_to_package)` but that will not help relative import failures which depend on `__package__` having the right value. Pandas uses absolute while Sklearn uses relative. **Pandas**: <https://github.com/pandas-dev/pandas/blob/master/pandas/core/groupby/groupby.py> **Sklearn**: <https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/metrics/pairwise.py> ### How you write the imports determine success/failure `from folder.utils import add` vs `from utils import add` are two different ways of writing imports. The former searches for `folder` to import while the latter searches for `utils` . When things break down, look at `sys.path` to see whether what you are importing can be found in any of the paths. ### The current directory in terminal you run from, and which folder you open VSCode in, determines success/failure The current directory in terminal from which you run as module or script affects which path gets prepended to `sys.path` , which affects whether the imports in code (also depends on how you wrote them) can be found. The folder you open VSCode in affects the red underline warnings saying "xxx can’t be resolved". They are an eyesore and can be fixed by opening VSCode in the correct directory.(Assuming you are in the correct environment with imported packages installed) ### What is **init**.py Helps useful modules/functions bubble up from deep within a project hierarchy to the top level, so users or other modules can call them with a simpler `from folder import add` rather than the deeper `from folder.utils import add` . A similar concept can be seen from my other article (<https://towardsdatascience.com/whats-the-difference-between-pd-merge-and-df-merge-ab387bc20a2e>), where the `merge` function is being bubbled up from the `pandas.core.reshape.merge` module into the `pandas.core.reshape.api` namespace through the imports in that file, then further bubbled up to the top level of `pandas.merge` through the top level `__init__.py` in the pandas package at [https://github.com/pandas-dev/pandas/blob/v0.25.1/pandas/**init**.py\#L129-L143](https://github.com/pandas-dev/pandas/blob/v0.25.1/pandas/__init__.py#L129-L143). I welcome any feedback in the comments on corrections or unclear explanations, which I’ll use to update this article. References: 1. <https://stackoverflow.com/questions/16981921/relative-imports-in-python-3> 2. <https://stackoverflow.com/questions/14132789/relative-imports-for-the-billionth-time> 3. <https://stackoverflow.com/questions/22241420/execution-of-python-code-with-m-option-or-not> 4. <https://stackoverflow.com/questions/21233229/whats-the-purpose-of-the-package-attribute-in-python/> 5. <https://stackoverflow.com/questions/448271/what-is-init-py-for>