🕷️ Crawler Inspector

 .png) #### Discover more from daily.dev Personalized news feed, dev communities and search, much better than what’s out there. Maybe ;) [**Start reading** - Free forever](https://play.google.com/store/apps/details?id=dev.daily) [**Start reading** - Free forever](https://api.daily.dev/get) Continue reading \> [](https://daily.dev/?r=0)[Start reading **\- Free forever**](https://api.daily.dev/get)[Start now](https://app.daily.dev/?utm_source=landing&utm_medium=cta&utm_campaign=landing_conversions)  [Home](https://daily.dev/?r=0)[Blog](https://daily.dev/blog-superold)[Get into tech](https://daily.dev/categories/get-into-tech) # Get to know Asynchio: Multithreaded Python using async/await Apr 9, 2024  Author [Nimrod Kramer @NimrodKramer](https://app.daily.dev/kramer) Related tags on daily.dev  Table of contents  [Read more on daily.dev](https://app.daily.dev/?utm_source=landing&utm_medium=blog&utm_campaign=toc) 🎯 Discover the power of Asynchio, a technique for multithreaded programming in Python using async/await. Learn about asynchronous programming basics, evolution in Python, practical applications, and performance considerations. Discover the power of `Asynchio`, a technique for multithreaded programming in Python using async/await. This method can significantly speed up your Python applications by allowing them to perform multiple tasks simultaneously, much like a well-coordinated kitchen staff. Here's a quick rundown of what you'll learn: - **Asynchronous Programming Basics**: Understand the concept and why it's faster and more efficient. - **Evolution in Python**: How `asyncio` and `async`/`await` have simplified multitasking in Python. - **Using async/await**: Learn to write functions that can do several things at once without waiting in line. - **Multithreading and [asyncio](https://pymotw.com/3/asyncio/)**: How to combine them for even better performance. - **Practical Applications**: Examples including web crawling and database access. - **Performance Considerations**: Tips on measuring impact and avoiding common pitfalls. In simpler terms, this article will guide you through making your Python programs multitask efficiently, just like having multiple chefs in a kitchen, ensuring everything runs smoothly and swiftly. ### What is Asynchronous Programming? Imagine you're trying to do several chores at home. In asynchronous programming, instead of finishing one chore before starting the next, you start multiple chores and switch between them as needed. For example, while laundry is washing, you start cooking. This way, you get more done in less time. Here’s why asynchronous programming is great: - **It’s faster** - It lets your computer work on something else while waiting for a task to finish, like waiting for a webpage to load. - **It can handle more** - Your program can deal with lots of users or tasks at the same time without getting bogged down. - **It keeps things moving** - Even if one task is waiting, the rest of your program can keep running smoothly, making sure things like apps or websites don’t freeze up. Normally, programs do one thing at a time, waiting for each task to finish before starting the next. But with asynchronous programming, tasks can pause and pick up later, letting other tasks run in the meantime. ### The Evolution of Asynchronous Programming in Python Python used to handle tasks one after another or required some tricky coding to do many things at once. But then came `asyncio` with Python 3.4, introducing a simpler way to write code that can do multiple tasks at once without getting tangled up. For instance: ``` import asyncio async def fetch_data(): print('fetching...') await asyncio.sleep(2) # lets other tasks run while waiting print('done!') asyncio.run(fetch_data()) ``` With `asyncio`, writing code that can do several things at once became a lot easier. It’s like telling your Python program, "Keep doing other stuff while you’re waiting." This is super useful for things like web apps, where you want to handle lots of users at the same time without slowing down. ## Diving into Async/Await ### The async Keyword The `async` keyword helps you create functions in Python that can do several things at once, without waiting in line. Think of it like a chef who starts cooking multiple dishes simultaneously, not just one at a time. Here's a simple way to use `async`: ``` async def fetch_data(): response = await make_api_call() return response ``` Important points about `async`: - It marks a function as ready to handle several tasks together. - These functions can take a break with `await` when they're waiting for something. - When you call an async function, it prepares but doesn't start right away. - To actually run it, you use `asyncio.run()`. Good habits with `async`: - It's best for tasks that have to wait, like getting data from the internet. - Mix async with regular tasks to keep things balanced. - Run multiple async tasks at the same time with `asyncio.gather()`. ### The await Keyword You can only use `await` inside those special `async` functions. It's like hitting pause on a movie: the function waits right there until its task is done, letting other things happen in the meantime. Example of `await` in action: ``` async def get_data(): data = await fetch_api_data() # Waits here print(data) ``` Key points about `await`: - It's for calling functions that are also using `async`. - Your code will wait at each `await` until that task is finished. - This waiting lets other tasks keep going, so everything runs smoothly. - Don't use `await` in regular functions; it can cause jams. Good habits with `await`: - Use it for big tasks that take time, so they don't hold up everything else. - Too many `await` can slow things down, so balance is key. - To wait on several tasks at once, `asyncio.gather()` is your friend. ### Managing Asynchronous Tasks `asyncio` has tools for running and keeping an eye on many tasks at once. A handy tool is `asyncio.gather()`, which lets you run several async functions together: ``` import asyncio async def func1(): print('func1 complete') async def func2(): print('func2 complete') asyncio.run(asyncio.gather(func1(), func2())) ``` What `asyncio.gather()` does: - Runs many async functions at the same time. - Puts all the results in a list for you. - If something goes wrong, it lets you know. - You don't have to keep track of everything yourself. Other helpful `asyncio` tools: - `asyncio.create_task()`: Gets a task ready to go. - `asyncio.as_completed()`: Shows tasks as they finish. - `asyncio.wait()`: Waits for a bunch of tasks to finish. These tools make it easier to handle lots of tasks without losing track, keeping your program quick and responsive. ## Multithreading in Python ### Understanding Threads and Processes Think of threads and processes like workers in an office. Threads are like team members who share an office space (memory), working on different parts of the same project. Processes are like different teams, each with their own office space, working independently. **Multithreading** means having several threads (or workers) in one process (project) doing tasks at the same time. But, because of a rule in Python called the Global Interpreter Lock (GIL), these threads take turns rather than working all at once. **Multiprocessing** is when tasks run in their own processes, or separate teams, each doing its own thing without sharing space. This gets around the GIL rule and can do tasks at the same time on different CPU cores, but it's a bit more complex to set up and manage. For tasks that wait around a lot (like loading a webpage), both multithreading and multiprocessing can help. But for tasks that need a lot of CPU (like doing complex calculations), multiprocessing is usually the way to go. ### Integrating Asyncio with Multithreading Asyncio is like a smart organizer that helps manage tasks that need to wait (like sending emails or loading files) without blocking other tasks. It's great for handling lots of tasks that involve waiting, but it normally works in just one thread because of Python's rules. To mix asyncio with multithreading, you can use `asyncio.to_thread()` to move a task that's waiting too much to its own thread, letting your main program keep running smoothly: ``` import asyncio import threading async def main(): result = await asyncio.to_thread(blocking_func) print(result) asyncio.run(main()) ``` Another tool, `asyncio.run_coroutine_threadsafe()`, lets you run a task in a separate thread and wait for it to finish: ``` fut = asyncio.run_coroutine_threadsafe(coro_func(), loop) result = fut.result() ``` When mixing [threads](https://en.wikipedia.org/wiki/thread_\(computing\)) and asyncio, it's important to be careful with data that's shared between them. Using something like `asyncio.Queue` helps pass data safely from threads to the main program. Choosing the right mix of multiprocessing, multithreading, and asyncio depends on what your program needs. If it's doing a lot of heavy lifting (CPU work), multiprocessing might be best. For waiting around (like loading pages), asyncio is great. And for some cases, adding in some threads can help manage tasks that block others. ## Practical Applications and Examples ### Building a Simple Asynchronous Application Let's make a simple app that does tasks at the same time using Python's `asyncio`: - **Start by importing `asyncio` and making async functions:** Mark functions with `async def` to tell Python they can do things simultaneously. Like this: ``` import asyncio async def fetch_data(): print('Fetching data...') await asyncio.sleep(2) print('Done fetching') async def process_data(data): print(f'Processing {data}') ``` - **Run the async functions:** Use `asyncio.run()` to kick off these functions. It gets the whole thing moving: ``` data = asyncio.run(fetch_data()) asyncio.run(process_data(data)) ``` - **Do multiple things at once:** Use `asyncio.gather()` to run different tasks together: ``` asyncio.run(asyncio.gather( fetch_data(), process_data() )) ``` - **Handle mistakes:** If something goes wrong, catch the error with `try`/`except`: ``` try: data = asyncio.run(fetch_data()) except HTTPError as e: print(f'Error: {e}') ``` By following these steps, you can make your Python programs do a lot at once without getting stuck. ### Advanced Use Cases: Web Crawling and Database Access **Web Crawler** Here's how to visit many websites at the same time without waiting around: ``` async def crawl_page(url): page = await fetch_page(url) links = parse_links(page) print(f'Crawled {url}') return links tasks = [] for url in start_urls: tasks.append( asyncio.create_task(crawl_page(url)) ) await asyncio.gather(*tasks) ``` **Async Database with Asyncpg** This is how to ask a database for info without stopping your program: ``` import asyncpg async def get_rows(): conn = await asyncpg.connect(DSN) result = await conn.fetch('SELECT * FROM table') await conn.close() return result ``` Using `asyncpg`, your program can ask the database many things at once, keeping everything smooth and fast. ###### sbb-itb-bfaad5b ## Performance Considerations ### Measuring the Impact of Async/Await and Multithreading When you're checking how fast and efficient your Python code is, especially when using async/await or multithreading, here are some handy ways to do it: - **Use the `timeit` module** - This tool helps you see how long your code takes to run. It's great for comparing different ways of doing things. - **Profile with `cProfile`** - This digs into your code and shows you which parts are slow or use a lot of resources. - **Monitor system resources** - Keep an eye on how much CPU and memory your code uses with tools like `psutil` or the `top` command. Big jumps in usage might mean there's a problem. - **Load test** - Pretend you have a bunch of users at once to see how your code holds up. Check if using async keeps things speedy as more users come on board. - **Compare metrics** - Look at important numbers like how many requests you can handle per second, how long responses take, and how often errors happen. This tells you if your changes are really helping. Testing in real-life situations is key. How your code performs can change a lot based on things like how many users you have, what tasks you're doing, how big your data is, and the power of your computer. ### Common Pitfalls and How to Avoid Them Here are some usual mistakes when working with async/await and threads, and how to dodge them: - **Too many awaits** - Using `await` too much can slow things down. Try to group your data fetching and use `asyncio.gather()` for better flow. - **Shared mutable state** - When [threads](https://en.wikipedia.org/wiki/thread_\(computing\)) share data, it can lead to mix-ups. Use a queue or an [asyncio](https://pymotw.com/3/asyncio/) lock to keep things orderly. - **Blocking the event loop** - If your code does a lot of heavy lifting, it can stop your async tasks from running smoothly. Use `run_in_executor()` to handle this. - **Unclosed connections** - Always make sure to close your database and web connections. Using `async with` makes this easier by closing them automatically. - **Difficult debugging** - Finding problems in async code can be tricky. Turn on `asyncio`'s debug mode while you're working out the kinks to make this easier. Sticking to the best practices for async programming in Python can help you steer clear of these issues. Test your code under different conditions, watch out for unexpected resource use, and handle errors gracefully. Begin with simple setups and add complexity gradually once everything is running smoothly. ## Conclusion Using async programming with Python's tools like asyncio and async/await can really help make your apps quick to respond and able to handle a lot of work at once. It's like teaching your code to multitask efficiently, where it can pause a task that's waiting around without stopping everything else. Here's what you should remember: - Asyncio is like a control center for running async code. It lets a single program do many things at the same time without getting stuck. - The async/await keywords are used to point out which functions can take a break and wait without freezing the whole program. - Asyncio is great for tasks that have to wait for something else to happen, like getting data from the internet. If you have tasks that need a lot of computing power, using multiple processes might be a better choice. - Testing your code as if it's in the real world is crucial to make sure it actually works better. Keeping an eye on how much computer power and memory it uses can also help spot problems. - To avoid common mistakes, try to keep things simple. Make sure you're not stopping the flow of tasks, always close your connections properly, and don't overuse the pause (await) feature. As Python keeps getting better at handling many tasks at once, using async is becoming more important. We might see even better ways to mix it with other methods like multiprocessing in the future. For now, asyncio gives us a strong way to make apps that are fast and can do a lot at the same time. ## Appendices ### Appendix A: Asyncio’s Roots in Generators Asyncio is built on something called generator functions in Python. Think of generators like a magic trick that lets a program pause and then continue from where it left off, producing one result at a time instead of all at once. Here’s how coroutines, which are a big part of asyncio, are similar yet different from generators: - They use `async/await` instead of `yield/next` to pause and resume. - Coroutines can wait for each other using `await`, allowing them to run at the same time. - Asyncio adds a special system (event loops) to manage these coroutines. So, asyncio takes the pausing feature of generators and adds its own twist with async/await and event loops, making it a powerful tool for running tasks at the same time without slowing down. ### Appendix B: Comparison Tables **Synchronous vs Asynchronous vs Multithreading Execution** | Approach | Single Task Execution | Multiple Task Execution | |---|---|---| | Synchronous | Tasks are done one by one, with each waiting for the previous to finish. | Each task waits its turn, adding up to a long wait time. | | Asynchronous | Tasks are done one by one but can pause and let others go ahead. | Tasks overlap, so everything gets done faster. | | Multithreading | Each thread does tasks one by one. | Multiple threads mean tasks can be done at the same time, speeding things up. | **Async/Await vs Multithreading** | Factor | Async/Await | Multithreading | |---|---|---| | Concurrency | Yes, tasks can run at the same time on one thread with `await`. | Yes, tasks run at the same time but on different threads. | | Parallelism | No, since it’s all on one thread, tasks are really done one after another. | Kind of, but the GIL (a Python rule) limits true side-by-side work. | | Overhead | Very little extra work for switching tasks. | More work needed to switch between tasks on different threads. | | Shared State | Easy to manage within one thread. | Needs special care to avoid mix-ups between threads. | | Blocking Calls | Use `run_in_executor()` to keep the main flow going. | Each thread can wait without stopping others. | | Resource Usage | Uses less memory because it's all in one thread. | Uses more memory for each thread’s needs. | ## Related Questions ### Is asyncio multithreaded in Python? No, asyncio itself doesn't use multiple threads. Instead, it lets you do lots of things at once using a single-thread approach, which is like having a super-efficient to-do list for your computer. However, you can mix it with threads using special functions like `asyncio.to_thread` or `asyncio.run_coroutine_threadsafe` to run some parts of your code in different threads. ### What is the difference between asyncio and thread? Here's how asyncio and threading are different: - **Execution model**: Asyncio runs tasks one after another in a very organized way on a single thread, while threading splits tasks across multiple threads for real multitasking. - **State management**: With asyncio, managing shared data is easier because everything happens in order. With threading, you need to be careful about data access across threads. - **Blocking calls**: In asyncio, if one task stops to wait, everything waits. But with threads, one task can wait without stopping others. Basically, asyncio is great for tasks that wait a lot, like loading web pages, while threads are good when you have lots of tasks that don't need to wait for each other. ### What is the difference between asynchronous and multithreading and multiprocessing? - **Async** is about doing lots of tasks in order, but in a way that lets others jump in when there's waiting time. - **Threads** are about doing different tasks at the same time within the same app. - **Processes** are like having separate apps, each doing its own thing independently. Async is for when you're dealing with lots of waiting, threads are for a mix of waiting and working, and processes are for heavy-duty tasks that need all the power they can get. ### What are the advantages of asyncio? Asyncio's plus points include: - It makes apps that wait on things like downloads or loading pages faster by letting other tasks run in the meantime. - It keeps things simple since you're only dealing with one thread. - Code is easier to read and manage thanks to the async/await setup. - It's efficient at switching between tasks without using up too much memory or processing power. - It works well with libraries designed for asynchronous operations. So, asyncio is a good choice for apps that need to do a lot of waiting around for things like network requests or file operations. ## Related Blog Posts - [Introduction to Python for Data Science](https://daily.dev/blog/introduction-to-python-for-data-science/) - [Modern JavaScript Essentials for Developers](https://daily.dev/blog/modern-javascript-essentials-for-developers/) - [Apollo's Learning Resources for Developers](https://daily.dev/blog/apollos-learning-resources-for-developers/) - [Async GraphQL Basics for Beginners](https://daily.dev/blog/async-graphql-basics-for-beginners/) Author [Nimrod Kramer @NimrodKramer](https://app.daily.dev/kramer) Related tags on daily.dev  Table of contents [Read more on daily.dev](https://api.daily.dev/get) [Why not level up your reading with daily.dev?Stay up-to-date with the latest developer news every time you open a new tab.Start reading - Free forever](https://api.daily.dev/get) ## Why not level up your reading with daily.dev? Stay up-to-date with the latest developer news every time you open a new tab. [Start reading - Free forever](https://api.daily.dev/get) ## Read more [Get into tech PWA vs Native Apps vs Hybrid: Comparison Comparison of PWA, Native Apps, and Hybrid Apps in terms of cost, performance, device integration, cross-platform support, offline capabilities, distribution, and maintenance. Explore the pros and cons to choose the right approach.  Nimrod Kramer May 16, 2024](https://daily.dev/blog/pwa-vs-native-apps-vs-hybrid-comparison) [Get into tech Form React Hook Essentials for Developers Explore the essentials of Form React Hook, from installation to advanced techniques like dynamic forms and custom hooks. Learn how to build robust, high-performing forms in React applications.  Nimrod Kramer February 22, 2024](https://daily.dev/blog/form-react-hook-essentials-for-developers) [Get into tech Essential CSS Tools for Developers Explore essential CSS tools for developers, including SCSS Compiler, PostCSS, Tailwind CSS, Sass, PurgeCSS, CSS Doodle, CSS Linters, Bourbon, Koala App, and CSS Grid. Learn about their features, performance, and best use cases.  Nimrod Kramer March 19, 2024](https://daily.dev/blog/essential-css-tools-for-developers) [See more on daily.dev](https://api.daily.dev/get) [](https://daily.dev/?r=0) daily.dev is a professional network for developers to learn, collaborate, and grow together. [](https://apps.apple.com/app/daily-dev/id6740634400)[](https://play.google.com/store/apps/details?id=dev.daily) [](https://app.daily.dev/squads/daily_updates)[](https://twitter.com/dailydotdev)[](https://github.com/dailydotdev/daily)[](https://www.instagram.com/dailydotdev/)[](https://www.tiktok.com/@dailydotdev) Product [Web app](https://app.daily.dev/?utm_source=landing&utm_medium=cta&utm_campaign=landing_conversions)[Chrome](https://r.daily.dev/chrome) [Edge](https://microsoftedge.microsoft.com/addons/detail/dailydev-the-homepage-/cbdhgldgiancdheindpekpcbkccpjaeb?hl=en-GB) [Firefox add-ons](https://addons.mozilla.org/en-US/firefox/addon/daily/) [iOS](https://apps.apple.com/app/daily-dev/id6740634400)[Android](https://play.google.com/store/apps/details?id=dev.daily)[Status](https://www.dailydevstatus.com/) Community [Docs](https://docs.daily.dev/docs/intro)[Open source](https://github.com/dailydotdev/daily)[Events](https://www.meetup.com/the-monthly-dev-world-class-talks-by-expert-developers/)[Swag](https://store.daily.dev/)[Changelog](https://app.daily.dev/squads/daily_updates) Company [About](https://daily.dev/about-us)[Careers](https://daily.dev/careers)[Blog](https://daily.dev/blog-superold)[Brand](https://brand.daily.dev/)[Contact](https://daily.dev/contact) Business [For Advertisers](https://business.daily.dev/?ref=homepage)[For Recruiters](https://recruiter.daily.dev/?ref=homepage) © 2025 Daily Dev Ltd. [Terms](https://daily.dev/tos) [Privacy](https://daily.dev/privacy) [Guidelines](https://r.daily.dev/content-guidelines) Working remotely wherever we're happiest 🇮🇱 🇮🇹 🇵🇭 🇳🇱 🇬🇧 🇭🇷 🇱🇹 🇦🇺 🇦🇪 🇳🇴 🇦🇱 🇵🇹