Keywords: asyncio | requests | asynchronous | HTTP | Python
Abstract: This article explains how to handle parallel HTTP requests in Python's asyncio without blocking the event loop. It focuses on using the run_in_executor method to run the blocking requests library asynchronously, with examples in both Python 3.4 and 3.5+ syntax. Additional libraries like aiohttp are discussed for comparison, ensuring a comprehensive understanding of asynchronous programming concepts.
Introduction
In Python, the asyncio library provides a framework for writing concurrent code using the async/await syntax. However, many popular libraries, such as python-requests, are blocking and can stall the event loop. This article addresses how to integrate such libraries into asyncio for efficient parallel HTTP requests.
Using run_in_executor with asyncio
The BaseEventLoop.run_in_executor method allows running blocking functions in a separate thread, preventing them from blocking the asyncio event loop. This enables the use of libraries like requests in an asynchronous manner without modifying their core implementation.
Code Examples
Below is an example using Python 3.4 style coroutines:
import asyncio
import requests
@asyncio.coroutine
def main():
loop = asyncio.get_event_loop()
future1 = loop.run_in_executor(None, requests.get, 'http://www.google.com')
future2 = loop.run_in_executor(None, requests.get, 'http://www.google.co.uk')
response1 = yield from future1
response2 = yield from future2
print(response1.text)
print(response2.text)
loop = asyncio.get_event_loop()
loop.run_until_complete(main())For Python 3.5 and above, the modern async/await syntax can be used:
import asyncio
import requests
async def main():
loop = asyncio.get_event_loop()
future1 = loop.run_in_executor(None, requests.get, 'http://www.google.com')
future2 = loop.run_in_executor(None, requests.get, 'http://www.google.co.uk')
response1 = await future1
response2 = await future2
print(response1.text)
print(response2.text)
loop = asyncio.get_event_loop()
loop.run_until_complete(main())This approach allows multiple HTTP requests to be made in parallel without blocking the event loop, thereby improving application responsiveness.
Alternative Libraries
While run_in_executor works well, native asynchronous libraries like aiohttp offer better performance and built-in support for features such as HTTP proxies. As shown in other answers, aiohttp can handle proxies, and libraries like httpx provide similar functionality but may have slightly lower performance.
Conclusion
Using run_in_executor is an effective way to incorporate blocking HTTP libraries into asyncio for parallel requests. However, for new projects, considering native async libraries might be beneficial for optimal performance. Developers should choose the appropriate method based on specific requirements.