Async Support in LangChain
We’re excited to roll out initial asynchronous support in LangChain by leveraging the asyncio library. Asyncio uses uses coroutines and an event loop to perform non-blocking I/O operations; these coroutines are able to “pause” (
await) while waiting on their ultimate result and let other routines run in the meantime. To learn more about asyncio and how it compares to multithreading and multiprocessing, check out this awesome tutorial.
Since LangChain applications tend to be fairly I/O and network bound (calling LLM APIs and interacting with data stores), asyncio offers significant advantages by allowing you to run LLMs, chains, and agents concurrently: while one agent is waiting for an LLM call or tool to complete, another one can continue to make progress. Async support in LangChain also allows you to more seamlessly integrate your async chains and agents into frameworks that support asyncio, such as
Check out the async agent docs in particular to see how significantly concurrent execution can speed things up!
As a starting point, we’ve implemented async support for:
agenerate (see docs):
acall (see docs):
arun (see docs):
We’re just getting started with asyncio. In the near future, we hope to add:
- Async support for more LLMs, Chains, and Agent tools
- Ability to run multiple tools concurrently for a single action input
- Async support for callback handlers
- More seamless support with tracing