We’re excited to roll out initial asynchronous support in LangChain by leveraging the asyncio library. Asyncio uses uses coroutines and an event loop to perform non-blocking I/O operations; these coroutines are able to “pause” (await) while waiting on their ultimate result and let other routines run in the meantime. To learn more about asyncio and how it compares to multithreading and multiprocessing, check out this awesome tutorial.
Motivation
Since LangChain applications tend to be fairly I/O and network bound (calling LLM APIs and interacting with data stores), asyncio offers significant advantages by allowing you to run LLMs, chains, and agents concurrently: while one agent is waiting for an LLM call or tool to complete, another one can continue to make progress. Async support in LangChain also allows you to more seamlessly integrate your async chains and agents into frameworks that support asyncio, such as FastAPI.
Check out the async agent docs in particular to see how significantly concurrent execution can speed things up!
Usage
As a starting point, we’ve implemented async support for:
LLM via agenerate (see docs):
OpenAI
Chain via arun and acall (see docs):
LLMChainLLMMathChain
Agent and Tool via arun (see docs):
SerpAPIWrapperLLMMathChain
Up Next
We’re just getting started with asyncio. In the near future, we hope to add:
- Async support for more LLMs, Chains, and Agent tools
- Ability to run multiple tools concurrently for a single action input
- Async support for callback handlers
- More seamless support with tracing