Benchmarks¶

How much time and memory does the agent library itself cost? These compare pyox-agents with pydantic-ai, the OpenAI Agents SDK, Microsoft Agent Framework, LangGraph, LangChain, LlamaIndex and Agno on the same tasks, against the same fake OpenAI server, which answers instantly. So the numbers are the overhead of each library: called directly from Python, with each streaming mode, and behind a FastAPI streaming endpoint under load.

Called directly from Python¶

Python 3.13.17, Linux x86_64 (Intel(R) Xeon(R) Processor @ 2.80GHz). 500 runs per measurement, 100 at a time for throughput. Versions: pyox-agents 0.1.0, pydantic-ai 2.61.0, openai-agents 0.20.0, agent-framework 1.23.0, langgraph 1.2.18, langchain 1.4.7, llama-index 0.14.25, agno 3.1.12.

Latency p50 of one run, and in parentheses how many times the pyox-agents time it is. Fastest first.

Each scenario is one streamed agent run: text streams a long answer (~450 chunks), tool makes one tool call and parallel_tools five at once, then answer. Startup is importing the library and creating an agent. Throughput runs 100 at once in one process.

• Tools: pyox-agents is 2 to 6 times faster than every other library, serves 4 to 10 times more runs per second and uses a third to a fifth of the memory.
• Long streamed text: only 1.1 to 2 times faster with the default text_chunks="delta", because each TextDelta event costs about 0.25 ms. With text_chunks=10 it takes 15 ms, 7 to 14 times less than the others.
• Startup: 0.13 s vs 1.5 to 3.2 s.
• "First text" for text is close to the whole run for the OpenAI Agents SDK, LangChain, LangGraph and LlamaIndex. This is most likely an artifact of the instant fake model: they pass pieces through a queue between asyncio tasks.

Streaming modes (text_chunks)¶

pyox-agents can group streamed text into fewer TextDelta events. Cost of each mode on the text scenario:

The cost is per event, not per character. With a real model, the wait for the first TextDelta grows with the group size (the whole answer with "message"), so 10 is the safe choice for live UIs.

Serving a stream: run.batches() and run.ai_sdk()¶

Three ways to turn a run into an AI SDK stream for a web response: one Python step per event, one per batch of waiting events (run.batches()), or the whole stream made in Rust (run.ai_sdk()). Run it with uv run stream_bench.py (stream_bench.py).

Intel(R) Xeon(R) Processor @ 2.80GHz.

Instant model (the cost of the Python side alone):

Paced model (100 pieces, 10 ms apart):

Each Python step costs far more than the event it carries. A batch holds the events that were already waiting, so batching only pays off when events pile up: a model that answers faster than Python reads it (cached answers, a fast local model), or a busy server that falls behind. A real model streaming one piece every few milliseconds gives one event per step either way, and the three ways cost about the same.

Sync code: agent.run_sync()¶

The sync API runs the agent in Rust with no Python event loop. Here against the async API and against asyncio.run around it, what sync code (a Django view, a Celery task) would do without it. 8 at a time means 8 tasks on one event loop for the async API, and 8 threads, like a Gunicorn gthread worker, for the other two (sync_bench.py).

Intel(R) Xeon(R) Processor @ 2.80GHz. tool scenario, 1000 runs per row.

A sync run takes half the CPU of an async one: each event is a plain hand-over to a waiting thread, where the async API goes through the event loop. A sync run still holds its thread until it ends, so for hundreds of runs at once an async server is the better fit.

Behind a FastAPI streaming endpoint¶

Each library serves POST /chat from one uvicorn process (server.py), and k6 users each send requests in a loop, reading the whole streamed answer (load.js). "First byte" is when the first text arrives; "Failed" counts requests without the full answer.

Python 3.13.17, one uvicorn process, Intel(R) Xeon(R) Processor @ 2.80GHz. Each measurement: 20s of load.

text: one long streamed answer¶

tool: one tool call, then the answer¶

• One process is CPU-bound with every library, so throughput stops growing at about 10 users. pyox-agents gets 2 to 6 times more requests per second on streamed text and 4 to 10 times more with a tool call.
• With 100 users, pyox-agents starts every answer within about 50 ms and streams them side by side. The others make most answers wait 1 to 10 seconds before they start. Agno's p50 is as good as pyox-agents' on streamed text, but it runs a started answer to the end while the others wait, so its p90 and p99 are about 4 times worse. The average is the fairest single number.

Behind a Django streaming endpoint¶

The same load test with Django instead of FastAPI: an async view returning a StreamingHttpResponse, served by one uvicorn process (ASGI), with async tools (django_server.py). Only pyox-agents, Agno (the fastest other library above) and pydantic-ai. Run it with uv run load_test.py --server django.

Python 3.13.18, one uvicorn process, Intel(R) Xeon(R) Processor @ 2.10GHz. Each measurement: 20s of load.

text: one long streamed answer¶

tool: one tool call, then the answer¶

• Same picture as FastAPI: pyox-agents serves 3.5 times more requests per second than Agno with a tool call (232 vs 67 with 100 users) and 8 times more than pydantic-ai, and about 2 times more than Agno on streamed text.
• With 100 users it starts answers within 40 to 90 ms; Agno and pydantic-ai take 0.6 to 3.7 s.
• Async tools run on the event loop. A sync tool (@tool def ...) also runs on the loop and blocks it while it runs; mark slow ones @tool(run_in_thread=True) to run them with asyncio.to_thread.

Responses API over WebSocket¶

pyox-agents (Client(transport="websocket")) and the OpenAI Agents SDK (use_responses_websocket=True), each also over HTTP, on a tool loop of 1, 5 or 20 calls, one per turn, against a fake Responses API (responses_server.py). A local server cannot reproduce OpenAI's per-connection response cache, where most of WebSocket mode's speedup comes from, so this is each library's own overhead and the upload it saves: over WebSocket, turns after the first send only the new tool result. The Agents SDK chains turns with auto_previous_response_id=True and runs one request at a time per connection, so its concurrent runs wait for each other.

Python 3.13.17, Intel(R) Xeon(R) Processor @ 2.10GHz, the fake server, model delay 0 ms per turn. 500 runs per measurement, 100 at a time for throughput.

Latency p50 of one run, by number of tool calls (one per turn). Fastest first.

With the fake model waiting 10 ms before each answer:

Python 3.13.17, Intel(R) Xeon(R) Processor @ 2.10GHz, the fake server, model delay 10 ms per turn. 500 runs per measurement, 100 at a time for throughput.

Latency p50 of one run, by number of tool calls (one per turn). Fastest first.

How it is measured, and how to run the benchmarks yourself, is in benchmarks/.