What breaks in document AI

Many users at once on one GPU

18 of 18 quotes

• “I am currently using llama.cpp, although I'm just now working on switching to vllm since it is better for concurrent inference, since it can interleave prompt-processing and decode.”

• “uploading a few image-heavy PDFs at once drives the server process to 100 % of its CPU limit and keeps it there.”

• “Llama.cpp, fix the bugs that cause horrible speed decay in Qwen3.8-Flash-Next in longer context and with multiple users.”

• “I think there could be an issue with KV cache I had with llama.cpp that randomly threw in characters from other thread's KV cache...”

• “in an environment with the ability to define multiple users, multiple API keys, it is not abnormal to have this type of situations.”

• Hit this one too? We are building for it. Join the waitlist.

• “Three concurrent `POST /v1/chat/completions` requests for three different models on a cold router (no model loaded): one request succeeds, the other two fail immediately”

• “unlocked chat-blob read-modify-write silently drops messages under concurrency (chat_message table diverges from chat.chat)”

• “I have experienced broken tool calling from VSCode to Qwen3-Coder-Next-80b-A3b and Qwen3.6-27b-MTP and I had a Hermes agent go haywire thanks to this issue. I just applied the patch referenced above and everything is rosy now!”

• “Under parallel mixed load (`-np 4 --kv-unified`), `llama-server` on an integrated HIP GPU (Strix Halo / gfx1151, 128 GB UMA) returns complete responses that verbatim belong to a different, earlier request.”

• “I've been fighting similar issues, but happens on `parallel=1` / `no-kv-unified=true` as well”

• “server -np 3 decode drops ~4.4x (166 -> 38 t/s) whenever the two busy slots are the non-contiguous pair {0,2}”

• “When testing qwen3.6:35b with OLLAMA_NUM_PARALLEL=3 configured, launching prompts via 3 separate CMD windows executes them strictly sequentially (one after another)”

• “I found that it's optimal to run 3 workers concurrently even though Ollama processes 1 request at a time.”

• “The ~25GB KV Cache is not released when a session ends or a "New Conversation" is triggered in the upstream app (Dify/RAGFlow).”

• “This is most visible on concurrent agent / RAG / chat workloads where two clients share the same system prompt + few-shot prefix but happen to land on different slots due to slot availability.”

• “When 10 concurrent requests are made, 504 and 502 timeout errors occur.”

• “The V1 engine core silently deadlocks under concurrent load using default configuration.”

• “save embedding to vector DB freezes the whole application”

• Hit this one too? We are building for it. Join the waitlist.