What breaks in document AI

Stale answers after updates

15 of 15 quotes

• “When using the WebDAV data source with automatic sync enabled, the sync only works for deletions. It fails to add new files and also fails to update modified files.”

• “one of the pricing pages changed its numbers back in march, but my vectors obviously still had the old version. retrieval kept returning it with high similarity for weeks.”

• “I also experienced some data loss during the import because I was running a concurrent reindex.”

• “Found out from a support ticket where a user said the information "seemed outdated." Took us a while to trace it, a document update had shifted our embedding distribution just enough that top-k was pulling adjacent but wrong chunks.”

• “Confluence data source file synchronization may produce duplicate files.”

• Hit this one too? We are building for it. Join the waitlist.

• “it was answering correctly based on the wrong document. was blaming the model the whole time while the actual problem was vector search not knowing what a version number is.”

• “Then we connected it to real production data, and the quiet failures started showing up: outdated documents being retrieved, conflicting information between sources, numbers changing slightly in responses, incomplete context producing very confident answers.”

• “Stale index is sneaky, docs were getting updated but I hadn't set up automatic re-indexing.”

• “when I run my script to ingest documents that are already indexed, I see that some chunks are updated (added and deleted)”

• “The CEO was changed and I already updated the data, but every time I ask the chatbot “Who is the CEO?”, it still returns the old one instead of the new one.”

• “watching someone ask about a reimbursement policy and the system confidently pull last year’s PDF.”

• “all subsequent sync cycles fail to detect any new files**”

• “It was because it found something real from the wrong version of the document. Old policy PDF still sitting in the index.”

• “Now I need to change the embedding model to another one, which is not retrieving the information related to the old indices.”

• “删除了的文件切片检索还能搜到显示在"reference"”

• Hit this one too? We are building for it. Join the waitlist.