Why does my RAG pipeline return irrelevant chunks even with a high similarity threshold?

A high cosine similarity score means chunks are semantically close to the query embedding, not that they contain the specific answer — this is a known limitation of pure vector search, especially on short, ambiguous queries.

Fix

Add a re-ranking step (a cross-encoder model scoring query-chunk pairs directly) after initial retrieval, and consider hybrid retrieval combining vector and keyword search for domain-specific terminology your embedding model wasn’t trained on.