Turso FTS5 in RAG without 500s

In the post on hybrid search I showed that, on a benchmark close to technical documentation, keyword search added no recall. FTS5 still stays in my RAG, because of the questions an API portal gets: endpoint names, error codes, the exact title of a page. Vectors are good with meaning, and for exact words I use FTS5.

The work is keeping it from turning into 500s, empty results or leaks.

Tokenizer: accents must not split results

CREATE VIRTUAL TABLE chunks_fts USING fts5(
  chunk_id UNINDEXED,
  text,
  tokenize = "unicode61 remove_diacritics 2"
);

With remove_diacritics 2, "orientações" and "orientacoes" land on the same index term. That matters in Portuguese, where lots of people type without accents. chunk_id is UNINDEXED because it only exists for the JOIN.

The user's question never goes straight to MATCH

MATCH has its own syntax. An ordinary question breaks the parser: a trailing ?, quotes pasted from somewhere, C++, a stray "a AND". The result is fts5: syntax error, and both obvious reactions are bad: return a 500 for a mistyped question, or retry without the filter that failed.

So the expression is built, never passed through:

const TERM = /[\p{L}\p{N}]+/gu;

export function toMatchExpression(query: string): string {
  const terms = [...new Set([...query.matchAll(TERM)].map((m) => m[0].toLowerCase()))];
  if (terms.length === 0) throw new InvalidQueryError("the question has no searchable term");
  return terms.map((t) => `"${t.replace(/"/g, '""')}"`).join(" OR ");
}

Every run of letters and digits becomes a quoted string. Inside quotes, or is the word "or" and not the operator, so no question can change the shape of the query. \p{L} matters: with a non-Unicode \w, "orientações" would become "orienta" and "es".

Users lose search syntax, so nobody can type title:pix or pix* through this path. For a chatbot, that's fine.

Why OR and not AND

Between bare terms, FTS5 uses AND. On a natural-language question, that makes the lexical leg return zero rows almost every time. A hybrid search won't notice: the vector search still answers, the fusion still returns 20 chunks, and half the system is dead without anyone knowing. With OR, it keeps contributing candidates and bm25() handles the ranking.

The exact phrase, with weight

OR finds scattered terms. For a question that really is a phrase, a second query looks for the phrase in order:

export function phraseExpression(query: string): string | null {
  const tokens = (query.toLowerCase().match(/[\p{L}\p{N}_]+/gu) ?? []).slice(0, 24);
  return tokens.length < 2 ? null : `"${tokens.join(" ")}"`;
}

This leg enters the fusion with weight 2. The reason is in the post on weighted RRF.

When FTS5 refuses anyway

If the generated expression is ever refused, the answer must be "invalid question" (4xx), never 500. The driver gives no structured code, only the message, and not every FTS5 message starts with fts5:. Measured on SQLite 3.45.1:

input	message
()	fts5: syntax error near ")"
" AND *	unterminated string
*	unknown special query:

const FTS5_REFUSAL = /fts5:|unterminated string|unknown special query/i;

A classifier that only checked the prefix would treat two of the three as server errors.

Writing to the index without leaving junk

The FTS table is standalone (no content=), and that has consequences on the write path:

• Don't use INSERT OR REPLACE on chunks. It changes the rowid, the index keeps pointing at the old one, and the new text is never indexed. Use ON CONFLICT DO UPDATE.
• The cascade doesn't reach FTS. Deleting the document deletes its chunks through the foreign key, but the index entries stay. Delete from both tables, by chunk_id, in the same batch().
• The transaction is the client's batch(). In an HTTP pipeline with manual BEGIN and COMMIT, the failing statement doesn't stop the ones after it, COMMIT runs, and half a document stays written. batch() is atomic.

I tested all of this on the real database. None of these failures raise an error; search just gets a little worse with each ingestion.