Building a Simple TF-IDF Q&A Search Engine
After understanding how TF-IDF works in my previous note, I tried using it to build something real: an FAQ search engine. A user types a free-form question, the system finds the most similar question in an FAQ database, and returns its answer.
Sounds simple. But once I started testing it with odd, off-topic questions, I found a bug that turned out to matter quite a bit for how "similarity" gets measured.
The FAQ dataset used here is entirely fictional — the store name and every Q&A pair are just examples for demonstrating the technique, not real business data.
How It Works
Preprocessing — every question and answer is lowercased, stripped of punctuation, and each word is stemmed down to its root form using Sastrawi (an Indonesian stemming library).
Vectorization — all the preprocessed text is converted into TF-IDF vectors.
Search — when a user types a question, the system computes a similarity score between the user's question and every question in the database, then returns the answer from whichever is most similar.
The Bug I Found: A Fragile Distance Threshold
The first version of this system measured "similarity" using Euclidean distance (the straight-line distance between two vector points) with a fixed threshold: if the distance was below 1.0, it counted as a match.
I threw four test questions at it — three relevant, and one deliberately unrelated:
| Test Question | Euclidean Distance | Cosine Similarity |
|---|---|---|
| "how long does shipping take" | 0.8294 | 0.6560 |
| "how do I return a damaged item" | 0.9124 | 0.5838 |
| "is there any discount" | 0.8952 | 0.5993 |
| "good fried rice recipe" (off-topic) | 1.0000 | 0.0000 |
Look at the last row. A question about a fried rice recipe — clearly unrelated to a furniture FAQ — produced a Euclidean distance of exactly 1.0, sitting precisely on the threshold line. Not "almost" — genuinely right at the boundary. If the comparison had come out even marginally different, the system would have confidently treated it as a "match" and returned an irrelevant answer.
Compare that with cosine similarity (which measures the angle between two vectors, not their absolute distance): for the same question, the result is 0.0000 — a far clearer signal that there's genuinely no match at all.
Why does this happen? Euclidean distance is sensitive to the length/density of text vectors, so two texts that are both "short and sparse" can end up looking close together in distance even when their content has nothing to do with each other. Cosine similarity doesn't have this problem, since it only measures the angle between vectors.
The Fix I Applied
Switched the default metric from Euclidean distance to cosine similarity, using a minimum similarity threshold (0.15) instead of a distance threshold.
Made sure the user's question goes through the exact same preprocessing function as the training data — in the earlier version, the user's question was vectorized directly with no preprocessing at all, even though every FAQ entry had already been lowercased and stemmed.
Honest note: that 0.15 threshold was picked manually based on these 4 test questions, not from systematic tuning across many phrasings. For "good fried rice recipe" specifically, the rejection was clean (similarity score of exactly zero) — but that's not a guarantee every off-topic question will be rejected this cleanly. A real production system would need testing against a much wider range of questions before relying on a single fixed threshold.
Try It Yourself
git clone https://github.com/arielshakaramiro/faq-search-engine-tfidf-sastrawi-arielshakaramiro.git
cd faq-search-engine-tfidf-sastrawi-arielshakaramiro
pip install -r requirements.txt
python train.py
python app.py
The full code, exploration notebook, and further audit details are available in this GitHub repo.
If you haven't read the TF-IDF fundamentals this project builds on, there's a previous note covering Bag of Words, TF-IDF, and Word Embedding from the ground up. Another way of putting TF-IDF to work — intent classification with Logistic Regression — is covered in a separate note.
https://shaka-ai.hashnode.dev/text-vectorizer-bow-tfidf-word-embedding-explained
https://shaka-ai.hashnode.dev/intent-classification-tfidf-logistic-regression
Part of my AI Engineering study notes.
