Intent Classification with TF-IDF + Logistic Regression
The previous two notes used TF-IDF to measure similarity between texts (FAQ search). This time the approach is different: TF-IDF is used as a feature to train a classifier that predicts the intent behind a sentence — whether it's a greeting, an identity question, or a time question.
The model scored 100% accuracy. But that number turned out to be less straightforward than it looks.
The dataset used contains 94 Indonesian sentences with 3 labels:
greeting,siapa_anda(who are you),sekarang_jam_berapa(what time is it) — bootcamp practice material, not production data.
How It Works
Preprocessing — lowercase, strip punctuation, stem with Sastrawi (same as the earlier FAQ search engine project).
Training —
TfidfVectorizer+LogisticRegressioncombined in a singlesklearn.Pipeline.Evaluation — accuracy and classification report on the test set.
100% Accuracy — Coincidence or Real?
The first split (80% train, 20% test) produced 100% accuracy on 19 test samples. A number that good is easy to get excited about, but also easy to be misled by — 19 samples is small, and a result this clean could just be luck.
To check, I ran 5-fold stratified cross-validation — training and testing the model 5 times with different data splits:
Scores per fold: [1. 1. 1. 1. 1.]
Mean: 1.0000 (std: 0.0000)
Still 100% across every fold. So this isn't a fluke from one lucky split. But it's also not proof the technique itself is remarkably powerful — a more plausible explanation: the three intents in this dataset are lexically very distinct. Sentences about time ("jam", "sekarang", "pukul") almost never share words with greetings ("halo", "selamat") or identity questions ("siapa", "kamu"). TF-IDF + Logistic Regression will always look "perfect" when the classes are already this far apart in vocabulary — it's not a representative benchmark for intents that are more similar to each other.
Testing the Real Limits: Cases Outside the Training Data
To actually find the model's limits, I tried sentences that don't appear in the training data at all — slang, ambiguous phrasing, and completely off-topic sentences:
Two interesting findings here:
"kamu tau ga sekarang tanggal berapa"(asking about the date, not the time) still gets classified assekarang_jam_berapa. The model doesn't actually distinguish the concepts of "date" and "time" — it just recognizes the word pattern "sekarang" + "berapa" (now + how much/what), which happens to resemble the time-intent pattern from training."random kalimat tidak jelas maksudnya apa"(a sentence with no clear topic at all) still gets forced into one of the 3 classes (greeting, with a confidence of just 0.439 — barely above a random guess among 3 classes). This model has no "unknown" class, so no matter the input, it will always return one of the 3 existing labels.
This mirrors the Euclidean distance bug from the earlier FAQ search engine note: a system built without considering out-of-scope input will always force an answer rather than admit it doesn't know. The mechanism is different — there it was a distance threshold, here it's the lack of a rejection class — but the lesson is the same.
Summary
| Aspect | Finding |
|---|---|
| Single-split accuracy | 100% (19 test samples) — easy to misread |
| 5-fold CV accuracy | 100% (consistent) — not a fluke, but because the classes are lexically far apart |
| "Date" vs "time" case | Misclassified — the model matches word patterns, not concepts |
| Off-topic case | Forced into one of 3 classes, low confidence, no "unknown" class |
Main takeaway: high accuracy on a small dataset with lexically distinct classes doesn't automatically mean the model is reliable for real-world cases that are more ambiguous or fall outside the training data's scope.
The full code and re-executed notebook are available in this GitHub repo.
Part of my AI Engineering study notes.
