# Intent Classification with TF-IDF + Logistic Regression

The previous two notes used TF-IDF to *measure similarity* between texts (FAQ search). This time the approach is different: TF-IDF is used as a feature to **train a classifier** that predicts the intent behind a sentence — whether it's a greeting, an identity question, or a time question.

The model scored 100% accuracy. But that number turned out to be less straightforward than it looks.

> The dataset used contains 94 Indonesian sentences with 3 labels: `greeting`, `siapa_anda` (who are you), `sekarang_jam_berapa` (what time is it) — bootcamp practice material, not production data.

## How It Works

1.  **Preprocessing** — lowercase, strip punctuation, stem with Sastrawi (same as the earlier FAQ search engine project).
    
2.  **Training** — `TfidfVectorizer` + `LogisticRegression` combined in a single `sklearn.Pipeline`.
    
3.  **Evaluation** — accuracy and classification report on the test set.
    

## 100% Accuracy — Coincidence or Real?

The first split (80% train, 20% test) produced 100% accuracy on 19 test samples. A number that good is easy to get excited about, but also easy to be misled by — 19 samples is small, and a result this clean could just be luck.

To check, I ran **5-fold stratified cross-validation** — training and testing the model 5 times with different data splits:

```plaintext
Scores per fold: [1. 1. 1. 1. 1.]
Mean: 1.0000 (std: 0.0000)
```

Still 100% across every fold. So this **isn't a fluke from one lucky split**. But it's also not proof the technique itself is remarkably powerful — a more plausible explanation: the three intents in this dataset are **lexically very distinct**. Sentences about time ("jam", "sekarang", "pukul") almost never share words with greetings ("halo", "selamat") or identity questions ("siapa", "kamu"). TF-IDF + Logistic Regression will always look "perfect" when the classes are already this far apart in vocabulary — it's not a representative benchmark for intents that are more similar to each other.

## Testing the Real Limits: Cases Outside the Training Data

To actually find the model's limits, I tried sentences that don't appear in the training data at all — slang, ambiguous phrasing, and completely off-topic sentences:

![Model confidence on hard test cases](https://raw.githubusercontent.com/arielshakaramiro/intent-classification-tfidf-logistic-arielshakaramiro/main/images/edge-case-confidence.png align="center")

Two interesting findings here:

1.  `"kamu tau ga sekarang tanggal berapa"` (asking about the **date**, not the **time**) still gets classified as `sekarang_jam_berapa`. The model doesn't actually distinguish the concepts of "date" and "time" — it just recognizes the word pattern "sekarang" + "berapa" (now + how much/what), which happens to resemble the time-intent pattern from training.
    
2.  `"random kalimat tidak jelas maksudnya apa"` (a sentence with no clear topic at all) still gets forced into one of the 3 classes (`greeting`, with a confidence of just 0.439 — barely above a random guess among 3 classes). This model **has no "unknown" class**, so no matter the input, it will always return one of the 3 existing labels.
    

> **This mirrors the Euclidean distance bug from the earlier FAQ search engine note:** a system built without considering out-of-scope input will always force an answer rather than admit it doesn't know. The mechanism is different — there it was a distance threshold, here it's the lack of a rejection class — but the lesson is the same.

## Summary

| Aspect | Finding |
| --- | --- |
| Single-split accuracy | 100% (19 test samples) — easy to misread |
| 5-fold CV accuracy | 100% (consistent) — not a fluke, but because the classes are lexically far apart |
| "Date" vs "time" case | Misclassified — the model matches word patterns, not concepts |
| Off-topic case | Forced into one of 3 classes, low confidence, no "unknown" class |

**Main takeaway:** high accuracy on a small dataset with lexically distinct classes doesn't automatically mean the model is reliable for real-world cases that are more ambiguous or fall outside the training data's scope.

The full code and re-executed notebook are available in [this GitHub repo](https://github.com/arielshakaramiro/intent-classification-tfidf-logistic-arielshakaramiro).

* * *

*Part of my AI Engineering study notes.*