# Text Vectorizer Explained: BoW, N-gram, TF-IDF, and Word Embedding

Computers can't read words the way humans do — everything has to be converted into numbers before a machine learning model can process it. That conversion is what's called a **text vectorizer**. In this note, I re-ran four vectorizer techniques with real numbers and charts from actual execution — and along the way, found two real bugs in the bootcamp material that hadn't been caught before.

## 1\. Bag of Words (BoW): Counting, Not Understanding

The simplest way to turn text into numbers: count how many times each word appears, and completely ignore the order.

Take three example sentences, one of which deliberately repeats a word:

*   Document 1: `"saya suka machine learning learning"` (I like machine learning learning)
    
*   Document 2: `"machine learning adalah masa depan"` (machine learning is the future)
    
*   Document 3: `"natural language processing adalah masa depan"` (natural language processing is the future)
    

After running `CountVectorizer`, the `"learning"` column for Document 1 shows a value of **2** — because the word genuinely appears twice. This is what sets BoW apart from a simple "present/absent" check: BoW actually counts frequency.

One thing worth noting: if a fitted vectorizer is used to transform a new sentence containing a word outside its vocabulary (say, `"apple"`), that word is **silently ignored** with no warning at all. Not an error, but a quiet side effect worth watching for in production.

## 2\. N-gram: Keeping Phrases Intact

Standard BoW breaks text into single words (*unigrams*). N-grams break it into combinations of **consecutive words**, so a phrase like `"machine learning"` stays treated as one unit instead of two separate words.

Using `CountVectorizer(ngram_range=(2, 2))` on the three documents above, `"machine learning"` becomes its own feature, distinct from `"learning adalah"` or `"suka machine"`. The trade-off: the vocabulary grows larger and the data gets sparser, but phrases with combined meaning are preserved.

## 3\. TF-IDF: Common Words Get a Lower Weight

TF-IDF fixes BoW's weakness: words that appear in many documents (common) get a lower weight, while words that only appear in a few documents (unique) get a higher weight.

![TF-IDF weight comparison](https://raw.githubusercontent.com/arielshakaramiro/text-vectorizer-bow-tfidf-word2vec-arielshakaramiro/main/images/tfidf-weight-comparison.png align="center")

The words `"machine"` and `"learning"` — which appear in both documents — get a weight of roughly 0.36–0.41. Words that only appear in one document get a higher weight (0.50–0.58). TF-IDF automatically "knows" which words are more distinctive for telling one document apart from another — something plain BoW can't do.

## 4\. Word Embedding with spaCy — and Two Bugs I Found

I originally assumed the technique used in this session was Word2Vec (gensim), since that's what showed up as example code in the slides. But after checking the actual practice notebook from class, the technique genuinely used was **spaCy**. And that's where I found something fairly significant.

### Bug #1: The Model Used Has No Word Vectors At All

The original notebook loads `en_core_web_sm` and displays `.vector` for words as if it were a meaningful representation. I tested whether that's actually true:

```python
print(len(nlp.vocab.vectors))  # result: 0
```

**Zero.** `en_core_web_sm` is spaCy's **small** model, and small models are designed to **not include real word vectors at all** — to keep the file size down. The `.vector` values shown aren't semantic representations like Word2Vec/GloVe; they're just internal tensors from the tagger/parser/NER components.

spaCy itself throws an explicit warning when `.similarity()` is used under these conditions:

> *"The model you're using has no word vectors loaded... may not give useful similarity judgements."*

> **Why this matters:** if the original notebook displays vector and similarity numbers without this warning attached, a reader could easily assume it's a valid semantic representation — when technically it isn't. For real word vectors, spaCy has larger models (`en_core_web_md` or `en_core_web_lg`) that do ship with pretrained vectors.

### Bug #2: An Indonesian Lemmatizer That's Always Empty

The original notebook also tries Indonesian tokenization + lemmatization using `spacy.lang.id.Indonesian()`. I ran it exactly as written, and got:

```plaintext
Pipeline components: []
Original tokenizer result: ['', '', '', '', '', '', '', '', '']
```

Completely empty — not just the result, but the pipeline itself. `Indonesian()` in spaCy is a **blank pipeline**: just a tokenizer, with no lemmatizer component whatsoever. That's why `token.lemma_` always returns an empty string. This isn't a mistake in how the code was run — it's a genuine limitation of that blank pipeline.

**The fix:** replace `token.lemma_` with **Sastrawi** — the Indonesian stemming library I'd already used in the FAQ search engine project. The result:

```plaintext
Fixed tokenizer result: ['pdip', 'resmi', 'calon', 'gubernur', 'dki', 'jakarta', 'jokowi', 'calon', 'presiden']
```

## 5\. Bonus: Word2Vec (gensim) for Comparison

Since Word2Vec/gensim still appears as illustrative example code in the source slides, I've included it here too — as a point of comparison, not a replacement for section 4.

> **Honest note:** the corpus used is just 2 short sentences, so the resulting similarity scores (all below 0.1) aren't semantically meaningful yet — similar to the "zero vectors" issue with spaCy's `en_core_web_sm`, just for a different reason. Word2Vec needs a corpus of millions of words for valid results, while spaCy's `en_core_web_sm` simply doesn't include vectors by design.

Two models, two different ways of failing quietly — but the lesson is the same: **always check whether the "vector" you're using actually represents meaning**, rather than assuming it does just because the numbers are there.

## Summary

| Technique | Strength | Weakness / What to Watch For |
| --- | --- | --- |
| Bag of Words | Simple, captures word frequency | Ignores order & meaning; out-of-vocabulary words are silently dropped |
| N-gram | Preserves phrases/word combinations | Vocabulary grows fast, data gets sparser |
| TF-IDF | Common words automatically get lower weight | Still ignores context and word order |
| Word Embedding (spaCy `en_core_web_sm`) | Convenient, one line of code | Small model **has no real word vectors** |
| Word2Vec (gensim) | Can be trained on domain-specific data | Needs a large corpus; small corpora give meaningless vectors |

The full code and re-executed notebook — including both bugs above and their fixes — are available in [this GitHub repo](https://github.com/arielshakaramiro/text-vectorizer-bow-tfidf-word2vec-arielshakaramiro).

Once the fundamentals were in place, I used TF-IDF directly for two hands-on projects: a similarity-based FAQ search system, and an intent classifier using Logistic Regression.

> [https://shaka-ai.hashnode.dev/text-vectorizer-bow-tfidf-word-embedding-explained](https://shaka-ai.hashnode.dev/text-vectorizer-bow-tfidf-word-embedding-explained)
> 
> [https://shaka-ai.hashnode.dev/intent-classification-tfidf-logistic-regression](https://shaka-ai.hashnode.dev/intent-classification-tfidf-logistic-regression)

* * *

*Part of my AI Engineering study notes.*