Table of Contents

• What Is Deep Learning?

• Machine Learning vs Deep Learning

• Artificial Neurons — The Basics

• Neural Network Architecture

• Why PyTorch?

• Hands-On: Digit Classification with PyTorch

• Recap Checklist

• Mini Quiz

• Further Exercises

• Closing Thoughts

What Is Deep Learning?

Deep Learning is a subfield of Machine Learning that uses Artificial Neural Networks (ANNs) with many layers (multi-layer).

Two things set it apart from conventional ML:

• It can learn complex patterns from large, unstructured data.

• Feature representation happens automatically — features don't need to be manually engineered or selected like in conventional Machine Learning.

Deep Learning shows up heavily in Computer Vision, Natural Language Processing (NLP), Speech Recognition, and Recommendation Systems.

Machine Learning vs Deep Learning

The most fundamental difference is in the process flow:

In ML, a human decides which features matter before feeding data into the model. In DL, feature extraction and classification happen together inside the network — which is why DL needs more data, but can also handle far more complex patterns.

Artificial Neurons — The Basics

Artificial neurons are inspired by biological ones: dendrites receive signals, the nucleus processes them, the axon sends signals onward. The math version:

• Each input (x₁, x₂, …, xₙ) is multiplied by its own weight (w).

• All the products are summed (Σ), then a bias (b) is added.

• The result passes through an activation function φ(.) to produce the output (y).

Z = W · X + b  →  Activation Function φ(Z)

🤔 Guess First: is ReLU typically used in the hidden layer or the output layer?

Answer: the hidden layer. ReLU (max(0, x)) is the standard activation between hidden layers because it's computationally cheap and helps the network learn non-linear patterns. For classification outputs, what's typically used is Softmax (multi-class, one label per sample) or Sigmoid per class (multi-label, multiple labels active at once) — not ReLU.

Neural Network Architecture

A Deep Neural Network is built from three types of layers:

1. Input Layer — receives the raw data as the initial input.

2. Hidden Layers — one or more hidden layers where complex patterns get learned; the more/deeper the layers, the more "deep" the network is.

3. Output Layer — produces the final result (prediction/classification).

Typically, every neuron in one layer connects to every neuron in the next (fully connected).

Why PyTorch?

A few reasons PyTorch is a solid choice for learning deep learning:

• Pythonic — the syntax feels natural and is easy for beginners to pick up.

• Widely used in both industry and academic research.

• Dynamic by design (define-by-run), which makes debugging much easier than with a static graph.

• Backed by a large open-source community, with a strong ecosystem: Torchvision, Torchtext, PyTorch Lightning, Torchserve.

The standard workflow for a Deep Learning project with PyTorch usually looks like this:

Load Dataset → Preprocessing → Build Model → Loss Function & Optimizer → Training Loop → Evaluation → Inference/Deployment

Let's put this workflow into practice.

Hands-On: Digit Classification with PyTorch

We'll build a simple neural network (SimpleNN) to recognize handwritten digits 0–9 from the MNIST dataset — the full pipeline, from loading data to a model ready for prediction.

1. Setup & Imports

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader
from torchvision import datasets, transforms, models

print('PyTorch version:', torch.__version__)

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print('Using device:', device)

# Set seed for reproducibility
SEED = 42
torch.manual_seed(SEED)
if torch.cuda.is_available():
    torch.cuda.manual_seed_all(SEED)

💡 torch.manual_seed() matters for reproducibility — without it, weight initialization and data shuffling are random, so results will vary slightly every time the notebook re-runs.

2. Load Dataset & DataLoader

transform = transforms.ToTensor()

train_dataset = datasets.MNIST(root='data', train=True, transform=transform, download=True)
test_dataset  = datasets.MNIST(root='data', train=False, transform=transform, download=True)

train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True)
test_loader  = DataLoader(test_dataset, batch_size=64, shuffle=False)

len(train_dataset), len(test_dataset)

(60000, 10000)

The data is split into train (60,000 images) and test (10,000 images) right away — so evaluation later genuinely measures how well the model generalizes to data it's never seen, instead of data it's already memorized during training.

3. Looking at Sample Data

import matplotlib.pyplot as plt

images, labels = next(iter(train_loader))
print('Batch shape:', images.shape)
print('Labels:', labels[:10])

plt.figure()
plt.imshow(images[0].squeeze(), cmap='gray')
plt.title(f'Label: {labels[0].item()}')
plt.axis('off')
plt.show()

Each image is 28×28 pixels, grayscale (1 channel), labeled with a digit from 0 to 9.

4. Building a Simple Neural Network

class SimpleNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc = nn.Sequential(
            nn.Flatten(), # out: 784 (28 * 28)
            nn.Linear(28 * 28, 128),
            nn.ReLU(),
            nn.Linear(128, 10)
        )

    def forward(self, x):
        return self.fc(x)

model = SimpleNN().to(device)
model

SimpleNN(
  (fc): Sequential(
    (0): Flatten(start_dim=1, end_dim=-1)
    (1): Linear(in_features=784, out_features=128, bias=True)
    (2): ReLU()
    (3): Linear(in_features=128, out_features=10, bias=True)
  )
)

The flow: Flatten turns the 28×28 image into a 784-length vector → the first Linear layer compresses it into 128 hidden features → ReLU adds non-linearity → the second Linear layer produces 10 outputs (one per digit, 0–9).

Notice the model does not end with a Softmax layer. That's intentional — nn.CrossEntropyLoss(), used in the next step, already applies log-softmax internally. Adding a manual Softmax here would apply that activation twice, which would mess up training.

5. Loss Function & Optimizer

criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=1e-3)

6. Training Loop

def train(model, loader, criterion, optimizer, device):
    model.train()
    running_loss = 0.0
    for images, labels in loader:
        images, labels = images.to(device), labels.to(device)

        outputs = model(images)
        loss = criterion(outputs, labels)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        running_loss += loss.item() * images.size(0)

    return running_loss / len(loader.dataset)

num_epochs = 10
for epoch in range(num_epochs):
    loss = train(model, train_loader, criterion, optimizer, device)
    print(f'Epoch [{epoch+1}/{num_epochs}] - Loss: {loss:.4f}')

Epoch [1/10] - Loss: 0.3421
Epoch [2/10] - Loss: 0.1532
Epoch [3/10] - Loss: 0.1044
Epoch [4/10] - Loss: 0.0789
Epoch [5/10] - Loss: 0.0630
Epoch [6/10] - Loss: 0.0508
Epoch [7/10] - Loss: 0.0417
Epoch [8/10] - Loss: 0.0337
Epoch [9/10] - Loss: 0.0284
Epoch [10/10] - Loss: 0.0238

Loss drops steadily from 0.3421 in the first epoch to 0.0238 in the last — a good sign the model is genuinely learning, not stagnating or getting worse.

A technical note: running_loss += loss.item() * images.size(0), divided by len(loader.dataset) at the end, gives the average loss over a full epoch — not just the loss from the last batch. Printing loss.item() at the end of the loop alone would only reflect the very last batch, which can be misleading as a progress indicator.

7. Evaluating Accuracy on Test Data

def evaluate(model, loader, device):
    model.eval()
    correct, total = 0, 0
    with torch.no_grad():
        for images, labels in loader:
            images, labels = images.to(device), labels.to(device)
            outputs = model(images)
            _, preds = outputs.max(1)
            total += labels.size(0)
            correct += (preds == labels).sum().item()
    return correct / total

test_acc = evaluate(model, test_loader, device)
print(f'Test accuracy: {test_acc:.4f}')

🤔 Guess First: how accurate do you think this simple model (just 1 hidden layer, 10 epochs) is on test data?

Answer: 97.86%. Pretty impressive for such a simple architecture — it shows MNIST is a relatively "easy" benchmark dataset, and shows a basic fully-connected network is already strong enough for image classification tasks that aren't too complex (at least for this one experiment — not necessarily true across every dataset).

model.eval() and torch.no_grad() matter here: eval() switches the model's mode (relevant if there were layers like Dropout/BatchNorm), and no_grad() disables gradient computation since evaluation doesn't need backward() — making the process faster and more memory-efficient. One thing to watch closely: loader here must be test_loader, not train_loader — evaluating on the same data used for training would produce a biased accuracy that doesn't reflect the model's actual ability to generalize.

8. Inference on a Single Image

model.eval()
image, label = test_dataset[0]
plt.figure()
plt.imshow(image.squeeze(), cmap='gray')
plt.title(f'Ground truth: {label}')
plt.axis('off')
plt.show()

with torch.no_grad():
    x = image.unsqueeze(0).to(device)
    output = model(x)
    pred = output.argmax(1).item()

print('Model prediction:', pred)

image.unsqueeze(0) adds a batch dimension up front (turning [1, 28, 28] into [1, 1, 28, 28]) — because the model expects data in batch form, even when we only want a prediction for a single image.

9. Saving & Reloading the Model

There are two ways to save a model in PyTorch:

# Option 1: save only the model weights (most common, recommended)
torch.save(model.state_dict(), "model_mnist.pth")

# Option 2: save the entire model object (rarely used, but still possible)
torch.save(model, "full_model_mnist.pth")

To reload the model from saved weights:

model = SimpleNN().to(device)
model.load_state_dict(torch.load("model_mnist.pth", map_location=device))
model.eval()

💡 map_location=device matters if you save a model from a GPU machine and want to load it on a machine without a GPU (or vice versa) — without it, torch.load() can throw an error trying to map tensors to a device that isn't available.

Recap Checklist

• [ ] Understand the difference between ML (manual feature extraction) and DL (automatic)

• [ ] Can explain the basic neuron formula: Z = W·X + b → φ(Z)

• [ ] Know the three layer types: input, hidden, output

• [ ] Understand why ReLU belongs in hidden layers, not the output

• [ ] Can lay out the PyTorch workflow: dataset → model → loss/optimizer → training loop → evaluation → inference

• [ ] Understand why model.eval() + torch.no_grad() are used during evaluation/inference

• [ ] Know why evaluation should use test_loader, not train_loader

• [ ] Can save and reload a model with state_dict()

Mini Quiz

1. Why doesn't the SimpleNN model above end with a Softmax layer?

Because nn.CrossEntropyLoss() in PyTorch already applies log-softmax internally. Adding a manual Softmax to the model would apply that activation twice, which disrupts training.

2. What happens if you evaluate the model using train_loader instead of test_loader?

The resulting accuracy would be biased — reflecting how well the model "memorized" the training data, rather than how well it generalizes to new, unseen data.

3. What does torch.no_grad() do during evaluation?

It disables gradient computation, since evaluation/inference never calls backward(). This makes the process faster and more memory-efficient than keeping gradient tracking on.

Further Exercises

To explore further from this notebook:

1. Change the hidden layer size from 128 to 256 or 64 — observe the effect on accuracy.

2. Add a new fully-connected layer (3 layers instead of 2).

3. Swap the optimizer from Adam to SGD, and compare convergence speed.

4. Add Dropout to the model and observe its effect on overfitting.

Closing Thoughts

The thing that stuck with me most from this material: deep learning isn't magic — it's just a simple neuron (Z = W·X + b) stacked in layers, trained through a highly structured trial-and-error process (forward pass → compute loss → backward pass → update weights). What makes it powerful is the scale and repetition, not the complexity of any single neuron.

The full, end-to-end verified notebook is available on this GitHub repo.

Part of my AI Engineering learning notes series — AI Notes & Engineering

Daftar Isi

• Apa Itu Deep Learning?

• Machine Learning vs Deep Learning

• Neuron Buatan — Konsep Dasar

• Arsitektur Neural Network

• Kenapa PyTorch?

• Praktik: Klasifikasi Angka dengan PyTorch

• Checklist Recap

• Kuis Mini

• Latihan Lanjutan

• Penutup

Apa Itu Deep Learning?

Deep Learning adalah subbidang Machine Learning yang menggunakan Artificial Neural Network (ANN) dengan banyak layer (multi-layer).

Dua hal yang membedakannya dari ML konvensional:

• Mampu mempelajari pola kompleks dari data besar & tidak terstruktur.

• Representasi fitur terjadi secara otomatis — fitur tidak perlu dibuat/dipilih manual seperti pada Machine Learning konvensional.

Deep Learning banyak dipakai di Computer Vision, Natural Language Processing (NLP), Speech Recognition, dan Recommendation System.

Machine Learning vs Deep Learning

Perbedaan paling mendasar ada di alur prosesnya:

Pada ML, manusia yang menentukan fitur apa yang penting sebelum masuk ke model. Pada DL, proses ekstraksi fitur dan klasifikasi terjadi sekaligus di dalam network — itu sebabnya DL butuh data lebih banyak, tapi bisa menangani pola yang jauh lebih kompleks.

Neuron Buatan — Konsep Dasar

Neuron buatan terinspirasi dari neuron biologis: dendrites menerima sinyal, nucleus mengolahnya, axon mengirim sinyal keluar. Versi matematisnya:

• Setiap input (x₁, x₂, …, xₙ) dikalikan dengan bobotnya masing-masing (w).

• Seluruh hasil perkalian dijumlahkan (Σ), lalu ditambah bias (b).

• Hasilnya diproses lewat activation function φ(.) untuk menghasilkan output (y).

Z = W · X + b  →  Activation Function φ(Z)

🤔 Coba Tebak Dulu: fungsi aktivasi ReLU itu biasanya dipasang di layer mana — hidden layer atau output layer?

Jawaban: hidden layer. ReLU (max(0, x)) adalah aktivasi standar di antara layer tersembunyi karena murah secara komputasi dan membantu network belajar pola non-linear. Untuk output klasifikasi, yang biasa dipakai justru Softmax (multi-class, satu label per data) atau Sigmoid per kelas (multi-label, beberapa label aktif sekaligus) — bukan ReLU.

Arsitektur Neural Network

Sebuah Deep Neural Network tersusun atas tiga jenis layer:

1. Input Layer — menerima data mentah sebagai masukan awal.

2. Hidden Layers — satu atau lebih layer tersembunyi tempat pola-pola kompleks dipelajari; semakin banyak/dalam layer, semakin "deep" network tersebut.

3. Output Layer — menghasilkan hasil akhir (prediksi/klasifikasi).

Umumnya setiap neuron di satu layer terhubung ke seluruh neuron di layer berikutnya (fully connected).

Kenapa PyTorch?

Beberapa alasan PyTorch jadi framework pilihan untuk belajar deep learning:

• Pythonic — sintaksnya natural, mudah dipahami pemula.

• Banyak dipakai di industri maupun riset akademik.

• Bersifat dinamis (define-by-run), jadi proses debugging lebih mudah dibanding graph statis.

• Didukung komunitas open source yang besar, dengan ekosistem kuat: Torchvision, Torchtext, PyTorch Lightning, Torchserve.

Alur kerja standar sebuah proyek Deep Learning dengan PyTorch biasanya begini:

Load Dataset → Preprocessing → Buat Model → Loss Function & Optimizer → Training Loop → Evaluation → Inference/Deployment

Sekarang kita praktikkan alur ini langsung.

Praktik: Klasifikasi Angka dengan PyTorch

Kita akan membangun neural network sederhana (SimpleNN) untuk mengenali angka tulisan tangan 0–9 dari dataset MNIST, lengkap dari load data sampai model bisa dipakai untuk prediksi.

1. Setup & Import Library

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader
from torchvision import datasets, transforms, models

print('PyTorch version:', torch.__version__)

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print('Using device:', device)

# Set seed untuk reproducibility
SEED = 42
torch.manual_seed(SEED)
if torch.cuda.is_available():
    torch.cuda.manual_seed_all(SEED)

💡 torch.manual_seed() penting supaya hasil training bisa direproduksi — tanpa ini, tiap kali notebook dijalankan ulang, hasilnya bisa sedikit berbeda karena inisialisasi bobot dan urutan shuffle data bersifat acak.

2. Load Dataset & DataLoader

transform = transforms.ToTensor()

train_dataset = datasets.MNIST(root='data', train=True, transform=transform, download=True)
test_dataset  = datasets.MNIST(root='data', train=False, transform=transform, download=True)

train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True)
test_loader  = DataLoader(test_dataset, batch_size=64, shuffle=False)

len(train_dataset), len(test_dataset)

(60000, 10000)

Data langsung dipisah jadi train (60.000 gambar) dan test (10.000 gambar) sejak awal — supaya nanti evaluasi benar-benar mengukur kemampuan model pada data yang belum pernah dilihat, bukan data yang sudah dihafal saat training.

3. Melihat Contoh Data

import matplotlib.pyplot as plt

images, labels = next(iter(train_loader))
print('Batch shape:', images.shape)
print('Labels:', labels[:10])

plt.figure()
plt.imshow(images[0].squeeze(), cmap='gray')
plt.title(f'Label: {labels[0].item()}')
plt.axis('off')
plt.show()

Setiap gambar berukuran 28×28 piksel, grayscale (1 channel), dengan label angka 0–9.

4. Membuat Model Neural Network Sederhana

class SimpleNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc = nn.Sequential(
            nn.Flatten(), # out: 784 (28 * 28)
            nn.Linear(28 * 28, 128),
            nn.ReLU(),
            nn.Linear(128, 10)
        )

    def forward(self, x):
        return self.fc(x)

model = SimpleNN().to(device)
model

SimpleNN(
  (fc): Sequential(
    (0): Flatten(start_dim=1, end_dim=-1)
    (1): Linear(in_features=784, out_features=128, bias=True)
    (2): ReLU()
    (3): Linear(in_features=128, out_features=10, bias=True)
  )
)

Alurnya: Flatten meratakan gambar 28×28 jadi vektor 784 → Linear pertama memampatkannya jadi 128 fitur tersembunyi → ReLU menambahkan non-linearitas → Linear kedua menghasilkan 10 output (satu untuk tiap digit 0–9).

Perhatikan: model ini tidak diakhiri dengan Softmax. Ini disengaja — nn.CrossEntropyLoss() di langkah berikutnya sudah menerapkan log-softmax secara internal, jadi menambahkan Softmax manual di sini justru bikin aktivasi diterapkan dua kali.

5. Menentukan Loss Function & Optimizer

criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=1e-3)

6. Training Loop

def train(model, loader, criterion, optimizer, device):
    model.train()
    running_loss = 0.0
    for images, labels in loader:
        images, labels = images.to(device), labels.to(device)

        outputs = model(images)
        loss = criterion(outputs, labels)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        running_loss += loss.item() * images.size(0)

    return running_loss / len(loader.dataset)

num_epochs = 10
for epoch in range(num_epochs):
    loss = train(model, train_loader, criterion, optimizer, device)
    print(f'Epoch [{epoch+1}/{num_epochs}] - Loss: {loss:.4f}')

Epoch [1/10] - Loss: 0.3421
Epoch [2/10] - Loss: 0.1532
Epoch [3/10] - Loss: 0.1044
Epoch [4/10] - Loss: 0.0789
Epoch [5/10] - Loss: 0.0630
Epoch [6/10] - Loss: 0.0508
Epoch [7/10] - Loss: 0.0417
Epoch [8/10] - Loss: 0.0337
Epoch [9/10] - Loss: 0.0284
Epoch [10/10] - Loss: 0.0238

Loss turun konsisten dari 0.3421 di epoch pertama ke 0.0238 di epoch terakhir — tanda model benar-benar belajar, bukan stagnan atau malah memburuk.

Catatan teknis: running_loss += loss.item() * images.size(0) lalu dibagi len(loader.dataset) di akhir adalah cara menghitung rata-rata loss satu epoch penuh — bukan cuma loss dari batch terakhir. Kalau cuma print(loss.item()) di akhir loop, angka yang muncul cuma mewakili batch paling akhir, yang bisa menyesatkan kalau dijadikan indikator progres training.

7. Evaluasi Akurasi di Data Test

def evaluate(model, loader, device):
    model.eval()
    correct, total = 0, 0
    with torch.no_grad():
        for images, labels in loader:
            images, labels = images.to(device), labels.to(device)
            outputs = model(images)
            _, preds = outputs.max(1)
            total += labels.size(0)
            correct += (preds == labels).sum().item()
    return correct / total

test_acc = evaluate(model, test_loader, device)
print(f'Akurasi di data test: {test_acc:.4f}')

🤔 Coba Tebak Dulu: menurutmu, kira-kira berapa akurasi model sesederhana ini (cuma 1 hidden layer, 10 epoch) di data test?

Jawaban: 97.86%. Cukup mengejutkan untuk arsitektur yang sangat sederhana — ini menunjukkan MNIST memang relatif "mudah" sebagai dataset benchmark, dan menunjukkan fully-connected network dasar sudah cukup kuat untuk kasus klasifikasi gambar yang tidak terlalu kompleks (setidaknya untuk satu eksperimen ini — belum tentu berlaku sama di dataset lain).

model.eval() dan torch.no_grad() di sini penting: eval() mengubah mode model (relevan kalau ada layer seperti Dropout/BatchNorm), dan no_grad() menonaktifkan perhitungan gradien karena saat evaluasi kita tidak perlu backward() — hasilnya proses jadi lebih cepat dan hemat memori. Satu hal yang wajib diperhatikan: loader di sini harus test_loader, bukan train_loader — kalau evaluasi dilakukan di data yang sama dengan training, angka akurasinya jadi bias dan tidak mencerminkan kemampuan generalisasi model.

8. Inference pada Satu Gambar

model.eval()
image, label = test_dataset[0]
plt.figure()
plt.imshow(image.squeeze(), cmap='gray')
plt.title(f'Ground truth: {label}')
plt.axis('off')
plt.show()

with torch.no_grad():
    x = image.unsqueeze(0).to(device)
    output = model(x)
    pred = output.argmax(1).item()

print('Prediksi model:', pred)

image.unsqueeze(0) menambahkan dimensi batch di depan (dari [1, 28, 28] jadi [1, 1, 28, 28]) — karena model dilatih untuk menerima data dalam bentuk batch, walaupun kita cuma mau prediksi satu gambar.

9. Menyimpan & Memuat Ulang Model

Ada dua cara menyimpan model di PyTorch:

# Cara 1: simpan hanya bobot model (paling umum & direkomendasikan)
torch.save(model.state_dict(), "model_mnist.pth")

# Cara 2: simpan seluruh objek model (jarang dipakai, tapi tetap bisa)
torch.save(model, "full_model_mnist.pth")

Untuk memuat ulang model dari bobot yang disimpan:

model = SimpleNN().to(device)
model.load_state_dict(torch.load("model_mnist.pth", map_location=device))
model.eval()

💡 map_location=device penting kalau kamu menyimpan model dari mesin dengan GPU lalu ingin memuatnya di mesin tanpa GPU (atau sebaliknya) — tanpa ini, torch.load() bisa melempar error karena mencoba memuat tensor ke device yang tidak tersedia.

Checklist Recap

• [ ] Paham beda alur ML (manual feature extraction) vs DL (otomatis)

• [ ] Bisa jelaskan rumus dasar satu neuron: Z = W·X + b → φ(Z)

• [ ] Tahu tiga jenis layer: input, hidden, output

• [ ] Paham kenapa ReLU dipakai di hidden layer, bukan output

• [ ] Bisa susun workflow PyTorch: dataset → model → loss/optimizer → training loop → evaluasi → inference

• [ ] Ngerti kenapa model.eval() + torch.no_grad() dipakai saat evaluasi/inference

• [ ] Tahu kenapa evaluasi harus pakai test_loader, bukan train_loader

• [ ] Bisa menyimpan & memuat ulang model dengan state_dict()

Kuis Mini

1. Kenapa model SimpleNN di atas tidak diberi layer Softmax di akhir?

Karena nn.CrossEntropyLoss() di PyTorch sudah menerapkan log-softmax secara internal. Menambahkan Softmax manual di model akan membuat aktivasi tersebut diterapkan dua kali, yang justru mengacaukan proses training.

2. Apa akibatnya kalau evaluasi model dilakukan memakai train_loader alih-alih test_loader?

Angka akurasi yang dihasilkan akan bias — mencerminkan seberapa baik model "menghafal" data training, bukan seberapa baik model bisa generalisasi ke data baru yang belum pernah dilihat.

3. Fungsi torch.no_grad() saat evaluasi itu untuk apa?

Menonaktifkan perhitungan gradien, karena saat evaluasi/inference kita tidak melakukan backward(). Ini membuat proses lebih cepat dan hemat memori dibanding kalau gradien tetap dihitung.

Latihan Lanjutan

Kalau mau eksplorasi lebih jauh dari notebook ini:

1. Ubah ukuran hidden layer dari 128 menjadi 256 atau 64 — amati pengaruhnya ke akurasi.

2. Tambahkan layer fully-connected baru (jadi 3 layer, bukan 2).

3. Ganti optimizer dari Adam ke SGD, bandingkan kecepatan konvergensinya.

4. Tambahkan Dropout di model untuk melihat efeknya terhadap overfitting.

Penutup

Dari materi ini, hal yang paling nempel buat saya adalah: deep learning itu bukan sihir — cuma neuron sederhana (Z = W·X + b) yang disusun berlapis-lapis, dilatih lewat proses trial-and-error yang sangat terstruktur (forward pass → hitung loss → backward pass → update bobot). Yang bikin powerful adalah skala dan pengulangannya, bukan kerumitan satu neuronnya.

Notebook lengkap (sudah diverifikasi end-to-end) bisa dicek di GitHub repo ini.

Bagian dari seri catatan belajar AI Engineering saya — AI Notes & Engineering

Table of Contents

• What Is Deep Learning?

• Biological Neurons vs Artificial Neurons

• Why "Deep"?

• Neural Network Architecture

• Activation Functions

• How AI Learns (The Training Process)

• Loss Function & Optimizer

• Machine Learning vs Deep Learning

• Real-World Applications

• Practice: Building My First Neural Network

• Level Up: Fashion-MNIST

• Mini Challenge: Hyperparameter Experiments

• Common Mistakes to Avoid

• Quiz: Test Your Understanding

• Conclusion

What Is Deep Learning?

Deep learning is an AI technique that processes information through many layers — similar to how the human brain has different levels of understanding. Each layer processes the output of the layer before it, so understanding gets deeper step by step: from simple lines, to more complex shapes, to eventually recognizing a whole object.

The core idea is simple: the more layers, the deeper the understanding a model can build.

Biological Neurons vs Artificial Neurons

Neural networks really are inspired by how the human brain works. A biological neuron receives an electrical signal, processes it, then passes it on to other neurons. An artificial neuron mimics the same flow — just with numbers instead of electrical signals.

The scale is wildly different — the human brain has around 86 billion neurons, while an artificial neural network usually runs anywhere from a few dozen to millions of neurons, depending on how complex the problem is.

Why "Deep"?

🤔 Try to Guess First: what do you think separates a 1-layer network from a many-layer network in terms of the kinds of patterns each can recognize?

A 1-layer network can only separate simple, linear patterns — picture drawing a single straight line to split two groups of data. A network with many layers can recognize complex, non-linear patterns — faces, voices, language, even emotional tone in text. Every extra layer gives the model a new "vocabulary" for describing more complicated patterns.

That's the reason for the word "deep" in deep learning, not just "learning" — depth itself (the number of layers) is what lets a model capture that complexity.

Neural Network Architecture

Data flows through three kinds of layers:

This flow — data in, processed through hidden layers, out as a prediction — is called forward propagation.

At the level of a single neuron, here's what happens:

1. Input — receives raw data (image pixels, numbers, etc.)

2. Processing — each input is multiplied by its weight, summed, and a bias is added

3. Activation Function — decides whether this neuron "fires" or not

4. Output — the result is sent to neurons in the next layer

Activation Functions

An activation function is basically a smart on/off switch inside each neuron — like a brain deciding whether an idea is worth passing along.

How AI Learns (The Training Process)

Training repeats over and over (each full pass is called an epoch) until the model reaches the accuracy you're aiming for:

Loss Function & Optimizer

A loss function measures how far off a prediction is from the actual value — the smaller the loss, the better the model. For a classification task like MNIST, I used Sparse Categorical Crossentropy (the sparse version, since the dataset's labels are plain integers 0–9, not one-hot encoded).

To steer the model toward the lowest possible loss, we use an optimizer:

Machine Learning vs Deep Learning

Choose Machine Learning when:

• [ ] Your dataset is small (thousands to tens of thousands of rows)

• [ ] You need results that are easy to explain

• [ ] Compute resources are limited

• [ ] The problem is well-defined with clear features

Choose Deep Learning when:

• [ ] Your dataset is very large (hundreds of thousands to millions of rows)

• [ ] The data is complex: images, video, audio, text

• [ ] You have access to a GPU/TPU for training

• [ ] You need the highest accuracy possible

Since the goal here is image recognition (MNIST), the hands-on section below leans on deep learning.

Real-World Applications

The same concepts behind MNIST — just with far bigger architectures and far more data — power a lot of real-world use cases:

From recognizing a simple digit to technology reshaping entire industries, the underlying concepts are the same ones practiced below.

Practice: Building My First Neural Network

Everything below was run end-to-end in Google Colab — so every number you see here is a real, executed result, not an estimate.

Step 1 — Load the MNIST Dataset

MNIST contains 70,000 images of handwritten digits (0–9), each 28×28 pixels — a classic dataset for learning deep learning.

from tensorflow import keras

(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()

Step 2 — Build the Model

model = keras.Sequential([
    keras.layers.Flatten(input_shape=(28, 28)),
    keras.layers.Dense(128, activation='relu'),
    keras.layers.Dense(64, activation='relu'),
    keras.layers.Dense(10, activation='softmax')
])

• Flatten — turns the 2D image (28×28) into a 1D array (784 elements)

• Dense 128 — first hidden layer, 128 neurons, learns basic features

• Dense 64 — second hidden layer, 64 neurons, extracts more abstract features

• Dense 10 — output layer, one neuron per digit (0–9), softmax for probabilities

Total trainable parameters: 109,386.

Step 3 — Compile the Model

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

Step 4 — Train the Model

history = model.fit(x_train, y_train, epochs=5, validation_split=0.2)

The AI "goes to school" on 48,000 images (80% of the 60,000 training images, with the rest held out as a validation set). Each epoch is one full pass through all the training data.

Accuracy climbs consistently epoch over epoch while loss drops — a sign the training is going smoothly.

Step 5 — Evaluate the Model

test_loss, test_accuracy = model.evaluate(x_test, y_test)
print(f'Test accuracy: {test_accuracy:.4f}')

The model is tested on 10,000 images it has never seen before — like a final exam.

Result: Test accuracy 97.39%, Test loss 0.0899.

Step 6 — The Moment AI Recognizes a Digit

predictions = model.predict(x_test[:12])

No manual rule was ever written to recognize each digit — the model learned this entirely on its own from thousands of examples.

Step 7 — When AI Gets It Wrong

Out of 10,000 test images, the model misclassified 261 images (≈2.6%). A few examples below — many of these are genuinely ambiguous handwriting, even to a human eye:

Level Up: Fashion-MNIST

After digits, the next challenge is recognizing clothing. Fashion-MNIST has 10 categories (T-shirt, Trouser, Pullover, Dress, Coat, Sandal, and more) — a harder task, since the differences between categories are more subtle than they are for digits.

I used the exact same architecture (Flatten → Dense 128 → Dense 64 → Dense 10) to show that the concept is universal — only the data changes.

Result: Test accuracy 87.30%, Test loss 0.3463 — lower than MNIST, which makes sense since clothing visuals are inherently more ambiguous than digit shapes.

Mini Challenge: Hyperparameter Experiments

The material included a few "mini challenges" — I actually ran all three, instead of just imagining what might happen:

🤔 Try to Guess First: of these three experiments, which do you think had the biggest impact on accuracy?

Based on the numbers above, increasing the number of neurons (256) gave the biggest accuracy boost compared to adding more layers or training longer. But there's an important caveat: in the "more epochs" experiment (10 epochs), validation loss started fluctuating after epoch 5, while training loss kept dropping steadily — an early sign of overfitting. So "training longer" doesn't automatically mean "better."

Common Mistakes to Avoid

Quiz: Test Your Understanding

1. Why use sparse_categorical_crossentropy instead of categorical_crossentropy for the MNIST dataset?

Because the labels from keras.datasets.mnist.load_data() are plain integers (0–9), not one-hot encoded. sparse_categorical_crossentropy is designed to accept integer labels directly, so no extra encoding step is needed.

2. If training accuracy keeps climbing but validation accuracy stalls or drops, what's happening?

That's a sign of overfitting — the model is fitting the training data so closely that it's losing its ability to generalize to new data.

3. Why does Fashion-MNIST end up with lower accuracy (87.30%) than MNIST (97.39%) even though the architecture is exactly the same?

Because the data itself is harder. The visual patterns of clothing (e.g. telling a "Shirt" apart from a "Coat" based on silhouette alone) are far more ambiguous than the shapes of handwritten digits, which tend to be more structurally distinct.

Conclusion

A few things I'm taking away from this:

• A neural network is an artificial brain — layers of neurons working together to recognize complex patterns, inspired by how the human brain works.

• Deep learning shines on complex data — images, audio, text — because it can learn features on its own without manual feature engineering.

• Good training numbers don't guarantee good real-world numbers — a validation set and watching for overfitting are non-negotiable.

• Small experiments give real insight — the mini challenge taught me that "bigger" (more neurons) isn't automatically weaker than "deeper" (more layers) or "longer" (more epochs) — it really depends on the case.

The same concepts in this notebook — layers, activation functions, loss, backpropagation — are the foundation for more advanced architectures like CNNs for images or Transformers for text, which will likely be the next thing I dig into.

💻 Full code and notebook, ready to run, is on GitHub: github.com/arielshakaramiro/deep-learning-mnist-neural-network-arielshakaramiro

Practice material from the Fullstack Bangalore AI Engineer Bootcamp.

Daftar Isi

• Apa Itu Deep Learning?

• Neuron Biologis vs Neuron Buatan

• Kenapa Harus "Deep"?

• Arsitektur Neural Network

• Activation Function

• Bagaimana AI Belajar (Training Process)

• Loss Function & Optimizer

• Machine Learning vs Deep Learning

• Praktik: Membangun Neural Network Pertama

• Level Up: Fashion-MNIST

• Mini Challenge: Eksperimen Hyperparameter

• Kesalahan Umum yang Perlu Dihindari

• Quiz: Uji Pemahaman

• Kesimpulan

Apa Itu Deep Learning?

Deep learning adalah teknik AI yang memproses informasi lewat banyak lapisan (layers) — mirip cara otak manusia punya berbagai tingkat pemahaman. Setiap layer memproses hasil dari layer sebelumnya, jadi pemahamannya makin lama makin dalam: dari mengenali garis sederhana, ke bentuk yang lebih kompleks, sampai akhirnya mengenali objek utuh.

Intinya sederhana: makin banyak layer, makin dalam pemahaman yang bisa dibangun model.

Neuron Biologis vs Neuron Buatan

Neural network memang terinspirasi dari cara kerja otak manusia. Neuron biologis menerima sinyal listrik, memprosesnya, lalu meneruskan ke neuron lain. Neuron buatan meniru alur yang sama, cuma bahasanya angka, bukan sinyal listrik.

Skalanya jauh beda — otak manusia punya sekitar 86 miliar neuron, sedangkan neural network buatan biasanya "cuma" mulai dari puluhan sampai jutaan neuron, tergantung seberapa rumit masalah yang mau diselesaikan.

Kenapa Harus "Deep"?

🤔 Coba Tebak Dulu: menurutmu, apa yang membedakan network 1 layer dengan network banyak layer dalam hal jenis pola yang bisa dikenali?

Network 1 layer cuma bisa memisahkan pola yang sederhana dan linear — bayangkan menarik satu garis lurus untuk memisahkan dua kelompok data. Network dengan banyak layer bisa mengenali pola yang kompleks dan non-linear — wajah, suara, bahasa, bahkan pola emosi dalam teks. Setiap layer tambahan memberi model "kosakata" baru untuk mendeskripsikan pola yang lebih rumit.

Ini alasan kenapa istilahnya "deep" learning, bukan sekadar "learning" — kedalaman (jumlah layer) itu sendiri yang membuat model bisa menangkap kompleksitas.

Arsitektur Neural Network

Data mengalir lewat tiga jenis layer:

Alur ini — data masuk, diolah di hidden layers, keluar sebagai prediksi — disebut forward propagation.

Di level neuron, prosesnya begini:

1. Input — menerima data mentah (pixel gambar, angka, dll)

2. Pemrosesan — tiap input dikalikan weight-nya, dijumlahkan, ditambah bias

3. Activation Function — menentukan apakah neuron ini "aktif" atau tidak

4. Output — hasil dikirim ke neuron di layer berikutnya

Activation Function

Activation function itu semacam saklar on/off yang pintar di tiap neuron — mirip otak yang memutuskan apakah suatu ide layak diteruskan atau tidak.

Bagaimana AI Belajar (Training Process)

Proses training itu diulang berkali-kali (disebut epoch) sampai model mencapai akurasi yang diinginkan:

Loss Function & Optimizer

Loss function mengukur seberapa jauh prediksi model dari nilai sebenarnya — makin kecil loss, makin bagus modelnya. Untuk klasifikasi seperti MNIST, saya pakai Sparse Categorical Crossentropy (versi sparse karena label datasetnya berupa angka integer 0–9, bukan format one-hot).

Untuk mengarahkan model menuju loss terkecil, dipakai optimizer:

Machine Learning vs Deep Learning

Pilih Machine Learning kalau:

• [ ] Dataset kecil (ribuan–puluhan ribu baris)

• [ ] Butuh hasil yang bisa dijelaskan dengan jelas

• [ ] Resource komputasi terbatas

• [ ] Problem sudah well-defined dengan fitur yang jelas

Pilih Deep Learning kalau:

• [ ] Dataset sangat besar (ratusan ribu–jutaan baris)

• [ ] Data kompleks: gambar, video, audio, teks

• [ ] Ada akses ke GPU/TPU untuk training

• [ ] Butuh akurasi setinggi mungkin

Karena mau tackle problem image recognition (MNIST), bagian praktik di bawah ini fokus ke deep learning.

Contoh Penerapan di Dunia Nyata

Konsep yang sama seperti di MNIST — cuma dengan arsitektur yang jauh lebih besar dan data yang jauh lebih banyak — dipakai untuk berbagai use case nyata:

Dari mengenali digit sederhana sampai teknologi yang mengubah banyak industri, fondasi konsepnya sama seperti yang dipraktikkan di bawah ini.

Praktik: Membangun Neural Network Pertama

Semua kode di bawah ini saya jalankan penuh di Google Colab — jadi semua angka yang muncul memang hasil eksekusi nyata, bukan perkiraan.

Step 1 — Load Dataset MNIST

MNIST berisi 70.000 gambar angka tulisan tangan (0–9) berukuran 28×28 pixel — dataset klasik untuk belajar deep learning.

from tensorflow import keras

(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()

Step 2 — Build Model

model = keras.Sequential([
    keras.layers.Flatten(input_shape=(28, 28)),
    keras.layers.Dense(128, activation='relu'),
    keras.layers.Dense(64, activation='relu'),
    keras.layers.Dense(10, activation='softmax')
])

• Flatten — ubah gambar 2D (28×28) jadi array 1D (784 elemen)

• Dense 128 — hidden layer pertama, 128 neuron, belajar fitur dasar

• Dense 64 — hidden layer kedua, 64 neuron, ekstraksi fitur lebih abstrak

• Dense 10 — output layer, satu neuron per digit (0–9), softmax untuk probabilitas

Total parameter yang dilatih: 109.386.

Step 3 — Compile Model

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

Step 4 — Train Model

history = model.fit(x_train, y_train, epochs=5, validation_split=0.2)

AI "sekolah" dari 48.000 gambar (80% dari 60.000 data training, sisanya jadi validation set). Tiap epoch adalah satu putaran penuh melewati semua data training.

Accuracy naik konsisten dari epoch ke epoch, loss turun — tanda training berjalan sehat.

Step 5 — Evaluate Model

test_loss, test_accuracy = model.evaluate(x_test, y_test)
print(f'Test accuracy: {test_accuracy:.4f}')

Model diuji dengan 10.000 gambar yang belum pernah dilihat sebelumnya — seperti ujian akhir.

Hasil: Test accuracy 97.39%, Test loss 0.0899.

Step 6 — Momen AI Mengenali Angka

predictions = model.predict(x_test[:12])

Tidak ada rule manual yang ditulis untuk mengenali tiap angka — model belajar sendiri dari ribuan contoh.

Step 7 — Saat AI Salah Menebak

Dari 10.000 gambar test, model salah menebak 261 gambar (≈2.6%). Beberapa contohnya di bawah — banyak yang memang tulisan tangan ambigu, bahkan buat mata manusia:

Level Up: Fashion-MNIST

Setelah berhasil dengan angka, tantangan berikutnya: mengenali jenis pakaian. Fashion-MNIST punya 10 kategori (T-shirt, Trouser, Pullover, Dress, Coat, Sandal, dll) — lebih sulit karena perbedaan antar kategori lebih subtle dibanding angka.

Saya pakai arsitektur persis sama (Flatten → Dense 128 → Dense 64 → Dense 10) untuk menunjukkan bahwa konsepnya universal, cuma datanya yang beda.

Hasil: Test accuracy 87.30%, Test loss 0.3463 — turun dibanding MNIST, wajar karena pola visual pakaian memang lebih ambigu daripada pola angka.

Mini Challenge: Eksperimen Hyperparameter

Materinya kasih beberapa "mini challenge" — saya coba jalankan ketiganya beneran, bukan cuma dibayangkan:

🤔 Coba Tebak Dulu: dari ketiga eksperimen ini, mana yang menurutmu paling ampuh naikkin akurasi?

Dari data di atas, menambah jumlah neuron (256) memberi peningkatan akurasi paling besar dibanding menambah layer atau menambah epoch. Tapi ada catatan penting: pada eksperimen "tambah epoch" (10 epoch), validation loss mulai naik-turun tidak stabil setelah epoch ke-5, sementara training loss terus turun — ini sinyal awal overfitting. Jadi "lebih lama training" nggak selalu berarti "lebih baik".

Kesalahan Umum yang Perlu Dihindari

Quiz: Uji Pemahaman

1. Kenapa dipakai sparse_categorical_crossentropy, bukan categorical_crossentropy, untuk dataset MNIST?

Karena label dari keras.datasets.mnist.load_data() berupa angka integer (0–9), bukan format one-hot encoded. sparse_categorical_crossentropy didesain untuk menerima label integer langsung, jadi tidak perlu encoding tambahan.

2. Kalau accuracy training terus naik tapi validation accuracy malah stagnan atau turun, apa yang sedang terjadi?

Ini tanda overfitting — model terlalu menyesuaikan diri dengan data training sampai kehilangan kemampuan generalisasi ke data baru.

3. Kenapa Fashion-MNIST punya akurasi lebih rendah (87.30%) dibanding MNIST (97.39%) padahal arsitekturnya sama persis?

Karena tingkat kesulitan datanya beda. Pola visual pakaian (misalnya membedakan "Shirt" dan "Coat" hanya dari siluet) jauh lebih ambigu dibanding pola angka tulisan tangan yang bentuknya relatif lebih terstruktur.

Kesimpulan

Beberapa hal yang saya bawa pulang dari sesi ini:

• Neural network = otak buatan — layer-layer neuron bekerja sama mengenali pola kompleks, terinspirasi dari cara kerja otak manusia.

• Deep learning unggul untuk data kompleks — gambar, suara, teks — karena bisa belajar fitur sendiri tanpa perlu feature engineering manual.

• Angka bagus di training tidak menjamin angka bagus di dunia nyata — validation set dan pemantauan overfitting itu wajib.

• Eksperimen kecil kasih insight nyata — dari mini challenge, saya belajar bahwa "lebih besar" (neuron) belum tentu kalah efektif dibanding "lebih dalam" (layer) atau "lebih lama" (epoch), tergantung kasusnya.

Konsep yang sama di notebook ini — layer, activation function, loss, backpropagation — jadi fondasi untuk arsitektur yang lebih kompleks seperti CNN untuk gambar atau Transformer untuk teks, yang kemungkinan jadi bahasan lanjutan berikutnya.

💻 Kode lengkap dan notebook yang bisa langsung dijalankan ada di GitHub: github.com/arielshakaramiro/deep-learning-mnist-neural-network-arielshakaramiro

Materi latihan dari Fullstack Bangalore AI Engineer Bootcamp.

In Part 1, I covered the theory behind Supervised vs Unsupervised Learning, and in Part 2 I put the supervised side into practice — training a model to recognize Iris species from labels it already had. Now it's time for the side that hasn't been practiced yet: Unsupervised Learning. This time the model isn't told what species anything is — it has to figure out the grouping entirely on its own, just from patterns in the data.

Same Iris flowers, a different angle. Every piece of code in this post has been run and verified myself.

Table of Contents

• From Supervised to Unsupervised

• Setup & Load Data

• Clustering with KMeans

• Evaluating Clustering: Adjusted Rand Index

• Visualization: Clusters vs Ground Truth

• Experiment 1: Random Forest vs Logistic Regression

• Experiment 2: Effect of Cluster Count on ARI

• Experiment 3: Effect of Scaling on Clustering

• Experiment 4: Another Dataset — Wine

• Source Code

• Lessons From All the Experiments

• Checklist

• Summary

From Supervised to Unsupervised

In Part 2, the flow was clear: we had X (features) and y (species labels), the model learned the relationship between them, then got tested on new data.

In Unsupervised Learning, y is completely hidden from the model. It only gets X, and its job is to find structure or groupings on its own — purely from similarity between data points. Only after clustering is done do we "reveal" the true labels to check how close the model's groupings are to reality.

Setup & Load Data

import numpy as np
import matplotlib.pyplot as plt

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, classification_report, adjusted_rand_score
from sklearn.cluster import KMeans
from sklearn.decomposition import PCA

iris = load_iris()
X = iris.data
y = iris.target
feature_names = iris.feature_names
target_names = iris.target_names

The dataset and X, y variables here are exactly the same as what we used in Part 2 — 150 samples, 4 features, 3 species.

Clustering with KMeans

n_clusters = 3  # we know Iris has 3 classes, but KMeans doesn't know the class names
kmeans = KMeans(n_clusters=n_clusters, random_state=42, n_init=10)
cluster_labels = kmeans.fit_predict(X)

print("Unique cluster labels:", np.unique(cluster_labels))

Notice: KMeans.fit_predict(X) only takes X — no y at all. KMeans works by iteratively placing 3 "center points" (centroids), grouping each sample to its nearest centroid, and repeating until the centroid positions stabilize.

The result is a set of cluster labels (0, 1, 2) — but importantly, these numbers don't automatically correspond to the true species label order (0=setosa, 1=versicolor, 2=virginica). KMeans's cluster 0 isn't necessarily the same as class 0 in the original data.

Evaluating Clustering: ARI

Since cluster labels don't automatically map to the true labels, we can't just use regular accuracy_score. This is where Adjusted Rand Index (ARI) comes in — a metric that measures how similar two groupings are, regardless of how the labels are numbered.

ari = adjusted_rand_score(y, cluster_labels)
print("Adjusted Rand Index (ARI):", round(ari, 4))

Verified result: ARI = 0.7302

ARI ranges from -1 to 1: the closer to 1.0, the more similar the clustering is to the true labels. A score of 0.73 is fairly good — a sign that the natural structure in the Iris data actually aligns well with the true species split, even though the model was never "told" any species names.

🤔 Guess First: why isn't ARI a perfect 1.0, when Logistic Regression in [Part 2](https://shaka-ai.hashnode.dev/hands-on-iris-classification-logistic-regression-vs-decision-tree) hit 96.67% accuracy?

Because the tasks are different. Logistic Regression in Part 2 was given the answers through labels during training — it just had to learn the boundary between classes. KMeans here has zero idea which of the 3 species is which — it only groups based on distance between data points. Since two species (versicolor and virginica) have naturally similar and slightly overlapping traits, KMeans can misplace a few boundary samples — the exact same kind of mistake that also showed up in Part 2's supervised model.

Visualization: Clusters vs Ground Truth

pca = PCA(n_components=2)
X_2d = pca.fit_transform(X)

plt.figure(figsize=(10, 4))

plt.subplot(1, 2, 1)
plt.title("KMeans Clustering (Unsupervised)")
scatter1 = plt.scatter(X_2d[:, 0], X_2d[:, 1], c=cluster_labels, alpha=0.7)
plt.xlabel("PCA 1")
plt.ylabel("PCA 2")
plt.colorbar(scatter1, label="Cluster ID")

plt.subplot(1, 2, 2)
plt.title("True Labels (Ground Truth)")
scatter2 = plt.scatter(X_2d[:, 0], X_2d[:, 1], c=y, alpha=0.7)
plt.xlabel("PCA 1")
plt.ylabel("PCA 2")
plt.colorbar(scatter2, label="True Class")

plt.tight_layout()
plt.show()

PCA here is used purely for visualization — reducing Iris's 4 features down to 2 dimensions so it can be plotted as a scatter plot. Placed side by side, the two plots' color patterns look strikingly similar — a visual confirmation of that 0.73 ARI score above: one cluster (setosa) sits far apart and perfectly separated, while the other two (versicolor–virginica) overlap slightly in both plots.

Experiment 1: Random Forest vs Logistic Regression

Beyond the unsupervised practice, I also explored a few follow-up questions. First: does a more complex model automatically perform better?

Update: the original version of this section compared two models' raw test accuracy directly to decide which was better — the exact same methodology problem a reader (Ahmet Özel) flagged in the Part 2 comments, just in "model selection" form instead of "hyperparameter selection" form. Once you compare several models via test scores and declare a winner, that test score stops being an honest estimate. The correct approach: compare via cross-validation on the training set first, and only touch the test set once, with the winning model, at the end.

Using the exact same train/test split as Part 2:

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

Step 1 — Compare via cross-validation (training set only):

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import StratifiedKFold, cross_val_score

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

lr_scores = cross_val_score(LogisticRegression(max_iter=200), X_train, y_train, cv=cv)
rf_scores = cross_val_score(RandomForestClassifier(random_state=42), X_train, y_train, cv=cv)

print("Logistic Regression — CV accuracy:", round(lr_scores.mean(), 4))
print("Random Forest — CV accuracy:", round(rf_scores.mean(), 4))

Based on this — without touching the test set at all — Logistic Regression wins.

Step 2 — Touch the test set once, with the winning model:

final_model = LogisticRegression(max_iter=200)
final_model.fit(X_train, y_train)
final_acc = accuracy_score(y_test, final_model.predict(X_test))
print("Final test accuracy:", round(final_acc, 4))

Result: 96.67% — the same Logistic Regression number already reported in Part 2, but this time the "Logistic Regression beats Random Forest" decision came purely from the training set.

Interestingly, the final conclusion doesn't change from the old approach (Random Forest still loses), but that's a coincidence — and it's exactly why the process still has to be correct: on a different dataset where the result happened to differ, the flawed approach (comparing directly on the test set) would never reveal its own mistake until the model was deployed and underperformed expectations.

🤔 For reference: what happens if we still compare directly on the test set (the discouraged way)?

Random Forest = 90.0% on the test set, Logistic Regression = 96.67% on the test set. Same conclusion, but that 90.0% Random Forest number isn't fully trustworthy as an estimate, technically speaking, since it came from the same process used to "compare and choose" — not from a clean final evaluation.

Experiment 2: Effect of Cluster Count on ARI

In the real world, we don't always know the "correct" number of groups. What happens if we change n_clusters?

for k in range(2, 7):
    km_k = KMeans(n_clusters=k, random_state=42, n_init=10)
    labels_k = km_k.fit_predict(X)
    ari_k = adjusted_rand_score(y, labels_k)
    print(f"n_clusters={k}: ARI = {round(ari_k, 4)}")

The highest ARI consistently lands exactly at n_clusters=3 — matching the true number of species. Both decreasing and increasing from that number drops the ARI. This confirms how important it is to validate the number of clusters (e.g. via elbow method or silhouette score), rather than guessing a number.

Experiment 3: Effect of Scaling on Clustering

from sklearn.preprocessing import StandardScaler

X_scaled = StandardScaler().fit_transform(X)
kmeans_scaled = KMeans(n_clusters=3, random_state=42, n_init=10)
cluster_labels_scaled = kmeans_scaled.fit_predict(X_scaled)
ari_scaled = adjusted_rand_score(y, cluster_labels_scaled)

print("ARI without scaling:", round(ari, 4))
print("ARI with scaling   :", round(ari_scaled, 4))

Verified result: ARI without scaling = 0.7302, with scaling = 0.6201.

This surprised me before I ran it — scaling is usually treated as a default "best practice" before clustering. But in Iris's case, scaling actually decreased ARI. The reason: all four Iris features are already on the same unit scale (cm), so forcing every feature into a standardized scale actually erased a bit of relative information that was previously helping KMeans distinguish clusters.

Experiment 4: Another Dataset — Wine

To see whether the pattern above holds generally, I tried a dataset whose features span a much wider range of scales.

from sklearn.datasets import load_wine

wine = load_wine()
X_wine, y_wine = wine.data, wine.target

# Clustering without scaling
kmeans_wine = KMeans(n_clusters=3, random_state=42, n_init=10)
cluster_wine = kmeans_wine.fit_predict(X_wine)
ari_wine = adjusted_rand_score(y_wine, cluster_wine)

# Clustering with scaling
X_wine_scaled = StandardScaler().fit_transform(X_wine)
kmeans_wine_scaled = KMeans(n_clusters=3, random_state=42, n_init=10)
cluster_wine_scaled = kmeans_wine_scaled.fit_predict(X_wine_scaled)
ari_wine_scaled = adjusted_rand_score(y_wine, cluster_wine_scaled)

print("Wine ARI without scaling:", round(ari_wine, 4))
print("Wine ARI with scaling   :", round(ari_wine_scaled, 4))

A stark contrast. The proline feature in the Wine dataset is on a scale of hundreds, while other features stay under 10 — so without scaling, KMeans effectively only "sees" the proline feature when computing distances between points, drowning out other genuinely informative features. After scaling, ARI jumped to 0.8975.

As an additional note, Logistic Regression on the Wine dataset (without scaling) also triggered a ConvergenceWarning — a sign that this feature-scale issue doesn't only affect distance-based clustering, but also gradient-based models like Logistic Regression.

Source Code

The full notebook behind this post — complete with real output, visualizations, and all 4 independent experiments — is available to run yourself via this GitHub repo:

🔗 github.com/arielshakaramiro/supervised-unsupervised-learning-praktik-arielshakaramiro

No dataset download needed — both Iris and Wine ship built into scikit-learn. Just pip install -r requirements.txt and run the notebook top to bottom.

Lessons From All the Experiments

📋 Click to reveal the full summary

• A more complex model ≠ automatically more accurate — default Random Forest underperformed Logistic Regression on the small Iris test set

• The number of clusters needs to be validated, not guessed — ARI consistently drops when n_clusters moves away from the true number of groups

• Scaling isn't an automatic win — it can help dramatically (Wine) or actually hurt slightly (Iris), depending on how the original features are distributed across scales

• The only honest way to know what's right: try it, measure it, compare it — not assume from theory alone

Checklist

• [ ] Understand the difference between the supervised workflow (uses labels) and unsupervised workflow (no labels)

• [ ] Understand why accuracy_score doesn't work for clustering, and why ARI is used instead

• [ ] Can explain why PCA is used here purely for visualization, not for training

• [ ] Understand that model complexity and preprocessing (scaling) need to be tested case by case, not assumed to always help

Summary

Unsupervised Learning proved something interesting: without being told a single species name, KMeans still managed to find structure similar to the true grouping (ARI 0.73) — purely from distance patterns in the data. But this result doesn't stand alone; the string of experiments above shows that model performance, cluster count, and preprocessing all interact with each other, and no single rule applies across every case.

Across this three-part series — theory (Part 1), supervised practice (Part 2), and unsupervised practice plus experiments (this Part 3) — one thread stays the same: don't trust a number until you've measured it yourself.

Part of my AI Engineering study notes. Read Part 1 (theory) and Part 2 (supervised classification practice) for full context. Full source code: GitHub.

Di Part 1 saya bahas teori Supervised vs Unsupervised Learning, dan di Part 2 saya praktikkan sisi supervised-nya — melatih model buat mengenali spesies Iris dari label yang sudah ada. Sekarang giliran sisi yang belum pernah dipraktikkan: Unsupervised Learning. Kali ini modelnya nggak dikasih tahu spesies apa itu — dia harus cari sendiri pengelompokannya cuma dari pola data.

Bunga Iris yang sama, sudut pandang yang berbeda. Semua kode di post ini sudah saya jalankan sendiri dan angkanya terverifikasi.

Daftar Isi

• Dari Supervised ke Unsupervised

• Setup & Load Data

• Clustering dengan KMeans

• Evaluasi Clustering: Adjusted Rand Index

• Visualisasi: Cluster vs Label Asli

• Eksperimen 1: Random Forest vs Logistic Regression

• Eksperimen 2: Efek Jumlah Cluster terhadap ARI

• Eksperimen 3: Efek Scaling pada Clustering

• Eksperimen 4: Dataset Lain — Wine

• Source Code

• Pelajaran dari Semua Eksperimen

• Checklist

• Ringkasan

Dari Supervised ke Unsupervised

Di Part 2, alur kerjanya jelas: kita punya X (fitur) dan y (label spesies), model belajar hubungan keduanya, lalu diuji ke data baru.

Di Unsupervised Learning, y-nya kita sembunyikan sepenuhnya dari model. Model cuma dikasih X, dan tugasnya adalah menemukan struktur atau pengelompokan sendiri — murni dari kemiripan antar data. Setelah clustering selesai, baru kita "buka" label aslinya buat mengecek seberapa dekat hasil pengelompokan model dengan kenyataan.

Setup & Load Data

import numpy as np
import matplotlib.pyplot as plt

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, classification_report, adjusted_rand_score
from sklearn.cluster import KMeans
from sklearn.decomposition import PCA

iris = load_iris()
X = iris.data
y = iris.target
feature_names = iris.feature_names
target_names = iris.target_names

Dataset dan variabel X, y di sini sama persis dengan yang dipakai di Part 2 — 150 sampel, 4 fitur, 3 spesies.

Clustering dengan KMeans

n_clusters = 3  # kita tahu Iris punya 3 kelas, tapi KMeans tidak tahu nama kelas
kmeans = KMeans(n_clusters=n_clusters, random_state=42, n_init=10)
cluster_labels = kmeans.fit_predict(X)

print("Cluster label unik:", np.unique(cluster_labels))

Perhatikan: KMeans.fit_predict(X) cuma menerima X, nggak ada y sama sekali. KMeans bekerja dengan cara menempatkan 3 "titik pusat" (centroid) secara iteratif, lalu mengelompokkan tiap sampel ke centroid terdekat, sampai posisi centroid-nya stabil.

Hasilnya adalah label cluster (0, 1, 2) — tapi penting dicatat, angka ini tidak otomatis cocok dengan urutan label spesies asli (0=setosa, 1=versicolor, 2=virginica). Cluster 0 dari KMeans belum tentu sama dengan kelas 0 di data asli.

Evaluasi Clustering: ARI

Karena label cluster nggak otomatis nyambung ke label asli, kita nggak bisa pakai accuracy_score biasa. Di sinilah Adjusted Rand Index (ARI) dipakai — metrik yang mengukur seberapa mirip dua cara pengelompokan, terlepas dari penomoran labelnya.

ari = adjusted_rand_score(y, cluster_labels)
print("Adjusted Rand Index (ARI):", round(ari, 4))

Hasil terverifikasi: ARI = 0.7302

Skala ARI dari -1 sampai 1: semakin dekat ke 1.0, semakin mirip hasil clustering dengan label asli. Nilai 0.73 tergolong cukup baik — tandanya struktur alami di data Iris memang cukup selaras dengan pembagian spesies aslinya, meskipun modelnya sama sekali nggak pernah "diberi tahu" nama spesiesnya.

🤔 Coba Tebak Dulu: kenapa ARI-nya nggak sampai 1.0 sempurna, padahal di [Part 2](https://shaka-ai.hashnode.dev/praktik-klasifikasi-iris-logistic-regression-vs-decision-tree) Logistic Regression bisa dapat akurasi 96.67%?

Karena tugasnya beda. Logistic Regression di Part 2 dikasih tahu jawabannya lewat label saat training — dia tinggal belajar batas antar kelas. KMeans di sini sama sekali nggak tahu ada 3 spesies yang mana — dia cuma mengelompokkan berdasarkan jarak antar titik data. Kalau dua spesies (versicolor dan virginica) punya ciri yang mirip dan sedikit tumpang tindih secara alami, KMeans bisa saja salah menempatkan beberapa sampel di batas itu — persis seperti kesalahan yang juga muncul di model supervised Part 2.

Visualisasi: Cluster vs Label Asli

pca = PCA(n_components=2)
X_2d = pca.fit_transform(X)

plt.figure(figsize=(10, 4))

plt.subplot(1, 2, 1)
plt.title("Clustering KMeans (Unsupervised)")
scatter1 = plt.scatter(X_2d[:, 0], X_2d[:, 1], c=cluster_labels, alpha=0.7)
plt.xlabel("PCA 1")
plt.ylabel("PCA 2")
plt.colorbar(scatter1, label="Cluster ID")

plt.subplot(1, 2, 2)
plt.title("Label Asli (Ground Truth)")
scatter2 = plt.scatter(X_2d[:, 0], X_2d[:, 1], c=y, alpha=0.7)
plt.xlabel("PCA 1")
plt.ylabel("PCA 2")
plt.colorbar(scatter2, label="True Class")

plt.tight_layout()
plt.show()

PCA dipakai di sini murni buat visualisasi — menyederhanakan 4 fitur Iris jadi 2 dimensi biar bisa digambar di scatter plot. Kalau dua plot ini ditaruh bersebelahan, pola warnanya kelihatan sangat mirip — bukti visual dari angka ARI 0.73 di atas: satu cluster (setosa) terpisah jauh dan sempurna, sementara dua cluster lain (versicolor-virginica) sedikit tumpang tindih di keduanya.

Eksperimen 1: Random Forest vs Logistic Regression

Selain praktik unsupervised, saya juga eksplorasi beberapa pertanyaan lanjutan. Pertama: apakah model yang lebih kompleks otomatis lebih baik?

Update: versi awal section ini membandingkan langsung akurasi test dua model buat menyimpulkan mana yang lebih baik — dan itu masalah metodologi yang sama persis dengan yang diangkat pembaca (Ahmet Özel) di kolom komentar Part 2, cuma di sini bentuknya "pilih model" bukan "pilih hyperparameter". Begitu kita bandingkan beberapa model lewat skor test lalu simpulkan mana yang menang, skor test itu udah nggak murni lagi jadi estimasi jujur. Cara yang benar: bandingkan dulu pakai cross-validation di training set, baru model yang menang disentuh ke test set sekali di akhir.

Split train/test-nya sama persis dengan yang dipakai di Part 2:

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

Langkah 1 — Bandingkan lewat cross-validation (cuma di training set):

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import StratifiedKFold, cross_val_score

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

lr_scores = cross_val_score(LogisticRegression(max_iter=200), X_train, y_train, cv=cv)
rf_scores = cross_val_score(RandomForestClassifier(random_state=42), X_train, y_train, cv=cv)

print("Logistic Regression — CV accuracy:", round(lr_scores.mean(), 4))
print("Random Forest — CV accuracy:", round(rf_scores.mean(), 4))

Berdasarkan angka ini — belum menyentuh test set sama sekali — Logistic Regression yang menang.

Langkah 2 — Sentuh test set sekali, pakai model yang menang:

final_model = LogisticRegression(max_iter=200)
final_model.fit(X_train, y_train)
final_acc = accuracy_score(y_test, final_model.predict(X_test))
print("Akurasi test final:", round(final_acc, 4))

Hasil: 96.67% — sama dengan angka Logistic Regression yang sudah dilaporkan di Part 2, tapi sekarang keputusan "Logistic Regression lebih baik dari Random Forest" diambil murni dari training set.

Menariknya, kesimpulan akhirnya nggak berubah dibanding cara yang lama (Random Forest tetap kalah), tapi itu kebetulan — dan itu justru bukti kenapa prosesnya tetap harus benar: kalau di dataset lain hasilnya kebetulan berbeda, cara yang salah (bandingkan langsung di test set) nggak akan pernah ketahuan salahnya sampai model di-deploy dan performanya meleset dari yang diharapkan.

🤔 Referensi: apa yang terjadi kalau kita tetap bandingkan langsung di test set (cara yang tidak disarankan)?

Random Forest = 90.0% di test set, Logistic Regression = 96.67% di test set. Kesimpulannya sama, tapi angka Random Forest 90.0% ini secara teknis bukan estimasi yang bisa dipercaya penuh, karena diambil dari proses yang sama dengan yang dipakai buat "membandingkan dan memilih" — bukan dari evaluasi final yang bersih.

Eksperimen 2: Efek Jumlah Cluster terhadap ARI

Di real-world, kita nggak selalu tahu berapa jumlah kelompok yang "benar." Gimana kalau n_clusters di-ubah?

for k in range(2, 7):
    km_k = KMeans(n_clusters=k, random_state=42, n_init=10)
    labels_k = km_k.fit_predict(X)
    ari_k = adjusted_rand_score(y, labels_k)
    print(f"n_clusters={k}: ARI = {round(ari_k, 4)}")

ARI tertinggi konsisten muncul persis di n_clusters=3 — sesuai jumlah spesies asli. Baik dikurangi maupun ditambah dari angka itu, ARI-nya turun. Ini menegaskan pentingnya validasi jumlah cluster (misalnya lewat elbow method atau silhouette score), bukan asal tebak angka.

Eksperimen 3: Efek Scaling pada Clustering

from sklearn.preprocessing import StandardScaler

X_scaled = StandardScaler().fit_transform(X)
kmeans_scaled = KMeans(n_clusters=3, random_state=42, n_init=10)
cluster_labels_scaled = kmeans_scaled.fit_predict(X_scaled)
ari_scaled = adjusted_rand_score(y, cluster_labels_scaled)

print("ARI tanpa scaling :", round(ari, 4))
print("ARI dengan scaling:", round(ari_scaled, 4))

Hasil terverifikasi: ARI tanpa scaling = 0.7302, dengan scaling = 0.6201.

Ini di luar dugaan saya sebelum dijalankan — biasanya scaling dianggap "praktik baik" secara default sebelum clustering. Tapi di kasus Iris, scaling justru menurunkan ARI. Penyebabnya: keempat fitur Iris sudah berada di skala satuan yang sama (cm), jadi memaksa semua fitur ke skala yang seragam malah menghilangkan sedikit informasi relatif yang tadinya membantu KMeans membedakan cluster.

Eksperimen 4: Dataset Lain — Wine

Buat lihat apakah pola di atas berlaku umum, saya coba dataset lain yang fitur-fiturnya jauh lebih beragam skalanya.

from sklearn.datasets import load_wine

wine = load_wine()
X_wine, y_wine = wine.data, wine.target

# Clustering tanpa scaling
kmeans_wine = KMeans(n_clusters=3, random_state=42, n_init=10)
cluster_wine = kmeans_wine.fit_predict(X_wine)
ari_wine = adjusted_rand_score(y_wine, cluster_wine)

# Clustering dengan scaling
X_wine_scaled = StandardScaler().fit_transform(X_wine)
kmeans_wine_scaled = KMeans(n_clusters=3, random_state=42, n_init=10)
cluster_wine_scaled = kmeans_wine_scaled.fit_predict(X_wine_scaled)
ari_wine_scaled = adjusted_rand_score(y_wine, cluster_wine_scaled)

print("ARI Wine tanpa scaling:", round(ari_wine, 4))
print("ARI Wine dengan scaling:", round(ari_wine_scaled, 4))

Kontras yang jelas. Fitur proline di dataset Wine punya skala ratusan, sementara fitur lain di bawah 10 — jadi tanpa scaling, KMeans praktis cuma "melihat" fitur proline doang saat menghitung jarak antar titik, dan mengabaikan fitur lain yang sebenarnya informatif. Setelah di-scale, ARI-nya melonjak ke 0.8975.

Sebagai catatan tambahan, Logistic Regression di dataset Wine (tanpa scaling) juga sempat memunculkan ConvergenceWarning — tanda bahwa masalah skala fitur ini nggak cuma berdampak ke clustering berbasis jarak, tapi juga ke model berbasis gradient seperti Logistic Regression.

Source Code

Seluruh notebook di post ini — lengkap dengan output asli, visualisasi, dan 4 eksperimen mandiri — bisa dijalankan sendiri lewat repo GitHub berikut:

🔗 github.com/arielshakaramiro/supervised-unsupervised-learning-praktik-arielshakaramiro

Nggak perlu download dataset apa pun — Iris dan Wine sudah built-in di scikit-learn. Tinggal pip install -r requirements.txt lalu jalankan notebook-nya dari atas ke bawah.

Pelajaran dari Semua Eksperimen

📋 Klik untuk lihat rangkuman lengkap

• Model lebih kompleks ≠ otomatis lebih akurat — Random Forest default performanya di bawah Logistic Regression di test set Iris yang kecil

• Jumlah cluster harus divalidasi, bukan ditebak — ARI selalu turun kalau n_clusters jauh dari jumlah kelompok asli

• Scaling bukan langkah otomatis yang selalu menguntungkan — bisa membantu drastis (Wine) atau justru sedikit merugikan (Iris), tergantung sebaran skala fitur aslinya

• Cara paling jujur buat tahu mana yang benar: coba, ukur, bandingkan — bukan asumsi dari teori semata

Checklist

• [ ] Paham bedanya alur kerja supervised (pakai label) vs unsupervised (tanpa label)

• [ ] Ngerti kenapa accuracy_score nggak bisa dipakai buat clustering, dan kenapa ARI dipakai sebagai gantinya

• [ ] Bisa jelasin kenapa PCA dipakai di sini murni untuk visualisasi, bukan untuk training

• [ ] Paham bahwa model kompleks dan preprocessing (scaling) itu perlu diuji per kasus, bukan diasumsikan selalu membantu

Ringkasan

Unsupervised Learning membuktikan sesuatu yang menarik: tanpa dikasih tahu satu pun nama spesies, KMeans tetap berhasil menemukan struktur yang mirip dengan pengelompokan asli (ARI 0.73) — murni dari pola jarak antar data. Tapi hasil ini nggak berdiri sendiri; deretan eksperimen di atas nunjukin kalau performa model, jumlah cluster, dan preprocessing semuanya saling memengaruhi, dan nggak ada satu aturan yang berlaku di semua kasus.

Dari tiga part seri ini — teori (Part 1), praktik supervised (Part 2), dan praktik unsupervised plus eksperimen (Part 3 ini) — satu benang merahnya sama: jangan percaya angka sebelum diukur sendiri.

Bagian dari catatan belajar AI Engineering saya. Baca Part 1 (teori) dan Part 2 (praktik klasifikasi) untuk konteks lengkap. Source code lengkap: GitHub.

In Part 1, I covered the theory behind Supervised vs Unsupervised Learning and how to measure model performance. Now it's time to go hands-on: training an actual model that can recognize flower species from petal measurements, then comparing two different algorithms.

Every piece of code in this post has been run and verified myself — these aren't guessed numbers.

Table of Contents

• Problem: Iris Species Classification

• The Short ML Pipeline

• 1. Load the Dataset

• 2. Split the Data

• 3. First Model: Logistic Regression

• 4. Evaluating Model 1

• 5. Second Model: Decision Tree

• 6. Evaluating Model 2

• Comparing Results

• Mini Practice: Effect of Changing Parameters

• Common Mistakes to Avoid

• Summary

Problem

Dataset: Iris (built into scikit-learn) Task: classify flower species — setosa / versicolor / virginica Input: sepal & petal measurements (4 numeric features)

This is a classic Supervised Learning — Classification example (not regression), since the target is a discrete category (flower species), not a continuous number.

The Short ML Pipeline

Data → Split → Train → Test → Predict → Evaluate

X = features (input to the model). y = labels (the "answer key") we're trying to predict.

1. Load the Dataset

The Iris dataset ships built-in with scikit-learn, so there's no manual download needed:

from sklearn.datasets import load_iris

iris = load_iris(as_frame=True)
df = iris.frame
df.head()

2. Split the Data

The data is split into a training set (for learning) and a test set (for evaluation, unseen by the model):

from sklearn.model_selection import train_test_split

X = df[iris.feature_names]
y = df["target"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

With test_size=0.2 out of the Iris dataset's 150 total rows, the result is: 120 rows for training, 30 rows for testing. The stratify=y parameter matters here — it makes sure each species is proportionally represented in both the train and test sets, instead of a purely random split.

3. First Model: Logistic Regression

from sklearn.linear_model import LogisticRegression

clf = LogisticRegression(max_iter=200)
clf.fit(X_train, y_train)

Intuition behind how it works:

• The model computes a score: z = w·x + b

• That score is converted into a probability via the sigmoid function: p = sigmoid(z)

• .fit() searches for the best w and b values so the model's predictions get closer to the true labels (the loss goes down)

4. Evaluating Model 1

from sklearn.metrics import accuracy_score

pred = clf.predict(X_test)
acc = accuracy_score(y_test, pred)
print(acc)

Verified result: Logistic Regression accuracy = 0.9667 (96.67%)

Out of 30 test samples, the model only got 1 wrong. Here's the confusion matrix:

from sklearn.metrics import ConfusionMatrixDisplay

ConfusionMatrixDisplay.from_predictions(
    y_test, pred, display_labels=iris.target_names, cmap="Blues"
)

The only mistake: 1 versicolor flower predicted as virginica. Setosa is predicted perfectly every time — which makes sense, since morphologically setosa is the easiest of the three species to tell apart.

5. Second Model: Decision Tree

from sklearn.tree import DecisionTreeClassifier

tree = DecisionTreeClassifier(max_depth=3, random_state=42)
tree.fit(X_train, y_train)

Intuition behind how it works:

• The tree picks questions like: feature ≤ threshold?

• Splits are chosen to "purify" the classes — impurity goes down (measured with Gini or Entropy)

• max_depth controls tree complexity: too shallow → underfitting, too deep → overfitting

6. Evaluating Model 2

pred_tree = tree.predict(X_test)
acc_tree = accuracy_score(y_test, pred_tree)
print(acc_tree)

Verified result: Decision Tree accuracy = 0.9667 (96.67%)

Interestingly, the Decision Tree's confusion matrix is exactly identical to Logistic Regression's — both make the same single mistake, mispredicting 1 versicolor as virginica. On a dataset as clean as Iris, where the class boundaries are fairly distinct, two very different algorithms can still land on the exact same result.

Comparing Results

For this case, both are tied. That doesn't mean these two algorithms are always equivalent on other datasets, though — on more complex or non-linear data, results can differ significantly. That's exactly why the model selection process (trying several models, comparing their metrics) matters — not just picking one algorithm and stopping there.

A note on test set size (added after Ahmet Özel flagged this in the comments): the test set here is only 30 samples, so a single sample switching sides moves accuracy by roughly 3.3 percentage points. That means small differences between models on a dataset this size — even ones that look like a 1-2 point gap — are usually within noise, not evidence that one model is genuinely better. It's also why cross-validation (covered in Mini Practice) is more trustworthy than any single test-set number.

Mini Practice

Update: this section originally just showed the effect of changing test_size and max_depth by looking directly at test set accuracy — and there's a methodology problem there worth correcting. If we pick the "best" hyperparameter based on the test set score, that score stops being an honest estimate of performance on new data, because we've quietly "peeked" at the test set to make a decision. This is a subtler form of data leakage than just "don't train on test data" — and it's easy to miss. (Credit to Ahmet Özel for raising this in the comments on Part 1 — a sharp catch worth acting on.)

The correct approach: separate the tuning process (choosing hyperparameters) from the final evaluation. Tuning happens via cross-validation inside the training set only — the test set stays untouched until the decision has already been made.

On the test_size experiment (0.1 / 0.2 / 0.3): this is purely an illustration of how test set size affects the variance of the result, not something to be chosen based on which gives the highest accuracy. A smaller test set means fewer samples to measure performance on, so the number swings more from sampling luck, not because the model is actually better or worse.

Step 1 — Tune max_depth using cross-validation (training set only):

from sklearn.model_selection import StratifiedKFold, cross_val_score

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

for depth in [1, 2, 3, 5, None]:
    tree = DecisionTreeClassifier(max_depth=depth, random_state=42)
    scores = cross_val_score(tree, X_train, y_train, cv=cv)
    print(f"max_depth={depth}: CV accuracy = {scores.mean():.4f}")

These numbers haven't touched the test set at all. Based on this, max_depth=3 is the best choice.

Step 2 — Now touch the test set, once, for the final number:

final_tree = DecisionTreeClassifier(max_depth=3, random_state=42)
final_tree.fit(X_train, y_train)
final_acc = accuracy_score(y_test, final_tree.predict(X_test))
print(final_acc)

Result: 96.67% — exactly the same number reported earlier in the Decision Tree evaluation section, but this time arrived at through the correct process: the max_depth=3 decision came purely from the training set, and the test set was only used once at the end for confirmation.

🤔 Guess First: why is accuracy at max_depth=1 so much lower (66.67%) compared to max_depth=3?

With max_depth=1, the tree is only allowed to make one split question before deciding. That's enough to separate setosa from the other two species (since setosa is quite distinct), but not enough to separate versicolor from virginica, whose characteristics are much more similar. The model becomes underfit — too simple to capture the pattern that actually exists in the data.

Interestingly, max_depth=None (letting the tree grow freely) scores the same as max_depth=5, and slightly below max_depth=3 — a small early sign of overfitting once the tree is allowed to grow too deep.

🤔 Why does the correct tuning process still matter, even though the final answer happened to be the same?

In this case, both the correct approach (CV on the training set) and the "peeking" approach (looking at test scores) point to the same max_depth=3. But that's a coincidence, not a guarantee. On a different dataset, the two approaches could point to different hyperparameters. If we choose based on the test score, we'd never know whether the final accuracy is purely the model's real performance, or whether it's been quietly overfit to the test set without realizing it. Only by separating the tuning and evaluation processes can the final number be trusted as an honest estimate.

Common Mistakes

A few traps that commonly make ML results misleading:

• Train-test leakage — this comes in two forms. The obvious one: test data leaks into the training process itself. The subtler one (and the one that's easier to miss): choosing hyperparameters, features, or a stopping point based on test set performance — once that happens, the test score is no longer valid as an honest estimate (see the full discussion in Mini Practice above)

• Evaluating without the right metric — using accuracy alone when the data is imbalanced (can lead to wrong conclusions — see the Precision/Recall discussion in Part 1)

• Overfitting — the model "memorizes" the training data and fails to generalize to new data (visible in the max_depth=None experiment above)

If your model's score looks bad, check the data and your evaluation method first — don't jump straight to blaming the algorithm.

Summary

• Load a built-in scikit-learn dataset → split with stratify → train → predict → evaluate

• Logistic Regression and Decision Tree can produce the exact same accuracy (96.67%) on a dataset as simple as Iris

• Parameters like test_size and max_depth have a real effect on the final result — there's no single "correct" number, it all needs experimentation

• A confusion matrix gives more detailed insight than accuracy alone — in this case, the model's error consistently sits at the boundary between versicolor and virginica

From an example this simple, you can see how the model learning process actually works: give examples, minimize error, test on new data, measure, then iterate. That's the essence of Supervised Learning covered in Part 1.

In Part 3, I continue into the side not yet covered: Unsupervised Learning — clustering the same Iris flowers, this time with no labels given at all.

Part of my AI Engineering study notes. Read the theory in Part 1. Continue to Part 3 for hands-on unsupervised learning — full source code on GitHub.

Di Part 1, saya bahas teori Supervised vs Unsupervised Learning dan cara mengukur performa model. Sekarang saatnya praktik langsung: melatih model beneran yang bisa mengenali spesies bunga dari ukuran kelopaknya, lalu membandingkan dua algoritma berbeda.

Semua kode di post ini sudah saya jalankan sendiri dan angkanya terverifikasi — bukan hasil tebakan.

Daftar Isi

• Problem: Klasifikasi Spesies Iris

• Pipeline ML Singkat

• 1. Load Dataset

• 2. Split Data

• 3. Model Pertama: Logistic Regression

• 4. Evaluasi Model 1

• 5. Model Kedua: Decision Tree

• 6. Evaluasi Model 2

• Perbandingan Hasil

• Mini Practice: Efek Mengubah Parameter

• Kesalahan Umum yang Perlu Dihindari

• Ringkasan

Problem

Dataset: Iris (built-in di scikit-learn) Task: klasifikasi spesies bunga — setosa / versicolor / virginica Input: ukuran sepal & petal (4 fitur numerik)

Ini adalah contoh klasik Supervised Learning — Klasifikasi (bukan regresi), karena target-nya adalah kategori diskrit (spesies bunga), bukan angka kontinu.

Pipeline ML Singkat

Data → Split → Train → Test → Predict → Evaluate

X = fitur (ciri) yang jadi input model. y = label (kunci jawaban) yang mau diprediksi.

1. Load Dataset

Dataset Iris sudah tersedia built-in di scikit-learn, jadi nggak perlu download manual:

from sklearn.datasets import load_iris

iris = load_iris(as_frame=True)
df = iris.frame
df.head()

2. Split Data

Data dibagi jadi training set (buat belajar) dan test set (buat diuji, model belum pernah lihat data ini):

from sklearn.model_selection import train_test_split

X = df[iris.feature_names]
y = df["target"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

Dengan test_size=0.2 dari total 150 baris data Iris, hasilnya: 120 baris untuk training, 30 baris untuk testing. Parameter stratify=y penting di sini — ini memastikan proporsi tiap spesies di data train dan test tetap seimbang, bukan asal random.

3. Model Pertama: Logistic Regression

from sklearn.linear_model import LogisticRegression

clf = LogisticRegression(max_iter=200)
clf.fit(X_train, y_train)

Intuisi cara kerjanya:

• Model hitung skor: z = w·x + b

• Skor itu diubah jadi probabilitas lewat fungsi sigmoid: p = sigmoid(z)

• .fit() mencari nilai w dan b terbaik supaya prediksi model semakin mendekati label asli (loss-nya turun)

4. Evaluasi Model 1

from sklearn.metrics import accuracy_score

pred = clf.predict(X_test)
acc = accuracy_score(y_test, pred)
print(acc)

Hasil terverifikasi: akurasi Logistic Regression = 0.9667 (96.67%)

Dari 30 data test, model salah memprediksi 1 saja. Confusion matrix-nya:

from sklearn.metrics import ConfusionMatrixDisplay

ConfusionMatrixDisplay.from_predictions(
    y_test, pred, display_labels=iris.target_names, cmap="Blues"
)

Satu-satunya kesalahan: 1 bunga versicolor diprediksi sebagai virginica. Kelas setosa selalu terprediksi sempurna — masuk akal, karena secara morfologi setosa memang paling gampang dibedakan dari dua spesies lainnya.

5. Model Kedua: Decision Tree

from sklearn.tree import DecisionTreeClassifier

tree = DecisionTreeClassifier(max_depth=3, random_state=42)
tree.fit(X_train, y_train)

Intuisi cara kerjanya:

• Tree memilih pertanyaan seperti: feature ≤ threshold?

• Split dipilih untuk "memurnikan" kelas — impurity-nya turun (diukur pakai Gini atau Entropy)

• max_depth mengontrol kompleksitas tree: terlalu dangkal → underfit, terlalu dalam → overfit

6. Evaluasi Model 2

pred_tree = tree.predict(X_test)
acc_tree = accuracy_score(y_test, pred_tree)
print(acc_tree)

Hasil terverifikasi: akurasi Decision Tree = 0.9667 (96.67%)

Menariknya, confusion matrix Decision Tree persis sama dengan Logistic Regression — sama-sama cuma salah di 1 versicolor yang diprediksi virginica. Untuk dataset sesederhana Iris yang batas antar kelasnya cukup jelas, dua algoritma dengan pendekatan berbeda bisa saja "landing" di hasil akhir yang sama.

Perbandingan Hasil

Untuk kasus ini, keduanya setara. Tapi ini bukan berarti dua algoritma ini selalu setara di dataset lain — di data yang lebih kompleks atau nggak linear, hasilnya bisa jauh berbeda. Makanya proses model selection (coba beberapa model, bandingkan metriknya) itu penting, bukan cuma pakai satu algoritma dan berhenti di situ.

Catatan soal ukuran test set (ditambahkan setelah Ahmet Özel mengingatkan lewat komentar): test set di sini cuma 30 sampel, jadi 1 sampel yang berpindah sisi = sekitar 3.3 poin persentase akurasi. Itu artinya perbedaan kecil antar model di dataset sesederhana ini — bahkan yang kelihatan "beda" 1-2 poin — biasanya masih dalam rentang noise, bukan bukti satu model beneran lebih baik. Ini juga alasan kenapa cross-validation (dibahas di Mini Practice) lebih bisa dipercaya dibanding satu angka test tunggal.

Mini Practice

Update: section ini awalnya cuma nunjukin efek ubah test_size dan max_depth dengan lihat langsung akurasi di test set — dan ada masalah metodologi di situ yang perlu diluruskan. Kalau kita memilih hyperparameter "terbaik" berdasarkan angka test set, angka itu berhenti jadi estimasi jujur soal performa di data baru, karena kita sudah diam-diam "mengintip" test set buat bikin keputusan. Ini bentuk data leakage yang lebih halus dibanding sekadar "jangan training pakai data test" — dan gampang banget kelewat. (Terima kasih ke Ahmet Özel yang mengangkat poin ini lewat komentar di Part 1 — kritik yang tajam dan layak ditindaklanjuti.)

Cara yang benar: pisahkan proses tuning (pilih hyperparameter) dari proses evaluasi final. Tuning dilakukan pakai cross-validation di dalam training set saja — test set nggak disentuh sama sekali sampai keputusan sudah diambil.

Soal eksperimen test_size (0.1 / 0.2 / 0.3): ini murni ilustrasi bagaimana ukuran test set memengaruhi variasi hasil, bukan sesuatu yang dipilih berdasarkan akurasi mana yang tertinggi. Semakin kecil test set, semakin sedikit sampel buat mengukur performa, jadi angkanya lebih gampang naik-turun karena kebetulan sampel, bukan karena modelnya beneran lebih baik atau lebih buruk.

Langkah 1 — Tuning max_depth pakai cross-validation (cuma di training set):

from sklearn.model_selection import StratifiedKFold, cross_val_score

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

for depth in [1, 2, 3, 5, None]:
    tree = DecisionTreeClassifier(max_depth=depth, random_state=42)
    scores = cross_val_score(tree, X_train, y_train, cv=cv)
    print(f"max_depth={depth}: CV accuracy = {scores.mean():.4f}")

Angka-angka ini sama sekali belum menyentuh test set. Berdasarkan ini, max_depth=3 adalah pilihan terbaik.

Langkah 2 — Baru sentuh test set, sekali, buat angka final:

final_tree = DecisionTreeClassifier(max_depth=3, random_state=42)
final_tree.fit(X_train, y_train)
final_acc = accuracy_score(y_test, final_tree.predict(X_test))
print(final_acc)

Hasil: 96.67% — persis sama dengan angka yang saya laporkan di bagian evaluasi Decision Tree sebelumnya, tapi sekarang didapat lewat proses yang benar: keputusan max_depth=3 murni berasal dari training set, dan test set cuma dipakai sekali di akhir buat konfirmasi.

🤔 Coba Tebak Dulu: kenapa akurasi di max_depth=1 jauh lebih rendah (66.67%) dibanding max_depth=3?

Dengan max_depth=1, tree cuma boleh bikin satu pertanyaan split sebelum harus memutuskan. Itu cukup buat misahin setosa dari dua spesies lain (karena setosa emang beda jauh), tapi nggak cukup buat misahin versicolor dari virginica yang ciri-cirinya lebih mirip. Model jadi underfit — terlalu simpel buat menangkap pola yang sebenarnya ada di data.

Menariknya, max_depth=None (tree dibiarkan tumbuh sebebas-bebasnya) malah CV accuracy-nya sama dengan max_depth=5, dan sedikit di bawah max_depth=3 — tanda kecil overfitting mulai muncul begitu tree dibiarkan terlalu dalam.

🤔 Kenapa proses tuning yang benar tetap penting, padahal hasil akhirnya kebetulan sama?

Di kasus ini, baik cara yang benar (CV di training set) maupun cara yang "mengintip" test set sama-sama menunjuk ke max_depth=3. Tapi itu kebetulan, bukan jaminan. Di dataset lain, dua pendekatan ini bisa saja menunjuk ke hyperparameter yang berbeda. Kalau kita memilih berdasarkan angka test, kita nggak akan pernah tahu apakah akurasi final itu murni performa model, atau sudah diam-diam "disesuaikan" ke test set tanpa sadar. Cuma dengan memisahkan proses tuning dan evaluasi, angka akhir bisa dipercaya sebagai estimasi yang jujur.

Kesalahan Umum

Beberapa jebakan yang sering bikin hasil ML jadi menyesatkan:

• Train-test leakage — ada dua bentuk. Yang jelas: data test ikut "bocor" ke proses training. Yang lebih halus (dan lebih sering kejadian tanpa disadari): memilih hyperparameter, fitur, atau titik berhenti training berdasarkan performa test set — begitu itu terjadi, angka test-nya nggak lagi valid sebagai estimasi jujur (lihat pembahasan lengkap di Mini Practice di atas)

• Evaluasi tanpa metrik yang tepat — cuma pakai accuracy padahal datanya imbalanced (bisa bikin salah kesimpulan, lihat lagi bahasan Precision/Recall di Part 1)

• Overfitting — model terlalu "hapal" data training, gagal generalisasi ke data baru (kelihatan di eksperimen max_depth=None di atas)

Kalau skor model jelek, cek dulu datanya dan cara evaluasinya — jangan buru-buru nyalahin algoritmanya.

Ringkasan

• Load dataset built-in scikit-learn → split dengan stratify → train → predict → evaluate

• Logistic Regression dan Decision Tree bisa menghasilkan akurasi yang sama persis (96.67%) di dataset sesederhana Iris

• Parameter seperti test_size dan max_depth punya efek nyata ke hasil akhir — nggak ada angka "pasti benar," semua perlu dieksperimenkan

• Confusion matrix kasih insight lebih detail dibanding accuracy doang — di kasus ini, kesalahan modelnya konsisten selalu di batas antara versicolor dan virginica

Dari contoh sesederhana ini, kelihatan gimana proses belajar model sebenarnya bekerja: kasih contoh, minimalkan error, uji ke data baru, ukur, lalu iterasi. Itulah inti dari Supervised Learning yang dibahas teorinya di Part 1.

Di Part 3, saya lanjutkan ke sisi yang belum disentuh: Unsupervised Learning — clustering bunga Iris yang sama, kali ini tanpa dikasih label sama sekali.

Bagian dari catatan belajar AI Engineering saya. Baca teorinya di Part 1. Lanjut ke Part 3 untuk praktik unsupervised learning — source code lengkap ada di GitHub.

Whenever people hear "Machine Learning," the mental image often jumps straight to robots or AI that "thinks for itself." The reality is much simpler: a computer learns patterns from data, without us writing every rule by hand.

This post is my study notes on the two foundational approaches in Machine Learning — Supervised and Unsupervised Learning — plus how to actually measure whether a trained model is good, or just memorizing. Part 2 of this series goes hands-on: building a real classifier using the Iris dataset.

Table of Contents

• What is Machine Learning?

• The Big-Picture ML Process

• 4 Categories of ML

• Supervised Learning

• Unsupervised Learning

• Strategic Comparison Table

• Model Evaluation: How Do We Know a Model Is Good?

• Real-World Business Case Studies

• Quick Recall Cheat Sheet

• Summary & Next: Part 2

What is Machine Learning?

Machine Learning (ML) is a branch of Artificial Intelligence that lets computers learn automatically from data — without being explicitly programmed for every case.

With ML, systems can:

• Recognize patterns in historical data

• Make predictions on new data

• Make decisions based on experience rather than hardcoded rules

Everyday examples: e-commerce product recommendations, face detection on your phone, AI-assisted medical diagnosis, weather prediction systems.

The Big-Picture ML Process

Almost every ML project — no matter how simple — follows the same flow:

Data Collection & Preparation → Model Training → Model Evaluation → Prediction / Deployment

1. Data Collection & Preparation — gather and clean data so it's ready to use

2. Model Training — train the algorithm to understand relationships between features

3. Model Evaluation — measure model performance with specific metrics

4. Prediction / Deployment — apply the model to new data to produce real predictions

4 Categories of ML

Based on how they learn, ML splits into four major categories:

This note focuses on the two most commonly encountered when starting out with ML: Supervised and Unsupervised Learning.

🤔 Guess First: if your data has no "answer key" at all, which category does it fall under?

Unsupervised Learning. If there's no known label/target output, the model has to find patterns on its own from the structure of the data — not from a "correct answer" it was given.

Supervised Learning

Concept: the model is trained using data that already has labels. Every input has a known output, so the algorithm can learn the relationship between the two to produce accurate predictions on new data.

Analogy: similar to how a human learns under a teacher's (supervisor's) guidance — the teacher provides example questions with answers, and the student learns the pattern.

Characteristics:

• Training data has labels or targets

• The model learns by comparing its predictions against the true labels (error-based learning)

2 Main Problem Types:

Workflow:

Labelled Data → Algorithms (trained with Training Data + Desired Output, under Supervisor Intervention) → Process → Output

Example algorithms:

• Classification: Support Vector Machines, Discriminant Analysis, Naive Bayes, Nearest Neighbor

• Regression: Linear Regression/GLM, SVR/GPR, Ensemble Methods, Decision Trees, Neural Networks

Unsupervised Learning

Concept: used to analyze and discover hidden patterns in data that has no labels. The algorithm automatically groups data based on similarity or structure — without human guidance on "what group this belongs to."

Analogy: like someone trying to understand something with no direct instructions — just by observing and finding patterns in the information available.

Characteristics:

• Training data has no labels or target output

• The model tries to find relationships, similarities, or hidden patterns

2 Main Problem Types:

Workflow:

Raw Data → Interpretation → Algorithms → Process → Output

Example clustering algorithms: K-Means, K-Medoids, Fuzzy C-Means, Hierarchical, Gaussian Mixture, Neural Networks, Hidden Markov Model

Strategic Comparison Table

Model Evaluation

Having a model isn't enough — we need to know how good it actually is, and more importantly: whether it's only good on training data, or genuinely reliable on new data.

Evaluation works by comparing the model's predictions against ground truth (the actual outcome).

Confusion Matrix

A Confusion Matrix compares the model's predictions against reality, producing four possible outcomes:

Forest fire detection example:

• TP — actually a fire, predicted fire ✅

• TN — actually no fire, predicted no fire ✅

• FP — actually no fire, but predicted fire ❌ (Type I Error)

• FN — actually a fire, but predicted no fire ❌ (Type II Error, usually the more dangerous one)

Accuracy, Precision, Recall

Accuracy — percentage of correct predictions out of all observations:

Accuracy = (TP + TN) / (TP + FP + TN + FN)

Precision — of everything predicted positive, how much was actually positive:

Precision = TP / (TP + FP)

Recall — of everything actually positive, how much did the model successfully catch:

Recall = TP / (TP + FN)

Quick way to remember: Precision is about "how exact am I when I say positive" (avoiding FP). Recall is about "how complete am I at catching everything that's actually positive" (avoiding FN).

🤔 Guess First: for cancer detection, which should be prioritized — Precision or Recall?

Recall. A False Negative (patient is sick but predicted healthy) is far more dangerous than a False Positive (patient is healthy but predicted sick, then re-checked). Better to "over-suspect" than to miss a real case.

Model Selection

Several models are tried on the same input data, then each is evaluated using relevant metrics (Model 1 → Metrics 1, Model 2 → Metrics 2, etc.). This evaluation result becomes the basis for choosing the best model to deploy.

Real-World Business Case Studies

Supervised — Classification:

• Banking fraud detection: historical transaction data labeled "fraud"/"not fraud" is used to predict new transactions. Impact: prevents financial loss in real time.

• Content moderation: classifying user-uploaded content as "acceptable" vs "policy violation," based on data previously labeled by moderators. Impact: maintains platform quality without manual review of everything.

Supervised — Regression:

• Property price prediction: based on land size, location, number of rooms, building age → continuous numeric output (price). Impact: helps sellers/buyers set fair prices.

Unsupervised — Clustering:

• Customer segmentation: with no "type A/B/C" label from the start, the model groups customers into segments ("budget shoppers," "high-value loyal customers") based on spending patterns. Impact: more targeted campaigns.

• Cybersecurity anomaly detection: monitoring login/traffic patterns without an explicit "normal/attack" label — patterns that deviate significantly are flagged as potential threats. Impact: faster prevention of data breaches.

Precision vs Recall in real business:

• Credit card fraud needs balance (usually via F1-Score) — low Precision means many good customers complain about blocked transactions; low Recall means a lot of fraud slips through.

• Product recommendation systems — multiple models (collaborative filtering, content-based, hybrid) are tried, evaluated using metrics like precision@k, then the best one gets deployed.

Quick Recall Cheat Sheet

📋 Click to reveal the quick memorization guide

• Supervised = HAS A TEACHER → has labels, has a supervisor, goal is prediction

• Unsupervised = NO TEACHER → no labels, finds patterns on its own, goal is exploration

• Classification = category (discrete) | Regression = number (continuous)

• Clustering = similar groups | Dimensional Reduction = simplify features

• The keyword "label" is the main distinguisher between Supervised and Unsupervised

• Confusion Matrix: True/False = correct/incorrect prediction, Positive/Negative = what was predicted

• Precision = "of what I said was positive, how much was actually positive?"

• Recall = "of what was actually positive, how much did I manage to catch?"

Checklist Before Moving to Practice

• [ ] Understand the difference between labeled vs unlabeled data

• [ ] Can distinguish when to use classification vs regression

• [ ] Can distinguish when to use clustering vs dimensional reduction

• [ ] Understand why Recall matters more in medical cases, and why both matter in fraud cases

• [ ] Understand the TP/TN/FP/FN terms in a confusion matrix

Summary

Supervised Learning learns from examples that already have answers — a good fit when the goal is prediction. Unsupervised Learning learns with no answers at all — a good fit when the goal is exploration and discovering structure we didn't already know about.

But theory alone isn't enough to really understand ML. This series continues with Part 2 (hands-on Iris classification — Logistic Regression vs Decision Tree) and Part 3 (hands-on clustering with no labels using KMeans, plus follow-up experiments), complete with source code.

Part of my AI Engineering study notes. Part 2: hands-on Iris classification. Part 3: hands-on unsupervised learning — full source code on GitHub.

Setiap kali orang dengar "Machine Learning," bayangannya sering langsung lompat ke robot atau AI yang "mikir sendiri." Padahal intinya jauh lebih sederhana: komputer belajar pola dari data, tanpa kita tulis semua aturannya satu-satu secara manual.

Post ini adalah catatan belajar saya soal dua pendekatan dasar dalam Machine Learning — Supervised dan Unsupervised Learning — plus cara mengukur apakah model yang kita latih itu beneran bagus atau cuma menghafal. Part 2 dari seri ini akan masuk ke praktik langsung: bikin classifier beneran pakai dataset Iris.

Daftar Isi

• Apa itu Machine Learning?

• Alur Besar Proses ML

• 4 Kategori ML

• Supervised Learning

• Unsupervised Learning

• Tabel Perbandingan Strategis

• Model Evaluation: Gimana Cara Tahu Model Kita Bagus?

• Studi Kasus Nyata di Dunia Kerja

• Cheat Sheet Menghafal

• Ringkasan & Lanjut ke Part 2

Apa itu Machine Learning?

Machine Learning (ML) adalah cabang dari Artificial Intelligence yang memungkinkan komputer belajar secara otomatis dari data — tanpa perlu diprogram secara eksplisit untuk setiap kasus.

Dengan ML, sistem bisa:

• Mengenali pola dari data historis

• Membuat prediksi terhadap data baru

• Mengambil keputusan berdasarkan pengalaman, bukan aturan hardcode

Contoh yang paling gampang ditemuin sehari-hari: rekomendasi produk di e-commerce, deteksi wajah di kamera HP, diagnosis medis berbantuan AI, sampai prediksi cuaca.

Alur Besar Proses ML

Hampir semua proyek ML — sesederhana apa pun — mengikuti alur yang sama:

Data Collection & Preparation → Model Training → Model Evaluation → Prediction / Deployment

1. Data Collection & Preparation — kumpulin dan bersihin data biar siap dipakai

2. Model Training — latih algoritma supaya paham hubungan antar fitur

3. Model Evaluation — ukur performa model pakai metrik tertentu

4. Prediction / Deployment — pakai model ke data baru buat hasilkan prediksi nyata

4 Kategori ML

Berdasarkan cara belajarnya, ML dibagi jadi 4 kategori besar:

Fokus catatan ini ada di dua yang paling sering ditemuin di awal belajar ML: Supervised dan Unsupervised Learning.

🤔 Coba Tebak Dulu: kalau data kamu nggak punya "jawaban" sama sekali, itu masuk kategori yang mana?

Jawabannya Unsupervised Learning. Kalau nggak ada label/target output yang diketahui, model harus cari pola sendiri dari struktur data — bukan dari "jawaban benar" yang dikasih ke dia.

Supervised Learning

Konsep: model dilatih pakai data yang sudah punya label. Tiap data input punya output yang diketahui, jadi algoritma bisa belajar hubungan antara input dan output tersebut untuk menghasilkan prediksi akurat pada data baru.

Analogi: mirip proses belajar manusia di bawah bimbingan guru (supervisor) — guru kasih contoh soal beserta jawabannya, murid belajar polanya.

Ciri-ciri:

• Data training punya label atau target

• Model belajar dengan cara membandingkan prediksi terhadap label asli (error-based learning)

2 Jenis Masalah Utama:

Alur kerja:

Labelled Data → Algorithms (dilatih dengan Training Data + Desired Output, di bawah Supervisor Intervention) → Process → Output

Contoh algoritma:

• Classification: Support Vector Machines, Discriminant Analysis, Naive Bayes, Nearest Neighbor

• Regression: Linear Regression/GLM, SVR/GPR, Ensemble Methods, Decision Trees, Neural Networks

Unsupervised Learning

Konsep: dipakai untuk menganalisis dan menemukan pola tersembunyi dari data yang tidak punya label. Algoritma secara otomatis mengelompokkan data berdasarkan kemiripan atau struktur tertentu — tanpa bantuan manusia buat kasih tahu "ini kelompok apa."

Analogi: mirip orang yang coba memahami sesuatu tanpa petunjuk langsung — cuma dengan mengamati dan mencari pola dari informasi yang ada.

Ciri-ciri:

• Data training tidak punya label atau target output

• Model berusaha mencari hubungan, kesamaan, atau pola tersembunyi

2 Jenis Masalah Utama:

Alur kerja:

Raw Data → Interpretation → Algorithms → Process → Output

Contoh algoritma clustering: K-Means, K-Medoids, Fuzzy C-Means, Hierarchical, Gaussian Mixture, Neural Networks, Hidden Markov Model

Tabel Perbandingan Strategis

Model Evaluation

Punya model aja nggak cukup — kita harus tahu seberapa bagus model itu, dan lebih penting lagi: apakah dia cuma bagus di data training, atau beneran bisa diandalkan di data baru.

Evaluasi dilakukan dengan membandingkan hasil prediksi model dengan ground truth (data sebenarnya).

Confusion Matrix

Confusion Matrix membandingkan hasil prediksi model dengan realita, menghasilkan 4 kemungkinan:

Contoh kasus deteksi kebakaran hutan:

• TP — realita ada api, diprediksi ada api ✅

• TN — realita tidak ada api, diprediksi tidak ada api ✅

• FP — realita tidak ada api, tapi diprediksi ada api ❌ (Type I Error)

• FN — realita ada api, tapi diprediksi tidak ada api ❌ (Type II Error, biasanya paling berbahaya)

Accuracy, Precision, Recall

Accuracy — persentase prediksi yang benar dari semua pengamatan:

Akurasi = (TP + TN) / (TP + FP + TN + FN)

Precision — dari semua yang diprediksi positif, berapa persen yang beneran positif:

Presisi = TP / (TP + FP)

Recall — dari semua yang beneran positif, berapa persen yang berhasil ditangkap model:

Recall = TP / (TP + FN)

Cara gampang bedain: Presisi fokus ke "seberapa tepat aku pas bilang positif" (menghindari FP). Recall fokus ke "seberapa lengkap aku nangkep semua yang beneran positif" (menghindari FN).

🤔 Coba Tebak Dulu: buat deteksi penyakit kanker, mana yang harus lebih diprioritaskan — Precision atau Recall?

Recall. False Negative (pasien sakit tapi diprediksi sehat) jauh lebih berbahaya daripada False Positive (pasien sehat tapi diprediksi sakit, lalu dicek ulang). Lebih baik "curiga berlebihan" daripada melewatkan kasus nyata.

Model Selection

Beberapa model dicoba di input data yang sama, lalu masing-masing dievaluasi pakai metrik yang relevan (Model 1 → Metrics 1, Model 2 → Metrics 2, dst). Hasil evaluasi ini jadi dasar milih model terbaik yang akan dipakai.

Studi Kasus Nyata

Supervised — Klasifikasi:

• Fraud Detection perbankan: data transaksi historis yang sudah dilabeli "fraud"/"bukan fraud" dipakai buat prediksi transaksi baru. Dampak: mencegah kerugian finansial secara real-time.

• Content Moderation: klasifikasi konten upload user sebagai "layak tayang" vs "melanggar kebijakan," berdasarkan data yang sudah dilabeli moderator. Dampak: jaga kualitas platform tanpa cek manual satu-satu.

Supervised — Regresi:

• Prediksi harga properti: berdasarkan luas tanah, lokasi, jumlah kamar, usia bangunan → output angka kontinu (harga). Dampak: bantu penjual/pembeli menentukan harga wajar.

Unsupervised — Clustering:

• Segmentasi pelanggan: tanpa label "tipe A/B/C" dari awal, model mengelompokkan pelanggan jadi segmen ("pembeli hemat," "pelanggan loyal bernilai tinggi") berdasarkan pola belanja. Dampak: campaign lebih tepat sasaran.

• Deteksi anomali cybersecurity: memantau pola login/traffic tanpa label "normal/serangan" eksplisit — pola yang menyimpang jauh dianggap potensi ancaman. Dampak: cegah kebocoran data lebih cepat.

Precision vs Recall dalam bisnis nyata:

• Fraud kartu kredit butuh keseimbangan (biasanya pakai F1-Score) — Precision rendah = banyak nasabah baik komplain karena transaksinya diblokir; Recall rendah = banyak fraud lolos.

• Sistem rekomendasi produk — beberapa model (collaborative filtering, content-based, hybrid) dicoba, dievaluasi pakai metrik seperti precision@k, lalu yang terbaik di-deploy.

Cheat Sheet Menghafal

📋 Klik untuk lihat cara cepat menghafal

• Supervised = ADA GURU → ada label, ada supervisor, tujuannya memprediksi

• Unsupervised = TANPA GURU → tanpa label, cari pola sendiri, tujuannya eksplorasi

• Klasifikasi = kategori (diskrit) | Regresi = angka (kontinu)

• Clustering = kelompok mirip | Dimensional Reduction = sederhanakan fitur

• Kata kunci "label" adalah pembeda utama antara Supervised dan Unsupervised

• Confusion Matrix: True/False = benar/salah prediksi, Positive/Negative = apa yang diprediksi

• Presisi = "dari yang aku bilang positif, berapa yang beneran positif?"

• Recall = "dari yang beneran positif, berapa yang berhasil aku tangkap?"

Checklist Sebelum Lanjut ke Praktik

• [ ] Paham bedanya data berlabel vs tanpa label

• [ ] Bisa bedain kapan pakai klasifikasi vs regresi

• [ ] Bisa bedain kapan pakai clustering vs dimensional reduction

• [ ] Ngerti kenapa Recall lebih penting dari Precision di kasus medis, dan kenapa keduanya penting di kasus fraud

• [ ] Paham istilah TP/TN/FP/FN di confusion matrix

Ringkasan

Supervised Learning belajar dari contoh yang sudah ada jawabannya — cocok kalau tujuan kita memang untuk memprediksi. Unsupervised Learning belajar tanpa jawaban sama sekali — cocok kalau tujuan kita untuk eksplorasi dan menemukan struktur yang belum kita tahu sebelumnya.

Tapi teori doang belum cukup buat ngerti ML. Seri ini berlanjut ke Part 2 (praktik klasifikasi Iris — Logistic Regression vs Decision Tree) dan Part 3 (praktik clustering tanpa label dengan KMeans, plus eksperimen lanjutan), lengkap dengan source code-nya.

Bagian dari catatan belajar AI Engineering saya. Part 2: praktik klasifikasi Iris. Part 3: praktik unsupervised learning — source code lengkap ada di GitHub.

If you think baldness is purely a matter of genetics, here's a plot twist: a few surprisingly sensible patterns show up once you dig into age, smoking habits, and stress levels together.

This post walks through the full journey — from a genuinely messy raw dataset to a working regression model that predicts someone's baldness probability — along with the reasoning behind every technical decision made along the way.

🇮🇩 Baca versi Bahasa Indonesia: Memprediksi Probabilitas Kebotakan · 💻 Full code & dataset: GitHub

Table of Contents

• About the Dataset

• Data Cleaning: Taming the Mess

• EDA: Patterns Hiding in the Data

• Outlier Handling with IQR

• Modeling & Cross-Validation

• Best Model & a Sample Prediction

• Cheat Sheet: Missing Value Strategy

• Quiz: Test Yourself

• Takeaways

About the Dataset

The dataset (botak_kotor.csv, Indonesian for "dirty baldness data") is synthetic — not a real survey or medical dataset — with 7,917 rows across 14 columns: age, gender, occupation, province, salary, marital status, family history of baldness, weight & height, shampoo brand, smoking habit, education level, stress level, and a botak_prob (baldness probability) target column. It was built as a data-cleaning exercise for the Fullstack Bangalore AI Engineer Bootcamp, deliberately "dirtied up" with missing values spread across every column (~1% each) and dozens of duplicate rows.

⚠️ Important note: since the data is synthetic, every pattern and correlation discussed here reflects the simulated dataset only — not a medical or scientific claim about what actually causes baldness. Treat this as a case study in data cleaning, EDA, and machine learning technique, not health research.

🤔 Guess first: what percentage of the data do you think is missing?

Missing values turned out to be spread fairly evenly across all 14 columns, roughly 1% each. After dropping rows with an empty target, 79 rows had to go because their botak_prob label was missing — filling a label with a median would just contaminate the training data.

Data Cleaning: Taming the Mess

The missing-value strategy was split by column type:

Median was chosen over mean for numeric columns because it's more resistant to outliers — a handful of rows had extreme salary or weight values that would have dragged the mean away from the actual center of the data.

Next steps:

• 81 duplicate rows found and removed

• Text columns cleaned up (whitespace stripped, province names title-cased)

• Binary columns and age cast back to integers (they'd become floats because of the NaNs)

After this stage, 7,757 rows remained — ready for exploratory data analysis (EDA), before outlier cleanup.

EDA: Patterns Hiding in the Data

Feature Correlation with Baldness Probability

Ranking each numeric feature's correlation with botak_prob (strongest first):

1. Family history (is_keturunan) — 0.44

2. Age — 0.35

3. Stress level — 0.29

4. Smoking habit — 0.26

5. Marital status — 0.12 (weak)

6. Salary (0.06), height (0.01), weight (near zero) — essentially no correlation

Interestingly, salary and height/weight barely correlate with baldness probability at all — so those columns (along with province, which isn't logically relevant either) were deliberately left out of the model's feature set.

🤔 Guess first: which feature do you think correlates most strongly with baldness probability?

It's family history (is_keturunan), at 0.44 — which tracks with how genetics is commonly associated with baldness patterns (again, this is a pattern in simulated data, not a medical claim).

The High-Risk Group (Baldness Probability ≥ 70%)

Out of 7,757 people at this stage, 1,760 (22.7%) fall into the "high-risk" bucket (botak_prob >= 0.7). Comparing that group to the general population:

The pattern lines up with the correlation results: the high-risk group skews older, smokes more, and has more than double the rate of family history compared to the general population.

Outlier Handling with IQR

After EDA, outliers in the height and weight columns were cleaned up using the IQR (Interquartile Range) method — values outside the Q1−1.5×IQR to Q3+1.5×IQR range were dropped. This step removed 326 rows, leaving 7,431 clean rows for modeling.

Why remove outliers after EDA instead of before? So the data's original patterns (extreme values included) could still be observed during exploration, before deciding what actually needed to go for modeling purposes.

Modeling & Cross-Validation

Four regression algorithms were compared: Linear Regression, Decision Tree, Random Forest, and Support Vector Regression (SVR). Numeric features were scaled with MinMaxScaler — but not just once upfront. Instead, scaling was wrapped inside a Pipeline alongside each model, so the scaler gets re-fit on every fold of 5-fold cross-validation. That matters because it keeps the test fold fully isolated from the fitting process, avoiding data leakage.

Cross-validation results (average RMSE, lower is better):

Best Model & a Sample Prediction

Random Forest was automatically selected as the model with the lowest cross-validation RMSE. After retraining on the full training set and evaluating on a held-out test set it had never seen:

• RMSE (test set): 0.0675

• R² (test set): 0.856 — the model explains roughly 85.6% of the variance in baldness probability using the available features

As an example: for a new data point (40-year-old male, married, private employee, family history present, 65kg, 170cm, master's degree, smoker, stress level 7), the model predicts a baldness probability of about 64.1%.

Cheat Sheet: Missing Value Strategy

Quiz: Test Yourself

1. Why is scaling done inside a Pipeline instead of once before the split?

So the MinMaxScaler gets re-fit on every cross-validation fold, keeping the test fold genuinely isolated from the fitting process — which prevents data leakage that would otherwise make evaluation results look better than the model's real-world performance.

2. Why were rows with a missing target (botak_prob) dropped instead of imputed with the median?

Because the target is the label the model is trying to learn. Filling it with a synthetic value like the median would contaminate training — the model would end up learning from values that never actually existed in the original data.

3. Which columns were deliberately excluded as model features, and why?

salary and province — salary's correlation with botak_prob is only 0.06 (essentially no relationship), and neither is logically relevant as a predictor of baldness anyway.

Takeaways

Three things worth carrying forward from this exercise:

1. Careful cleaning matters — even small decisions (median vs. mean, dropping vs. imputing the target) ripple through to model quality down the line.

2. EDA earns its keep as a feature filter — near-zero correlations (salary, height, weight) are a solid reason to leave features out rather than force them in.

3. Cross-validation is more trustworthy than a single train-test split, especially when choosing between several candidate models.

Random Forest won this particular case study, but the more valuable part is the process that got there: from messy data, to defensible cleaning decisions, to insight from EDA, to rigorous model evaluation.

💻 Full code, dataset, and a runnable notebook are on GitHub: github.com/arielshakaramiro/baldness-probability-prediction

Kalau kamu pikir kebotakan cuma soal gen dari orang tua, coba tunggu dulu — dari eksplorasi data kali ini, muncul beberapa pola menarik (dan cukup masuk akal) dari kombinasi umur, kebiasaan merokok, dan tingkat stres.

Tulisan ini merangkum proses penuh dari data mentah yang berantakan sampai model machine learning yang bisa memprediksi probabilitas kebotakan seseorang — lengkap dengan alasan di balik tiap keputusan teknisnya.

🇬🇧 Baca versi Bahasa Inggris: Predicting Baldness Probability · 💻 Kode & dataset lengkap: GitHub

Daftar Isi

• Sekilas Tentang Dataset

• Data Cleaning: Merapikan yang Kotor

• EDA: Pola yang Muncul dari Data

• Outlier Handling dengan IQR

• Modeling & Cross-Validation

• Model Terbaik & Contoh Prediksi

• Cheat Sheet: Strategi Isi Missing Value

• Quiz: Uji Pemahaman

• Kesimpulan

Sekilas Tentang Dataset

Dataset yang dipakai (botak_kotor.csv) adalah dataset sintetis — bukan data survei atau data medis nyata — berisi 7.917 baris dengan 14 kolom: umur, jenis kelamin, pekerjaan, provinsi, gaji, status menikah, riwayat keturunan botak, berat & tinggi badan, merek sampo, kebiasaan merokok, pendidikan, tingkat stres, hingga probabilitas kebotakan (botak_prob) sebagai target. Dataset ini jadi materi latihan data cleaning untuk Fullstack Bangalore AI Engineer Bootcamp, sengaja "dikotori" dengan missing value yang tersebar di semua kolom (~1% per kolom) dan puluhan baris duplikat.

⚠️ Catatan penting: Karena datanya sintetis, semua pola dan korelasi yang dibahas di tulisan ini murni pola dari data simulasi — bukan klaim medis atau ilmiah tentang penyebab kebotakan di dunia nyata. Anggap ini studi kasus teknik data cleaning, EDA, dan machine learning, bukan riset kesehatan.

🤔 Coba Tebak Dulu: menurutmu berapa persen data yang hilang (missing value) di dataset ini?

Missing value ternyata tersebar cukup merata di semua 14 kolom, masing-masing sekitar 1%. Setelah baris dengan target kosong dibuang, ada 79 baris yang harus di-drop karena label (botak_prob)-nya kosong — mengisi label dengan median cuma akan mencemari data training.

Data Cleaning: Merapikan yang Kotor

Strategi isi missing value dibedakan berdasarkan tipe kolom:

Median dipilih untuk kolom numerik (bukan mean) karena lebih tahan terhadap outlier — beberapa baris punya nilai gaji atau berat badan ekstrem yang bisa "menarik" nilai mean menjauh dari pusat data yang sebenarnya.

Langkah selanjutnya:

• 81 baris duplikat ditemukan dan dihapus

• Teks dirapikan (strip whitespace, title-case untuk nama provinsi)

• Tipe data kolom biner & umur dikembalikan ke integer (sempat berubah jadi float akibat NaN)

Setelah tahap ini, dataset tersisa 7.757 baris — siap dipakai untuk eksplorasi data (EDA) sebelum masuk ke pembersihan outlier.

EDA: Pola yang Muncul dari Data

Korelasi Fitur terhadap Probabilitas Botak

Urutan korelasi tiap fitur numerik terhadap botak_prob (dari yang paling kuat):

1. Riwayat keturunan (is_keturunan) — 0,44

2. Umur — 0,35

3. Tingkat stres — 0,29

4. Kebiasaan merokok — 0,26

5. Status menikah — 0,12 (lemah)

6. Gaji (0,06), tinggi (0,01), berat (mendekati 0) — praktis tidak berkorelasi

Menariknya, gaji dan tinggi/berat badan nyaris tidak berkorelasi sama sekali dengan probabilitas kebotakan — jadi kedua kolom itu (plus provinsi, yang secara logis juga tidak relevan) sengaja tidak dipakai sebagai fitur model nantinya.

🤔 Coba Tebak Dulu: menurutmu fitur apa yang paling berkorelasi dengan probabilitas botak?

Jawabannya riwayat keturunan (is_keturunan), dengan korelasi 0,44 — cukup masuk akal karena secara umum faktor keturunan sering dikaitkan dengan pola kebotakan (ingat, ini pola dari data simulasi, bukan klaim medis).

Kelompok Risiko Tinggi (Probabilitas Botak ≥ 70%)

Dari 7.757 individu pada tahap ini, 1.760 orang (22,7%) masuk kategori "risiko tinggi" (botak_prob >= 0,7). Beberapa perbandingan populasi umum vs kelompok ini:

Polanya konsisten dengan hasil korelasi: kelompok risiko tinggi cenderung lebih tua, proporsi perokoknya lebih besar, dan proporsi yang punya riwayat keturunan botak lebih dari dua kali lipat dibanding populasi umum.

Outlier Handling dengan IQR

Setelah EDA, baru dilakukan pembersihan outlier pada kolom tinggi dan berat memakai metode IQR (Interquartile Range) — nilai di luar rentang Q1−1,5×IQR sampai Q3+1,5×IQR dibuang. Tahap ini menghapus 326 baris, menyisakan 7.431 baris bersih yang dipakai untuk modeling.

Kenapa outlier baru dibuang setelah EDA, bukan sebelumnya? Supaya pola asli di data (termasuk nilai ekstrem) masih bisa diamati dulu saat eksplorasi, sebelum diputuskan mana yang benar-benar perlu dibuang untuk keperluan modeling.

Modeling & Cross-Validation

Untuk pemodelan, 4 algoritma regresi dibandingkan: Regresi Linear, Decision Tree, Random Forest, dan Support Vector Regression (SVR). Fitur numerik diskalakan dengan MinMaxScaler — tapi bukan sekali di awal, melainkan dibungkus dalam Pipeline bersama tiap model, supaya proses scaling di-fit ulang di setiap fold 5-fold cross-validation. Pendekatan ini penting supaya data test benar-benar terisolasi dari proses training (menghindari data leakage).

Hasil cross-validation (RMSE rata-rata, makin kecil makin baik):

Model Terbaik & Contoh Prediksi

Random Forest otomatis terpilih sebagai model dengan RMSE cross-validation terendah. Setelah di-retrain di seluruh data train dan dievaluasi di data test yang belum pernah dilihat model sebelumnya:

• RMSE (test set): 0,0675

• R² (test set): 0,856 — model bisa menjelaskan sekitar 85,6% variasi probabilitas botak dari fitur yang tersedia

Sebagai contoh: untuk data baru (laki-laki, 40 tahun, sudah menikah, pekerja swasta, punya riwayat keturunan, berat 65kg, tinggi 170cm, pendidikan S2, perokok, level stres 7), model memprediksi probabilitas botak sekitar 64,1%.

Cheat Sheet: Strategi Isi Missing Value

Quiz: Uji Pemahaman

1. Kenapa scaling dilakukan di dalam Pipeline, bukan sekali di awal sebelum split?

Supaya MinMaxScaler di-fit ulang di setiap fold cross-validation, memastikan data test benar-benar terisolasi dari proses fitting — mencegah data leakage yang bisa membuat hasil evaluasi terlihat lebih bagus dari performa sebenarnya.

2. Kenapa baris dengan target (botak_prob) kosong di-drop, bukan diisi median?

Karena target adalah label yang dipelajari model. Mengisi label dengan nilai buatan (median) akan mencemari proses training — model akan belajar dari nilai yang sebenarnya tidak pernah ada di data asli.

3. Kolom apa yang sengaja tidak dipakai sebagai fitur model, dan kenapa?

gaji dan provinsi — korelasi gaji terhadap botak_prob cuma 0,06 (praktis tidak ada hubungan), dan keduanya juga secara logis tidak relevan sebagai prediktor kebotakan.

Kesimpulan

Dari eksplorasi ini, tiga hal utama yang bisa dibawa pulang:

1. Cleaning yang hati-hati itu penting — keputusan sekecil apa pun (median vs mean, drop vs isi target) bisa memengaruhi kualitas model di ujung proses.

2. EDA membantu menyaring fitur sebelum modeling — korelasi yang nyaris nol (gaji, tinggi, berat) jadi alasan kuat untuk tidak memaksakan fitur itu masuk model.

3. Cross-validation lebih meyakinkan dibanding satu kali train-test split — terutama saat harus memilih di antara beberapa kandidat model.

Random Forest menang di studi kasus ini, tapi yang lebih penting adalah proses sampai ke sana: dari data kotor, ke keputusan cleaning yang masuk akal, ke insight dari EDA, sampai evaluasi model yang ketat.

💻 Kode lengkap, dataset, dan notebook yang bisa langsung dijalankan ada di GitHub: github.com/arielshakaramiro/baldness-probability-prediction

Learning notes with code that actually runs — six core data preprocessing techniques you'll reach for before feeding data into a Machine Learning model, backed by real execution output instead of made-up numbers.

There's a saying that gets repeated a lot in data science: "80% of AI work is data, not the model" (the exact number varies by source, but the general point tends to hold up). What people usually mean is this part of the pipeline: checking for missing values, catching and handling outliers, turning categories and text into numbers, compressing dimensions, and keeping a version history of your data. Every technique below was run with actual Python code (Pandas + Scikit-learn), against two kinds of data — a real 10,000-row customer dataset, and a few small synthetic examples to make the concepts concrete.

Note on the code blocks: variable names are kept exactly as they were in the original notebook (Indonesian words like nama, umur, gaji, batas_bawah) rather than translated, so the code here is byte-for-byte reproducible against the source. Only the surrounding explanation is in English.

📂 The full notebook (already executed end-to-end, dataset included) is on GitHub: github.com/arielshakaramiro/data-preprocessing-praktik-arielshakaramiro

Table of Contents

• 1. Data Collection & Storage: Quick Concept

• 2. Missing Values: Check First, Don't Assume

• 3. Detecting & Handling Outliers with IQR

• 4. Feature Engineering: Turning Categories into Numbers

• 5. Text Preprocessing: Bag of Words & TF-IDF

• 6. Reducing Dimensions with PCA

• 7. The Data Versioning Concept

• 8. Summary & Next Steps

• 9. Cheat Sheet: Preprocessing Checklist

• 10. Quiz Check

1. Data Collection & Storage: Quick Concept

Before diving into practice, there are two types of data worth knowing:

• Unstructured Data — data with "free-form" shape, like images, text, or audio. Humans can grasp its meaning directly without any special structure (e.g., we instantly recognize a picture of a cat).

• Structured Data — data neatly organized into rows and columns, like transaction records or customer data. Its meaning depends heavily on business context, so it's not always understandable "at a glance" to a non-expert.

Both types need storage that's secure, easily accessible, and versionable (keeps a history of changes) — this versioning concept is covered further in section 7.

2. Missing Values: Check First, Don't Assume

There are two common ways to handle missing (NaN) values:

• Dropna — remove rows/columns with missing data. Works well when only a small amount is missing.

• Fillna — fill the gap with a substitute: mean, median, or mode (the most frequent value).

The step beginners skip most often: check first, don't assume the data is dirty.

sample_data = pd.read_csv('customers-10000.csv')
print(sample_data.isnull().sum())

(The original code loads the dataset from a Google Drive link; it's written here as a local file for easy reproduction — the content is identical, a public 10,000-row dataset from Datablist.)

Real execution result on the customer dataset (10,000 rows, 12 columns — Index, Customer Id, First Name, Last Name, Company, City, Country, Phone 1, Phone 2, Email, Subscription Date, Website):

Missing values per column: 0 (every column, 0.0%)

🤔 Guess First: How much of a public-facing customer dataset like this do you think is typically missing?

Reveal Answer

In this case: 0% — this is a synthetic dataset generated with Faker for practice purposes, so it was built clean on purpose, with no missing values at all. That's actually the real lesson here: never assume a dataset is "obviously dirty." Always run .isnull().sum() first before deciding on a strategy — if it turns out clean, you just saved yourself time; if something is missing, you know exactly which column needs attention.

To actually see both handling techniques in action, a tiny synthetic dataset was built with intentional gaps:

data = {
    'nama': ['Andi', 'Budi', 'Citra', 'Dewi', 'Eka'],
    'umur': [25, np.nan, 30, 22, np.nan],
    'gaji': [5000000, 6000000, np.nan, 4500000, 5200000]
}
df = pd.DataFrame(data)

(nama = name, umur = age, gaji = salary — kept in the original language since that's the actual column names in the code.)

Missing count and percentage per column (real output of df.isnull().sum() and df.isnull().sum() / len(df) * 100):

Option 1 — dropna():

df_dropped = df.dropna()

Budi, Citra, and Eka all get dropped since each has one empty cell — 5 rows shrink down to 2. Effective when the missing amount is small and those rows aren't critical; wasteful when the data is valuable but just happens to have one blank column.

Option 2 — fillna() with mean/median:

df_filled = df.copy()
df_filled['umur'] = df_filled['umur'].fillna(df_filled['umur'].mean())
df_filled['gaji'] = df_filled['gaji'].fillna(df_filled['gaji'].median())

Missing ages get filled with the average of the ones present, and missing salaries get filled with the median. (Added context beyond the original code: median is generally chosen for salary because it's more robust against extreme values than the mean — this will make more sense after the outlier section below.)

3. Detecting & Handling Outliers with IQR

An outlier is a data point that sits far outside the majority. The decision rule:

• Outliers that are rare and don't impact the business → safe to drop.

• Outliers that do disrupt business processes → don't just drop them, handle them more carefully (at production scale, one advanced approach is an Autoencoder — a small model that learns the pattern of normal data, so data that deviates far from that pattern produces a large reconstruction error and can be flagged as an outlier).

For beginners, one of the most common and straightforward methods is IQR (Interquartile Range). The simulation: 1,000 normally-distributed salary (gaji) values (mean Rp5,000,000, standard deviation Rp1,000,000), with 1 extreme outlier of Rp50,000,000 injected on top.

np.random.seed(42)
gaji = np.random.normal(5000000, 1000000, 1000).tolist()
gaji.append(50000000)  # outlier: a salary far larger than the rest
df_gaji = pd.DataFrame({'gaji': gaji})

A quick look before computing anything — the boxplot code:

plt.figure(figsize=(6, 4))
plt.boxplot(df_gaji['gaji'])
plt.title('Boxplot Gaji (perhatikan ada titik yang jauh di atas)')
plt.ylabel('Gaji')
plt.show()

(The title/label strings are left in Indonesian, exactly as they were actually run — "Boxplot Gaji (perhatikan ada titik yang jauh di atas)" means "Salary boxplot, notice the point far above.")

Then the bounds get computed mathematically with IQR:

Q1 = df_gaji['gaji'].quantile(0.25)
Q3 = df_gaji['gaji'].quantile(0.75)
IQR = Q3 - Q1
batas_bawah = Q1 - 1.5 * IQR
batas_atas = Q3 + 1.5 * IQR

outliers = df_gaji[(df_gaji['gaji'] < batas_bawah) | (df_gaji['gaji'] > batas_atas)]

(batas_bawah = lower bound, batas_atas = upper bound.)

Real execution results (with random seed 42, so the numbers are reproducible on a re-run):

🤔 Guess First: Only 1 fake outlier (Rp50M) was injected. Do you think IQR flags exactly 1 outlier too, or could it be more?

Reveal Answer

It flagged 9 outliers, not just 1. Besides the injected Rp50M value, 8 other data points — pure products of the normal distribution, not intentional — happened to land outside the 1.5×IQR range, and they sit on both sides: 4 below the lower bound (roughly Rp1.76M–2.38M) and 4 above the upper bound (roughly Rp7.63M–8.85M). The lesson: the IQR method is purely statistical, it has no idea which values are "intentional anomalies" and which are just "extreme by coincidence" due to the natural spread of the data. That's why, before dropping anything IQR flags, it's still worth checking the business context — an Rp8M salary in this dataset could genuinely be a director's valid salary, not a data-entry mistake, and an Rp1.8M salary might just be an intern.

After removal:

df_bersih = df_gaji[(df_gaji['gaji'] >= batas_bawah) & (df_gaji['gaji'] <= batas_atas)]

The clean remainder: 992 out of 1,001 rows.

4. Feature Engineering: Turning Categories into Numbers

Machine Learning models only understand numbers, not text categories like "Apple" or "Chicken." Two common ways to convert them:

Example data:

food = pd.DataFrame({
    'Food Name': ['Apple', 'Chicken', 'Broccoli'],
    'Calories': [95, 231, 50]
})

Label Encoding (real LabelEncoder output):

Notice the numbers follow alphabetical order (Apple=0, Broccoli=1, Chicken=2) — not calories or any meaningful ranking. This is a classic source of bugs: applied to categories with no real order, a model can "read" Chicken (2) as somehow twice as large or important as Broccoli (1), when it's really just an alphabetical index.

One Hot Encoding (real pd.get_dummies output):

No implied ordering — each category gets its own column. The trade-off: with hundreds or thousands of categories (postal codes, for example), the column count can explode.

5. Text Preprocessing: Bag of Words & TF-IDF

Just like categories, text also needs to become a numeric vector (vectorization). Example corpus (3 short Indonesian sentences):

corpus = [
    "rumah ini bagus",        # d1 - "the house is nice"
    "rumah saya makan nasi",  # d2 - "my house, I eat rice"
    "saya makan nasi"         # d3 - "I eat rice"
]

Bag of Words — counts how many times each word appears in each document:

vectorizer = CountVectorizer()
bow_matrix = vectorizer.fit_transform(corpus)

bow_df = pd.DataFrame(
    bow_matrix.toarray(),
    columns=vectorizer.get_feature_names_out(),
    index=['d1', 'd2', 'd3']
)

Real execution result (bow_df):

BoW has a weakness: words that appear in every document get counted as heavily as any other word, even though they don't help tell documents apart.

TF-IDF (Term Frequency–Inverse Document Frequency) fixes this — words that show up across many documents get a lower weight, while rarer, more unique words get a higher weight:

tfidf_vectorizer = TfidfVectorizer()
tfidf_matrix = tfidf_vectorizer.fit_transform(corpus)

tfidf_df = pd.DataFrame(
    tfidf_matrix.toarray(),
    columns=tfidf_vectorizer.get_feature_names_out(),
    index=['d1', 'd2', 'd3']
).round(3)

Real execution result (tfidf_df, rounded to 3 decimals):

🤔 Guess First: "rumah" appears in both d1 and d2, while "bagus" only appears in d1. Do you think the TF-IDF weight of "bagus" in d1 is higher or lower than "rumah" in d1?

Reveal Answer

Higher (0.623 vs 0.474). "rumah" appears in 2 out of 3 documents, so it's considered less distinctive → its weight gets pulled down. "bagus" only appears in 1 document (d1) → it's considered more distinctive for that document → its weight is higher. That's the core idea behind IDF: the rarer a word is across the whole corpus, the more "distinguishing power" it carries for the document it appears in.

💡 Beyond the core material: There's a more advanced method called Word2Vec — a small neural network that learns word meaning from surrounding words, rather than just counting frequency. The result: words with similar meanings end up with vector positions close to each other (for instance, "King" and "Queen" end up close together along a dimension roughly representing "power/rank"). This is a natural next step after mastering BoW and TF-IDF.

6. Reducing Dimensions with PCA

Sometimes data has too many columns/features, making it hard to visualize or slowing a model down. PCA (Principal Component Analysis) compresses data into fewer dimensions while preserving as much of the important information as possible.

Example: the Iris flower dataset (4 features — petal/sepal length & width) compressed down to 2 dimensions.

iris = load_iris()
X = iris.data   # 4 features: petal/sepal length & width
y = iris.target # flower species

pca = PCA(n_components=2)
X_pca = pca.fit_transform(X)

plt.figure(figsize=(6, 5))
scatter = plt.scatter(X_pca[:, 0], X_pca[:, 1], c=y, cmap='viridis')
plt.xlabel('Principal Component 1')
plt.ylabel('Principal Component 2')
plt.title('Data Iris setelah PCA (4 dimensi -> 2 dimensi)')
plt.legend(handles=scatter.legend_elements()[0], labels=list(iris.target_names))
plt.show()

print(f"Dimensi awal: {X.shape[1]} fitur")
print(f"Dimensi setelah PCA: {X_pca.shape[1]} fitur")
print(f"Informasi yang berhasil dijaga: {pca.explained_variance_ratio_.sum()*100:.1f}%")

(The plot title and print labels are left exactly as run — "Dimensi awal" = original dimensions, "Dimensi setelah PCA" = dimensions after PCA, "Informasi yang berhasil dijaga" = information preserved.)

Output of the three print lines above, plus the per-component breakdown (pca.explained_variance_ratio_):

In other words: the original 4 columns get cut down to 2, but only about 2.2% of the information is lost — a trade-off that's usually well worth it for visualization or speeding up model training. The scatter plot also shows it clearly: the setosa species (cyan) is already cleanly separated from the other two even with just 2 dimensions, while versicolor (blue) and virginica (purple) overlap a bit in the middle.

7. The Data Versioning Concept

A simple analogy: saving a document as report_v1, then report_v2 after a revision — so if the new version turns out wrong, you can still go back to the old one. Data versioning applies the same idea to datasets.

Why it matters for Machine Learning:

• If a model suddenly gets worse after a data update, you can roll back to the previous data version.

• Model development becomes easier to track — which matters a lot for debugging and audits.

Commonly used real-world tools:

The simplest possible simulation: saving each dataset "version" as a separate file.

import os
os.makedirs('data_versions', exist_ok=True)

df_v1 = pd.DataFrame({'nama': ['Apple', 'Chicken'], 'kalori': [95, 231]})
df_v1.to_csv('data_versions/dataset_v1.csv', index=False)

df_v2 = df_v1.copy()
df_v2.loc[len(df_v2)] = ['Broccoli', 50]
df_v2.to_csv('data_versions/dataset_v2.csv', index=False)

(kalori = calories.)

dataset_v1.csv holds 2 rows (Apple, Chicken); dataset_v2.csv holds 3 rows (Apple, Chicken, Broccoli) — a small ("micro") change gets saved as a separate file, so the old version stays accessible at any time.

⚠️ Transparency note: the example above is just a concept simulation using manually separated files — it's not how real versioning tools like DVC actually work under the hood (they use content hashing and storage separate from Git). It's simplified here so the core idea is easy to grasp without extra installation.

8. Summary & Next Steps

A few next steps worth trying on your own:

• Swap in your own dataset instead of the examples above.

• Explore the gensim library to go deeper into Word2Vec.

• Try an open-source tool like DVC for real, production-grade data versioning.

9. Cheat Sheet: Preprocessing Checklist

• [ ] Check for missing values first with .isnull().sum() before assuming the data is dirty

• [ ] Small, non-critical gaps → dropna(); valuable data or large gaps → fillna() (mean/median/mode)

• [ ] Detect outliers with IQR, but check the business context before dropping — a statistical outlier isn't automatically a data error, and outliers can show up on either side of the distribution

• [ ] Ordered categories (low–medium–high) → Label Encoding; unordered categories → One Hot Encoding

• [ ] Short/simple text → Bag of Words; need to distinguish "distinctive" words from common ones → TF-IDF

• [ ] Too many features → consider PCA, and check how much information is actually retained

• [ ] Keep a version history of your data (not just your code) — so you can roll back if a model suddenly gets worse

10. Quiz Check

1. Why check .isnull().sum() first, before jumping straight to dropna/fillna?

Because not every dataset is automatically "dirty" — in the customer dataset example above, the result was 0% missing values across every column. Skipping the check and applying a cleaning strategy anyway risks wasting time (or worse, discarding perfectly fine data) on a problem that didn't actually exist.

2. Why was the missing salary filled with the median instead of the mean?

The median is more robust against extreme values/outliers than the mean. If one salary is far larger than the rest, the mean gets pulled upward and becomes less representative, while the median stays stable near the center of the data.

3. IQR flagged 9 outliers even though only 1 was injected intentionally, and they sat on both sides of the distribution. What does that mean?

Statistical methods like IQR have no way of distinguishing "intentionally anomalous" data from values that just happen to sit at either tail of a normal distribution. Everything outside the 1.5×IQR range gets flagged the same way. So outlier detection results still need to be verified against business context before deciding whether to drop or keep them.

4. When should you avoid Label Encoding for categories like city names or food names?

When the category has no natural order/ranking. Label Encoding assigns sequential numbers (0, 1, 2, ...) that a model can accidentally "read" as a mathematical relationship — for instance, treating category 2 as "greater than" category 1, when that ordering was really just alphabetical coincidence. One Hot Encoding is the safer choice here.

5. Why do words that appear in many documents get a lower TF-IDF weight?

Because TF-IDF's whole purpose is measuring how distinctive a word is for telling one document apart from another. A word that shows up in most or all documents (like "rumah" here) doesn't help distinguish between them much, so its IDF weight is pulled down. A rarer word carries more information about the specific document it appears in, so it gets a higher weight.

Every number and chart in this post comes from actually running the code (not a manual simulation), with the random seed for the outlier section noted so the results are reproducible.

Catatan belajar plus kode langsung jalan — enam teknik inti data preprocessing yang paling sering dipakai sebelum data masuk ke model Machine Learning, lengkap dengan angka hasil eksekusi asli, bukan contoh yang cuma ditempel.

Ada ungkapan yang sering diulang-ulang di dunia data science: "80% kerjaan AI itu ngurusin data, bukan model" (angka pastinya bervariasi tergantung sumber, tapi intinya sering disepakati). Biasanya yang dimaksud ya bagian ini: mengecek data kosong, membuang/menangani outlier, mengubah kategori dan teks jadi angka, memampatkan dimensi, dan menyimpan riwayat versi data. Semua praktik di bawah ini dijalankan langsung dengan Python (Pandas + Scikit-learn), pakai dua jenis data: dataset pelanggan sungguhan berisi 10.000 baris, dan beberapa contoh sintetis untuk memperjelas konsep.

📂 Notebook lengkapnya (sudah dieksekusi penuh, plus dataset-nya) ada di GitHub: github.com/arielshakaramiro/data-preprocessing-praktik-arielshakaramiro

Daftar Isi

• 1. Data Collection & Storage: Konsep Singkat

• 2. Missing Values: Cek Dulu, Jangan Asumsi

• 3. Mendeteksi & Menangani Outlier dengan IQR

• 4. Feature Engineering: Kategori Jadi Angka

• 5. Text Preprocessing: Bag of Words & TF-IDF

• 6. Mengurangi Dimensi Data dengan PCA

• 7. Konsep Data Versioning

• 8. Rangkuman & Langkah Selanjutnya

• 9. Cheat Sheet: Checklist Preprocessing

• 10. Quiz Check

1. Data Collection & Storage: Konsep Singkat

Sebelum masuk ke praktik, ada dua jenis data yang perlu dikenali:

• Unstructured Data — data yang "bentuknya bebas", misalnya gambar, teks, audio. Manusia bisa langsung memahami maknanya tanpa struktur khusus (contoh: kita bisa langsung mengenali gambar kucing).

• Structured Data — data yang rapi dalam bentuk tabel/baris-kolom, misalnya data transaksi atau data pelanggan. Maknanya sangat tergantung konteks bisnis, sehingga tidak selalu bisa dipahami "sekilas lihat" oleh orang awam.

Kedua jenis data ini butuh tempat penyimpanan yang aman, mudah diakses, dan bisa di-versioning (disimpan riwayat perubahannya) — konsep versioning ini dibahas lebih lanjut di bagian 7.

2. Missing Values: Cek Dulu, Jangan Asumsi

Ada dua pendekatan umum menangani nilai kosong (NaN):

• Dropna — buang baris/kolom yang datanya kosong. Cocok kalau yang hilang sedikit.

• Fillna — isi nilai kosong dengan pengganti: mean, median, atau modus (nilai paling sering muncul).

Langkah pertama yang sering dilewatkan pemula: cek dulu, jangan langsung asumsi datanya kotor.

sample_data = pd.read_csv('customers-10000.csv')
print(sample_data.isnull().sum())

(Kode aslinya memuat dataset lewat link Google Drive; di sini ditulis seolah file lokal supaya gampang dicoba ulang — isinya persis sama, dataset publik 10.000 baris dari Datablist.)

Hasil eksekusi pada dataset pelanggan (10.000 baris, 12 kolom — Index, Customer Id, First Name, Last Name, Company, City, Country, Phone 1, Phone 2, Email, Subscription Date, Website):

Missing value per kolom: 0 (semua kolom, 0.0%)

🤔 Coba Tebak Dulu: Menurutmu, dataset pelanggan publik seperti ini biasanya berapa persen datanya yang kosong?

Lihat Jawaban

Di kasus ini: 0% — dataset ini memang dataset sintetis untuk latihan (dibuat dengan Faker), jadi sengaja dibuat rapi tanpa nilai kosong. Ini justru pelajaran penting: jangan pernah asumsikan sebuah dataset "pasti kotor". Selalu jalankan .isnull().sum() dulu sebelum memutuskan strategi apa pun — kalau ternyata bersih, kamu hemat waktu; kalau ternyata ada yang kosong, kamu tahu persis di kolom mana.

Untuk melihat kedua teknik penanganannya, dipakai data sintetis kecil yang sengaja dibuat berlubang:

data = {
    'nama': ['Andi', 'Budi', 'Citra', 'Dewi', 'Eka'],
    'umur': [25, np.nan, 30, 22, np.nan],
    'gaji': [5000000, 6000000, np.nan, 4500000, 5200000]
}
df = pd.DataFrame(data)

Jumlah & persentase missing value per kolom (hasil eksekusi df.isnull().sum() dan df.isnull().sum() / len(df) * 100):

Opsi 1 — dropna():

df_dropped = df.dropna()

Baris Budi, Citra, Eka ikut terbuang karena masing-masing punya satu sel kosong — dari 5 baris tersisa 2 baris. Efektif kalau data yang hilang cuma sedikit dan baris yang hilang tidak penting; boros kalau data berharga tapi kebetulan ada satu kolom bolong.

Opsi 2 — fillna() dengan mean/median:

df_filled = df.copy()
df_filled['umur'] = df_filled['umur'].fillna(df_filled['umur'].mean())
df_filled['gaji'] = df_filled['gaji'].fillna(df_filled['gaji'].median())

Umur kosong diisi rata-rata umur yang ada, gaji kosong diisi nilai tengah (median). (Tambahan penjelasan di luar kode aslinya: median dipilih untuk gaji karena secara umum lebih tahan terhadap nilai ekstrem dibanding mean — poin ini akan lebih jelas maknanya setelah bagian outlier di bawah.)

3. Mendeteksi & Menangani Outlier dengan IQR

Outlier adalah data yang nilainya jauh berbeda dari mayoritas. Aturan pengambilan keputusannya:

• Outlier jarang terjadi dan tidak berdampak ke bisnis → boleh dibuang.

• Outlier mengganggu proses bisnis → jangan dibuang begitu saja, perlu ditangani lebih hati-hati (di skala production, salah satu pendekatan lanjutan adalah Autoencoder — model kecil yang belajar pola data normal, sehingga data yang menyimpang jauh dari pola itu punya error rekonstruksi besar dan bisa ditandai sebagai outlier).

Untuk pemula, salah satu metode paling umum dan sederhana adalah IQR (Interquartile Range). Simulasinya: 1000 data gaji berdistribusi normal (rata-rata Rp5.000.000, standar deviasi Rp1.000.000), lalu disisipkan 1 outlier ekstrem sebesar Rp50.000.000.

# Membuat contoh data gaji, dengan 1 outlier yang sangat besar
np.random.seed(42)
gaji = np.random.normal(5000000, 1000000, 1000).tolist()
gaji.append(50000000)  # outlier: gaji yang jauh lebih besar dari yang lain
df_gaji = pd.DataFrame({'gaji': gaji})

Visualisasi cepat sebelum dihitung — kode boxplot-nya:

plt.figure(figsize=(6, 4))
plt.boxplot(df_gaji['gaji'])
plt.title('Boxplot Gaji (perhatikan ada titik yang jauh di atas)')
plt.ylabel('Gaji')
plt.show()

Baru dihitung batasnya secara matematis dengan IQR:

# Deteksi outlier dengan metode IQR
Q1 = df_gaji['gaji'].quantile(0.25)
Q3 = df_gaji['gaji'].quantile(0.75)
IQR = Q3 - Q1
batas_bawah = Q1 - 1.5 * IQR
batas_atas = Q3 + 1.5 * IQR

outliers = df_gaji[(df_gaji['gaji'] < batas_bawah) | (df_gaji['gaji'] > batas_atas)]

Hasil eksekusi nyata (dengan random seed 42, jadi angkanya konsisten kalau dijalankan ulang):

🤔 Coba Tebak Dulu: Cuma disisipkan 1 outlier buatan (Rp50 juta). Menurutmu, IQR bakal mendeteksi tepat 1 outlier juga, atau bisa lebih?

Lihat Jawaban

Terdeteksi 9 outlier, bukan cuma 1. Selain angka Rp50 juta yang memang disisipkan, ada 8 data lain (hasil distribusi normal biasa, bukan disengaja) yang kebetulan jatuh di luar rentang 1,5×IQR — dan posisinya di dua sisi: 4 di bawah batas bawah (sekitar Rp1,76–2,38 juta) dan 4 di atas batas atas (sekitar Rp7,63–8,85 juta). Ini pelajaran penting: metode IQR itu murni statistik, dia tidak tahu mana outlier yang "disengaja/anomali" dan mana yang cuma "kebetulan ekstrem karena sebaran data". Makanya sebelum membuang data yang ditandai IQR, tetap perlu dicek konteks bisnisnya — misalnya gaji Rp8 juta di dataset ini mungkin memang gaji direktur yang valid, bukan kesalahan input, dan gaji Rp1,8 juta mungkin memang pegawai magang.

Setelah dibuang, sisa data bersih:

df_bersih = df_gaji[(df_gaji['gaji'] >= batas_bawah) & (df_gaji['gaji'] <= batas_atas)]

992 dari 1.001 baris tersisa.

4. Feature Engineering: Kategori Jadi Angka

Model Machine Learning cuma mengerti angka, bukan teks kategori seperti "Apple" atau "Chicken". Dua cara paling umum mengonversinya:

Data contoh:

food = pd.DataFrame({
    'Food Name': ['Apple', 'Chicken', 'Broccoli'],
    'Calories': [95, 231, 50]
})

Label Encoding (hasil eksekusi LabelEncoder):

Perhatikan angkanya diurutkan alfabetis (Apple=0, Broccoli=1, Chicken=2) — bukan berdasarkan kalori atau urutan logis apa pun. Ini sumber bug klasik: kalau dipakai untuk kategori tanpa urutan, model bisa salah "mengira" Chicken (2) lebih besar/penting dua kali lipat dari Broccoli (1), padahal itu cuma nomor urut alfabet.

One Hot Encoding (hasil eksekusi pd.get_dummies):

Tidak ada urutan tersirat — setiap kategori dapat kolom sendiri. Trade-off-nya: kalau kategorinya ratusan/ribuan (misalnya kode pos), jumlah kolom bisa meledak.

5. Text Preprocessing: Bag of Words & TF-IDF

Sama seperti kategori, teks juga perlu diubah jadi vektor angka (vektorisasi). Korpus contoh (3 kalimat pendek Bahasa Indonesia):

corpus = [
    "rumah ini bagus",        # d1
    "rumah saya makan nasi",  # d2
    "saya makan nasi"         # d3
]

Bag of Words — menghitung berapa kali tiap kata muncul di tiap dokumen:

vectorizer = CountVectorizer()
bow_matrix = vectorizer.fit_transform(corpus)

bow_df = pd.DataFrame(
    bow_matrix.toarray(),
    columns=vectorizer.get_feature_names_out(),
    index=['d1', 'd2', 'd3']
)

Hasil eksekusi (bow_df):

BoW punya kelemahan: kata yang sering muncul di semua dokumen dianggap "penting" secara nilai, padahal belum tentu informatif untuk membedakan dokumen satu dengan lainnya.

TF-IDF (Term Frequency–Inverse Document Frequency) memperbaiki ini — kata yang muncul di banyak dokumen diberi bobot lebih rendah, kata yang unik/jarang diberi bobot lebih tinggi:

tfidf_vectorizer = TfidfVectorizer()
tfidf_matrix = tfidf_vectorizer.fit_transform(corpus)

tfidf_df = pd.DataFrame(
    tfidf_matrix.toarray(),
    columns=tfidf_vectorizer.get_feature_names_out(),
    index=['d1', 'd2', 'd3']
).round(3)

Hasil eksekusi (tfidf_df):

🤔 Coba Tebak Dulu: Kata "rumah" muncul di d1 dan d2, sedangkan "bagus" cuma muncul di d1. Menurutmu, bobot TF-IDF kata "bagus" di d1 lebih tinggi atau lebih rendah dibanding bobot "rumah" di d1?

Lihat Jawaban

Lebih tinggi (0.623 vs 0.474). Karena "rumah" muncul di 2 dari 3 dokumen, ia dianggap kurang khas/kurang membedakan → bobotnya diturunkan. "Bagus" cuma muncul di 1 dokumen (d1) → dianggap lebih khas untuk dokumen itu → bobotnya lebih tinggi. Inilah inti IDF: makin jarang sebuah kata muncul di seluruh korpus, makin tinggi "nilai pembeda"-nya untuk dokumen tempat ia muncul.

💡 Tambahan di luar materi utama: Ada metode yang lebih canggih bernama Word2Vec — neural network kecil yang mempelajari makna kata dari kata-kata di sekitarnya, bukan sekadar menghitung frekuensi. Hasilnya, kata-kata bermakna mirip akan punya posisi vektor yang berdekatan (misalnya "Raja" dan "Ratu" berdekatan pada dimensi tertentu yang merepresentasikan "kekuasaan"). Ini topik lanjutan setelah menguasai BoW dan TF-IDF.

6. Mengurangi Dimensi Data dengan PCA

Kadang data punya terlalu banyak kolom/fitur, sehingga sulit divisualisasikan atau bikin model lambat. PCA (Principal Component Analysis) memampatkan data ke dimensi lebih kecil sambil mempertahankan informasi paling penting.

Contoh: dataset bunga Iris (4 fitur: panjang & lebar kelopak/mahkota) dimampatkan jadi 2 dimensi.

iris = load_iris()
X = iris.data   # 4 fitur: panjang & lebar kelopak/mahkota bunga
y = iris.target # jenis bunga

pca = PCA(n_components=2)
X_pca = pca.fit_transform(X)

plt.figure(figsize=(6, 5))
scatter = plt.scatter(X_pca[:, 0], X_pca[:, 1], c=y, cmap='viridis')
plt.xlabel('Principal Component 1')
plt.ylabel('Principal Component 2')
plt.title('Data Iris setelah PCA (4 dimensi -> 2 dimensi)')
plt.legend(handles=scatter.legend_elements()[0], labels=list(iris.target_names))
plt.show()

print(f"Dimensi awal: {X.shape[1]} fitur")
print(f"Dimensi setelah PCA: {X_pca.shape[1]} fitur")
print(f"Informasi yang berhasil dijaga: {pca.explained_variance_ratio_.sum()*100:.1f}%")

Output dari ketiga baris print di atas, plus rincian per komponen (pca.explained_variance_ratio_):

Artinya: dari 4 kolom asli dipangkas jadi 2 kolom, tapi cuma kehilangan sekitar 2,2% informasi — trade-off yang biasanya sangat sepadan untuk keperluan visualisasi atau mempercepat training model. Dari scatter plot-nya juga terlihat: spesies setosa (cyan) sudah terpisah jelas dari dua spesies lain bahkan cuma dengan 2 dimensi, sementara versicolor (biru) dan virginica (ungu) sedikit tumpang tindih di tengah.

7. Konsep Data Versioning

Analogi sederhana: menyimpan dokumen sebagai laporan_v1, lalu laporan_v2 setelah direvisi — supaya kalau versi baru ternyata salah, masih bisa kembali ke versi lama. Data versioning menerapkan ide yang sama untuk dataset.

Kenapa penting untuk Machine Learning:

• Kalau model tiba-tiba memburuk setelah data diperbarui, bisa rollback ke versi data sebelumnya.

• Proses pengembangan model jadi lebih tertrack (mudah dilacak riwayatnya) — penting untuk debugging dan audit.

Contoh tools yang lazim dipakai di dunia nyata:

Simulasi paling sederhana: menyimpan tiap "versi" dataset sebagai file terpisah.

import os
os.makedirs('data_versions', exist_ok=True)

df_v1 = pd.DataFrame({'nama': ['Apple', 'Chicken'], 'kalori': [95, 231]})
df_v1.to_csv('data_versions/dataset_v1.csv', index=False)

df_v2 = df_v1.copy()
df_v2.loc[len(df_v2)] = ['Broccoli', 50]
df_v2.to_csv('data_versions/dataset_v2.csv', index=False)

dataset_v1.csv berisi 2 baris (Apple, Chicken), dataset_v2.csv berisi 3 baris (Apple, Chicken, Broccoli) — perubahan kecil ("micro change") tersimpan sebagai file terpisah, sehingga versi lama tetap bisa diakses kapan saja.

⚠️ Catatan transparansi: contoh di atas cuma simulasi konsep dengan menyimpan file terpisah manual — bukan cara kerja tool versioning sungguhan seperti DVC (yang pakai sistem hashing dan penyimpanan terpisah dari Git). Ini disederhanakan supaya intinya mudah dipahami tanpa perlu instalasi tambahan.

8. Rangkuman & Langkah Selanjutnya

Langkah lanjutan yang bisa dicoba sendiri:

• Ganti contoh data di atas dengan dataset sendiri.

• Eksplorasi library gensim untuk mempelajari Word2Vec lebih lanjut.

• Coba tools open-source seperti DVC untuk praktik data versioning yang sesungguhnya.

9. Cheat Sheet: Checklist Preprocessing

• [ ] Cek missing value dulu dengan .isnull().sum() sebelum asumsi data kotor

• [ ] Data hilang sedikit & tidak krusial → dropna(); data berharga/hilang banyak → fillna() (mean/median/modus)

• [ ] Deteksi outlier dengan IQR, tapi cek konteks bisnis sebelum membuang — outlier statistik ≠ otomatis kesalahan data, dan outlier bisa muncul di dua sisi (atas maupun bawah)

• [ ] Kategori berurutan (rendah–sedang–tinggi) → Label Encoding; kategori tanpa urutan → One Hot Encoding

• [ ] Teks pendek/simpel → Bag of Words; butuh membedakan kata "khas" vs kata umum → TF-IDF

• [ ] Fitur terlalu banyak → pertimbangkan PCA, cek berapa persen informasi yang masih terjaga

• [ ] Simpan riwayat versi data (bukan cuma versi kode) — supaya bisa rollback kalau model tiba-tiba memburuk

10. Quiz Check

1. Kenapa mengecek .isnull().sum() lebih dulu itu penting, sebelum langsung pakai dropna/fillna?

Karena tidak semua dataset otomatis "kotor" — pada contoh dataset pelanggan di atas, hasilnya 0% missing value di semua kolom. Kalau langsung menjalankan strategi pembersihan tanpa cek dulu, bisa jadi kamu membuang waktu (atau lebih parah, membuang data yang sebenarnya baik-baik saja) untuk masalah yang tidak ada.

2. Kenapa gaji yang kosong diisi dengan median, bukan mean?

Median lebih tahan (robust) terhadap nilai ekstrem/outlier dibanding mean. Kalau ada satu gaji yang jauh lebih besar dari yang lain, mean akan "tertarik" ke atas dan jadi kurang representatif, sedangkan median tetap stabil di tengah data.

3. Metode IQR mendeteksi 9 outlier padahal cuma 1 yang disisipkan sengaja, dan posisinya tersebar di dua sisi. Apa artinya?

Metode statistik seperti IQR tidak tahu mana data yang "sengaja aneh" dan mana yang "kebetulan berada di ujung distribusi normal" — baik di sisi atas maupun bawah. Semua yang berada di luar rentang 1,5×IQR ditandai sama. Karena itu, hasil deteksi outlier tetap perlu diverifikasi dengan pemahaman konteks bisnis sebelum diputuskan dibuang atau dipertahankan.

4. Kapan sebaiknya menghindari Label Encoding untuk kategori seperti nama kota atau nama makanan?

Ketika kategori itu tidak punya urutan/tingkatan alami. Label Encoding memberi angka berurutan (0, 1, 2, ...) yang secara tidak sengaja bisa "dibaca" model sebagai hubungan matematis (misalnya kategori bernomor 2 dianggap "lebih besar" dari kategori bernomor 1), padahal urutan itu cuma kebetulan alfabetis. One Hot Encoding lebih aman untuk kasus ini.

5. Kenapa bobot TF-IDF kata yang muncul di banyak dokumen justru lebih rendah?

Karena tujuan TF-IDF adalah mengukur seberapa "khas" sebuah kata untuk membedakan satu dokumen dari dokumen lain. Kata yang muncul di banyak/semua dokumen (seperti "rumah" di kasus ini) tidak banyak membantu membedakan dokumen, jadi diberi bobot (IDF) lebih rendah. Kata yang jarang muncul justru lebih informatif untuk membedakan dokumen tempat ia muncul, sehingga diberi bobot lebih tinggi.

Semua angka dan chart pada tulisan ini adalah hasil eksekusi kode secara langsung (bukan simulasi manual), dengan random seed yang dicantumkan pada bagian outlier supaya bisa direproduksi.

This is a case study from a project I built as an Individual Assignment at Rework Academy — AI Engineering Bootcamp, in the Data Engineering: Pipeline & Preparation session. I built an ETL pipeline for a deliberately "dirtied" automobile dataset, complete with unit tests and CI. Every number in this post comes straight from running the pipeline — none of it is estimated.

Full source code is on GitHub: assignment-data-pipeline-arielshakaramiro

Table of Contents

• The Case: A "Dirty" Automobile Dataset

• Stage 1 — Extract & Inspect

• Stage 2 — Data Cleaning

• Stage 3 — Data Transformation

• Wiring It All Into an ETL Pipeline

• Testing & CI: Making Sure the Pipeline Doesn't Quietly Break

• Final Result

• Checklist: Data Cleaning Before Modeling

• Closing Thoughts

The Case: A "Dirty" Automobile Dataset

The dataset is automobileEDA_dirty_training.csv — 205 rows of vehicle specs (make, dimensions, engine, price, fuel economy) intentionally corrupted for practice. The rule was simple but strict: the raw dataset can never be overwritten or edited in place. Every output has to be a new file, generated automatically by the script — not copied from an "answer key."

Initial inspection turned up 7 distinct data quality issues:

1. Missing values — 17 empty cells spread across 7 columns

2. Duplicate records — 4 fully identical rows

3. Inconsistent categories — mixed casing across several columns

4. Excess whitespace — trailing spaces stuck to some text values

5. Mixed date formats — 4 different formats in a single column

6. Categorical columns that are actually numeric — numbers spelled out as words

7. Unscaled numeric ranges — raw units, ready to bias any model toward large-magnitude features

Stage 1 — Extract & Inspect

Before cleaning anything, the pipeline looks at the data first. inspect_data() prints the first five rows, row/column counts, dtypes per column, missing value counts, duplicate counts, and unique values for categorical columns — every single time the pipeline runs, not just once during manual exploration.

Here's what inspection found:

Columns with missing values:

The most interesting finding came from checking unique values in categorical columns. The make column, which should only contain 22 brand names, actually had 27 distinct text variants because of things like:

See the category inconsistencies that were found

• make: ALFA-ROMERO, Audi, BMW vs alfa-romero, audi, bmw — plus values like dodge / porsche with trailing whitespace

• body-style: SEDAN, Sedan, sedan — three different spellings of the same thing

• drive-wheels: AWD, RWD vs fwd, rwd

• fuel-system: MPFI, Mpfi, mpfi

If this gets one-hot encoded without cleaning first, ALFA-ROMERO and alfa-romero get counted as two different brands. The model ends up learning from noise instead of signal.

def inspect_data(df: pd.DataFrame) -> None:
    """Print an initial inspection of the dataset to the terminal."""
    print("-- Missing values per column (only > 0) --")
    miss = df.isna().sum()
    print(miss[miss > 0] if (miss > 0).any() else "No missing values")

    print("-- Unique values in relevant categorical columns --")
    for col in ["make", "body-style", "drive-wheels", "fuel-system",
                "num-of-doors", "aspiration", "horsepower-binned"]:
        if col in df.columns:
            uniques = sorted(df[col].dropna().astype(str).unique().tolist())
            print(f"  {col} ({len(uniques)}): {uniques}")

Stage 2 — Data Cleaning

Every cleaning decision is documented with a reason — not just "fill in the missing value," but why that specific method was chosen for that specific column.

The date column deserves a bit more explanation. transaction_date mixes four formats in the same column: 2025-01-01, 02/01/2025, 01-03-2025, 04-Jan-2025. The pipeline tries each format in sequence until one matches:

def _parse_mixed_dates(series: pd.Series) -> pd.Series:
    """Convert a date column with mixed formats into a uniform datetime."""
    formats = ["%Y-%m-%d", "%d/%m/%Y", "%m-%d-%Y", "%d-%b-%Y"]
    parsed = pd.Series(pd.NaT, index=series.index, dtype="datetime64[ns]")
    remaining = series.copy()
    for fmt in formats:
        mask = parsed.isna() & remaining.notna()
        if not mask.any():
            break
        attempt = pd.to_datetime(remaining[mask], format=fmt, errors="coerce")
        parsed.loc[attempt.notna().index[attempt.notna()]] = attempt.dropna()
    return parsed

🤔 Guess first: of the 7 columns with missing values, which one was deliberately left empty after cleaning?

The answer: transaction_date — and that was a deliberate call, not a bug.

Every other numeric column (stroke, horsepower, price) got filled with the median, and categorical columns got filled with the mode. But a transaction date is a different kind of thing — it's factual, historical data, not a statistical quantity. A car's price has a "reasonable value" you can approximate from the median. A transaction date has exactly one correct value, and there's no statistical pattern that can safely guess it without risking a fabricated fact.

The right data engineering practice for a missing ground-truth field is to go back to the data owner and confirm — not to guess through imputation. So those 2 rows were honestly left blank: better empty-but-correct than filled-but-wrong. This column also isn't used as a modeling feature, so it doesn't affect any downstream analysis.

Cleaning result: 205 → 201 rows (4 duplicates removed), missing values 17 → 0 (aside from the 2 transaction_date rows left blank on purpose).

Stage 3 — Data Transformation

Cleaning makes the data correct. Transformation makes it model-ready. Two different things — and beginners mix them up more often than you'd expect.

That last row is the one people tend to skip: not every categorical column belongs in one-hot encoding. If make (22 categories) also got one-hot encoded, that's 22 new columns, most of them zeros for any given row — sparse and wasteful. Frequency encoding is the middle ground: each brand becomes a single number representing how often it shows up in the dataset — one column, still informative.

Real before/after example (Min-Max Scaling on horsepower):

Worth remembering: Min-Max Scaling only changes the axis range, not the shape of the distribution. Look at the price histogram before and after — the spread pattern is identical, only the x-axis shifts from raw USD to a 0–1 range.

New columns produced: horsepower_ordinal, make_freq, and every one-hot column (body-style_*, drive-wheels_*, aspiration_*, engine-type_*, engine-location_*, fuel-system_*). After all this, the column count goes from 30 to 51.

Wiring It All Into an ETL Pipeline

Every step above is packaged into 5 separate functions, wired together by one orchestrator:

def run_pipeline() -> pd.DataFrame:
    """Run the entire ETL pipeline start to finish."""
    df_raw = read_data(RAW_PATH)         # Extract
    inspect_data(df_raw)                 # Transform: inspection
    df_clean = clean_data(df_raw)        # Transform: cleaning
    df_final = transform_data(df_clean)  # Transform: transformation
    save_data(df_final, PROCESSED_PATH)  # Load
    return df_final

if __name__ == "__main__":
    run_pipeline()

One small but useful detail: file paths are resolved with Path(__file__).resolve().parent.parent, not a path relative to the current working directory. That means the pipeline runs correctly whether it's invoked from the project root or from inside src/ — it doesn't depend on where the script happens to be launched from.

Testing & CI: Making Sure the Pipeline Doesn't Quietly Break

Getting a pipeline to run correctly once is the easy part. The harder part is making sure it's still correct six months later, after it's been edited, or after the source data shifts slightly. That's what tests are for.

There are 15 tests in tests/test_pipeline.py, organized around the ETL stages themselves:

def test_no_duplicates_after_clean(raw_df):
    cleaned = pipeline.clean_data(raw_df)
    assert cleaned.duplicated().sum() == 0

def test_scaled_columns_within_0_1(processed_df):
    """Every Min-Max scaled column must fall within [0, 1]."""
    for col in pipeline.NUMERIC_TO_SCALE:
        assert processed_df[col].min() >= -1e-9
        assert processed_df[col].max() <= 1 + 1e-9

def test_raw_dataset_untouched(raw_df):
    """Running cleaning must not mutate the original raw object."""
    before = raw_df.shape
    pipeline.clean_data(raw_df)
    assert raw_df.shape == before

That last test is my favorite: it turns the assignment's own rule ("never modify the raw dataset") into code that can actually fail, instead of just a promise in a README that's easy to break without noticing.

All of this also runs automatically via GitHub Actions on every push or pull request to main:

- name: Run ETL pipeline
  run: python src/pipeline.py

- name: Verify processed dataset was generated
  run: test -f data/processed/automobileEDA_processed.csv

- name: Run tests
  run: pytest -v

Here's the actual terminal output (not a mockup) from running the pipeline:

Final Result

Checklist: Data Cleaning Before Modeling

A short checklist I use myself on any new tabular dataset — feel free to copy it:

• [ ] Check shape, dtypes, and the first 5 rows before doing anything else

• [ ] Count missing values per column, not just the total

• [ ] Check duplicated().sum() before and after cleaning

• [ ] For every categorical column: run .unique() and look for casing or whitespace variants

• [ ] For date columns: check whether the format is consistent across all rows, not just the first few

• [ ] Decide a missing-value strategy per column, not one strategy for everything (numeric → median/mean, categorical → mode, ground-truth fields → don't impute unless you're sure)

• [ ] Only scale/normalize after cleaning, never before

• [ ] Choose an encoding method based on cardinality, not one-hot for every categorical column by default

• [ ] Write at least a handful of assert/tests that validate the pipeline's output, not just print statements

Closing Thoughts

The hardest part of this project wasn't the Pandas syntax — drop_duplicates() and pd.get_dummies() are already muscle memory at this point. The hard part was making defensible decisions: why median instead of mean, why one column gets left empty while others get imputed, why one categorical column gets one-hot encoded while another gets frequency encoding instead.

That's what I tried to show here: a good ETL pipeline isn't just code that runs without errors — it's a chain of decisions you can actually defend, one line at a time.

Full source, including the test suite and CI config, is on GitHub: assignment-data-pipeline-arielshakaramiro

If you'd have handled any part of this differently — especially the imputation calls or the encoding choices — the comments are open.

This project was built as part of the Rework Academy — AI Engineering Bootcamp*, Data Engineering: Pipeline & Preparation session. The automobile dataset (automobileEDA) was provided as practice material for that session.*

Tulisan ini adalah studi kasus dari proyek yang saya kerjakan sebagai Individual Assignment di Rework Academy — AI Engineering Bootcamp, sesi Data Engineering: Pipeline & Preparation. Saya membangun ETL pipeline untuk dataset otomotif yang sengaja "dikotori", lengkap dengan unit test dan CI. Semua angka di tulisan ini diambil langsung dari hasil eksekusi pipeline-nya — bukan estimasi.

Source code lengkap ada di GitHub: assignment-data-pipeline-arielshakaramiro

Daftar Isi

• Studi Kasus: Dataset Otomotif yang "Kotor"

• Tahap 1 — Extract & Inspect

• Tahap 2 — Data Cleaning

• Tahap 3 — Data Transformation

• Menyusun Semuanya Jadi Pipeline ETL

• Testing & CI: Memastikan Pipeline Tidak Diam-Diam Rusak

• Hasil Akhir

• Checklist: Data Cleaning Sebelum Modeling

• Penutup

Studi Kasus: Dataset Otomotif yang "Kotor"

Dataset yang dipakai adalah automobileEDA_dirty_training.csv — 205 baris spesifikasi kendaraan (merek, dimensi, mesin, harga, konsumsi bahan bakar) yang sengaja dibuat kotor untuk latihan. Aturan mainnya sederhana tapi tegas: dataset mentah tidak boleh ditimpa atau diubah langsung. Semua hasil harus keluar sebagai file baru, dihasilkan otomatis oleh script — bukan disalin dari file "jawaban".

Setelah pemeriksaan awal, ada 7 masalah data yang harus ditangani:

1. Missing values — 17 sel kosong tersebar di 7 kolom

2. Duplicate records — 4 baris identik penuh

3. Kategori tidak konsisten — huruf besar/kecil campur di beberapa kolom

4. Whitespace berlebih — spasi menempel di beberapa nilai teks

5. Format tanggal campuran — 4 format berbeda dalam satu kolom

6. Kolom kategorikal yang sebenarnya numerik — angka ditulis sebagai kata

7. Skala numerik belum seragam — satuan asli, siap bikin model bias ke fitur dengan angka besar

Tahap 1 — Extract & Inspect

Sebelum membersihkan apa pun, pipeline dulu melihat datanya. Fungsi inspect_data() menampilkan lima baris pertama, jumlah baris/kolom, tipe data per kolom, jumlah missing values, jumlah duplikat, dan nilai unik pada kolom kategorikal — semuanya dicetak ke terminal setiap kali pipeline dijalankan, bukan cuma sekali waktu eksplorasi manual.

Ini hasil pemeriksaannya:

Kolom dengan missing values:

Yang paling menarik justru dari pengecekan nilai unik kolom kategorikal. Kolom make yang seharusnya cuma berisi 22 nama merek, ternyata punya 27 varian teks karena hal-hal seperti ini:

Lihat contoh inkonsistensi kategori yang ditemukan

• make: ALFA-ROMERO, Audi, BMW vs alfa-romero, audi, bmw — dan beberapa nilai seperti dodge / porsche yang punya spasi menempel di akhir

• body-style: SEDAN, Sedan, sedan — tiga cara berbeda untuk hal yang sama

• drive-wheels: AWD, RWD vs fwd, rwd

• fuel-system: MPFI, Mpfi, mpfi

Kalau ini langsung di-one-hot encode tanpa dibersihkan dulu, ALFA-ROMERO dan alfa-romero akan dihitung sebagai dua merek berbeda. Model jadi belajar dari noise, bukan dari sinyal.

def inspect_data(df: pd.DataFrame) -> None:
    """Menampilkan pemeriksaan awal dataset ke terminal."""
    print("-- Missing values per kolom (hanya > 0) --")
    miss = df.isna().sum()
    print(miss[miss > 0] if (miss > 0).any() else "Tidak ada missing values")

    print("-- Nilai unik kolom kategorikal relevan --")
    for col in ["make", "body-style", "drive-wheels", "fuel-system",
                "num-of-doors", "aspiration", "horsepower-binned"]:
        if col in df.columns:
            uniques = sorted(df[col].dropna().astype(str).unique().tolist())
            print(f"  {col} ({len(uniques)}): {uniques}")

Tahap 2 — Data Cleaning

Setiap keputusan cleaning didokumentasikan dengan alasannya — bukan sekadar "hapus missing value", tapi kenapa metode itu yang dipilih untuk kolom itu.

Bagian kolom tanggal ini butuh sedikit penjelasan. Kolom transaction_date punya 4 format berbeda tercampur dalam satu kolom: 2025-01-01, 02/01/2025, 01-03-2025, 04-Jan-2025. Pipeline mencoba setiap format satu per satu sampai salah satu cocok:

def _parse_mixed_dates(series: pd.Series) -> pd.Series:
    """Mengubah kolom tanggal dengan format campuran menjadi datetime seragam."""
    formats = ["%Y-%m-%d", "%d/%m/%Y", "%m-%d-%Y", "%d-%b-%Y"]
    parsed = pd.Series(pd.NaT, index=series.index, dtype="datetime64[ns]")
    remaining = series.copy()
    for fmt in formats:
        mask = parsed.isna() & remaining.notna()
        if not mask.any():
            break
        attempt = pd.to_datetime(remaining[mask], format=fmt, errors="coerce")
        parsed.loc[attempt.notna().index[attempt.notna()]] = attempt.dropna()
    return parsed

🤔 Coba tebak dulu: dari 7 kolom yang punya missing values, kolom mana yang sengaja dibiarkan tetap kosong setelah cleaning?

Jawabannya: transaction_date, dan itu keputusan sadar, bukan bug.

Semua kolom numerik lain (stroke, horsepower, price) diisi pakai median, dan kolom kategorikal diisi pakai modus. Tapi tanggal transaksi itu beda karakternya — dia data faktual/historis, bukan besaran statistik. Harga mobil punya "nilai yang masuk akal" yang bisa didekati dari median. Tanggal transaksi cuma punya satu nilai yang benar, dan itu tidak bisa ditebak dari pola statistik tanpa berisiko memalsukan fakta kejadian.

Praktik data engineering yang tepat untuk ground-truth field yang hilang adalah mengonfirmasi ulang ke data owner — bukan menebak lewat imputasi. Jadi 2 baris itu dibiarkan kosong secara jujur: lebih baik kosong-tapi-benar daripada terisi-tapi-salah. Kolom ini juga tidak dipakai sebagai fitur pemodelan, jadi tidak memengaruhi hasil analisis nantinya.

Hasil cleaning: 205 → 201 baris (4 duplikat dihapus), missing values 17 → 0 (di luar 2 baris transaction_date yang sengaja dibiarkan).

Tahap 3 — Data Transformation

Cleaning membuat data benar. Transformation membuat data siap dipakai model. Dua hal yang berbeda, dan sering tertukar oleh pemula.

Poin terakhir ini yang sering dilewatkan orang: tidak semua kolom kategorikal cocok di-one-hot. Kalau make (22 kategori) di-one-hot juga, itu nambah 22 kolom baru yang sebagian besar isinya nol — sparse dan boros. Frequency encoding jadi jalan tengah: setiap merek direpresentasikan sebagai proporsi kemunculannya di dataset, tetap 1 kolom, tetap informatif.

Contoh nyata sebelum vs sesudah Min-Max Scaling (horsepower):

Yang penting dipahami: Min-Max Scaling hanya mengubah rentang sumbu, bukan bentuk distribusi. Perhatikan histogram price sebelum dan sesudah — pola sebarannya identik, cuma sumbu-x yang berubah dari satuan USD ke rentang 0–1.

Kolom baru yang dihasilkan: horsepower_ordinal, make_freq, dan seluruh kolom one-hot (body-style_*, drive-wheels_*, aspiration_*, engine-type_*, engine-location_*, fuel-system_*). Setelah semua transformasi ini, jumlah kolom naik dari 30 menjadi 51.

Menyusun Semuanya Jadi Pipeline ETL

Semua langkah di atas dikemas jadi 5 function terpisah, dirangkai oleh satu orchestrator:

def run_pipeline() -> pd.DataFrame:
    """Menjalankan seluruh pipeline ETL dari awal sampai akhir."""
    df_raw = read_data(RAW_PATH)         # Extract
    inspect_data(df_raw)                 # Transform: inspeksi
    df_clean = clean_data(df_raw)        # Transform: cleaning
    df_final = transform_data(df_clean)  # Transform: transformasi
    save_data(df_final, PROCESSED_PATH)  # Load
    return df_final

if __name__ == "__main__":
    run_pipeline()

Satu detail kecil yang berguna: path file dihitung pakai Path(__file__).resolve().parent.parent, bukan path relatif ke current working directory. Artinya pipeline tetap jalan benar baik dipanggil dari root folder maupun dari dalam folder src/ — tidak bergantung dari mana script dieksekusi.

Testing & CI: Memastikan Pipeline Tidak Diam-Diam Rusak

Pipeline yang jalan sekali dengan benar itu gampang. Yang lebih sulit: memastikan dia tetap benar setelah diubah enam bulan kemudian, atau ketika dataset sumbernya berubah sedikit. Di sinilah test masuk.

Ada 15 test di tests/test_pipeline.py, dibagi mengikuti tahap ETL-nya sendiri:

def test_no_duplicates_after_clean(raw_df):
    cleaned = pipeline.clean_data(raw_df)
    assert cleaned.duplicated().sum() == 0

def test_scaled_columns_within_0_1(processed_df):
    """Semua kolom hasil Min-Max Scaling harus berada di rentang [0, 1]."""
    for col in pipeline.NUMERIC_TO_SCALE:
        assert processed_df[col].min() >= -1e-9
        assert processed_df[col].max() <= 1 + 1e-9

def test_raw_dataset_untouched(raw_df):
    """Menjalankan cleaning tidak boleh mengubah objek raw asli."""
    before = raw_df.shape
    pipeline.clean_data(raw_df)
    assert raw_df.shape == before

Test terakhir itu yang saya suka: dia memvalidasi ketentuan dari assignment-nya sendiri ("dataset mentah tidak boleh diubah") sebagai kode yang bisa gagal, bukan cuma janji di README yang gampang dilanggar tanpa sadar.

Semua test ini juga jalan otomatis lewat GitHub Actions setiap kali ada push atau pull request ke main:

- name: Run ETL pipeline
  run: python src/pipeline.py

- name: Verify processed dataset was generated
  run: test -f data/processed/automobileEDA_processed.csv

- name: Run tests
  run: pytest -v

Hasil eksekusi nyata (bukan simulasi) waktu pipeline ini dijalankan:

Hasil Akhir

Checklist: Data Cleaning Sebelum Modeling

Checklist ringkas yang saya pakai sendiri setiap kali dapat dataset tabular baru — silakan disalin:

• [ ] Cek shape, dtypes, dan 5 baris pertama sebelum melakukan apa pun

• [ ] Hitung missing values per kolom, bukan cuma total

• [ ] Cek duplicated().sum() sebelum dan sesudah cleaning

• [ ] Untuk tiap kolom kategorikal: .unique() dan lihat apakah ada variasi huruf besar/kecil atau whitespace

• [ ] Untuk kolom tanggal: cek apakah formatnya konsisten di seluruh baris, bukan cuma di beberapa baris pertama

• [ ] Tentukan strategi missing value per kolom, bukan satu strategi untuk semua (numerik → median/mean, kategorikal → modus, ground-truth field → jangan diimputasi kalau tidak yakin)

• [ ] Baru lakukan scaling/normalisasi setelah cleaning, bukan sebelum

• [ ] Pilih metode encoding berdasarkan kardinalitas kolom, bukan pakai one-hot untuk semua kolom kategorikal

• [ ] Tulis minimal beberapa assert/test yang memvalidasi output pipeline, bukan cuma print statement

Penutup

Bagian tersulit dari proyek ini bukan di sintaks Pandas-nya — drop_duplicates() dan pd.get_dummies() sudah jadi hal yang dihafal luar kepala. Bagian tersulit adalah membuat keputusan yang bisa dipertanggungjawabkan: kenapa median dan bukan mean, kenapa satu kolom dibiarkan kosong sementara yang lain diimputasi, kenapa satu kolom kategorikal di-one-hot sementara yang lain pakai frequency encoding.

Itu yang saya coba tunjukkan lewat tulisan ini: ETL pipeline yang baik bukan cuma soal kode yang jalan tanpa error, tapi soal reasoning yang bisa dipertanggungjawabkan di setiap baris keputusan.

Source code lengkap, termasuk test suite dan CI config-nya, ada di GitHub: assignment-data-pipeline-arielshakaramiro

Kalau ada bagian yang menurutmu bisa ditangani dengan cara lain — terutama soal keputusan imputasi atau pemilihan metode encoding — komentar di bawah selalu terbuka untuk didiskusikan.

Proyek ini dikerjakan sebagai bagian dari Rework Academy — AI Engineering Bootcamp, sesi Data Engineering: Pipeline & Preparation. Dataset otomotif (automobileEDA) disediakan sebagai materi latihan pada sesi tersebut.

Part 2 of 2: continuing from Part 1, which covered Data Cleaning. Here we move into Data Transformation, how to save a clean dataset, Data Versioning the MLOps way, Best Practices, and the big-picture Data Pipeline Flow.

Quick recap: in Part 1 we covered why preprocessing matters (the 80/20 formula), how data is collected & stored, the basics of Pandas, and how to clean data — from missing values and duplicates to outliers and adversarial data. Now it's time to turn that clean data into a format that Machine Learning algorithms can actually "digest."

Table of Contents

• 1. Data Transformation

• 2. Saving a Clean Dataset

• 3. Data Versioning & Management Strategies

• 4. Best Practices, Common Mistakes & Error Handling

• 5. Before vs After & Data Pipeline Flow

• 6. Quiz Check — Part 2

• 7. Preprocessing Checklist Cheat Sheet

1. Data Transformation

Most ML algorithms only accept numeric data, so non-numeric data needs to be turned into a numeric representation (vectorization / encoding). This is the core of data transformation.

1.1 Sorting & Filtering

• Sorting — the "Order before logic" principle: sorted data is easier for the model to analyze and process.

• Filtering — lets you focus on a specific subset of data, for example only transactions from Asia, or only products priced above Rp 100,000.

# Sorting
df_sorted = df.sort_values('price')                    # ascending
df_sorted = df.sort_values('price', ascending=False)    # descending

# Filtering
df_asia = df[df['region'] == 'Asia']
df_filtered = df[(df['region'] == 'Asia') & (df['price'] > 50000)]              # AND
df_filtered = df[(df['region'] == 'Asia') | (df['region'] == 'Europe')]         # OR
df_filtered = df[df['region'].isin(['Asia', 'Europe', 'Africa'])]               # isin()

1.2 Encoding Categorical Data

🤔 Guess First: Clothing size (S, M, L) and hair color are both categories. Do you think both should be encoded the same way?

See the Answer

No. Clothing size has a ranking order (S < M < L), so it's ordinal — it can be converted directly into numbers. Hair color has no order, so it's nominal — it's better to use One-Hot Encoding.

• Nominal: categories without order/ranking. Examples: hair color, ethnicity, vehicle type, gender → best encoded with One-Hot Encoding so the model doesn't assume any order exists.

• Ordinal: categories with a meaningful order/ranking. Examples: clothing size (S, M, L), education level, satisfaction level → can be converted directly into numeric values.

1.3 Dimensionality Reduction

Goal: reduce the number of features without losing important information, to address the Curse of Dimensionality (too many features → slow training, complexity, decreased performance).

Number of Dimensionality = the number of features/variables in a dataset.

i. Data Transformation Techniques

• PCA (Principal Component Analysis) — finds the directions with the largest data variation (principal components); widely used on numeric data to compress correlated features.

• t-SNE — preserves proximity between data points (local structure); good for visualizing clusters in high-dimensional data.

• UMAP — similar goal to t-SNE but faster & more scalable for large datasets; can also be used for feature engineering.

ii. Removing Less Relevant Features

• .corr() (Pandas) — measures the linear correlation between features, or between a feature and the target.

• Predictive Power Score (PPS / ppscore) — measures a feature's ability to predict the target; capable of detecting non-linear relationships (unlike ordinary correlation).

iii. Feature Engineering (Creating New Features)

• Combining two or more features into one new feature.

• Creating ratios, differences, or specific transformations from existing features.

• Extracting new information from time data (date, hour, day, month, etc.).

iv. Kernelization (Expanding Dimensions)

• Concept: mapping data into a higher-dimensional space so that data which is originally not linearly separable becomes separable. This can't be done with linear methods — it requires a kernel function.

• Kernel functions: Linear Kernel, Polynomial Kernel, Radial Basis Function (RBF) Kernel, Sigmoid Kernel.

• Goal: make data linearly separable so classification algorithms (Logistic Regression, SVM) can build a better decision boundary.

Kernelization vs TF-IDF — often mixed up, here's the difference

Kernelization is used in ML (e.g., SVM) to map data into higher dimensions, while TF-IDF is used to represent text as a weighted vector. Two different techniques with different purposes, even though both "change the representation of data."

1.4 Normalization / Scaling

Definition: the process of rescaling data to a specific range (usually 0–1 or -1 to 1) so that every feature contributes to the model in a balanced way.

Why it's needed:

• AI models are sensitive to data scale.

• Features with large values can dominate.

• Speeds up convergence during training.

• Improves model accuracy.

from sklearn.preprocessing import MinMaxScaler, StandardScaler

scaler = MinMaxScaler()
df['price_scaled'] = scaler.fit_transform(df[['price']])

scaler = StandardScaler()
df['price_std'] = scaler.fit_transform(df[['price']])

• When normalization is needed: especially for algorithms sensitive to scale, such as K-Nearest Neighbors (KNN), Neural Networks, and distance-based algorithms.

• Benefits for AI: training speed (the model converges faster), model performance (Neural Networks/KNN/SVM are very scale-sensitive), and fair interpretation (all features contribute in a balanced way without one dominating).

⚠️ Don't get the order wrong — see the Best Practices section below: normalization must be done after the train-test split, not before!

1.5 Text Vectorization for NLP

Computers can't directly understand words/sentences, so text needs to be turned into a numeric representation (vectors). Before being vectorized, text is usually cleaned first by removing links/URLs, tags (mentions/hashtags), and stopwords (the, a, an, or, for, etc.) — see Part 1's NLP-Specific Data Cleaning section — because these elements carry no useful information and only add computational load.

a) One-Hot Encoding

• How it works: every word is turned into a binary vector; each word has its own position (index), with only one value being 1 and the rest 0.

• Advantages: simple, easy to implement, good for a small vocabulary.

• Disadvantages: the vector dimension becomes very large as the number of words grows; doesn't capture relationships/meaning between words.

Corpus = the collection of text data that forms the basis for building a vocabulary. In modern NLP, a word is often called a token.

Example corpus (5 words): aku (I), makan (eat), nasi (rice), pakai (with), tempe (tempeh)

The sentence "aku makan nasi" (I eat rice) → the combined vector of aku + makan + nasi. Each word is treated as standing on its own, without regard to the meaning relationships between words.

b) Bag of Words (BoW)

Represents a document based on how many times each word appears, according to the vocabulary built from all documents.

Example — D1: rumah ini bagus (this house is nice) | D2: rumah saya makan nasi (my house I eat rice) | D3: saya makan nasi (I eat rice)

Only the count of each word matters; word order within a sentence is ignored. Simple, but doesn't yet understand the sentence's context/meaning.

c) TF-IDF (Term Frequency – Inverse Document Frequency)

• Formula: TF-IDF = TF × IDF

• TF (Term Frequency): measures how often a word appears in a single document — the more often it appears, the higher its TF value.

• IDF (Inverse Document Frequency): measures how unique/rare a word is across the entire document corpus. Common words (and, the, of) → low IDF because they're less informative; rare words → high IDF because they better distinguish document content.

🤔 Guess First: If you don't use IDF at all (only TF), what happens to common words like "the" or "of" that appear in ALMOST EVERY document?

See the Answer

Without IDF, words that appear frequently across many documents will always get a high weight even though they're not informative, making it hard for the model to distinguish the content between documents. This is exactly why IDF is needed — to "penalize" overly common words.

• Calculation process: must use every document in the dataset (the IDF value depends on the entire corpus), then TF and IDF are multiplied for each word in each document.

• Final result: a TF-IDF matrix sized (Number of Documents × Number of Unique Words in the Corpus). Example: 3 documents × 6 unique words → a 3×6 matrix.

• Uses: document classification, information retrieval, search engines, text mining.

d) Word2Vec

• Concept: represents words using dense vectors (embeddings) learned by a neural network; capable of capturing semantic relationships between words (words with similar meaning → vectors close together).

🤔 Guess First: Complete this famous word-vector equation: King − Man + Woman ≈ ?

See the Answer

Queen. This is the classic example showing that Word2Vec captures semantic relationships between words — not just word occurrence like BoW/TF-IDF.

• How it works: the model is trained to predict a missing word (masked word) or the surrounding words based on context — it's not generative, but rather word classification based on the context before & after.

• Hidden Dimension: the representation space (embedding space) that stores each word's features. Example: Hidden Dimension = 512 → each word is represented as a 512-feature vector.

• Visualization: embedding results can be visualized with PCA, t-SNE, or UMAP — words with similar meaning will form clusters (e.g., "King" and "Queen" ending up close together).

1.6 Images as Numeric Data

With image data, the computer essentially already receives the data in numeric form.

a) Representing an Image as a Matrix

• Images are stored as binary data, then represented as a pixel matrix.

• Grayscale: a single intensity value (0–255).

• RGB: three values — Red, Green, Blue.

b) CNN as a Feature Extractor

• How it works: a Convolutional Neural Network (CNN) automatically learns important image characteristics: edges, lines, textures, object shapes, patterns, and object parts.

• The deeper the CNN layer: the more complex the features it learns — early layers recognize lines/edges, deeper layers recognize complex objects (faces, vehicles, animals).

• Computer Vision tasks: Image Classification, Object Detection, Image Segmentation, Face Recognition, Image Retrieval.

2. Saving a Clean Dataset

Once all the cleaning and transformation steps are done, it's time to save the clean dataset — think of it as the "prepared ingredients" ready to be handed to the AI.

df_clean.to_csv('retail_data_clean.csv', index=False)
df_clean.to_excel('retail_data_clean.xlsx', index=False)
df_clean.to_pickle('retail_data_clean.pkl')  # format Python

Note: always save the original dataset and the cleaned dataset under different file names — never overwrite the original data.

3. Data Versioning & Management Strategies

A. Data Versioning

• Concept: one of the important concepts in MLOps.

• Goal: to support the development lifecycle by providing a checkpoint at every stage, so every change can be tracked.

• Rollback benefit: if an error occurs or an experiment's results don't meet expectations, a developer can roll back to a previous version without redoing the entire process from scratch.

• End result: the development process becomes more structured, well-documented, and easy to trace.

Requirements for a Versioning System:

• Able to label every version.

• Able to create a checkpoint / freeze progress at every development stage.

• Allows a developer to roll back to a specific version at any time.

B. Version Numbering Scheme

• vX.X.X (e.g., v1.0.0): consists of Major Version (fundamental/total changes), Minor Version (fairly large changes that don't change the whole system), and Patch/Revision or Micro Changing (small changes that don't affect the core structure).

• vX.X.X.X: adds one more Patch component, specifically for bug fixing or minor repairs.

C. Version Change Categories

🤔 Guess First: You just fixed 5 mislabeled data points in your dataset. Which version category does that fall under — Major, Minor, Micro, or Patch?

See the Answer

Patch (Bug Fix) — fixing labeling errors is a classic Patch example, since it doesn't change the system's main behavior, it just corrects a small mistake.

D. GitLab as a Data Versioning Tool

E. DataOps and MLOps Tools

4. Best Practices, Common Mistakes & Error Handling

A. Do & Don't

B. Trust But Verify

Principle: "Trust but verify" — always double-check the result of every cleaning step.

C. Error Handling

Note: an error is a learning moment, not a failure.

5. Before vs After & Data Pipeline Flow

A. Before vs After: From Chaos to Clarity

• Impact of a raw dataset: the AI gets confused, errors occur during training, and predictions are inaccurate.

• Impact of a clean dataset: the AI can learn well, training runs smoothly, and predictions are accurate.

B. Data Pipeline Flow (End-to-End)

1. Data Collection — gathering data from various sources (database, API, files).

2. Data Loading — reading data into the Python environment.

3. Data Exploration — understanding the structure, distribution, and issues in the data.

4. Data Cleaning — handling missing values, duplicates, and errors.

5. Data Transformation — sorting, filtering, normalization, feature engineering.

6. Data Storage — saving the clean dataset for the next stage (model training).

6. Quiz Check — Part 2

Q1. You have a dataset with extreme outliers in the salary column. Which scaling method is the most resistant to outliers: Min-Max, Standardization, or Robust Scaling?

Answer: Robust Scaling. This method uses the median and IQR (instead of mean/min/max), making it far more resistant to outliers than either Min-Max Scaling or Standardization.

Q2. Why MUST normalization be done after the train-test split, rather than before?

Answer: Because the scaler (e.g., MinMaxScaler) must be fit only on the training data. If it's fit before the split (on the entire dataset including the test set), information from the test set "leaks" into the training process — this is called data leakage, and it makes the model evaluation dishonest.

Q3. Your code throws a `KeyError` when trying to access `df['price']`. What's the most likely cause and solution?

Answer: The most likely cause is a wrong or nonexistent column name — maybe the actual column name is Price or harga (different capitalization). Solution: check the exact column name with df.columns.

Q4. You just added one new object class to an object detection dataset, plus thousands of new images. What kind of version change is this?

Answer: Minor Version. Adding a new class and a large change to the dataset both fall under the Minor Version category — a fairly large change, but one that doesn't yet change the whole system/ML task.

7. Preprocessing Checklist Cheat Sheet

Keep this checklist as a quick reference before your data goes into the training stage:

Exploration

• [ ] Have you run df.head(), df.info(), df.describe(), df.shape?

• [ ] Do you know the data type of every column (df.dtypes)?

Cleaning

• [ ] Have missing values been checked (df.isnull().sum()) and handled based on their percentage?

• [ ] Have duplicates been checked (df.duplicated().sum()) and removed?

• [ ] Have outliers been detected (Boxplot/IQR/Z-Score) and a decision made: remove or treat?

• [ ] Have data types been converted correctly (astype, to_numeric, to_datetime)?

• [ ] Has text been tidied up (strip, lowercase, replace) and made format-consistent?

• [ ] Are column names snake_case and consistent?

• [ ] Has the index been reset after dropping rows (reset_index(drop=True))?

Transformation

• [ ] Has categorical data been encoded according to its type (nominal → One-Hot, ordinal → direct numeric)?

• [ ] Has dimensionality reduction been considered if there are too many features?

• [ ] Is the scaler fit only on the training data, after the train-test split?

• [ ] Is the target variable (y) not normalized for regression cases?

Saving & Versioning

• [ ] Is the original dataset saved separately from the clean dataset (different file names)?

• [ ] Has the clean dataset been given a version (Major/Minor/Micro/Patch) matching the type of change?

• [ ] Is the dataset stored somewhere that supports versioning (S3/GCS/DVC/etc.)?

See Part 1 for the full discussion on Data Collection, Storage, and Data Cleaning.

Part 1 of 2: why preprocessing matters, how data is collected & stored, an introduction to Pandas, data exploration, and a deep dive into the cleaning process.

Before an AI model can learn anything, it has to "eat" data first. The problem is, raw data is almost always messy — some of it is missing, some is duplicated, some is inconsistently formatted. Part 1 of this note covers the foundations of data handling through to the full cleaning process. Part 2 will continue with transformation, versioning, and best practices.

Table of Contents

• 1. Why Does Data Preprocessing Matter?

• 2. Data Collection & Storage

• 3. Pandas: The Main Data Handling Tool

• 4. Exploring Data Before Cleaning

• 5. Data Cleaning (Deep Dive)

• 6. Quiz Check — Part 1

1. Why Does Data Preprocessing Matter?

Even the most sophisticated AI model won't produce accurate predictions if it's trained on bad data. Preprocessing isn't just a technical step — it's the foundation of an AI project's success.

Golden rule: Garbage In, Garbage Out (GIGO) — if the data is dirty, the AI's output will keep being wrong.

🤔 Guess First: In your opinion, what percentage of a data science project's total time is spent on data preprocessing versus building the model?

See the Answer

80% of the time in a project is spent preparing & cleaning data, with only 20% left for building & training the model. This is the AI success formula that beginners often underestimate.

Note: the figures above (including the "80% unstructured data" statistic in the next section) are commonly-cited figures in data science material, not from a single academic study that can be directly referenced. Read them as an illustration of scale/urgency, not a precise statistic.

Real Examples: When Bad Data Breaks AI

Case 1 — A Failed Chatbot

A customer service chatbot was trained on conversation data full of typos & non-standard language → the bot couldn't understand customer questions, and its answers became irrelevant.

Case 2 — A Wrong Price Prediction

A property price prediction model was trained on data with a lot of missing values & duplicates → a house worth Rp 500 million was predicted at Rp 2 billion.

Key Definitions

• Data Preprocessing: the process of preparing data as well and as thoroughly as possible so it's ready to be used by AI.

• Data Cleaning: a part of Data Preprocessing responsible for cleaning data (missing values, duplicates, inconsistent formats, wrong data types, etc.).

The Data Preprocessing Workflow

1. Raw Data — a dirty dataset with various problems and inconsistencies.

2. Cleaning — removing nulls, fixing formats, handling outliers.

3. Transformation — normalization, encoding, feature engineering.

4. Model Ready — data ready to be used for training AI.

2. Data Collection & Storage

A. Data Collection

Data Collection is the scheme for gathering data for the purpose of building AI. There are two broad types:

1. Unstructured Data

• Characteristics: humans can understand the meaning of the data (e.g., recognizing an animal species) and have reasoning ability over that data (e.g., understanding the point of a paragraph).

• Data sources: generated data is allowed, but real-world data that matches the problem is preferred — make sure the data conditions match real-world conditions.

• Examples: text, images, video.

• Fact: around 80% of the world's data is unstructured data.

2. Structured Data

• Characteristics: ordinary people don't have the reasoning ability to interpret the data's meaning; it's unique to each case — similar cases can use different data (e.g., inventory optimization differs depending on the business objective).

• Examples: tabular data, CSV, columns and rows.

• Important note: if an end-to-end system isn't available yet, don't jump straight into using AI.

  • Build a Data Pipeline first so an end-to-end data flow is established (involve a Data Engineer if needed).

  • AI should be used as tooling, not as the primary weapon.

B. Data Storage

3. Pandas: The Main Data Handling Tool

Pandas is one of the most popular Python libraries for data manipulation and analysis — think of it as Excel, but with superpowers and coding flexibility.

• Fast — processes millions of rows in seconds.

• Powerful — complex operations with short code.

• Flexible — can handle various data formats (CSV, Excel, JSON, etc.).

Other Data Tool Ecosystems

The data world generally relies on just Python and SQL. Besides Pandas, Python has a few other libraries:

• PySpark — used when data is stored in Apache Spark.

• Polars — rewritten in Rust so it's faster.

• DuckDB — for online analytical processing (OLAP) databases.

Note: if your data is still in CSV format, Pandas alone is enough.

File Formats Supported by Pandas

Loading a Dataset

pd is an alias for Pandas, making the code shorter to write. Pandas can read almost every data format commonly used in data science.

import pandas as pd

df = pd.read_csv('retail_data_raw.csv')
print(f"Jumlah baris: {len(df)}")

Practical tips:

• Always check the encoding if you see odd characters.

• Use the sep parameter for a custom delimiter.

• The nrows parameter for previewing large data.

• Pay attention to the delimiter and decimal separator.

Note: the variable df stands for "DataFrame", Pandas' main data structure shaped like a table with rows & columns.

Data Structures: Series vs DataFrame

• Series: a 1-dimensional array with an index. Example: pd.Series([1, 2, 3, 4])

• DataFrame: a 2-dimensional table with rows and columns — the most commonly used structure.

4. Exploring Data Before Cleaning

The principle of "Understand before fix": an AI engineer must understand the "face" of their data — column structure, value types, and general patterns — before starting to clean it.

5. Data Cleaning (Deep Dive)

5.1 Common Problems in a Dataset

5.2 Data Types & Conversion

One classic data cleaning problem is the wrong data type — for example, numbers stored as text, or dates stored as strings. Data types need to be transformed without changing their original meaning (numeric data is ratio/interval in nature, handled differently from categorical data).

Note: if a price column is stored as object (text), you can't calculate its average or sum — and the AI will get confused too.

# Konversi ke numerik
df['quantity'] = df['quantity'].astype(int)
df['price'] = pd.to_numeric(df['price'], errors='coerce')

# Konversi ke datetime
df['date'] = pd.to_datetime(df['date'], format='%d/%m/%Y')

# Konversi ke string
df['customer_id'] = df['customer_id'].astype(str)

Note: the errors='coerce' parameter turns values that fail to convert into NaN, so the code doesn't throw an error.

5.3 Cleaning Text Data (Tabular)

Common string operations for tidying up text columns:

df['nama'] = df['nama'].str.strip()              # hapus whitespace
df['kota'] = df['kota'].str.lower()               # lowercase semua
df['telp'] = df['telp'].str.replace('-', '')      # replace karakter
df['nomor'] = df['text'].str.extract(r'(\d+)')    # extract angka saja

Text cleaning tips:

• Always standardize case (upper/lower).

• Strip leading & trailing whitespace.

• Remove special characters if unnecessary.

• Check for typos with value_counts().

• Stay consistent with format ("Jakarta" vs "jakarta").

Note: this is text cleaning for table columns (general string cleanup). It's different from preparing text for an NLP model (text vectorization) below, which requires extra steps.

5.4 NLP-Specific Data Cleaning

Before text is vectorized for an NLP model, there are words considered to carry little useful information that need to be removed:

• Links or URLs

• Tags (mentions or hashtags)

• Stopwords, such as: the, a, an, or, for, and other common words

Why: links/tags aren't relevant to model learning; stopwords appear repeatedly, add computational load, and don't help distinguish one document from another.

5.5 Missing Values

Missing values are one of the most common problems in real-world datasets — they can happen due to input errors, broken sensors, or the data simply not existing.

Why they're dangerous:

• AI can't learn from empty data.

• Can cause errors during model training.

• Reduces prediction accuracy.

Detection:

• df.isnull().sum() → number of nulls per column.

• df.isnull().mean() * 100 → percentage of missing values.

Strategies for handling missing values:

1. Drop rows (df.dropna()) — if missing values are few (<5%) and there's still plenty of data.

2. Drop a column (df.dropna(axis=1)) — if a column has too many missing values (>50%).

3. Fill with a specific value (df.fillna(...)) — use 0, mean, median, or mode depending on context; used when a lot of data is missing but the column is important.

4. Forward fill / backward fill — for time series data, fill with the value before/after it (method='ffill').

# 1. Hapus baris dengan null
df_clean = df.dropna()

# 2. Hapus kolom dengan >50% null
threshold = len(df) * 0.5
df_clean = df.dropna(axis=1, thresh=threshold)

# 3. Isi dengan nilai tertentu / mean
df['age'].fillna(0, inplace=True)
df['price'].fillna(df['price'].mean(), inplace=True)

# 4. Forward fill (time series)
df.fillna(method='ffill', inplace=True)

Principle: "No blanks for brains" — AI needs complete data to learn well.

5.6 Duplicate Data

Duplicate data happens when one observation appears more than once in a dataset — usually caused by input errors or careless dataset merging.

Dangers of duplicates: the AI model can overfit to certain data, bias in predictions, longer training time, and inaccurate evaluation metrics.

# Deteksi duplikat
df.duplicated().sum()
df[df.duplicated()]

# Hapus duplikat
df_clean = df.drop_duplicates()
df_clean = df.drop_duplicates(subset=['customer_id', 'date'])
df_clean = df.drop_duplicates(keep='first')  # atau 'last'

Best practice: always keep the original dataset separate from the cleaned dataset (different file names) — never overwrite the original data.

Principle: "One truth per row" — every row must represent one unique observation (unique and clean).

5.7 Outliers

Definition: data that is anomalous / whose value is far different from most of the other data. Example: employee salaries are generally 5–15 million, but there's one at 500 million — that's an outlier.

Simple detection: Boxplot, scatter plot, the IQR (Interquartile Range) method, the Z-Score method, or domain knowledge.

When to remove an outlier: when its likelihood of occurring is very small, or when its presence has no impact on the business process. If an outlier represents a real business condition, don't remove it right away — it needs treatment instead.

Advanced Technique: Outlier Detection with a Variational Autoencoder (VAE)

What is an Autoencoder?

A neural network trained with the input and output being the same, so it learns to reconstruct data and understand the distribution of clean (normal) data.

How it works for outlier detection:

• A VAE compares the reconstruction result against the original input.

• The bigger the difference (measured with Mean Squared Error / MSE), the more likely the data point is an outlier.

• Data outside the learned distribution → the reconstruction result differs greatly → high MSE value → an indicator of outlier data.

Other methods: Z-Score, Interquartile Range (IQR), Isolation Forest.

Hands-on Implementation — Detecting & Removing Outliers with IQR:

Q1 = df['price'].quantile(0.25)
Q3 = df['price'].quantile(0.75)
IQR = Q3 - Q1

lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR

df_clean = df[(df['price'] >= lower_bound) & (df['price'] <= upper_bound)]

Note: the IQR method is one of the most commonly used ways to detect outliers — but it still needs to be validated against business context (see "When to remove an outlier" above). The number 1.5 is a standard constant, but it can be adjusted as needed.

5.8 Data That Confuses the Model

Beyond numeric outliers, there's a more subtle category of "dirty data": data that actively confuses the model's learning process.

• The goal of cleaning here: removing data that rarely occurs and doesn't represent actual conditions.

• Adversarial attack: data that has been deliberately modified so the AI model produces a wrong prediction.

• Important note: removing a small part of an object within an image is still acceptable if that object is actually lowering the quality of the training data — the goal is to keep the model learning from the correct patterns.

🤔 Guess First: If there's one small, irrelevant object in the corner of a training photo that's actually causing the model to learn incorrectly, is it okay to remove it from the image?

See the Answer

Yes, it's okay. As long as the goal is to keep the model learning from correct patterns, and the object really is degrading the quality of the training data, removing a small part of an object is still acceptable.

5.9 Renaming Columns

Why it matters: column names are often inconsistent, spaces & special characters cause hassle, descriptive names are easier to understand, and it standardizes the naming convention.

Best practice: use snake_case (lowercase letters, underscores for spaces), clear and short names, avoid special characters.

# Rename kolom tertentu
df.rename(columns={'Nama Lengkap': 'nama', 'Umur (tahun)': 'usia'}, inplace=True)

# Rename semua kolom sekaligus
df.columns = ['nama', 'usia', 'gaji']

# Lowercase & hilangkan spasi
df.columns = df.columns.str.lower()
df.columns = df.columns.str.replace(' ', '_')

6. Quiz Check — Part 1

Q1. Why should a categorical data type like "hair color" be encoded using One-Hot Encoding rather than converted into simple ordered numbers (0, 1, 2, ...)?

Answer: Because "hair color" is nominal data — it has no order/ranking. If it's turned into simple ordered numbers, the model might mistakenly assume there's a ranking relationship between categories (e.g., the number 2 being seen as "bigger" than 1), even though no such order exists. (Full details are covered in Part 2.)

Q2. When should you drop an ENTIRE COLUMN instead of just the rows that have missing values?

Answer: When that column has more than 50% missing values. If the missing values are few (under 5%) and there's still plenty of data, it's enough to just drop the rows with df.dropna().

Q3. An outlier salary of Rp 500 million is found in an HR dataset. Should it automatically be removed?

Answer: Not automatically. You need to check first whether that value represents a real business condition (e.g., it really is a director's salary) or is purely an input error. If it represents a real business condition, don't remove it right away — it needs treatment first.

Q4. What's the difference between "cleaning text in a table column" and "cleaning text for NLP"?

Answer: Cleaning text in a table column focuses on general string cleanup (stripping whitespace, lowercasing, replacing characters). Cleaning text for NLP requires additional, more specific steps: removing links/URLs, tags (mentions/hashtags), and stopwords — because these elements aren't informative and add extra computational load to the language model.

Continue to Part 2 — we'll cover the full Data Transformation process (encoding, dimensionality reduction, normalization, text vectorization, images as numeric data), how to save a clean dataset, Data Versioning & Management Strategies, Best Practices & Error Handling, up to the big-picture end-to-end Data Pipeline Flow.

Bagian 2 dari 2: lanjutan dari Part 1 yang membahas Data Cleaning. Di sini kita masuk ke Data Transformation, cara menyimpan dataset bersih, Data Versioning ala MLOps, Best Practices, sampai gambaran besar Data Pipeline Flow.

Sekilas recap: di Part 1 kita sudah tahu kenapa preprocessing itu penting (formula 80/20), bagaimana data dikumpulkan & disimpan, dasar-dasar Pandas, dan cara membersihkan data — mulai dari missing values, duplikat, outlier, sampai adversarial data. Sekarang saatnya mengubah data yang sudah bersih itu menjadi format yang benar-benar bisa "dimakan" oleh algoritma Machine Learning.

Daftar Isi

• 1. Data Transformation

• 2. Menyimpan Dataset Bersih

• 3. Data Versioning & Management Strategies

• 4. Best Practices, Kesalahan Umum & Error Handling

• 5. Before vs After & Data Pipeline Flow

• 6. Quiz Check — Part 2

• 7. Cheat Sheet Checklist Preprocessing

1. Data Transformation

Sebagian besar algoritma ML hanya menerima data numerik, sehingga data non-numerik perlu diubah menjadi representasi angka (vectorization / encoding). Ini inti dari data transformation.

1.1 Sorting & Filtering

• Sorting — prinsip "Order before logic": data yang terurut lebih mudah dianalisis dan diproses oleh model AI.

• Filtering — memungkinkan kita fokus pada subset data tertentu, misalnya hanya transaksi dari Asia, atau hanya produk dengan harga di atas Rp 100.000.

# Sorting
df_sorted = df.sort_values('price')                    # ascending
df_sorted = df.sort_values('price', ascending=False)    # descending

# Filtering
df_asia = df[df['region'] == 'Asia']
df_filtered = df[(df['region'] == 'Asia') & (df['price'] > 50000)]              # AND
df_filtered = df[(df['region'] == 'Asia') | (df['region'] == 'Europe')]         # OR
df_filtered = df[df['region'].isin(['Asia', 'Europe', 'Africa'])]               # isin()

1.2 Encoding Data Kategorikal

🤔 Coba Tebak Dulu: Ukuran baju (S, M, L) dan warna rambut, dua-duanya kategori. Menurutmu, apa keduanya harus di-encode dengan cara yang sama?

Lihat Jawaban

Tidak. Ukuran baju punya urutan tingkatan (S < M < L) sehingga termasuk ordinal — bisa langsung dikonversi ke angka. Warna rambut tidak punya urutan sehingga termasuk nominal — sebaiknya pakai One-Hot Encoding.

• Nominal: kategori tanpa urutan/tingkatan. Contoh: warna rambut, ras, jenis kendaraan, jenis kelamin → sebaiknya pakai One-Hot Encoding agar model tidak menganggap ada urutan.

• Ordinal: kategori dengan urutan/tingkatan yang bermakna. Contoh: ukuran baju (S, M, L), tingkat pendidikan, tingkat kepuasan → bisa dikonversi langsung ke nilai numerik.

1.3 Dimensionality Reduction (Mengubah Dimensi Data)

Tujuan: mengurangi jumlah fitur tanpa menghilangkan informasi penting, untuk mengatasi Curse of Dimensionality (fitur terlalu banyak → training lambat, kompleks, performa turun).

Number of Dimensionality = banyaknya fitur/variabel dalam dataset.

i. Teknik Transformasi Data

• PCA (Principal Component Analysis) — mencari arah dengan variasi data terbesar (principal components); banyak dipakai pada data numerik untuk mengompresi fitur yang saling berkorelasi.

• t-SNE — mempertahankan kedekatan antar data (local structure); baik untuk visualisasi klaster pada data berdimensi tinggi.

• UMAP — tujuan mirip t-SNE tapi lebih cepat & skalabel untuk dataset besar; juga bisa dipakai untuk feature engineering.

ii. Menghapus Fitur yang Kurang Relevan

• .corr() (Pandas) — mengukur korelasi linear antar fitur maupun fitur dengan target.

• Predictive Power Score (PPS / ppscore) — mengukur kemampuan fitur memprediksi target; mampu mendeteksi hubungan non-linear (berbeda dari korelasi biasa).

iii. Feature Engineering (Membuat Fitur Baru)

• Menggabungkan dua atau lebih fitur menjadi satu fitur baru.

• Membuat rasio, selisih, atau transformasi tertentu dari fitur yang ada.

• Mengekstraksi informasi baru dari data waktu (tanggal, jam, hari, bulan, dst).

iv. Kernelization (Membesarkan Dimensi)

• Konsep: memetakan data ke ruang berdimensi lebih tinggi agar data yang awalnya tidak terpisah secara linear menjadi dapat dipisahkan. Tidak bisa dilakukan dengan metode linear, melainkan pakai kernel function.

• Kernel function: Linear Kernel, Polynomial Kernel, Radial Basis Function (RBF) Kernel, Sigmoid Kernel.

• Tujuan: membuat data linearly separable agar algoritma klasifikasi (Logistic Regression, SVM) bisa membangun decision boundary yang lebih baik.

Kernelization vs TF-IDF — sering ketuker, ini bedanya

Kernelization dipakai di ML (misalnya SVM) untuk memetakan dimensi lebih tinggi, sedangkan TF-IDF dipakai untuk merepresentasikan teks menjadi vektor berbobot. Dua teknik yang beda tujuan meskipun sama-sama "mengubah representasi data".

1.4 Normalisasi / Scaling

Definisi: proses mengubah skala data ke range tertentu (biasanya 0–1 atau -1 sampai 1) supaya semua fitur punya kontribusi yang seimbang ke model.

Kenapa perlu:

• Model AI sensitif terhadap skala data.

• Fitur dengan nilai besar bisa mendominasi.

• Mempercepat convergence saat training.

• Meningkatkan akurasi model.

from sklearn.preprocessing import MinMaxScaler, StandardScaler

scaler = MinMaxScaler()
df['price_scaled'] = scaler.fit_transform(df[['price']])

scaler = StandardScaler()
df['price_std'] = scaler.fit_transform(df[['price']])

• Kapan perlu normalisasi: terutama untuk algoritma yang sensitif terhadap skala, seperti K-Nearest Neighbors (KNN), Neural Networks, dan algoritma berbasis jarak.

• Manfaat untuk AI: kecepatan training (model converge lebih cepat), performa model (Neural Networks/KNN/SVM sangat sensitif skala), dan interpretasi yang adil (semua fitur berkontribusi seimbang tanpa didominasi satu fitur).

⚠️ Jangan sampai salah urutan — lihat bagian Best Practices di bawah: normalisasi harus dilakukan setelah train-test split, bukan sebelumnya!

1.5 Text Vectorization untuk NLP

Komputer tidak dapat langsung memahami kata/kalimat, sehingga teks perlu diubah menjadi representasi numerik (vektor). Sebelum divektorisasi, teks biasanya dibersihkan dulu dengan menghapus link/URL, tag (mention/hashtag), dan stopwords (the, a, an, or, for, dll) — lihat Part 1 bagian Data Cleaning Khusus NLP — karena elemen-elemen ini tidak memberi informasi penting dan hanya memperberat komputasi.

a) One-Hot Encoding

• Cara kerja: setiap kata diubah menjadi vektor biner; setiap kata punya posisi (indeks) sendiri, hanya satu nilai bernilai 1, sisanya 0.

• Kelebihan: sederhana, mudah diimplementasikan, cocok untuk vocabulary kecil.

• Kekurangan: dimensi vektor sangat besar jika jumlah kata banyak; tidak menangkap hubungan/makna antar kata.

Korpus = kumpulan data teks yang jadi dasar pembentukan vocabulary. Dalam NLP modern, kata sering disebut token.

Contoh korpus (5 kata): aku, makan, nasi, pakai, tempe

Kalimat "aku makan nasi" → gabungan vektor aku + makan + nasi. Setiap kata dianggap berdiri sendiri, tidak memperhatikan hubungan makna antar kata.

b) Bag of Words (BoW)

Merepresentasikan dokumen berdasarkan jumlah kemunculan tiap kata sesuai korpus yang terbentuk dari seluruh dokumen.

Contoh — D1: rumah ini bagus | D2: rumah saya makan nasi | D3: saya makan nasi

Yang diperhatikan hanya jumlah kemunculan kata; urutan kata dalam kalimat diabaikan. Sederhana, tetapi belum memahami konteks/makna kalimat.

c) TF-IDF (Term Frequency – Inverse Document Frequency)

• Rumus: TF-IDF = TF × IDF

• TF (Term Frequency): mengukur seberapa sering sebuah kata muncul dalam satu dokumen — makin sering muncul, makin tinggi nilai TF-nya.

• IDF (Inverse Document Frequency): mengukur seberapa unik/jarang suatu kata muncul di seluruh dokumen korpus. Kata umum (dan, yang, di) → IDF rendah karena kurang informatif; kata jarang muncul → IDF tinggi karena lebih membedakan isi dokumen.

🤔 Coba Tebak Dulu: Kalau tidak pakai IDF sama sekali (cuma pakai TF), apa yang terjadi pada kata umum seperti "yang" atau "di" yang sering muncul di HAMPIR SEMUA dokumen?

Lihat Jawaban

Tanpa IDF, kata yang sering muncul di banyak dokumen akan selalu berbobot tinggi meski tidak informatif, sehingga model jadi sulit membedakan isi antar dokumen. Inilah alasan IDF dibutuhkan — untuk "menghukum" kata yang terlalu umum.

• Proses perhitungan: harus memakai seluruh dokumen dalam dataset (nilai IDF bergantung pada seluruh korpus), lalu TF dan IDF dikalikan untuk tiap kata di tiap dokumen.

• Hasil akhir: matriks TF-IDF berukuran (Jumlah Dokumen × Jumlah Kata dalam Korpus). Contoh: 3 dokumen × 6 kata unik → matriks 3×6.

• Kegunaan: klasifikasi dokumen, information retrieval, search engine, text mining.

d) Word2Vec

• Konsep: representasi kata menggunakan dense vector (embedding) yang dipelajari oleh neural network; mampu menangkap hubungan semantik antar kata (kata bermakna mirip → vektor berdekatan).

🤔 Coba Tebak Dulu: Lengkapi rumus vektor kata terkenal ini: King − Man + Woman ≈ ?

Lihat Jawaban

Queen. Ini contoh klasik yang menunjukkan Word2Vec menangkap hubungan semantik antar kata — bukan cuma kemunculan kata seperti BoW/TF-IDF.

• Cara kerja: model dilatih memprediksi kata yang hilang (masked word) atau kata di sekitarnya berdasarkan konteks — bukan generatif, melainkan klasifikasi kata berdasarkan konteks sebelum & sesudah.

• Hidden Dimension: ruang representasi (embedding space) penyimpan fitur tiap kata. Contoh: Hidden Dimension = 512 → tiap kata direpresentasikan sebagai vektor 512 fitur.

• Visualisasi: hasil embedding bisa divisualisasikan dengan PCA, t-SNE, atau UMAP — kata bermakna serupa akan membentuk cluster (contoh: "Raja" dan "Ratu" saling berdekatan).

1.6 Image sebagai Data Numerik

Pada data gambar, komputer pada dasarnya sudah menerima data dalam bentuk numerik.

a) Representasi Gambar sebagai Matriks

• Gambar disimpan sebagai data biner, lalu direpresentasikan sebagai matriks piksel.

• Grayscale: satu nilai intensitas (0–255).

• RGB: tiga nilai — Red, Green, Blue.

b) CNN sebagai Feature Extractor

• Cara kerja: Convolutional Neural Network (CNN) secara otomatis mempelajari karakteristik penting gambar: tepi (edges), garis, tekstur, bentuk objek, pola, dan bagian-bagian objek.

• Semakin dalam layer CNN: semakin kompleks fitur yang dipelajari — layer awal mengenali garis/tepi, layer dalam mengenali objek kompleks (wajah, kendaraan, hewan).

• Tugas Computer Vision: Image Classification, Object Detection, Image Segmentation, Face Recognition, Image Retrieval.

2. Menyimpan Dataset Bersih

Setelah semua tahap cleaning dan transformation selesai, saatnya menyimpan dataset bersih — ibarat "bahan masakan" yang siap diberikan ke AI.

df_clean.to_csv('retail_data_clean.csv', index=False)
df_clean.to_excel('retail_data_clean.xlsx', index=False)
df_clean.to_pickle('retail_data_clean.pkl')  # format Python

Catatan: selalu simpan dataset asli dan dataset yang sudah dibersihkan dengan nama file berbeda — jangan overwrite data original.

3. Data Versioning & Management Strategies

A. Data Versioning

• Konsep: salah satu konsep penting dalam MLOps.

• Tujuan: membantu development lifecycle dengan menyediakan checkpoint di tiap tahap, sehingga setiap perubahan dapat dilacak.

• Manfaat rollback: jika terjadi error atau hasil eksperimen tidak sesuai harapan, developer bisa rollback ke versi sebelumnya tanpa mengulang seluruh proses dari awal.

• Hasil akhir: proses pengembangan menjadi lebih terstruktur, terdokumentasi, dan mudah ditelusuri.

Syarat Sistem Versioning:

• Bisa memberi label pada setiap versi.

• Bisa membuat checkpoint / freeze progress pada setiap tahap pengembangan.

• Memungkinkan pengembang rollback ke versi tertentu kapan saja.

B. Skema Penomoran Versi

• vX.X.X (contoh: v1.0.0): terdiri dari Major Version (perubahan fundamental/total), Minor Version (perubahan besar tapi tidak mengubah keseluruhan sistem), dan Patch/Revision atau Micro Changing (perubahan kecil yang tidak memengaruhi struktur utama).

• vX.X.X.X: menambahkan satu komponen Patch lagi, khusus untuk bug fixing atau perbaikan kecil.

C. Kategori Perubahan Versi

🤔 Coba Tebak Dulu: Kamu baru saja memperbaiki 5 data yang salah label di dataset. Itu masuk kategori versi apa — Major, Minor, Micro, atau Patch?

Lihat Jawaban

Patch (Bug Fix) — memperbaiki kesalahan labeling termasuk contoh Patch, karena tidak mengubah perilaku utama sistem, hanya membenarkan kesalahan kecil.

D. GitLab sebagai Data Versioning

E. Tools untuk DataOps dan MLOps

4. Best Practices, Kesalahan Umum & Error Handling

A. Do & Don't

B. Trust But Verify

Prinsip: "Trust but verify" — selalu double-check hasil setiap langkah cleaning.

C. Error Handling

Catatan: error adalah momen belajar, bukan kegagalan.

5. Before vs After & Data Pipeline Flow

A. Before vs After: From Chaos to Clarity

• Dampak dataset mentah: AI akan bingung, error saat training, dan prediksi tidak akurat.

• Dampak dataset bersih: AI bisa belajar dengan baik, training lancar, dan prediksi akurat.

B. Data Pipeline Flow (End-to-End)

1. Data Collection — mengumpulkan data dari berbagai sumber (database, API, file).

2. Data Loading — membaca data ke dalam environment Python.

3. Data Exploration — memahami struktur, distribusi, dan masalah dalam data.

4. Data Cleaning — menangani missing values, duplikat, dan error.

5. Data Transformation — sorting, filtering, normalisasi, feature engineering.

6. Data Storage — menyimpan dataset bersih untuk tahap selanjutnya (training model).

6. Quiz Check — Part 2

Q1. Kamu punya dataset dengan outlier ekstrem di kolom gaji. Metode scaling mana yang paling tahan terhadap outlier: Min-Max, Standardization, atau Robust Scaling?

Jawaban: Robust Scaling. Metode ini memakai median dan IQR (bukan mean/min/max), sehingga jauh lebih tahan terhadap outlier dibanding Min-Max Scaling maupun Standardization.

Q2. Kenapa normalisasi HARUS dilakukan setelah train-test split, bukan sebelumnya?

Jawaban: Karena scaler (misalnya MinMaxScaler) harus di-fit hanya pada training data. Kalau di-fit sebelum split (pada seluruh data termasuk test set), informasi dari test set "bocor" ke proses training — ini disebut data leakage dan membuat evaluasi model jadi tidak jujur.

Q3. Kode kamu error dengan pesan `KeyError` saat mencoba akses `df['harga']`. Apa penyebab paling mungkin dan solusinya?

Jawaban: Penyebab paling mungkin adalah nama kolom salah atau tidak ada — mungkin sebenarnya nama kolomnya price atau Harga (kapital berbeda). Solusi: cek nama kolom persis dengan df.columns.

Q4. Kamu baru saja menambahkan satu kelas objek baru ke dataset object detection, plus menambah ribuan gambar baru. Ini termasuk perubahan versi apa?

Jawaban: Minor Version. Penambahan kelas baru dan perubahan dataset dalam jumlah besar termasuk kategori Minor Version — perubahan cukup besar tapi belum mengubah keseluruhan sistem/task ML-nya.

7. Cheat Sheet Checklist Preprocessing

Simpan checklist ini sebagai referensi cepat sebelum data kamu masuk ke tahap training:

Eksplorasi

• [ ] Sudah jalankan df.head(), df.info(), df.describe(), df.shape?

• [ ] Sudah tahu tipe data tiap kolom (df.dtypes)?

Cleaning

• [ ] Missing values sudah dicek (df.isnull().sum()) dan ditangani sesuai persentasenya?

• [ ] Duplikat sudah dicek (df.duplicated().sum()) dan dihapus?

• [ ] Outlier sudah dideteksi (Boxplot/IQR/Z-Score) dan diputuskan: hapus atau treatment?

• [ ] Tipe data sudah dikonversi dengan benar (astype, to_numeric, to_datetime)?

• [ ] Teks sudah dirapikan (strip, lowercase, replace) dan konsisten formatnya?

• [ ] Nama kolom sudah snake_case dan konsisten?

• [ ] Index sudah di-reset setelah drop rows (reset_index(drop=True))?

Transformation

• [ ] Data kategorikal sudah di-encode sesuai jenisnya (nominal → One-Hot, ordinal → numerik langsung)?

• [ ] Dimensionality reduction dipertimbangkan jika fitur terlalu banyak?

• [ ] Scaler di-fit hanya pada training data, setelah train-test split?

• [ ] Target variable (y) tidak dinormalisasi untuk kasus regression?

Penyimpanan & Versioning

• [ ] Dataset asli disimpan terpisah dari dataset bersih (nama file berbeda)?

• [ ] Dataset bersih sudah diberi versi (Major/Minor/Micro/Patch) sesuai jenis perubahan?

• [ ] Dataset sudah disimpan di tempat yang mendukung versioning (S3/GCS/DVC/dll)?

Lihat Part 1 untuk pembahasan lengkap tentang Data Collection, Storage, dan Data Cleaning.

Bagian 1 dari 2: kenapa preprocessing penting, cara data dikumpulkan & disimpan, pengenalan Pandas, eksplorasi data, dan proses cleaning secara mendalam.

Sebelum model AI belajar apa pun, ia harus "makan" data dulu. Masalahnya, data mentah hampir selalu berantakan — ada yang kosong, ada yang duplikat, ada yang formatnya beda-beda. Bagian 1 catatan ini membahas fondasi data handling sampai tuntas proses cleaning-nya. Bagian 2 akan lanjut ke transformation, versioning, dan best practices.

Daftar Isi

• 1. Kenapa Data Preprocessing Itu Penting?

• 2. Data Collection & Storage

• 3. Pandas: Tools Utama Data Handling

• 4. Eksplorasi Data Sebelum Cleaning

• 5. Data Cleaning (Deep Dive)

• 6. Quiz Check — Part 1

1. Kenapa Data Preprocessing Itu Penting?

Model AI paling sophisticated sekalipun tidak akan menghasilkan prediksi akurat kalau dilatih dengan data yang buruk. Preprocessing bukan cuma langkah teknis — ini fondasi kesuksesan proyek AI.

Prinsip emas: Garbage In, Garbage Out (GIGO) — kalau datanya kotor, output AI-nya akan salah terus.

🤔 Coba Tebak Dulu: Menurutmu, dari total waktu sebuah proyek data science, berapa persen yang dihabiskan untuk preprocessing data dibanding membangun model?

Lihat Jawaban

80% waktu proyek dihabiskan untuk mempersiapkan & membersihkan data, hanya 20% untuk membangun & melatih model. Ini formula kesuksesan AI yang sering diremehkan pemula.