# Data Handling & Preprocessing (Part 1): Foundations, Collection, and Data Cleaning

> *Part 1 of 2: why preprocessing matters, how data is collected & stored, an introduction to Pandas, data exploration, and a deep dive into the cleaning process.*

Before an AI model can learn anything, it has to "eat" data first. The problem is, raw data is almost always messy — some of it is missing, some is duplicated, some is inconsistently formatted. Part 1 of this note covers the foundations of data handling through to the full cleaning process. Part 2 will continue with transformation, versioning, and best practices.

## Table of Contents

*   [1\. Why Does Data Preprocessing Matter?](#1-why-does-data-preprocessing-matter)
    
*   [2\. Data Collection & Storage](#2-data-collection--storage)
    
*   [3\. Pandas: The Main Data Handling Tool](#3-pandas-the-main-data-handling-tool)
    
*   [4\. Exploring Data Before Cleaning](#4-exploring-data-before-cleaning)
    
*   [5\. Data Cleaning (Deep Dive)](#5-data-cleaning-deep-dive)
    
*   [6\. Quiz Check — Part 1](#6-quiz-check--part-1)
    

* * *

## 1\. Why Does Data Preprocessing Matter?

Even the most sophisticated AI model won't produce accurate predictions if it's trained on bad data. Preprocessing isn't just a technical step — it's the foundation of an AI project's success.

**Golden rule:** *Garbage In, Garbage Out* (GIGO) — if the data is dirty, the AI's output will keep being wrong.

> 🤔 **Guess First:** In your opinion, what percentage of a data science project's total time is spent on data *preprocessing* versus building the model?
> 
> See the Answer
> 
> **80% of the time** in a project is spent preparing & cleaning data, with only **20%** left for building & training the model. This is the AI success formula that beginners often underestimate.

| Statistic | Meaning |
| --- | --- |
| 80% | Of total data science project time spent on preprocessing |
| 70% | Accuracy improvement achievable with good data cleaning |
| 90% | Success rate of AI projects that apply proper preprocessing |

> *Note: the figures above (including the "80% unstructured data" statistic in the next section) are commonly-cited figures in data science material, not from a single academic study that can be directly referenced. Read them as an illustration of scale/urgency, not a precise statistic.*

### Real Examples: When Bad Data Breaks AI

**Case 1 — A Failed Chatbot**

A customer service chatbot was trained on conversation data full of typos & non-standard language → the bot couldn't understand customer questions, and its answers became irrelevant.

**Case 2 — A Wrong Price Prediction**

A property price prediction model was trained on data with a lot of missing values & duplicates → a house worth Rp 500 million was predicted at Rp 2 billion.

### Key Definitions

*   **Data Preprocessing**: the process of preparing data as well and as thoroughly as possible so it's ready to be used by AI.
    
*   **Data Cleaning**: a part of Data Preprocessing responsible for cleaning data (missing values, duplicates, inconsistent formats, wrong data types, etc.).
    

### The Data Preprocessing Workflow

1.  **Raw Data** — a dirty dataset with various problems and inconsistencies.
    
2.  **Cleaning** — removing nulls, fixing formats, handling outliers.
    
3.  **Transformation** — normalization, encoding, feature engineering.
    
4.  **Model Ready** — data ready to be used for training AI.
    

* * *

## 2\. Data Collection & Storage

### A. Data Collection

Data Collection is the scheme for gathering data for the purpose of building AI. There are two broad types:

**1\. Unstructured Data**

*   Characteristics: humans can understand the meaning of the data (e.g., recognizing an animal species) and have *reasoning* ability over that data (e.g., understanding the point of a paragraph).
    
*   Data sources: *generated data* is allowed, but *real-world data* that matches the problem is preferred — make sure the data conditions match real-world conditions.
    
*   Examples: text, images, video.
    
*   Fact: **around 80% of the world's data** is unstructured data.
    

**2\. Structured Data**

*   Characteristics: ordinary people don't have the reasoning ability to interpret the data's meaning; it's unique to each case — similar cases can use different data (e.g., inventory optimization differs depending on the *business objective*).
    
*   Examples: tabular data, CSV, columns and rows.
    
*   **Important note**: if an end-to-end system isn't available yet, don't jump straight into using AI.
    
    *   Build a **Data Pipeline** first so an end-to-end data flow is established (involve a Data Engineer if needed).
        
    *   AI should be used as *tooling*, not as the primary weapon.
        

### B. Data Storage

| Data Type | Storage Medium | Examples |
| --- | --- | --- |
| Unstructured Data | File Storage System — supports versioning, easy access, secure | Amazon S3, Google Cloud Storage (GCS) |
| Structured Data | Database (SQL/NoSQL) — ideally with separate data specifically for ML purposes, following the same principles: easy to store, versioned, easy to access, secure | PostgreSQL, MariaDB |
| Object Storage | Medium for object-shaped data, generally large files; supports scalability, versioning, good security | Images, videos, documents, ML datasets |

* * *

## 3\. Pandas: The Main Data Handling Tool

Pandas is one of the most popular Python libraries for data manipulation and analysis — think of it as Excel, but with superpowers and coding flexibility.

*   **Fast** — processes millions of rows in seconds.
    
*   **Powerful** — complex operations with short code.
    
*   **Flexible** — can handle various data formats (CSV, Excel, JSON, etc.).
    

### Other Data Tool Ecosystems

The data world generally relies on just Python and SQL. Besides Pandas, Python has a few other libraries:

*   **PySpark** — used when data is stored in Apache Spark.
    
*   **Polars** — rewritten in Rust so it's faster.
    
*   **DuckDB** — for *online analytical processing* (OLAP) databases.
    

> Note: if your data is still in CSV format, Pandas alone is enough.

### File Formats Supported by Pandas

| Format | Pandas Function | Notes |
| --- | --- | --- |
| CSV | `pd.read_csv()` | One of the most common & lightweight formats for tabular data; TSV is similar, differing in the delimiter |
| Excel (XLSX/XLS) | `pd.read_excel()` | Spreadsheet files |
| JSON | `pd.read_json()` | Web data format; for unstructured datasets that need labeling |
| SQL Database | `pd.read_sql()` | Query directly from a database |

### Loading a Dataset

`pd` is an alias for Pandas, making the code shorter to write. Pandas can read almost every data format commonly used in data science.

```python
import pandas as pd

df = pd.read_csv('retail_data_raw.csv')
print(f"Jumlah baris: {len(df)}")
```

**Practical tips:**

*   Always check the *encoding* if you see odd characters.
    
*   Use the `sep` parameter for a custom delimiter.
    
*   The `nrows` parameter for previewing large data.
    
*   Pay attention to the delimiter and *decimal separator*.
    

> Note: the variable `df` stands for "DataFrame", Pandas' main data structure shaped like a table with rows & columns.

### Data Structures: Series vs DataFrame

*   **Series**: a 1-dimensional array with an index. Example: `pd.Series([1, 2, 3, 4])`
    
*   **DataFrame**: a 2-dimensional table with rows and columns — the most commonly used structure.
    

* * *

## 4\. Exploring Data Before Cleaning

The principle of **"Understand before fix"**: an AI engineer must understand the "face" of their data — column structure, value types, and general patterns — before starting to clean it.

| Method | Function |
| --- | --- |
| `df.head()` / `df.head(10)` | Preview the first rows of the dataset (default 5 rows) |
| `df.tail()` | Preview the last rows; sometimes there's a different pattern at the end |
| `df.info()` | Number of rows & columns, column names, data types, non-null counts, memory usage |
| `df.describe()` | Descriptive statistics for numeric columns: mean, median, min, max, standard deviation |
| `df.shape` | Dataset size in `(rows, columns)` format |
| `df.columns` | List of all column names in the dataset |

* * *

## 5\. Data Cleaning (Deep Dive)

### 5.1 Common Problems in a Dataset

| Problem | Description | Example / Impact |
| --- | --- | --- |
| Missing Values (Null) | Data that is missing or incomplete | Can cause model errors or biased predictions |
| Outliers | Values that are too extreme / don't make sense | Age 250 years, negative salary |
| Duplicates | The same row appears multiple times | Can cause the model to overfit to certain data |
| Inconsistent Format | Dates in various formats, extra spaces, random capitalization | "Jakarta" vs "jakarta" vs "JAKARTA " |
| Wrong Data Type | Numbers stored as text, or categories stored as numbers | A price column typed as object, can't be calculated |

### 5.2 Data Types & Conversion

One classic data cleaning problem is the wrong data type — for example, numbers stored as text, or dates stored as strings. Data types need to be transformed without changing their original meaning (numeric data is ratio/interval in nature, handled differently from categorical data).

| Data Type | Description |
| --- | --- |
| `int64` | Integer |
| `float64` | Decimal number |
| `object` | Text / String |
| `datetime64` | Date and time |
| `bool` | True / False |

> Note: if a price column is stored as `object` (text), you can't calculate its average or sum — and the AI will get confused too.

```python
# Konversi ke numerik
df['quantity'] = df['quantity'].astype(int)
df['price'] = pd.to_numeric(df['price'], errors='coerce')

# Konversi ke datetime
df['date'] = pd.to_datetime(df['date'], format='%d/%m/%Y')

# Konversi ke string
df['customer_id'] = df['customer_id'].astype(str)
```

> Note: the `errors='coerce'` parameter turns values that fail to convert into `NaN`, so the code doesn't throw an error.

### 5.3 Cleaning Text Data (Tabular)

Common string operations for tidying up text columns:

```python
df['nama'] = df['nama'].str.strip()              # hapus whitespace
df['kota'] = df['kota'].str.lower()               # lowercase semua
df['telp'] = df['telp'].str.replace('-', '')      # replace karakter
df['nomor'] = df['text'].str.extract(r'(\d+)')    # extract angka saja
```

**Text cleaning tips:**

*   Always standardize *case* (upper/lower).
    
*   Strip leading & trailing *whitespace*.
    
*   Remove special characters if unnecessary.
    
*   Check for typos with `value_counts()`.
    
*   Stay consistent with format ("Jakarta" vs "jakarta").
    

> Note: this is text cleaning for table columns (*general string cleanup*). It's different from preparing text for an NLP model (*text vectorization*) below, which requires extra steps.

### 5.4 NLP-Specific Data Cleaning

Before text is vectorized for an NLP model, there are words considered to carry little useful information that need to be removed:

*   **Links or URLs**
    
*   **Tags** (mentions or hashtags)
    
*   **Stopwords**, such as: *the, a, an, or, for*, and other common words
    

**Why:** links/tags aren't relevant to model learning; stopwords appear repeatedly, add computational load, and don't help distinguish one document from another.

### 5.5 Missing Values

Missing values are one of the most common problems in real-world datasets — they can happen due to input errors, broken sensors, or the data simply not existing.

**Why they're dangerous:**

*   AI can't learn from empty data.
    
*   Can cause errors during model training.
    
*   Reduces prediction accuracy.
    

**Detection:**

*   `df.isnull().sum()` → number of nulls per column.
    
*   `df.isnull().mean() * 100` → percentage of missing values.
    

**Strategies for handling missing values:**

1.  **Drop rows** (`df.dropna()`) — if missing values are few (<5%) and there's still plenty of data.
    
2.  **Drop a column** (`df.dropna(axis=1)`) — if a column has too many missing values (>50%).
    
3.  **Fill with a specific value** (`df.fillna(...)`) — use 0, mean, median, or mode depending on context; used when a lot of data is missing but the column is important.
    
4.  **Forward fill / backward fill** — for time series data, fill with the value before/after it (`method='ffill'`).
    

```python
# 1. Hapus baris dengan null
df_clean = df.dropna()

# 2. Hapus kolom dengan >50% null
threshold = len(df) * 0.5
df_clean = df.dropna(axis=1, thresh=threshold)

# 3. Isi dengan nilai tertentu / mean
df['age'].fillna(0, inplace=True)
df['price'].fillna(df['price'].mean(), inplace=True)

# 4. Forward fill (time series)
df.fillna(method='ffill', inplace=True)
```

> Principle: **"No blanks for brains"** — AI needs complete data to learn well.

### 5.6 Duplicate Data

Duplicate data happens when one observation appears more than once in a dataset — usually caused by input errors or careless dataset merging.

**Dangers of duplicates:** the AI model can overfit to certain data, bias in predictions, longer training time, and inaccurate evaluation metrics.

```python
# Deteksi duplikat
df.duplicated().sum()
df[df.duplicated()]

# Hapus duplikat
df_clean = df.drop_duplicates()
df_clean = df.drop_duplicates(subset=['customer_id', 'date'])
df_clean = df.drop_duplicates(keep='first')  # atau 'last'
```

> **Best practice**: always keep the original dataset separate from the cleaned dataset (different file names) — never *overwrite* the original data.
> 
> Principle: **"One truth per row"** — every row must represent one unique observation (*unique and clean*).

### 5.7 Outliers

**Definition:** data that is anomalous / whose value is far different from most of the other data. Example: employee salaries are generally 5–15 million, but there's one at 500 million — that's an outlier.

**Simple detection:** Boxplot, scatter plot, the IQR (Interquartile Range) method, the Z-Score method, or domain knowledge.

**When to remove an outlier:** when its likelihood of occurring is very small, or when its presence has no impact on the business process. **If an outlier represents a real business condition, don't remove it right away** — it needs *treatment* instead.

**Advanced Technique: Outlier Detection with a Variational Autoencoder (VAE)**

**What is an Autoencoder?**

A neural network trained with the input and output being the same, so it learns to reconstruct data and understand the distribution of clean (normal) data.

**How it works for outlier detection:**

*   A VAE compares the reconstruction result against the original input.
    
*   The bigger the difference (measured with *Mean Squared Error* / MSE), the more likely the data point is an outlier.
    
*   Data outside the learned distribution → the reconstruction result differs greatly → high MSE value → an indicator of outlier data.
    

Other methods: **Z-Score**, **Interquartile Range (IQR)**, **Isolation Forest**.

**Hands-on Implementation — Detecting & Removing Outliers with IQR:**

```python
Q1 = df['price'].quantile(0.25)
Q3 = df['price'].quantile(0.75)
IQR = Q3 - Q1

lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR

df_clean = df[(df['price'] >= lower_bound) & (df['price'] <= upper_bound)]
```

> Note: the IQR method is one of the most commonly used ways to detect outliers — but it still needs to be validated against business context (see "When to remove an outlier" above). The number **1.5** is a standard constant, but it can be adjusted as needed.

### 5.8 Data That Confuses the Model

Beyond numeric outliers, there's a more subtle category of "dirty data": data that actively confuses the model's learning process.

*   **The goal of cleaning here:** removing data that rarely occurs and doesn't represent actual conditions.
    
*   **Adversarial attack**: data that has been **deliberately modified** so the AI model produces a wrong prediction.
    
*   **Important note:** removing a small part of an object within an image is still acceptable if that object is actually lowering the quality of the training data — the goal is to keep the model learning from the correct patterns.
    

> 🤔 **Guess First:** If there's one small, irrelevant object in the corner of a training photo that's actually causing the model to learn incorrectly, is it okay to remove it from the image?
> 
> See the Answer
> 
> **Yes, it's okay.** As long as the goal is to keep the model learning from correct patterns, and the object really is degrading the quality of the training data, removing a small part of an object is still acceptable.

### 5.9 Renaming Columns

**Why it matters:** column names are often inconsistent, spaces & special characters cause hassle, *descriptive* names are easier to understand, and it standardizes the *naming convention*.

**Best practice:** use `snake_case` (lowercase letters, underscores for spaces), clear and short names, avoid special characters.

```python
# Rename kolom tertentu
df.rename(columns={'Nama Lengkap': 'nama', 'Umur (tahun)': 'usia'}, inplace=True)

# Rename semua kolom sekaligus
df.columns = ['nama', 'usia', 'gaji']

# Lowercase & hilangkan spasi
df.columns = df.columns.str.lower()
df.columns = df.columns.str.replace(' ', '_')
```

* * *

## 6\. Quiz Check — Part 1

**Q1. Why should a categorical data type like "hair color" be encoded using One-Hot Encoding rather than converted into simple ordered numbers (0, 1, 2, ...)?**

**Answer:** Because "hair color" is **nominal** data — it has no order/ranking. If it's turned into simple ordered numbers, the model might mistakenly assume there's a ranking relationship between categories (e.g., the number 2 being seen as "bigger" than 1), even though no such order exists. *(Full details are covered in Part 2.)*

**Q2. When should you drop an ENTIRE COLUMN instead of just the rows that have missing values?**

**Answer:** When that column has **more than 50%** missing values. If the missing values are few (under 5%) and there's still plenty of data, it's enough to just drop the rows with `df.dropna()`.

**Q3. An outlier salary of Rp 500 million is found in an HR dataset. Should it automatically be removed?**

**Answer:** **Not automatically.** You need to check first whether that value represents a real business condition (e.g., it really is a director's salary) or is purely an input error. If it represents a real business condition, don't remove it right away — it needs *treatment* first.

**Q4. What's the difference between "cleaning text in a table column" and "cleaning text for NLP"?**

**Answer:** Cleaning text in a table column focuses on *general string cleanup* (stripping whitespace, lowercasing, replacing characters). Cleaning text for NLP requires additional, more specific steps: removing **links/URLs**, **tags** (mentions/hashtags), and **stopwords** — because these elements aren't informative and add extra computational load to the language model.

* * *

**Continue to** [**Part 2**](https://shaka-ai.hashnode.dev/data-handling-preprocessing-part-2-en) — we'll cover the full Data Transformation process (encoding, dimensionality reduction, normalization, text vectorization, images as numeric data), how to save a clean dataset, Data Versioning & Management Strategies, Best Practices & Error Handling, up to the big-picture end-to-end Data Pipeline Flow.