Data Handling & Preprocessing (Part 1): Foundations, Collection, and Data Cleaning
Part 1 of 2: why preprocessing matters, how data is collected & stored, an introduction to Pandas, data exploration, and a deep dive into the cleaning process.
Before an AI model can learn anything, it has to "eat" data first. The problem is, raw data is almost always messy — some of it is missing, some is duplicated, some is inconsistently formatted. Part 1 of this note covers the foundations of data handling through to the full cleaning process. Part 2 will continue with transformation, versioning, and best practices.
Table of Contents
1. Why Does Data Preprocessing Matter?
Even the most sophisticated AI model won't produce accurate predictions if it's trained on bad data. Preprocessing isn't just a technical step — it's the foundation of an AI project's success.
Golden rule: Garbage In, Garbage Out (GIGO) — if the data is dirty, the AI's output will keep being wrong.
🤔 Guess First: In your opinion, what percentage of a data science project's total time is spent on data preprocessing versus building the model?
See the Answer
80% of the time in a project is spent preparing & cleaning data, with only 20% left for building & training the model. This is the AI success formula that beginners often underestimate.
| Statistic | Meaning |
|---|---|
| 80% | Of total data science project time spent on preprocessing |
| 70% | Accuracy improvement achievable with good data cleaning |
| 90% | Success rate of AI projects that apply proper preprocessing |
Note: the figures above (including the "80% unstructured data" statistic in the next section) are commonly-cited figures in data science material, not from a single academic study that can be directly referenced. Read them as an illustration of scale/urgency, not a precise statistic.
Real Examples: When Bad Data Breaks AI
Case 1 — A Failed Chatbot
A customer service chatbot was trained on conversation data full of typos & non-standard language → the bot couldn't understand customer questions, and its answers became irrelevant.
Case 2 — A Wrong Price Prediction
A property price prediction model was trained on data with a lot of missing values & duplicates → a house worth Rp 500 million was predicted at Rp 2 billion.
Key Definitions
Data Preprocessing: the process of preparing data as well and as thoroughly as possible so it's ready to be used by AI.
Data Cleaning: a part of Data Preprocessing responsible for cleaning data (missing values, duplicates, inconsistent formats, wrong data types, etc.).
The Data Preprocessing Workflow
Raw Data — a dirty dataset with various problems and inconsistencies.
Cleaning — removing nulls, fixing formats, handling outliers.
Transformation — normalization, encoding, feature engineering.
Model Ready — data ready to be used for training AI.
2. Data Collection & Storage
A. Data Collection
Data Collection is the scheme for gathering data for the purpose of building AI. There are two broad types:
1. Unstructured Data
Characteristics: humans can understand the meaning of the data (e.g., recognizing an animal species) and have reasoning ability over that data (e.g., understanding the point of a paragraph).
Data sources: generated data is allowed, but real-world data that matches the problem is preferred — make sure the data conditions match real-world conditions.
Examples: text, images, video.
Fact: around 80% of the world's data is unstructured data.
2. Structured Data
Characteristics: ordinary people don't have the reasoning ability to interpret the data's meaning; it's unique to each case — similar cases can use different data (e.g., inventory optimization differs depending on the business objective).
Examples: tabular data, CSV, columns and rows.
Important note: if an end-to-end system isn't available yet, don't jump straight into using AI.
Build a Data Pipeline first so an end-to-end data flow is established (involve a Data Engineer if needed).
AI should be used as tooling, not as the primary weapon.
B. Data Storage
| Data Type | Storage Medium | Examples |
|---|---|---|
| Unstructured Data | File Storage System — supports versioning, easy access, secure | Amazon S3, Google Cloud Storage (GCS) |
| Structured Data | Database (SQL/NoSQL) — ideally with separate data specifically for ML purposes, following the same principles: easy to store, versioned, easy to access, secure | PostgreSQL, MariaDB |
| Object Storage | Medium for object-shaped data, generally large files; supports scalability, versioning, good security | Images, videos, documents, ML datasets |
3. Pandas: The Main Data Handling Tool
Pandas is one of the most popular Python libraries for data manipulation and analysis — think of it as Excel, but with superpowers and coding flexibility.
Fast — processes millions of rows in seconds.
Powerful — complex operations with short code.
Flexible — can handle various data formats (CSV, Excel, JSON, etc.).
Other Data Tool Ecosystems
The data world generally relies on just Python and SQL. Besides Pandas, Python has a few other libraries:
PySpark — used when data is stored in Apache Spark.
Polars — rewritten in Rust so it's faster.
DuckDB — for online analytical processing (OLAP) databases.
Note: if your data is still in CSV format, Pandas alone is enough.
File Formats Supported by Pandas
| Format | Pandas Function | Notes |
|---|---|---|
| CSV | pd.read_csv() |
One of the most common & lightweight formats for tabular data; TSV is similar, differing in the delimiter |
| Excel (XLSX/XLS) | pd.read_excel() |
Spreadsheet files |
| JSON | pd.read_json() |
Web data format; for unstructured datasets that need labeling |
| SQL Database | pd.read_sql() |
Query directly from a database |
Loading a Dataset
pd is an alias for Pandas, making the code shorter to write. Pandas can read almost every data format commonly used in data science.
import pandas as pd
df = pd.read_csv('retail_data_raw.csv')
print(f"Jumlah baris: {len(df)}")
Practical tips:
Always check the encoding if you see odd characters.
Use the
sepparameter for a custom delimiter.The
nrowsparameter for previewing large data.Pay attention to the delimiter and decimal separator.
Note: the variable
dfstands for "DataFrame", Pandas' main data structure shaped like a table with rows & columns.
Data Structures: Series vs DataFrame
Series: a 1-dimensional array with an index. Example:
pd.Series([1, 2, 3, 4])DataFrame: a 2-dimensional table with rows and columns — the most commonly used structure.
4. Exploring Data Before Cleaning
The principle of "Understand before fix": an AI engineer must understand the "face" of their data — column structure, value types, and general patterns — before starting to clean it.
| Method | Function |
|---|---|
df.head() / df.head(10) |
Preview the first rows of the dataset (default 5 rows) |
df.tail() |
Preview the last rows; sometimes there's a different pattern at the end |
df.info() |
Number of rows & columns, column names, data types, non-null counts, memory usage |
df.describe() |
Descriptive statistics for numeric columns: mean, median, min, max, standard deviation |
df.shape |
Dataset size in (rows, columns) format |
df.columns |
List of all column names in the dataset |
5. Data Cleaning (Deep Dive)
5.1 Common Problems in a Dataset
| Problem | Description | Example / Impact |
|---|---|---|
| Missing Values (Null) | Data that is missing or incomplete | Can cause model errors or biased predictions |
| Outliers | Values that are too extreme / don't make sense | Age 250 years, negative salary |
| Duplicates | The same row appears multiple times | Can cause the model to overfit to certain data |
| Inconsistent Format | Dates in various formats, extra spaces, random capitalization | "Jakarta" vs "jakarta" vs "JAKARTA " |
| Wrong Data Type | Numbers stored as text, or categories stored as numbers | A price column typed as object, can't be calculated |
5.2 Data Types & Conversion
One classic data cleaning problem is the wrong data type — for example, numbers stored as text, or dates stored as strings. Data types need to be transformed without changing their original meaning (numeric data is ratio/interval in nature, handled differently from categorical data).
| Data Type | Description |
|---|---|
int64 |
Integer |
float64 |
Decimal number |
object |
Text / String |
datetime64 |
Date and time |
bool |
True / False |
Note: if a price column is stored as
object(text), you can't calculate its average or sum — and the AI will get confused too.
# Konversi ke numerik
df['quantity'] = df['quantity'].astype(int)
df['price'] = pd.to_numeric(df['price'], errors='coerce')
# Konversi ke datetime
df['date'] = pd.to_datetime(df['date'], format='%d/%m/%Y')
# Konversi ke string
df['customer_id'] = df['customer_id'].astype(str)
Note: the
errors='coerce'parameter turns values that fail to convert intoNaN, so the code doesn't throw an error.
5.3 Cleaning Text Data (Tabular)
Common string operations for tidying up text columns:
df['nama'] = df['nama'].str.strip() # hapus whitespace
df['kota'] = df['kota'].str.lower() # lowercase semua
df['telp'] = df['telp'].str.replace('-', '') # replace karakter
df['nomor'] = df['text'].str.extract(r'(\d+)') # extract angka saja
Text cleaning tips:
Always standardize case (upper/lower).
Strip leading & trailing whitespace.
Remove special characters if unnecessary.
Check for typos with
value_counts().Stay consistent with format ("Jakarta" vs "jakarta").
Note: this is text cleaning for table columns (general string cleanup). It's different from preparing text for an NLP model (text vectorization) below, which requires extra steps.
5.4 NLP-Specific Data Cleaning
Before text is vectorized for an NLP model, there are words considered to carry little useful information that need to be removed:
Links or URLs
Tags (mentions or hashtags)
Stopwords, such as: the, a, an, or, for, and other common words
Why: links/tags aren't relevant to model learning; stopwords appear repeatedly, add computational load, and don't help distinguish one document from another.
5.5 Missing Values
Missing values are one of the most common problems in real-world datasets — they can happen due to input errors, broken sensors, or the data simply not existing.
Why they're dangerous:
AI can't learn from empty data.
Can cause errors during model training.
Reduces prediction accuracy.
Detection:
df.isnull().sum()→ number of nulls per column.df.isnull().mean() * 100→ percentage of missing values.
Strategies for handling missing values:
Drop rows (
df.dropna()) — if missing values are few (<5%) and there's still plenty of data.Drop a column (
df.dropna(axis=1)) — if a column has too many missing values (>50%).Fill with a specific value (
df.fillna(...)) — use 0, mean, median, or mode depending on context; used when a lot of data is missing but the column is important.Forward fill / backward fill — for time series data, fill with the value before/after it (
method='ffill').
# 1. Hapus baris dengan null
df_clean = df.dropna()
# 2. Hapus kolom dengan >50% null
threshold = len(df) * 0.5
df_clean = df.dropna(axis=1, thresh=threshold)
# 3. Isi dengan nilai tertentu / mean
df['age'].fillna(0, inplace=True)
df['price'].fillna(df['price'].mean(), inplace=True)
# 4. Forward fill (time series)
df.fillna(method='ffill', inplace=True)
Principle: "No blanks for brains" — AI needs complete data to learn well.
5.6 Duplicate Data
Duplicate data happens when one observation appears more than once in a dataset — usually caused by input errors or careless dataset merging.
Dangers of duplicates: the AI model can overfit to certain data, bias in predictions, longer training time, and inaccurate evaluation metrics.
# Deteksi duplikat
df.duplicated().sum()
df[df.duplicated()]
# Hapus duplikat
df_clean = df.drop_duplicates()
df_clean = df.drop_duplicates(subset=['customer_id', 'date'])
df_clean = df.drop_duplicates(keep='first') # atau 'last'
Best practice: always keep the original dataset separate from the cleaned dataset (different file names) — never overwrite the original data.
Principle: "One truth per row" — every row must represent one unique observation (unique and clean).
5.7 Outliers
Definition: data that is anomalous / whose value is far different from most of the other data. Example: employee salaries are generally 5–15 million, but there's one at 500 million — that's an outlier.
Simple detection: Boxplot, scatter plot, the IQR (Interquartile Range) method, the Z-Score method, or domain knowledge.
When to remove an outlier: when its likelihood of occurring is very small, or when its presence has no impact on the business process. If an outlier represents a real business condition, don't remove it right away — it needs treatment instead.
Advanced Technique: Outlier Detection with a Variational Autoencoder (VAE)
What is an Autoencoder?
A neural network trained with the input and output being the same, so it learns to reconstruct data and understand the distribution of clean (normal) data.
How it works for outlier detection:
A VAE compares the reconstruction result against the original input.
The bigger the difference (measured with Mean Squared Error / MSE), the more likely the data point is an outlier.
Data outside the learned distribution → the reconstruction result differs greatly → high MSE value → an indicator of outlier data.
Other methods: Z-Score, Interquartile Range (IQR), Isolation Forest.
Hands-on Implementation — Detecting & Removing Outliers with IQR:
Q1 = df['price'].quantile(0.25)
Q3 = df['price'].quantile(0.75)
IQR = Q3 - Q1
lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR
df_clean = df[(df['price'] >= lower_bound) & (df['price'] <= upper_bound)]
Note: the IQR method is one of the most commonly used ways to detect outliers — but it still needs to be validated against business context (see "When to remove an outlier" above). The number 1.5 is a standard constant, but it can be adjusted as needed.
5.8 Data That Confuses the Model
Beyond numeric outliers, there's a more subtle category of "dirty data": data that actively confuses the model's learning process.
The goal of cleaning here: removing data that rarely occurs and doesn't represent actual conditions.
Adversarial attack: data that has been deliberately modified so the AI model produces a wrong prediction.
Important note: removing a small part of an object within an image is still acceptable if that object is actually lowering the quality of the training data — the goal is to keep the model learning from the correct patterns.
🤔 Guess First: If there's one small, irrelevant object in the corner of a training photo that's actually causing the model to learn incorrectly, is it okay to remove it from the image?
See the Answer
Yes, it's okay. As long as the goal is to keep the model learning from correct patterns, and the object really is degrading the quality of the training data, removing a small part of an object is still acceptable.
5.9 Renaming Columns
Why it matters: column names are often inconsistent, spaces & special characters cause hassle, descriptive names are easier to understand, and it standardizes the naming convention.
Best practice: use snake_case (lowercase letters, underscores for spaces), clear and short names, avoid special characters.
# Rename kolom tertentu
df.rename(columns={'Nama Lengkap': 'nama', 'Umur (tahun)': 'usia'}, inplace=True)
# Rename semua kolom sekaligus
df.columns = ['nama', 'usia', 'gaji']
# Lowercase & hilangkan spasi
df.columns = df.columns.str.lower()
df.columns = df.columns.str.replace(' ', '_')
6. Quiz Check — Part 1
Q1. Why should a categorical data type like "hair color" be encoded using One-Hot Encoding rather than converted into simple ordered numbers (0, 1, 2, ...)?
Answer: Because "hair color" is nominal data — it has no order/ranking. If it's turned into simple ordered numbers, the model might mistakenly assume there's a ranking relationship between categories (e.g., the number 2 being seen as "bigger" than 1), even though no such order exists. (Full details are covered in Part 2.)
Q2. When should you drop an ENTIRE COLUMN instead of just the rows that have missing values?
Answer: When that column has more than 50% missing values. If the missing values are few (under 5%) and there's still plenty of data, it's enough to just drop the rows with df.dropna().
Q3. An outlier salary of Rp 500 million is found in an HR dataset. Should it automatically be removed?
Answer: Not automatically. You need to check first whether that value represents a real business condition (e.g., it really is a director's salary) or is purely an input error. If it represents a real business condition, don't remove it right away — it needs treatment first.
Q4. What's the difference between "cleaning text in a table column" and "cleaning text for NLP"?
Answer: Cleaning text in a table column focuses on general string cleanup (stripping whitespace, lowercasing, replacing characters). Cleaning text for NLP requires additional, more specific steps: removing links/URLs, tags (mentions/hashtags), and stopwords — because these elements aren't informative and add extra computational load to the language model.
Continue to Part 2 — we'll cover the full Data Transformation process (encoding, dimensionality reduction, normalization, text vectorization, images as numeric data), how to save a clean dataset, Data Versioning & Management Strategies, Best Practices & Error Handling, up to the big-picture end-to-end Data Pipeline Flow.
