1. Import Libraries¶
Before performing any data analysis, we need to import the required Python libraries. These libraries provide functionalities for data manipulation, numerical computation, and visualization.
Libraries Used¶
- Pandas (
pd) – Data loading, cleaning, manipulation, and analysis. - NumPy (
np) – Efficient numerical operations and mathematical computations. - Matplotlib (
plt) – Data visualization using charts and plots. - Pathlib – Platform-independent file and directory handling.
Why are libraries important?¶
Instead of implementing algorithms from scratch, Python libraries provide optimized and well-tested implementations, allowing us to focus on solving machine learning problems efficiently.
Learning Outcomes¶
After completing this section, you will be able to:
- Import commonly used Python libraries.
- Understand the purpose of each library.
- Configure display settings for data exploration.
In [2]:
# ==========================================================
# ML Engineering Lab - Data Collection & Data Validation
# Dataset: Olist E-commerce
# ==========================================================
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from pathlib import Path
pd.set_option("display.max_columns", None)
pd.set_option("display.max_rows", 10)
2. Load Dataset¶
Machine Learning projects begin by collecting and loading data from one or more sources. In this lab, we will use the Olist Brazilian E-commerce Dataset, which consists of multiple relational tables such as customers, orders, products, payments, reviews, and sellers.
Data Collection Pipeline¶
CSV Files
↓
Read using Pandas
↓
DataFrame
↓
Exploration & Validation
Why load data into DataFrames?¶
A DataFrame is a two-dimensional tabular data structure provided by Pandas that enables efficient data manipulation.
If a dataset contains
- n rows (records)
- m columns (features)
then the dataset can be represented as
$ D \in \mathbb{R}^{n \times m} $
where
- (n) = number of observations
- (m) = number of attributes
Learning Outcomes¶
- Load CSV files using Pandas.
- Understand structured tabular datasets.
- Explore relational datasets.
In [3]:
DATA_PATH = "/kaggle/input/datasets/organizations/olistbr/brazilian-ecommerce/"
customers = pd.read_csv(DATA_PATH + "olist_customers_dataset.csv")
orders = pd.read_csv(DATA_PATH + "olist_orders_dataset.csv")
products = pd.read_csv(DATA_PATH + "olist_products_dataset.csv")
payments = pd.read_csv(DATA_PATH + "olist_order_payments_dataset.csv")
In [4]:
# TODO
# Load the following datasets
# sellers
# reviews
# order_items
# Write your code below
3. First Look at Data¶
Before cleaning or analyzing a dataset, it is essential to understand its structure. Exploratory Data Analysis (EDA) begins with answering questions such as:
- How many rows are present?
- How many columns exist?
- What are the feature names?
- Which columns contain numerical or categorical data?
Common Inspection Functions¶
head()tail()info()describe()
Why is this important?¶
Early inspection helps identify
- missing values
- incorrect data types
- unexpected columns
- potential quality issues
Learning Outcomes¶
- Inspect datasets.
- Interpret DataFrame information.
- Understand feature distributions.
In [5]:
customers.head()
Out[5]:
| customer_id | customer_unique_id | customer_zip_code_prefix | customer_city | customer_state | |
|---|---|---|---|---|---|
| 0 | 06b8999e2fba1a1fbc88172c00ba8bc7 | 861eff4711a542e4b93843c6dd7febb0 | 14409 | franca | SP |
| 1 | 18955e83d337fd6b2def6b18a428ac77 | 290c77bc529b7ac935b93aa66c333dc3 | 9790 | sao bernardo do campo | SP |
| 2 | 4e7b3e00288586ebd08712fdd0374a03 | 060e732b5b29e8181a18229c7b0b2b5e | 1151 | sao paulo | SP |
| 3 | b2b6027bc5c5109e529d4dc6358b12c3 | 259dac757896d24d7702b9acbbff3f3c | 8775 | mogi das cruzes | SP |
| 4 | 4f2d8ab171c80ec8364f7c12e35b23ad | 345ecd01c38d18a9036ed96c73b8d066 | 13056 | campinas | SP |
In [6]:
customers.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 99441 entries, 0 to 99440 Data columns (total 5 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 customer_id 99441 non-null object 1 customer_unique_id 99441 non-null object 2 customer_zip_code_prefix 99441 non-null int64 3 customer_city 99441 non-null object 4 customer_state 99441 non-null object dtypes: int64(1), object(4) memory usage: 3.8+ MB
In [7]:
customers.describe(include="all")
Out[7]:
| customer_id | customer_unique_id | customer_zip_code_prefix | customer_city | customer_state | |
|---|---|---|---|---|---|
| count | 99441 | 99441 | 99441.000000 | 99441 | 99441 |
| unique | 99441 | 96096 | NaN | 4119 | 27 |
| top | 274fa6071e5e17fe303b9748641082c8 | 8d50f5eadf50201ccdcedfb9e2ac8455 | NaN | sao paulo | SP |
| freq | 1 | 17 | NaN | 15540 | 41746 |
| mean | NaN | NaN | 35137.474583 | NaN | NaN |
| ... | ... | ... | ... | ... | ... |
| min | NaN | NaN | 1003.000000 | NaN | NaN |
| 25% | NaN | NaN | 11347.000000 | NaN | NaN |
| 50% | NaN | NaN | 24416.000000 | NaN | NaN |
| 75% | NaN | NaN | 58900.000000 | NaN | NaN |
| max | NaN | NaN | 99990.000000 | NaN | NaN |
11 rows × 5 columns
In [8]:
# TODO
# Display
# first 10 rows of orders
# information of products
# summary statistics of payments
4. Dataset Shape¶
The shape of a dataset represents its dimensions. If a dataset contains
- (n) rows
- (m) columns
then
$ Shape=(n,m) $
Example
(99441, 8)
means
- 99,441 records
- 8 features
Why is shape important?¶
The dimensions of the dataset help estimate
- computational cost
- memory usage
- model complexity
Learning Outcomes¶
- Determine dataset dimensions.
- Interpret rows and columns correctly.
In [9]:
print(customers.shape)
(99441, 5)
In [10]:
# TODO
# Print shape of
# orders
# sellers
# reviews
In [34]:
import numpy as np
import pandas as pd
def inject_olist_impurities(
df,
dataset_type="customers",
noise_ratio=0.05,
duplicate_ratio=0.02,
outlier_ratio=0.02,
seed=42,
):
"""Injects domain errors, missing values, duplicates, and IQR quartile outliers into Olist datasets."""
np.random.seed(seed)
df_dirty = df.copy()
# Helper: Sample random indices
def get_random_indices(sample_size):
return np.random.choice(
df_dirty.index,
size=min(sample_size, len(df_dirty)),
replace=False,
)
# Helper: Inject extreme upper-bound quartile outliers (Q3 + multiplier * IQR)
def inject_iqr_outlier(col, sample_size, multiplier=10.0):
valid_series = df_dirty[col].dropna()
if valid_series.empty or not pd.api.types.is_numeric_dtype(
valid_series
):
return
q1 = valid_series.quantile(0.25)
q3 = valid_series.quantile(0.75)
iqr = q3 - q1
spread = iqr if iqr > 0 else (valid_series.std() or 100.0)
extreme_val = q3 + (multiplier * spread)
outlier_idx = get_random_indices(sample_size)
df_dirty.loc[outlier_idx, col] = extreme_val
n_total_noise = int(len(df_dirty) * noise_ratio)
n_outliers = int(len(df_dirty) * outlier_ratio)
# 1. Dataset-Specific Domain Errors & Quartile Outliers
if dataset_type == "customers":
n_per_error = n_total_noise // 3
df_dirty.loc[get_random_indices(n_per_error), "customer_state"] = (
np.nan
)
df_dirty.loc[
get_random_indices(n_per_error), "customer_zip_code_prefix"
] = np.nan
city_idx = get_random_indices(n_per_error)
df_dirty.loc[city_idx, "customer_city"] = (
df_dirty.loc[city_idx, "customer_city"].astype(str) + " "
)
inject_iqr_outlier(
"customer_zip_code_prefix", n_outliers, multiplier=10.0
)
elif dataset_type == "orders":
n_per_error = n_total_noise // 2
df_dirty.loc[
get_random_indices(n_per_error), "order_delivered_customer_date"
] = "2010-01-01 00:00:00"
df_dirty.loc[
get_random_indices(n_per_error), "order_status"
] = "lost_in_transit"
elif dataset_type == "products":
n_per_error = n_total_noise // 2
df_dirty.loc[get_random_indices(n_per_error), "product_weight_g"] = (
-500.0
)
df_dirty.loc[
get_random_indices(n_per_error), "product_category_name"
] = np.nan
inject_iqr_outlier("product_weight_g", n_outliers, multiplier=15.0)
inject_iqr_outlier("product_length_cm", n_outliers, multiplier=15.0)
elif dataset_type == "payments":
n_per_error = n_total_noise // 2
df_dirty.loc[get_random_indices(n_per_error), "payment_value"] = -100.0
df_dirty.loc[
get_random_indices(n_per_error), "payment_type"
] = "crypto_token"
inject_iqr_outlier("payment_value", n_outliers, multiplier=15.0)
inject_iqr_outlier("payment_installments", n_outliers, multiplier=10.0)
# 2. Inject Duplicate Rows
if duplicate_ratio > 0:
n_duplicates = int(len(df_dirty) * duplicate_ratio)
if n_duplicates > 0:
dup_indices = get_random_indices(n_duplicates)
duplicate_rows = df_dirty.loc[dup_indices].copy()
df_dirty = pd.concat(
[df_dirty, duplicate_rows], ignore_index=True
)
return df_dirty
# Execution Example:
customers = inject_olist_impurities(
customers, "customers", noise_ratio=0.05, duplicate_ratio=0.02
)
payments = inject_olist_impurities(
payments, "payments", noise_ratio=0.05, duplicate_ratio=0.02
)
products = inject_olist_impurities(
products, "products", noise_ratio=0.05, duplicate_ratio=0.02
)
orders = inject_olist_impurities(
orders, "orders", noise_ratio=0.05, duplicate_ratio=0.02
)
In [35]:
# Overwrite your original DataFrames with corrupted data
customers = inject_olist_impurities(
customers, dataset_type="customers", noise_ratio=0.1
)
orders = inject_olist_impurities(
orders, dataset_type="orders", noise_ratio=0.1
)
products = inject_olist_impurities(
products, dataset_type="products", noise_ratio=0.1
)
payments = inject_olist_impurities(
payments, dataset_type="payments", noise_ratio=0.1
)
5. Missing Values¶
Real-world datasets often contain missing or unavailable values. Missing values may occur due to
- sensor failures
- incomplete user input
- transmission errors
- unavailable information
Mathematical Representation¶
Suppose
$ D = \{x_1,x_2,\ldots,x_n\} $
A missing value is represented as
$ x_i = \text{NaN} $
The percentage of missing values in a feature is
$ Missing(\%)= \frac{\text{Number of Missing Values}} {\text{Total Number of Records}} \times100 $
Why does it matter?¶
Missing values can
- reduce model accuracy
- introduce bias
- prevent algorithms from training
Learning Outcomes¶
- Detect missing values.
- Calculate missing percentages.
- Decide appropriate handling techniques.
In [36]:
customers.isnull().sum()
Out[36]:
customer_id 0 customer_unique_id 0 customer_zip_code_prefix 9489 customer_city 0 customer_state 6482 dtype: int64
In [37]:
customers.isnull().sum().sort_values(ascending=False)
Out[37]:
customer_zip_code_prefix 9489 customer_state 6482 customer_id 0 customer_unique_id 0 customer_city 0 dtype: int64
In [38]:
# TODO
# Find missing values in
# orders
# products
# reviews
In [39]:
missing = customers.isnull().mean()*100
missing.sort_values(ascending=False)
Out[39]:
customer_zip_code_prefix 9.171927 customer_state 6.265405 customer_id 0.000000 customer_unique_id 0.000000 customer_city 0.000000 dtype: float64
In [40]:
# TODO
# Calculate percentage missing values
# for products dataset
In [41]:
customers["customer_city"] = customers["customer_city"].fillna("Unknown")
In [42]:
# TODO
# Replace missing review comments
# with "No Review"
6. Duplicate Records¶
Duplicate records represent repeated observations in the dataset. These often arise from
- repeated logging
- synchronization failures
- system retries
- human errors
Mathematical Definition¶
Let
$ R_i=(x_1,x_2,\ldots,x_m) $
Two records are duplicates if
$ R_i = R_j $
where
$ i \neq j $
Why remove duplicates?¶
Duplicate samples
- distort statistics
- bias machine learning models
- increase storage requirements
Learning Outcomes¶
- Detect duplicate records.
- Remove redundant observations.
In [43]:
customers.duplicated().sum()
Out[43]:
np.int64(3601)
In [44]:
customers = customers.drop_duplicates()
In [45]:
# TODO
# Check duplicates in
# products
# reviews
# sellers
7. Data Types¶
Every feature in a dataset has an associated data type. Common types include
- Integer
- Float
- Boolean
- String
- Datetime
Why is datatype validation important?¶
Incorrect data types can lead to
- computation errors
- visualization issues
- incorrect model inputs
Example
"100"
is a string,
whereas
100
is an integer.
Learning Outcomes¶
- Identify feature data types.
- Convert incorrect data types.
- Parse datetime columns.
In [46]:
orders.dtypes
Out[46]:
order_id object customer_id object order_status object order_purchase_timestamp object order_approved_at object order_delivered_carrier_date object order_delivered_customer_date object order_estimated_delivery_date object dtype: object
In [47]:
orders["order_purchase_timestamp"] = pd.to_datetime(
orders["order_purchase_timestamp"]
)
In [48]:
# TODO
# Convert all timestamp columns
# into datetime
In [49]:
payments[payments["payment_value"] < 0]
Out[49]:
| order_id | payment_sequential | payment_type | payment_installments | payment_value | |
|---|---|---|---|---|---|
| 35 | b2bb080b6bc860118a246fd9b6fad6da | 1 | crypto_token | 1 | -100.0 |
| 87 | 198298d54f269d8c5fe89e85a859af61 | 1 | crypto_token | 10 | -100.0 |
| 97 | d60a89ab9bc1e68834465ae630c3343e | 1 | crypto_token | 2 | -100.0 |
| 114 | f090de1b2ed9f4e251662cb31e3c7127 | 1 | crypto_token | 2 | -100.0 |
| 127 | f51e3c7f79d1e2c076acbb6af1588b95 | 1 | credit_card | 1 | -100.0 |
| ... | ... | ... | ... | ... | ... |
| 108050 | 6d0940a8f5fba47562bb14cd97dfd6da | 1 | crypto_token | 10 | -100.0 |
| 108052 | 9eb4c52cf0399fb41b3e537d6acf99e0 | 1 | credit_card | 4 | -100.0 |
| 108065 | 1d88068495a8986f241f769664ac98de | 1 | crypto_token | 1 | -100.0 |
| 108071 | 7cc23d880ead6eb7ebecb6292435b6d6 | 1 | crypto_token | 1 | -100.0 |
| 108081 | 13a8db555fde78dd0b116b01b6515617 | 1 | credit_card | 1 | -100.0 |
9626 rows × 5 columns
In [50]:
# TODO
# Find
# invalid freight values
# invalid payment values
In [51]:
orders["order_status"].unique()
Out[51]:
array(['delivered', 'lost_in_transit', 'shipped', 'invoiced',
'unavailable', 'processing', 'canceled', 'created', 'approved'],
dtype=object)
In [52]:
# TODO
# Find unique values
# in customer_state
In [53]:
orders["order_status"].value_counts()
Out[53]:
order_status delivered 84222 lost_in_transit 16665 shipped 990 unavailable 517 canceled 513 invoiced 273 processing 271 created 4 approved 2 Name: count, dtype: int64
In [54]:
# TODO
# Top 10 customer cities
8. Outlier Detection¶
Outliers are observations that differ significantly from the majority of the data.
Interquartile Range (IQR)¶
$ IQR=Q_3-Q_1 $
where
- (Q_1) = First Quartile
- (Q_3) = Third Quartile
A value is considered an outlier if
$ x<Q_1-1.5(IQR) $
or
$ x>Q_3+1.5(IQR) $
Why detect outliers?¶
Outliers can
- skew averages
- affect regression models
- reduce prediction performance
Learning Outcomes¶
- Visualize outliers.
- Compute IQR.
- Identify abnormal observations.
In [55]:
payments.boxplot(column="payment_value")
plt.show()
In [56]:
Q1 = payments["payment_value"].quantile(0.25)
Q3 = payments["payment_value"].quantile(0.75)
IQR = Q3 - Q1
outliers = payments[
(payments["payment_value"] < Q1 - 1.5*IQR) |
(payments["payment_value"] > Q3 + 1.5*IQR)
]
outliers.head()
Out[56]:
| order_id | payment_sequential | payment_type | payment_installments | payment_value | |
|---|---|---|---|---|---|
| 26 | d0a945f85ba1074b60aac97ade7e240e | 1 | credit_card | 2 | 541.0000 |
| 27 | c39414c195d0f94c9d9c35e7c6ed4f1a | 1 | credit_card | 2 | 1902.2225 |
| 52 | 2e2c60b99754ae1e4d8b18846cfec9f2 | 1 | credit_card | 4 | 542.6600 |
| 54 | 95442deb81a5d91c97c0df96b431634a | 1 | boleto | 1 | 368.9800 |
| 61 | 3ab4eea6776259ae8dfbabaeffece352 | 1 | credit_card | 8 | 387.8000 |
In [57]:
# TODO
# Detect outliers
# in freight_value
9. Schema Validation¶
A schema defines the expected structure of a dataset. It specifies
- feature names
- data types
- constraints
Schema Definition¶
If
$ S=\{c_1,c_2,\ldots,c_m\} $
is the expected schema,
then
$ Columns(D)=S $
must hold for a valid dataset.
Why is schema validation important?¶
Schema validation prevents
- missing columns
- incorrect column names
- incompatible data types
Learning Outcomes¶
- Compare expected and actual schemas.
- Detect schema violations.
In [58]:
expected_columns = [
"order_id",
"customer_id",
"order_status",
"order_purchase_timestamp"
]
In [59]:
missing_columns = set(expected_columns) - set(orders.columns)
missing_columns
Out[59]:
set()
In [60]:
# TODO
# Check whether
# products dataset
# contains all expected columns
10. Cross-field Validation¶
Cross-field validation ensures logical consistency between multiple columns. Example:
For an e-commerce order, $ Purchase\ Date \leq Delivery\ Date $, must always hold.
If, $ Delivery\ Date < Purchase\ Date $, the record is invalid.
Why is this important?¶
Cross-field validation identifies
- impossible events
- timestamp inconsistencies
- business rule violations
Learning Outcomes¶
- Validate relationships across columns.
- Detect logically inconsistent records.
In [61]:
orders["order_purchase_timestamp"] = pd.to_datetime(
orders["order_purchase_timestamp"]
)
orders["order_delivered_customer_date"] = pd.to_datetime(
orders["order_delivered_customer_date"]
)
invalid = orders[
orders["order_delivered_customer_date"] <
orders["order_purchase_timestamp"]
]
invalid.head()
Out[61]:
| order_id | customer_id | order_status | order_purchase_timestamp | order_approved_at | order_delivered_carrier_date | order_delivered_customer_date | order_estimated_delivery_date | |
|---|---|---|---|---|---|---|---|---|
| 35 | 8563039e855156e48fccee4d611a3196 | 5f16605299d698660e0606f7eae2d2f9 | delivered | 2018-02-17 15:59:46 | 2018-02-17 16:15:34 | 2018-02-20 23:03:56 | 2010-01-01 | 2018-03-20 00:00:00 |
| 53 | 641fb0752bf5b5940c376b3a8bb9dc52 | f5afca14dfa9dc64251cf2b45c54c363 | lost_in_transit | 2017-12-15 00:06:10 | 2017-12-15 00:14:55 | 2017-12-19 01:58:00 | 2010-01-01 | 2018-01-16 00:00:00 |
| 70 | a6aeb116d2cb5013eb8a94585b71ffef | bb2f5e670f7155dc622c57e4b31d0a69 | lost_in_transit | 2017-09-13 14:27:11 | 2017-09-13 14:44:39 | 2017-09-15 18:42:29 | 2010-01-01 | 2017-09-25 00:00:00 |
| 87 | cadbb3657dac2dbbd5b84b12e7b78aad | 93ada7a24817edda9f4ab998fa823d16 | lost_in_transit | 2018-02-27 12:55:42 | 2018-03-01 02:48:54 | 2018-03-03 02:27:03 | 2010-01-01 | 2018-03-29 00:00:00 |
| 97 | 6a0a8bfbbe700284feb0845d95e0867f | 68451b39b1314302c08c65a29f1140fc | delivered | 2017-11-22 11:32:22 | 2017-11-22 11:46:50 | 2017-11-27 13:39:35 | 2010-01-01 | 2017-12-11 00:00:00 |
In [62]:
# TODO
# Find orders
# where estimated delivery date
# is before purchase date
11. Data Summary Report¶
A validation report summarizes the overall quality of a dataset. Typical metrics include
- Number of rows
- Number of columns
- Missing values
- Duplicate records
- Data types
- Unique values
- Outliers
Why generate a report?¶
A summary report provides a quick overview of data quality before machine learning begins.
Learning Outcomes¶
- Build automated validation reports.
- Assess dataset readiness for ML pipelines.
In [63]:
summary = pd.DataFrame({
"Missing Values": orders.isnull().sum(),
"Data Type": orders.dtypes,
"Unique Values": orders.nunique()
})
summary
Out[63]:
| Missing Values | Data Type | Unique Values | |
|---|---|---|---|
| order_id | 0 | object | 99441 |
| customer_id | 0 | object | 99441 |
| order_status | 0 | object | 9 |
| order_purchase_timestamp | 0 | datetime64[ns] | 98875 |
| order_approved_at | 166 | object | 90733 |
| order_delivered_carrier_date | 1850 | object | 81018 |
| order_delivered_customer_date | 2790 | datetime64[ns] | 87049 |
| order_estimated_delivery_date | 0 | object | 459 |
12. Lab-Excercise¶
In [39]:
# ==========================================================
# 🏆 Lab-Excercise: Data Collection & Data Validation
# ==========================================================
#
# Objective
# ---------
# Apply the concepts learned in this lab to perform a complete
# data validation workflow on one of the Olist e-commerce datasets.
#
# Choose ONE dataset:
# • customers
# • sellers
# • products
# • reviews
In [40]:
# ✅ Task 1: Load the selected dataset.
#
# ✅ Task 2: Perform initial exploration.
# - Display the first few records.
# - Check dataset shape.
# - View column names and data types.
#
# ✅ Task 3: Identify missing values.
# - Count missing values.
# - Calculate the percentage of missing values.
#
# ✅ Task 4: Detect and remove duplicate records.
#
# ✅ Task 5: Validate the dataset.
# - Verify data types.
# - Check for invalid or inconsistent values.
#
# ✅ Task 6: Detect outliers.
# - Use the IQR method.
# - Visualize using a boxplot.
#
# ✅ Task 7: Generate a Data Quality Report.
# Include:
# • Number of rows
# • Number of columns
# • Missing values
# • Duplicate records
# • Unique values
# • Data types
# • Number of detected outliers
#
# ✅ Task 8: Save the cleaned dataset as a new CSV file.
In [41]:
# Deliverables
# ------------
# • Cleaned dataset (.csv)
# • Data Quality Report
# • Well-documented notebook with explanations
13. Final Challenge¶
In [42]:
# Create a reusable function:
#
# validate_dataset(df)
#
# that automatically performs all validation checks and
# returns a comprehensive summary report for any dataset.
In [ ]: