Categorical Data

Represents labels or distinct groups, not numerical measurements

Examples

• Colors (Red, Green, Blue)
• Cities (New York, London, Tokyo)
• Customer Status (Active, Churned, Pending)

Most ML algorithms require numerical input

Translation into numbers needed

Two Types of Categorical Data

Nominal Data (No Order)

• Categories have no intrinsic ranking
• Apple, Banana, Orange: Apple is not "greater" than Banana

Ordinal Data (Ordered)

• Categories have meaningful order
• Low, Medium, High
• S, M, L, XL

Label / Ordinal Encoding

Maps each category to a unique integer

• Low: 1
• Medium: 2
• High: 3

Pros

• Easy to implement
• Extremely memory efficient
• Preserves order (great for ordinal data)

Cons

• Implies a false numerical relationship for nominal data
• Banana is not "greater" than Apple
• Or two times as big

One-Hot Encoding (OHE)

Creates a new binary column for every unique category

• Color_Red | Color_Green | Color_Blue
• Red becomes: [1, 0, 0]
• Green becomes: [0, 1, 0]
• Blue becomes: [0, 0, 1]

Pros

• Eliminates false ordinal relationships
• Great for nominal data

Cons

• Can lead to high-dimensional feature spaces
• If a column has 1,000 unique categories, OHE adds 1,000 columns
• Creates sparse matrices (lots of zeros)
• Increases computational complexity

Target / Mean Encoding

Replaces a category with the average value of the target variable for that category

Example 1: Classification (Customer Churn)

• Prediction whether a customer will churn (1 = Yes, 0 = No)
• You have a categorical feature City
• New York: 1,000 customers, 300 churned 30% churn rate
• London: 500 customers, 50 churned: 10% churn rate
• Tokyo: 200 customers, 80 churned: 40% churn rate
• Encode the cities with their respective churn rates (mean of target values)
• Tokyo has a higher churn rate than New York and London
• Model learns that Tokyo is a higher-risk city for customer churn

Example 2: Regression (House Prices)

• Prediction of house prices
• You have a categorical feature Neighborhood
• Downtown: Average house price = $500,000
• Suburbs: Average house price = $250,000
• Rural: Average house price = $150,000
• Replace the categories with the averages
• Sensible value for this feature

Pros

• Handles high cardinality beautifully
• Captures predictive power directly

Cons

• Data Leakage / Overfitting
• If not cross-validated properly, the model "sees" the target during training
• Requires smoothing to handle rare categories
• Can be sensitive to outliers in the target variable
• May not perform well with very small sample sizes

The "Rare Category" Problem & Smoothing

• Predicting Churn
• You have a Zip_Code feature
• Zip_Code_A, only has 1 customer, and that customer churned
• Zip_Code_A gets encoded as 100% churn rate
• Solution: blend the category's mean with the global mean
• Smoothing Formula: $$\frac{ (\text{Category_Count} \times \text{Category_Mean}) + (\text{Weight} \times \text{Global_Mean})} {\text{Category_Count} + \text{Weight}}$$
• Assuming Weight = 10 and Global_Mean = 0.2
• Zip_Code_A gets encoded as: $$\frac{1\times 1.0 + 10 \times 0.2}{1 + 10} = 0.27$$

Frequency / Count Encoding

Replaces categories with how often they appear in the dataset

• If "New York" appears 5,000 times in the dataset, it becomes 5000

Pros

• Simple, fast, and doesn't add extra columns
• Works well when frequency correlates with the target
• Helps to identify rare categories
• Can be used as a feature in its own right

Cons

• Different categories can end up with the same number if they have the same frequency
• Doesn't capture target relationship directly

Binary & Hashing Encoding

Binary Encoding

• Converts integers (from Label Encoding) into binary digits
• Creates one column per bit
• 30 categories: only 5 columns needed (vs. 30 for OHE)

Feature Hashing

• Applies a hash function to the category
• Maps the hashed value to a fixed number of columns, e.g. modulo 5
• Corresponding column number is set to 1
• Behaves like a compressed version of One-Hot Encoding

Pros

• Massive memory savings for extreme cardinality (e.g., IP addresses, User IDs)

Cons

• Irreversible (can't easily map back to original category)
• Hash collisions

How to Choose the Right Encoder

Best Practices & Pitfalls

Fit on Train, Transform on Test

• Always fit your encoder on the training data
• Then apply to test data to prevent data leakage

Handle unknown categories

• Test data might have categories not seen in training data
• Plan for them (e.g., map to an "Unknown" category)

Handle Missing Values

• Treat missing data as own category
• Impute missing values with a specific value (e.g., "Missing")
• Use domain knowledge to fill in missing values
• Consider the context and business logic when handling missing data

Use processing pipelines

• Ensure that all data preprocessing steps are applied consistently
• Automate the preprocessing workflow to reduce manual errors