Categorical Data

Represents labels or distinct groups, not numerical measurements

Examples

• Colors (Red, Green, Blue)
• Cities (New York, London, Tokyo)
• Customer Status (Active, Churned, Pending)

Most ML algorithms require numerical input

Translation into numbers needed

Two Types of Categorical Data

Nominal Data (No Order)

• Categories have no intrinsic ranking
• Apple, Banana, Orange: Apple is not "greater" than Banana

Ordinal Data (Ordered)

• Categories have meaningful sequence or hierarchy
• Low, Medium, High
• S, M, L, XL

Label / Ordinal Encoding

Maps each category to a unique integer

• Low: 1
• Medium: 2
• High: 3

Pros

• Easy to implement
• Extremely memory efficient
• Preserves order (great for Ordinal data)

Cons

• Implies a false numerical relationship for nominal data
• Banana is not "greater" than Apple
• Or two times as big

One-Hot Encoding (OHE)

Creates a new binary column for every unique category

• Color_Red | Color_Green | Color_Blue
• Red becomes: [1, 0, 0]
• Green becomes: [0, 1, 0]
• Blue becomes: [0, 0, 1]

Pros

• Eliminates false ordinal relationships
• Great for nominal data

Cons

• Can lead to high-dimensional feature spaces
• If a column has 1,000 unique categories, OHE adds 1,000 columns
• Creates sparse matrices (lots of zeros)
• Increases computational complexity

Target / Mean Encoding

Replaces a category with the average value of the target variable for that category

• If City="London" has a 60% churn rate, "London" is replaced with 0.60

Pros

• Handles high cardinality beautifully
• Captures predictive power directly

Cons

• Data Leakage / Overfitting
• If not cross-validated properly, the model "sees" the target during training
• Requires smoothing to handle rare categories
• Can be sensitive to outliers in the target variable
• May not perform well with very small sample sizes

Frequency / Count Encoding

Replaces categories with how often they appear in the dataset

• If "New York" appears 5,000 times in the dataset, it becomes 5000

Pros

• Simple, fast, and doesn't add extra columns
• Works well when frequency correlates with the target
• Helps to identify rare categories
• Can be used as a feature in its own right

Cons

• Different categories can end up with the same number if they have the same frequency
• Doesn't capture target relationship directly

Binary & Hashing Encoding

Binary Encoding

• Converts integers (from Label Encoding) into binary digits
• Creates one column per bit
• 30 categories: only 5 columns needed (vs. 30 for OHE)

Feature Hashing

• Applies a hash function to the category
• Maps the hashed value to a fixed number of columns, e.g. modulo 5
• Corresponding column number is set to 1
• Behaves like a compressed version of One-Hot Encoding

Pros

• Massive memory savings for extreme cardinality (e.g., IP addresses, User IDs)

Cons

• Irreversible (can't easily map back to original category)ns
• Hash collisions

How to Choose the Right Encoder

Best Practices & Pitfalls

Fit on Train, Transform on Test

• Always fit your encoder on the training data
• Then apply to test data to prevent data leakage

Handle unknown categories

• Test data might have categories not seen in training data
• Plan for them (e.g., map to an "Unknown" category)

Handle Missing Values

• Treat missing data as own category
• Impute missing values with a specific value (e.g., "Missing")
• Use domain knowledge to fill in missing values
• Consider the context and business logic when handling missing data

Use processing pipelines

• Ensure that all data preprocessing steps are applied consistently
• Automate the preprocessing workflow to reduce manual errors