❮ Previous Next ❯

K-Fold Cross Validation

  • This lesson explains K-Fold Cross Validation and how multiple folds help evaluate machine learning models more accurately.

  • Working Mechanism

    1. Divide the dataset into K equal folds

    2. Iterate K times:

      • Use K-1 folds for training

      • Use 1 fold for testing

    3. Repeat until every fold has been used as a test set exactly once

    4. Calculate performance metrics (accuracy, F1, R², etc.) for each fold

    5. Take the average → final model performance

    Visual Representation:

    Iteration

    Training Folds

    Testing Fold

    1

    2,3,4,5

    1

    2

    1,3,4,5

    2

    3

    1,2,4,5

    3

    4

    1,2,3,5

    4

    5

    1,2,3,4

    5


    Value of K

    K Value

    Notes

    K = 5 or 10

    Most commonly used, balances bias & variance

    K too small

    High bias, less reliable evaluation

    K too large (e.g., Leave-One-Out)

    High variance, slower computation

    Rule of Thumb

    Use 5–10 folds for medium datasets


    Advantages

    1. Better Performance Estimate → Uses multiple splits

    2. Reduces Variance → Less dependent on a single train-test split

    3. All Data Used → Each data point is used for training and testing

    4. Works Well for Small Datasets → Maximizes use of available data


    Limitations

    1. Computationally Expensive → Model trained K times

    2. Doesn’t Prevent Overfitting → Only evaluates model, doesn’t improve it

    3. Imbalanced Data Risk → Class distribution may vary per fold (use Stratified K-Fold)

    Python Example: 5-Fold Cross Validation

K-Fold Cross Validation Example in Python using Linear Regression

This Python example demonstrates how to evaluate a machine learning model using K-Fold Cross Validation with Linear Regression. The code loads the California Housing dataset, splits the data into 5 folds using KFold, trains the model on different training subsets, and evaluates it using the R² score. It prints the R² score for each fold and the average R² to measure the model’s overall performance. 

from sklearn.model_selection import KFold, cross_val_score
from sklearn.linear_model import LinearRegression
from sklearn.datasets import fetch_california_housing
import numpy as np

# Step 1: Load California Housing dataset
california = fetch_california_housing()
X, y = california.data, california.target

# Step 2: Create Model
model = LinearRegression()

# Step 3: K-Fold Setup
kf = KFold(n_splits=5, shuffle=True, random_state=42)

# Step 4: Perform Cross Validation
scores = cross_val_score(model, X, y, cv=kf, scoring='r2')

# Step 5: Results
print("R² for each fold:", scores)
print("Average R²:", np.mean(scores))

  • Output:

    R² for each fold: [0.57578771 0.61374822 0.60856043 0.62126494 0.5875292 ]

    Average R²: 0.6013781013684627

❮ Previous Next ❯