Support Vector Machine (SVM)
- Support Vector Machine (SVM) is a powerful machine learning algorithm that separates data using optimal hyperplanes for classification tasks.
Maximum Margin Classifier
SVM tries to find the hyperplane that maximizes the margin between two classes.
Margin = Distance between hyperplane and nearest data points of each class
Maximum margin → Better generalization
Hyperplane Equation (2D)
w⋅x+b=0w \cdot x + b = 0w⋅x+b=0
w = weight vector (slope)
b = bias (intercept)
Goal:
Maximize:
Margin=2∣∣w∣∣Margin = \frac{2}{||w||}Margin=∣∣w∣∣2
Nearest points that lie on margin → Support Vectors
Support Vectors
Support vectors are the data points closest to the hyperplane.
They determine the position of the hyperplane
If support vectors change → hyperplane changes
Other points do not affect hyperplane
Visual Example
Class A o o
\
Hyperplane
/
Class B x x
o and x closest to hyperplane → Support Vectors
Kernel Functions
Sometimes data is not linearly separable in original space.
Kernel functions map data to higher dimensions to make it linearly separable.
Common Kernels
How Kernel Works:
Instead of explicitly mapping data to high dimensions, it calculates inner products in the new space, which is computationally efficient.
Soft Margin vs Hard Margin
Hard Margin
No misclassification allowed
Only works if data is perfectly separable
Sensitive to outliers
Soft Margin
Allows some misclassification
Introduces C parameter to balance margin and errors
Robust to noise and outliers
C small → wider margin, more misclassificationsC large → narrow margin, fewer misclassificationsC \text{ small → wider margin, more misclassifications} C \text{ large → narrow margin, fewer misclassifications}C small → wider margin, more misclassificationsC large → narrow margin, fewer misclassifications
Example: SVM Classifier
Support Vector Machine (SVM) Classification Example in Python for Student Pass/Fail
This Python example demonstrates how to use a Support Vector Machine (SVM) classifier to predict whether a student will pass or fail based on study hours and attendance. The code creates a dataset, splits it into training and testing sets, trains an SVM model using a linear kernel, and evaluates the model by making predictions and calculating accuracy.
# Step 1: Import Libraries
import numpy as np
from sklearn.svm import SVC
from sklearn.model_selection import train_test_split
# Step 2: Create Dataset
X = np.array([
[2, 50],
[3, 60],
[5, 80],
[6, 90],
[1, 40]
]) # Features: [Study Hours, Attendance]
y = np.array([0, 0, 1, 1, 0]) # Labels: 0=Fail, 1=Pass
# Step 3: Split Data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Step 4: Create SVM Model
model = SVC(kernel='linear', C=1.0)
# Step 5: Train Model
model.fit(X_train, y_train)
# Step 6: Predict
prediction = model.predict(X_test)
print("Prediction:", prediction)
print("Accuracy:", model.score(X_test, y_test))Output:
Prediction: [0 1]
Accuracy: 0.5
Key Concepts