You are viewing a public preview
Sign in to save your progress, do assignments, join discussions and get feedback.
Find a straight line (or a plane when there are many features) that best fits the cloud of data points:
ŷ = w₁x₁ + w₂x₂ + … + wₙxₙ + b
Here w is the weight of each feature and b is the intercept. The model learns by finding the set of w, b that makes the error smallest — usually measured by MSE.
This is the biggest strength of linear regression. If a house-price model gives the area feature a coefficient of 0.05 (billion VND per m²), we understand immediately: holding all other factors fixed, each additional m² raises the price by 0.05 billion. Very few models give us such a clear explanation.
from sklearn.linear_model import LinearRegression
model = LinearRegression()
model.fit(x_train, y_train)
print(model.coef_) # weight of each feature
print(model.intercept_) # intercept
If the real relationship is a curve, the model will underfit no matter how much data you have.
Coefficients depend on the unit of measurement. A column measured in meters will have a very different coefficient once converted to centimeters, even though the meaning is unchanged. To compare the importance of features, you must first standardize them to the same scale.
Reading a large coefficient as an important feature before standardizing. A column measured in dong and a column measured in billions of dong will produce coefficients that differ by nine orders of magnitude, which says nothing about importance.
By the end of this lesson you can:
coef_ and intercept_Key idea: Linear Regression's biggest strength is that its coefficients are directly interpretable, but only if you respect its assumptions and compare coefficients on standardised features.
Linear Regression finds the straight line, or the plane when there are many features, that best fits the cloud of data points:
y_hat = w1x1 + w2x2 + ... + wn*xn + b
Each w is the weight (coefficient) of one feature and b is the intercept. The model learns the set of w and b that makes the error as small as possible, usually measured by the mean squared error (MSE).
This is where Linear Regression stands out. Few models give such a clear explanation.
Suppose a house-price model, with price in billion VND, gives the feature area a coefficient of 0.05 (billion VND per square metre). The interpretation is: holding all other factors fixed, each additional square metre raises the predicted price by 0.05 billion VND (50 million VND).
"Holding all other factors fixed" matters: in a multiple regression, each coefficient describes the effect of one feature while the others stay the same.
In scikit-learn:
from sklearn.linear_model import LinearRegression
model = LinearRegression()
model.fit(x_train, y_train)
print(model.coef_) # weight of each feature
print(model.intercept_) # intercept
If the real relationship is a clear curve, a straight line cannot follow it. The model will underfit no matter how much data you collect, because more data does not change the shape of the model. Plotting the data first helps you spot this.
A coefficient depends on the unit of measurement. If a length column is converted from metres to centimetres, its values become 100 times larger and its coefficient becomes 100 times smaller, yet the meaning of the feature has not changed at all.
So a large coefficient does not mean an important feature. To compare the importance of features, first standardise them to the same scale (for example with StandardScaler, which gives each feature mean 0 and standard deviation 1). After that, a larger absolute coefficient does mean a stronger effect per standard deviation.
A toy house-price dataset (made-up numbers) with area in square metres and rooms, price in billion VND:
import pandas as pd
from sklearn.linear_model import LinearRegression
data = pd.DataFrame({
"area": [50, 60, 70, 80, 100],
"rooms": [1, 2, 2, 3, 3],
"price": [2.6, 3.2, 3.6, 4.2, 5.1],
})
x = data[["area", "rooms"]]
y = data["price"]
model = LinearRegression().fit(x, y)
print(model.coef_.round(3), round(model.intercept_, 3))
x_cm = x.assign(area=x["area"] * 10000) # area in square centimetres
model_cm = LinearRegression().fit(x_cm, y)
print(model_cm.coef_)
What to expect and how to read it:
area, plus a coefficient for rooms. Read the area weight as "each extra square metre adds about that much, with rooms held fixed".area coefficient becomes 10,000 times smaller. Same meaning, very different number.model.coef_ and model.intercept_ show the learned weights.Flashcards · Linear Regression
8 wordsThe change in the predicted output for a one-unit increase in a feature, with other features held fixed.
The model's prediction when every feature equals zero.
The scikit-learn attribute holding one learned weight per feature of a linear model.
The assumption that the output changes in a straight-line way with the features.
The assumption that prediction errors have roughly the same spread across all predicted values.
A model too simple to capture the real pattern, so it performs poorly even on training data.
A coefficient's size changes with the unit of its feature, even though its meaning does not.
Put features on the same scale before using coefficient sizes to compare feature importance.