Đại học Kinh tế Quốc dân – National Economics UniversityMICRO4ASIA
🏆 Arena
Sign inSign up
MICRO4ASIAThe six micro-credentials

Courses are delivered on the Limio learning platform. Limio

← Fundamental of Machine Learning

You are viewing a public preview

Sign in to save your progress, do assignments, join discussions and get feedback.

Sign in
Module 2: Core Machine Learning Algorithms

Linear Regression

Print / Save as PDF

Linear Regression

The Idea

Find a straight line (or a plane when there are many features) that best fits the cloud of data points:

ŷ = w₁x₁ + w₂x₂ + … + wₙxₙ + b

Here w is the weight of each feature and b is the intercept. The model learns by finding the set of w, b that makes the error smallest — usually measured by MSE.

Reading the Coefficients

This is the biggest strength of linear regression. If a house-price model gives the area feature a coefficient of 0.05 (billion VND per m²), we understand immediately: holding all other factors fixed, each additional m² raises the price by 0.05 billion. Very few models give us such a clear explanation.

from sklearn.linear_model import LinearRegression
model = LinearRegression()
model.fit(x_train, y_train)
print(model.coef_)        # weight of each feature
print(model.intercept_)   # intercept

Assumptions to Know

  1. The relationship between the features and the output is linear
  2. The observations are independent of each other
  3. The errors have constant variance

If the real relationship is a curve, the model will underfit no matter how much data you have.

Compare Coefficients Only After Standardizing

Coefficients depend on the unit of measurement. A column measured in meters will have a very different coefficient once converted to centimeters, even though the meaning is unchanged. To compare the importance of features, you must first standardize them to the same scale.

The Most Common Mistake

Reading a large coefficient as an important feature before standardizing. A column measured in dong and a column measured in billions of dong will produce coefficients that differ by nine orders of magnitude, which says nothing about importance.

Study notes

By the end of this lesson you can:

  • Write the Linear Regression model with weights and an intercept
  • Interpret a coefficient in plain language, "holding all other factors fixed"
  • Read the learned weights from a scikit-learn model with coef_ and intercept_
  • State the three main assumptions of Linear Regression and what happens when linearity fails
  • Explain why coefficients can only be compared after standardisation

Key idea: Linear Regression's biggest strength is that its coefficients are directly interpretable, but only if you respect its assumptions and compare coefficients on standardised features.

The idea

Linear Regression finds the straight line, or the plane when there are many features, that best fits the cloud of data points:

y_hat = w1x1 + w2x2 + ... + wn*xn + b

Each w is the weight (coefficient) of one feature and b is the intercept. The model learns the set of w and b that makes the error as small as possible, usually measured by the mean squared error (MSE).

Reading the coefficients

This is where Linear Regression stands out. Few models give such a clear explanation.

Suppose a house-price model, with price in billion VND, gives the feature area a coefficient of 0.05 (billion VND per square metre). The interpretation is: holding all other factors fixed, each additional square metre raises the predicted price by 0.05 billion VND (50 million VND).

"Holding all other factors fixed" matters: in a multiple regression, each coefficient describes the effect of one feature while the others stay the same.

In scikit-learn:

from sklearn.linear_model import LinearRegression
model = LinearRegression()
model.fit(x_train, y_train)
print(model.coef_)        # weight of each feature
print(model.intercept_)   # intercept

Assumptions to know

  1. The relationship between the features and the output is linear.
  2. The observations are independent of each other.
  3. The errors have constant variance (they do not grow or shrink systematically across the range of predictions).

If the real relationship is a clear curve, a straight line cannot follow it. The model will underfit no matter how much data you collect, because more data does not change the shape of the model. Plotting the data first helps you spot this.

Compare coefficients only after standardising

A coefficient depends on the unit of measurement. If a length column is converted from metres to centimetres, its values become 100 times larger and its coefficient becomes 100 times smaller, yet the meaning of the feature has not changed at all.

So a large coefficient does not mean an important feature. To compare the importance of features, first standardise them to the same scale (for example with StandardScaler, which gives each feature mean 0 and standard deviation 1). After that, a larger absolute coefficient does mean a stronger effect per standard deviation.

Worked example

A toy house-price dataset (made-up numbers) with area in square metres and rooms, price in billion VND:

import pandas as pd
from sklearn.linear_model import LinearRegression

data = pd.DataFrame({
    "area":  [50, 60, 70, 80, 100],
    "rooms": [1, 2, 2, 3, 3],
    "price": [2.6, 3.2, 3.6, 4.2, 5.1],
})
x = data[["area", "rooms"]]
y = data["price"]

model = LinearRegression().fit(x, y)
print(model.coef_.round(3), round(model.intercept_, 3))

x_cm = x.assign(area=x["area"] * 10000)   # area in square centimetres
model_cm = LinearRegression().fit(x_cm, y)
print(model_cm.coef_)

What to expect and how to read it:

  1. The first model prints two coefficients: roughly 0.04 to 0.05 billion per square metre for area, plus a coefficient for rooms. Read the area weight as "each extra square metre adds about that much, with rooms held fixed".
  2. After converting area to square centimetres, the predictions are identical, but the area coefficient becomes 10,000 times smaller. Same meaning, very different number.
  3. Lesson: never rank features by raw coefficient size; standardise first.

Common mistakes

  • Reading a large coefficient as an important feature before standardising. A column in dong and a column in billions of dong give coefficients nine orders of magnitude apart, which says nothing about importance.
  • Dropping "holding other factors fixed". A coefficient in a multiple regression is not the total effect of a feature on its own.
  • Adding more data to fix underfitting on a curved relationship. The straight-line form is the limit; you need a different model or new features.
  • Ignoring the assumptions. Dependent observations or errors that grow with the prediction make the coefficients less trustworthy.

Summary

  • Model: y_hat = w1x1 + ... + wnxn + b, learned by minimising the error (usually MSE).
  • A coefficient is the change in the prediction per unit of a feature, with other features held fixed.
  • model.coef_ and model.intercept_ show the learned weights.
  • Assumptions: linear relationship, independent observations, constant error variance.
  • A curved relationship leads to underfitting regardless of data size.
  • Compare coefficients only after standardising features.

Flashcards · Linear Regression

8 words
WordMeaningExample
  • Coefficient (weight)

    The change in the predicted output for a one-unit increase in a feature, with other features held fixed.

    area coefficient 0.05: +0.05 billion VND per extra m2.

  • Intercept

    The model's prediction when every feature equals zero.

    model.intercept_

  • model.coef_

    The scikit-learn attribute holding one learned weight per feature of a linear model.

    print(model.coef_) -> [0.045 0.12]

  • Linearity assumption

    The assumption that the output changes in a straight-line way with the features.

    Violated if the scatter plot shows a clear curve.

  • Constant error variance

    The assumption that prediction errors have roughly the same spread across all predicted values.

    Errors should not grow larger for expensive houses.

  • Underfitting

    A model too simple to capture the real pattern, so it performs poorly even on training data.

    A straight line fitted to a clearly curved relationship.

  • Unit dependence of coefficients

    A coefficient's size changes with the unit of its feature, even though its meaning does not.

    Metres to centimetres makes the coefficient 100 times smaller.

  • Standardise before comparing

    Put features on the same scale before using coefficient sizes to compare feature importance.

    StandardScaler, then compare abs(model.coef_).