Đại học Kinh tế Quốc dân – National Economics UniversityMICRO4ASIA
🏆 Arena
Sign inSign up
MICRO4ASIAThe six micro-credentials

Courses are delivered on the Limio learning platform. Limio

← Fundamental of Machine Learning

You are viewing a public preview

Sign in to save your progress, do assignments, join discussions and get feedback.

Sign in
Module 1: Data Preparation and Model Foundations

1.10 Machine Learning Pipeline

Print / Save as PDF

1.10 Machine Learning Pipeline

After this unit you will: be able to list the order of the nine steps of a complete machine learning project.

Key points

  1. Data collection
  2. Statistics
  3. Data preprocessing
  4. Data visualization
  5. Model selection
  6. Model training
  7. Model validation
  8. Model testing
  9. Model deployment
  • The diagram follows the arrows: the top row runs left to right, dropping to the middle row it runs right to left, and dropping to the bottom row it runs left to right again.
  • You cannot select a model while the data is still dirty, and you should not deploy before testing.

Study notes

By the end of this lesson you can:

  • List the nine steps of the machine learning pipeline in the correct order.
  • Explain what happens in each step and which earlier lessons it uses.
  • Identify which steps form the modelling phase.
  • Tell the difference between model validation and model testing.

Key idea: Almost every machine learning project follows the same nine-step pipeline, from collecting data to deploying the model. Module 1 covers the data steps; the modelling steps come in Modules 2 and 3.

The nine steps

The lecture brings all of Module 1 together in one diagram. Read it by following the arrows: the top row runs left to right, the middle row right to left, and the bottom row left to right again.

#StepWhat you do
1Data collectionGather raw data from databases, APIs, sensors, surveys or other sources
2StatisticsBasic statistical analysis to understand structure: mean, median, range
3Data preprocessingHandle missing values, duplicates, wrong formats and outliers; scale numerical features; encode categorical features
4Data visualizationPlot charts to explore relationships and reveal patterns or problems that numbers hide
5Model selectionChoose which algorithms to try, based on problem type, data and goal
6Model trainingThe algorithm learns from the training data through the loop: prediction, loss, update, repeat
7Model validationCheck performance on the validation set and adjust the approach if needed
8Model testingGet a final, honest score on the held-out test set
9Model deploymentPut the finished model into production so it makes real predictions in an application or business process

The data phase (steps 1 to 4)

The first four steps are what Module 1 has been about: understanding what type of data you have, what type each feature is, and how to clean and prepare it. Notice the order. You cannot sensibly choose a model while the data is still dirty, and preprocessing decisions (for example, which scaler to use or how to treat an outlier) depend on the statistics you calculated first.

The modelling phase (steps 5 to 9)

The diagram groups model selection, training, validation, testing and deployment as the modelling phase. They connect back to earlier lessons:

  • Training is the loop from lesson 1.1: data, model, prediction, loss function, update.
  • Validation uses the validation set from lesson 1.6, the "practice quiz" for comparing algorithms and settings.
  • Testing uses the test set from lesson 1.6, the "final exam" that is touched once.
  • Deployment comes only after testing. A model that has not been honestly tested should not be used to make real decisions.

Module 2 starts building actual models; Module 3 puts the whole pipeline together end to end on a real dataset.

Worked example

A real-estate company wants to predict the selling price of apartments in Hanoi. Walk it through the pipeline.

  1. Data collection: export past listings and final sale prices from the company database.
  2. Statistics: compute the mean, median and range of price and square_feet; count how many rows have missing values.
  3. Data preprocessing: remove duplicate listings, standardise date and price formats, impute missing square_feet, check a suspicious 500 billion VND listing, scale numerical columns, one-hot encode district and ordinal-encode tier.
  4. Data visualization: plot price against area to see the trend and spot remaining outliers.
  5. Model selection: the output is a number, so this is regression; shortlist a few regression algorithms.
  6. Model training: fit each candidate on the train set.
  7. Model validation: compare the candidates on the validation set and tune their settings; keep the best one.
  8. Model testing: evaluate the chosen model once on the test set to get an honest error figure for management.
  9. Model deployment: connect the model to the listing website so agents get a suggested price for each new apartment.

If the test result in step 8 is poor, the team goes back to earlier steps, but it must then hold out fresh data for the next honest test.

Common mistakes

  • Jumping straight to model training. Skipping statistics, preprocessing and visualization usually means training on dirty data.
  • Treating validation and testing as the same step. Validation is for comparing and tuning; testing is the final, one-time check.
  • Deploying before testing. Without a test on unseen data you do not know how the model will behave in the real world.
  • Reading the diagram row by row left to right. The middle row runs right to left; follow the arrows.

Summary

  • The pipeline has nine steps: collection, statistics, preprocessing, visualization, selection, training, validation, testing, deployment.
  • Steps 1 to 4 are the data phase covered in Module 1.
  • Steps 5 to 9 form the modelling phase.
  • Validation tunes and compares; testing gives the final honest score; deployment comes last.
  • This pipeline is the master template for almost any machine learning project.

Flashcards · 1.10 Machine Learning Pipeline

9 words
WordMeaningExample
  • Machine learning pipeline

    The ordered sequence of steps in an ML project, from data collection to model deployment.

    Nine steps in the lecture's diagram.

  • Data collection

    Gathering raw data from sources such as databases, APIs, sensors or surveys.

    Exporting past sales from a company database.

  • Statistics (pipeline step)

    Basic statistical analysis to understand the data's structure before cleaning it.

    Mean, median and range of price.

  • Data preprocessing

    Cleaning and transforming data: missing values, duplicates, formats, outliers, scaling and encoding.

    Impute square_feet, one-hot encode city.

  • Data visualization

    Plotting data to explore relationships and reveal patterns or problems numbers may hide.

    A scatter plot of price against area.

  • Model selection

    Choosing which algorithms to try based on the problem type, the data and the goal.

    Numeric output, so shortlist regression models.

  • Model validation

    Checking and tuning model performance on the validation set; it sits between training and testing.

    Compare three candidate models on validation data.

  • Model testing

    Measuring final performance once on the held-out test set for an honest estimate.

    Report the test error before launch.

  • Model deployment

    Putting the finished, tested model into production so it makes real predictions.

    Suggesting prices on a listing website.