You are viewing a public preview
Sign in to save your progress, do assignments, join discussions and get feedback.
After this unit you will: be able to list the order of the nine steps of a complete machine learning project.
By the end of this lesson you can:
Key idea: Almost every machine learning project follows the same nine-step pipeline, from collecting data to deploying the model. Module 1 covers the data steps; the modelling steps come in Modules 2 and 3.
The lecture brings all of Module 1 together in one diagram. Read it by following the arrows: the top row runs left to right, the middle row right to left, and the bottom row left to right again.
| # | Step | What you do |
|---|---|---|
| 1 | Data collection | Gather raw data from databases, APIs, sensors, surveys or other sources |
| 2 | Statistics | Basic statistical analysis to understand structure: mean, median, range |
| 3 | Data preprocessing | Handle missing values, duplicates, wrong formats and outliers; scale numerical features; encode categorical features |
| 4 | Data visualization | Plot charts to explore relationships and reveal patterns or problems that numbers hide |
| 5 | Model selection | Choose which algorithms to try, based on problem type, data and goal |
| 6 | Model training | The algorithm learns from the training data through the loop: prediction, loss, update, repeat |
| 7 | Model validation | Check performance on the validation set and adjust the approach if needed |
| 8 | Model testing | Get a final, honest score on the held-out test set |
| 9 | Model deployment | Put the finished model into production so it makes real predictions in an application or business process |
The first four steps are what Module 1 has been about: understanding what type of data you have, what type each feature is, and how to clean and prepare it. Notice the order. You cannot sensibly choose a model while the data is still dirty, and preprocessing decisions (for example, which scaler to use or how to treat an outlier) depend on the statistics you calculated first.
The diagram groups model selection, training, validation, testing and deployment as the modelling phase. They connect back to earlier lessons:
Module 2 starts building actual models; Module 3 puts the whole pipeline together end to end on a real dataset.
A real-estate company wants to predict the selling price of apartments in Hanoi. Walk it through the pipeline.
price and square_feet; count how many rows have missing values.square_feet, check a suspicious 500 billion VND listing, scale numerical columns, one-hot encode district and ordinal-encode tier.If the test result in step 8 is poor, the team goes back to earlier steps, but it must then hold out fresh data for the next honest test.
Flashcards · 1.10 Machine Learning Pipeline
9 wordsThe ordered sequence of steps in an ML project, from data collection to model deployment.
Gathering raw data from sources such as databases, APIs, sensors or surveys.
Basic statistical analysis to understand the data's structure before cleaning it.
Cleaning and transforming data: missing values, duplicates, formats, outliers, scaling and encoding.
Plotting data to explore relationships and reveal patterns or problems numbers may hide.
Choosing which algorithms to try based on the problem type, the data and the goal.
Checking and tuning model performance on the validation set; it sits between training and testing.
Measuring final performance once on the held-out test set for an honest estimate.
Putting the finished, tested model into production so it makes real predictions.