1.11 Lab: Handling Missing Values and Pipelines
[markdown] content available only on the platform
Flashcards · 1.11 Lab: Handling Missing Values and Pipelines
SimpleImputer — scikit-learn class in sklearn.impute that fills missing values using a chosen strategy.
SimpleImputer(strategy="mean").fit(x_train)
strategy="median" — Imputes with the column median, which is not pulled upwards or downwards by outliers.
Values 1, 2, 9: the median is 2, the mean is 4.
strategy="most_frequent" — Imputes with the most common value; works for categorical columns.
["IT", "IT", nan, "Econ"] -> nan becomes "IT"
statistics_ — Fitted imputer attribute holding the fill value learned for each column.
imp.statistics_ -> [4.]
Fit on train only — Learn imputation and scaling statistics from training data; test and new data only get transform.
imp.fit(x_train); imp.transform(x_test)
fit_transform on test (mistake) — Refitting on the test set fills gaps with test data you will not have at deployment time.
Test [nan, 20] becomes [20, 20] instead of [4, 20].
Pipeline (imputer + scaler) — Chains imputation and scaling so both are fitted on train in order and applied consistently.
Pipeline([("imputer", SimpleImputer()), ("scaler", StandardScaler())])
ColumnTransformer — Applies separate pipelines to numerical and categorical columns of one table.
("cat", Pipeline([...OneHotEncoder()]), ["major"])
get_feature_names_out() — Lists the output column names produced by a fitted transformer.
['num__age' 'cat__major_Econ' 'cat__major_IT']
Placeholder missing values — Markers like "?" or "N/A" are not recognised as missing; convert them to NaN first.
df.replace("?", np.nan)
[file] /api/lesson-media/files/9b417aa2-5e9f-4ae3-a9db-2a3b7f03f372-1794890093151-04269fdcb529906d.zip
