Capstone Practice: Planning Preprocessing for a Real Dataset
[markdown] content available only on the platform
[markdown] content available only on the platform
Flashcards · Capstone Practice: Planning Preprocessing for a Real Dataset
Preprocessing plan — A column-by-column table stating each feature's type, how to impute it and how to scale or encode it.
tier: ordinal, most frequent, OrdinalEncoder.
Data audit — Checking a dataset for missing values, duplicates, wrong formats, outliers and low variance before modelling.
df.isna().sum(); df.duplicated().sum()
df.nunique() — Counts distinct values per column; helps spot near-constant columns and categorical features.
houses.nunique()
SimpleImputer — scikit-learn transformer that fills missing values with the mean, median, most frequent value or a constant.
SimpleImputer(strategy="median")
Per-column treatment — Choosing imputation and transformation separately for each feature according to its type.
Median plus StandardScaler for square_feet.
Fit on train only — All preprocessing statistics are learned from training rows and merely applied to other data.
prep.fit_transform(X_train); prep.transform(X_test)
ColumnTransformer — Runs a different preprocessing pipeline on each group of columns and joins the outputs.
("nom", OneHotEncoder(), ["city", "has_garden"])
Target separation — Keeping the target column out of the feature preprocessing so it is never scaled or encoded with inputs.
X = df.drop(columns="price"); y = df["price"]
