Đại học Kinh tế Quốc dân – National Economics UniversityMICRO4ASIA
🏆 Arena
Sign inSign up
MICRO4ASIAThe six micro-credentials

Courses are delivered on the Limio learning platform. Limio

← Data Cleaning and Preprocessing with Python

You are viewing a public preview

Sign in to save your progress, do assignments, join discussions and get feedback.

Sign in
Module 1 — Explore and Understand the Data

Period 2 · Loading and Inspecting Data with Pandas

Print / Save as PDF
Lecture video — Period 2: Loading and Inspecting Data with Pandas

Loading CSV and Excel files, checking structure with head, shape and dtypes, selecting columns and filtering transactions.

Lesson overview

In Period 1, we discussed why data quality matters. Today, we will take the next step and work with a sales table in Pandas. By the end of this lesson, you should be able to load a file, inspect its structure, select columns, and filter transactions. We will use the same small example throughout the lesson.

Study notes

By the end of this lesson you can:

  • Load a CSV or Excel file into a pandas DataFrame and keep a separate working copy.
  • Explain the difference between a DataFrame and a Series, and between an attribute and a method.
  • Inspect a table with head(), shape, info() and dtypes.
  • Select columns with a list of names and fix a KeyError.
  • Filter rows with one or two conditions using loc, & and |.

Key idea: Loading a file successfully does not prove the data are good. Look at the structure first, then select and filter with clear, checkable code.

The teaching dataset and pandas structures

The lecture uses five invented transactions with four columns: transaction_id, region, quantity and unit_price (in Vietnamese dong). Each row is one transaction for one product category. For example, T03 is a North transaction with a quantity of 5 and a unit price of 80,000 dong. Column names and region values stay in English so that the same code works in every language version.

pandas has two basic structures:

  • A DataFrame is a table with rows and columns. Here df is the whole sales table.
  • A Series is one-dimensional, with an index.

df["quantity"] (one pair of brackets) returns a Series. df[["quantity"]] (a list inside the brackets) returns a DataFrame with one column. The values look similar, but the types differ. The row index identifies rows inside the object; it is separate from the transaction_id column.

Loading the data

import pandas as pd
raw = pd.read_csv("sales.csv")
df = raw.copy()

read_csv returns a DataFrame. We store it in raw and work on a copy called df, which follows the habit from Period 1: preserve the original data. Notes on loading:

  • The example file uses commas and has a header row. If your file uses semicolons, add sep=";".
  • A FileNotFoundError usually means the file name or path is wrong. Put the file in the notebook's working folder.
  • For Excel, use pd.read_excel("sales.xlsx", sheet_name="Sales", engine="openpyxl"). The sheet name must match the workbook. If openpyxl is missing in a notebook, run %pip install openpyxl.

You only need one route (CSV or Excel). Either way, the example should give 5 rows and 4 columns.

Inspecting the structure

Take a quick look before calculating anything:

CodeKindWhat it shows
df.head() / df.head(3)method (needs parentheses)First 5 rows by default, or the first 3
df.shapeattribute (no parentheses)(rows, columns), here (5, 4)
df.info()methodColumns, data types and non-null counts
df.dtypesattributeData type of each column

quantity and unit_price should be numeric before you compare or calculate with them. Text columns may appear as object, str or string, depending on the pandas version and loading options. Do not judge a type only by how a value looks. A correct preview of five rows does not prove that every row of a large file is correct.

Selecting columns and filtering rows

To keep only some columns, pass a list of names: df[["transaction_id", "quantity"]]. The result keeps all rows (shape (5, 2)), and the order of the names controls the column order. It does not change df. Names must match exactly, including spaces and letter case; if you get a KeyError, print df.columns and compare.

To filter rows, build a Boolean mask: one True/False value per row.

  • df["quantity"] >= 3 keeps rows with quantity 3 or more. With >, a row with exactly 3 would be dropped.
  • Combine conditions with & (AND) or | (OR), and put each comparison in parentheses. Do not use Python's and/or with Series.
  • == compares; a single = assigns.
  • df.loc[rows, columns] selects rows first, columns second.

Worked example

The lecture's practice task: find South transactions with a quantity of at least 2, showing only the ID and unit price. The quantities are 2, 1, 5, 3, 4 for T01 to T05; T03 (North, 80,000) and T05 (South, 90,000) come from the lecture, and the other prices are illustrative.

import pandas as pd

raw = pd.DataFrame({
    "transaction_id": ["T01", "T02", "T03", "T04", "T05"],
    "region": ["North", "South", "North", "Central", "South"],
    "quantity": [2, 1, 5, 3, 4],
    "unit_price": [60000, 70000, 80000, 50000, 90000],
})
df = raw.copy()

print(df.shape)                      # (5, 4)
mask = (df["region"] == "South") & (df["quantity"] >= 2)
cols = ["transaction_id", "unit_price"]
result = df.loc[mask, cols]
print(result)
print(result.shape)                  # (1, 2)

Expected output: one row, T05 with unit price 90000, and shape (1, 2). T02 is also in the South but its quantity is only 1. In the same way, (df["region"] == "North") & (df["quantity"] >= 3) keeps only T03: T01 is North but has quantity 2, and T04 and T05 are in other regions.

Common mistakes

  • Writing df.shape(). shape is an attribute, so it has no parentheses; head() and info() are methods and need them.
  • Using and instead of &. With pandas Series use & and |, and wrap each comparison in parentheses, or you will get an error or the wrong rows.
  • Forgetting the inner list. df["transaction_id", "quantity"] raises a KeyError; you need df[["transaction_id", "quantity"]].
  • Trusting head(). Five good-looking rows say nothing about the rest of the file; check info() and later quality checks.

Summary

  • pd.read_csv (or pd.read_excel) loads a DataFrame; keep raw and work on df = raw.copy().
  • One pair of brackets gives a Series; a list of names gives a DataFrame.
  • Inspect with head(), shape, info() and dtypes before calculating.
  • Select columns with a list; check df.columns when you see a KeyError.
  • Filter with Boolean masks, &/|, parentheses and loc[rows, columns].
  • Loading a file is not the same as confirming its quality.

Flashcards · Loading and Inspecting Data with Pandas

10 words
WordMeaningExample
  • DataFrame

    A pandas table with labelled rows and columns.

    `df = pd.read_csv("sales.csv")`

  • Series

    A one-dimensional pandas object with an index, such as a single column.

    `df["quantity"]` returns a Series.

  • pd.read_csv()

    Reads a text file of comma-separated (or other separated) values into a DataFrame.

    `pd.read_csv("sales.csv", sep=";")` for a semicolon file.

  • df.copy()

    Creates an independent copy so that work does not change the original data.

    `df = raw.copy()`

  • df.shape

    An attribute giving the number of rows and columns as a tuple.

    `(5, 4)` means 5 rows and 4 columns.

  • df.info()

    A method that lists columns, data types and non-null counts.

    Shows that each column has 5 non-null values.

  • KeyError

    The error pandas raises when a column name in your code does not exist in the DataFrame.

    Check `df.columns` for `"qty"` versus `"quantity"`.

  • Boolean mask

    A Series of True/False values, one per row, used to select matching rows.

    `mask = df["quantity"] >= 3`

  • df.loc[]

    Selects rows and columns by label or mask, rows first and columns second.

    `df.loc[mask, ["transaction_id", "unit_price"]]`

  • & and | operators

    Element-wise AND and OR for combining conditions on pandas Series.

    `(df["region"] == "North") & (df["quantity"] >= 3)`