You are viewing a public preview
Sign in to save your progress, do assignments, join discussions and get feedback.
In Period 1, we discussed why data quality matters. Today, we will take the next step and work with a sales table in Pandas. By the end of this lesson, you should be able to load a file, inspect its structure, select columns, and filter transactions. We will use the same small example throughout the lesson.
By the end of this lesson you can:
head(), shape, info() and dtypes.KeyError.loc, & and |.Key idea: Loading a file successfully does not prove the data are good. Look at the structure first, then select and filter with clear, checkable code.
The lecture uses five invented transactions with four columns: transaction_id, region, quantity and unit_price (in Vietnamese dong). Each row is one transaction for one product category. For example, T03 is a North transaction with a quantity of 5 and a unit price of 80,000 dong. Column names and region values stay in English so that the same code works in every language version.
pandas has two basic structures:
df is the whole sales table.df["quantity"] (one pair of brackets) returns a Series. df[["quantity"]] (a list inside the brackets) returns a DataFrame with one column. The values look similar, but the types differ. The row index identifies rows inside the object; it is separate from the transaction_id column.
import pandas as pd
raw = pd.read_csv("sales.csv")
df = raw.copy()
read_csv returns a DataFrame. We store it in raw and work on a copy called df, which follows the habit from Period 1: preserve the original data. Notes on loading:
sep=";".FileNotFoundError usually means the file name or path is wrong. Put the file in the notebook's working folder.pd.read_excel("sales.xlsx", sheet_name="Sales", engine="openpyxl"). The sheet name must match the workbook. If openpyxl is missing in a notebook, run %pip install openpyxl.You only need one route (CSV or Excel). Either way, the example should give 5 rows and 4 columns.
Take a quick look before calculating anything:
| Code | Kind | What it shows |
|---|---|---|
df.head() / df.head(3) | method (needs parentheses) | First 5 rows by default, or the first 3 |
df.shape | attribute (no parentheses) | (rows, columns), here (5, 4) |
df.info() | method | Columns, data types and non-null counts |
df.dtypes | attribute | Data type of each column |
quantity and unit_price should be numeric before you compare or calculate with them. Text columns may appear as object, str or string, depending on the pandas version and loading options. Do not judge a type only by how a value looks. A correct preview of five rows does not prove that every row of a large file is correct.
To keep only some columns, pass a list of names: df[["transaction_id", "quantity"]]. The result keeps all rows (shape (5, 2)), and the order of the names controls the column order. It does not change df. Names must match exactly, including spaces and letter case; if you get a KeyError, print df.columns and compare.
To filter rows, build a Boolean mask: one True/False value per row.
df["quantity"] >= 3 keeps rows with quantity 3 or more. With >, a row with exactly 3 would be dropped.& (AND) or | (OR), and put each comparison in parentheses. Do not use Python's and/or with Series.== compares; a single = assigns.df.loc[rows, columns] selects rows first, columns second.The lecture's practice task: find South transactions with a quantity of at least 2, showing only the ID and unit price. The quantities are 2, 1, 5, 3, 4 for T01 to T05; T03 (North, 80,000) and T05 (South, 90,000) come from the lecture, and the other prices are illustrative.
import pandas as pd
raw = pd.DataFrame({
"transaction_id": ["T01", "T02", "T03", "T04", "T05"],
"region": ["North", "South", "North", "Central", "South"],
"quantity": [2, 1, 5, 3, 4],
"unit_price": [60000, 70000, 80000, 50000, 90000],
})
df = raw.copy()
print(df.shape) # (5, 4)
mask = (df["region"] == "South") & (df["quantity"] >= 2)
cols = ["transaction_id", "unit_price"]
result = df.loc[mask, cols]
print(result)
print(result.shape) # (1, 2)
Expected output: one row, T05 with unit price 90000, and shape (1, 2). T02 is also in the South but its quantity is only 1. In the same way, (df["region"] == "North") & (df["quantity"] >= 3) keeps only T03: T01 is North but has quantity 2, and T04 and T05 are in other regions.
df.shape(). shape is an attribute, so it has no parentheses; head() and info() are methods and need them.and instead of &. With pandas Series use & and |, and wrap each comparison in parentheses, or you will get an error or the wrong rows.df["transaction_id", "quantity"] raises a KeyError; you need df[["transaction_id", "quantity"]].head(). Five good-looking rows say nothing about the rest of the file; check info() and later quality checks.pd.read_csv (or pd.read_excel) loads a DataFrame; keep raw and work on df = raw.copy().head(), shape, info() and dtypes before calculating.df.columns when you see a KeyError.&/|, parentheses and loc[rows, columns].Flashcards · Loading and Inspecting Data with Pandas
10 wordsA pandas table with labelled rows and columns.
A one-dimensional pandas object with an index, such as a single column.
Reads a text file of comma-separated (or other separated) values into a DataFrame.
Creates an independent copy so that work does not change the original data.
An attribute giving the number of rows and columns as a tuple.
A method that lists columns, data types and non-null counts.
The error pandas raises when a column name in your code does not exist in the DataFrame.
A Series of True/False values, one per row, used to select matching rows.
Selects rows and columns by label or mask, rows first and columns second.
Element-wise AND and OR for combining conditions on pandas Series.