Testing

Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.

Brian W. Kernighan

Testing gives us a way to check that code behaves as expected. In this lesson we will use pytest with the gecs-venv book-analysis project.

The basic development loop is:

%%{init: {"flowchart": {"curve": "basis"}, "theme": "base", "themeVariables": {"primaryColor": "#dbeafe", "primaryBorderColor": "#2563eb", "lineColor": "#64748b", "fontFamily": "Arial"}}}%%
flowchart LR
    A(Write code):::step edge1@==> B(Run tests):::step
    edge1@{ animate: slow }
    B edge2@==> C(Read failures):::step
    edge2@{ animate: slow }
    C edge3@==> D(Fix code):::step
    edge3@{ animate: slow }
    D edge4@==> A
    edge4@{ animate: slow }
    classDef step fill:#cfe9e5,stroke:transparent,color:#000

It is a big expectation to adhere to this loop, but it is a good goal. Let’s start by writing a few tests for the existing code.

Work on a branch

Before changing the project, create a new Git branch called testing. A branch gives you a separate line of work, so you can experiment without changing the existing code on main.

First, clone the example repository and open it in VS Code:

git clone https://github.com/igorsdub/gecs-venv
cd gecs-venv
code .

In VS Code:

  1. Open the Source Control panel.
  2. Select the current branch name, usually main, in the Source Control view or status bar.
  3. Choose Create new branch.
  4. Enter testing and press Enter.

You can also open the Command Palette with Cmd/Ctrl+Shift+P, type Git: Create Branch, and select that command. Enter testing when VS Code asks for the branch name.

From the gecs-venv directory, run:

git switch -c testing
git branch --show-current

The second command should print testing.

Set up the project

The project contains books, analysis scripts, and generated word-count files and plots:

books/                         input novels
scripts/count_words.py         extracts and counts words
scripts/get_summary.py         prints book metadata
scripts/plot_counts.py         creates plots
scripts/run_book_analysis.py   combines the workflow

The project uses a virtual environment to isolate its dependencies. We will use the uv command from the gecs-venv project to manage the environment. If you have not intialized the environment yet, run:

uv sync

Right now, we only missing pytest for testing. Add pytest for development:

uv add --dev pytest

uv run runs commands inside the project environment:

uv run pytest

Unit tests

A unit test checks one small piece of behavior on its own. Each script in this project contains one or more units, such as a function that extracts words or writes a count file. We can write several tests for each unit.

The first units we will test are:

  • extract_words() in count_words.py;
  • count_word_frequencies() in count_words.py; and
  • parse_counts_file() in plot_counts.py.

The relationship looks like this:

%%{init: {"flowchart": {"curve": "basis"}, "theme": "base", "themeVariables": {"primaryColor": "#ecfdf5", "primaryBorderColor": "#16a34a", "lineColor": "#64748b", "fontFamily": "Arial"}}}%%
flowchart LR
    C[count_words.py]:::unit --> T[tests/test_count_words.py]:::unit
    P[plot_counts.py]:::unit --> U[tests/test_plot_counts.py]:::unit
    T --> T1[lowercasing]:::unit
    T --> T2[punctuation]:::unit
    T --> T3[file output]:::unit
    U --> U1[count-file parsing]:::unit
    classDef unit fill:#e6dff1,stroke:transparent,color:#000

Test-driven development

Smash first, analyze later.

Dating advice from a CERN researcher

This is also a useful way to think about test-driven development, or TDD. Before writing the implementation, write a test that describes the behavior you want. The test should fail at first because the code does not do that thing yet. Then write the smallest amount of code needed to make the test pass.

write a failing test β†’ write code β†’ run the test β†’ improve the code

For example, the current regular expression does not support accented words. Write a test for the behavior we would like to add:

from scripts.count_words import extract_words


def test_extract_words_handles_accented_words():
    assert extract_words("cafΓ©") == ["cafΓ©"]

Add this test to tests/test_count_words.py and run it. It should fail because the current pattern only matches ASCII letters and numbers. That failure is useful: it tells us what behavior to implement. After the test passes, we can improve the implementation while keeping the test as a safety net.

Now return to the simpler tests below. They describe behavior that the current implementation already supports.

Write a first test

The extract_words() function turns text into lowercase words:

import re


def extract_words(text):
    return re.findall(r"\b[a-z0-9]+\b", text.lower())

Create a tests directory and a file named tests/test_count_words.py:

from scripts.count_words import extract_words


def test_extract_words_lowercases_text():
    result = extract_words("The QUICK brown fox")

    assert result == ["the", "quick", "brown", "fox"]

The regular expression explains what the function accepts:

  • text.lower() makes matching case-insensitive;
  • \b identifies word boundaries; and
  • [a-z0-9]+ matches one or more ASCII letters or numbers.

Because the pattern uses ASCII letters, it does not currently handle accented words such as cafΓ©. That is a limitation we can document or fix later.

The assert statement compares the result with the behavior we expect. Pytest finds files named test_*.py and functions named test_* automatically.

Run the test from the repository root:

uv run pytest

Try changing one expected word and run the test again. The failure report shows where the test failed and compares the actual and expected values. Restore the correct expectation afterward.

Test file output safely

count_word_frequencies() reads a book and writes a count file. A test should not overwrite a real file in counts/. The pytest tmp_path fixture gives each test a fresh temporary directory.

Add this test to tests/test_count_words.py:

from scripts.count_words import count_word_frequencies


def test_count_word_frequencies_writes_sorted_counts(tmp_path):
    input_path = tmp_path / "example.txt"
    output_path = tmp_path / "example.tsv"
    input_path.write_text("Red blue red!", encoding="utf-8")

    count_word_frequencies(input_path, output_path)

    assert output_path.read_text(encoding="utf-8") == (
        "      2 red\n"
        "      1 blue\n"
    )

The test creates its own input file, runs the function, and checks the generated output. The temporary directory is removed after the test.

Test several examples

Sometimes one function should work for several different inputs. Instead of writing a separate test function for every example, give pytest a list of examples:

import pytest

from scripts.count_words import extract_words


@pytest.mark.parametrize(
    ("text", "expected"),
    [
        ("HELLO", ["hello"]),
        ("one, two", ["one", "two"]),
        ("version 3", ["version", "3"]),
    ],
)
def test_extract_words_examples(text, expected):
    assert extract_words(text) == expected

Pytest runs this test once for each row. If one example fails, pytest identifies that example in the failure report.

Test another unit

The plotting script has a parse_counts_file() function that reads a count file. It can be tested without opening a browser or creating a plot.

Create tests/test_plot_counts.py:

from scripts.plot_counts import parse_counts_file


def test_parse_counts_file_reads_counts(tmp_path):
    counts_path = tmp_path / "counts.tsv"
    counts_path.write_text(
        "      4 the\n"
        "      2 book\n",
        encoding="utf-8",
    )

    words, counts = parse_counts_file(counts_path)

    assert words == ["the", "book"]
    assert counts == [4, 2]

Now we have several tests for individual units. They are fast and help us find which part of the project has a problem.

Integration testing

Unit tests check individual pieces. For example, count_word_frequencies() is tested as part of count_words.py, and parse_counts_file() is tested as part of plot_counts.py.

An integration test checks that the two scripts work together: the counting script creates a count file, and the plotting script uses that file to create a figure.

%%{init: {"flowchart": {"curve": "basis"}, "theme": "base", "themeVariables": {"primaryColor": "#fff7ed", "primaryBorderColor": "#ea580c", "lineColor": "#64748b", "fontFamily": "Arial"}}}%%
flowchart LR
    U1[Unit test 1]:::part --> S1
    U2[Unit test 2]:::part --> S2
    U1 ~~~ U2
    subgraph integration[Integration test]
        S1[Script 1]:::part -->|Script 1 output| S2[Script 2]:::part
    end
    classDef part fill:#f2d6c9,stroke:transparent,color:#000

The diagram is misaligning the unit tests but I could not fix it 😒.

The unit tests sit outside the Integration test box and check one script each. The Integration test evaluates both scripts together, with the output of Script 1 becoming the input to Script 2.

Create tests/test_workflow.py:

from scripts.count_words import count_word_frequencies
from scripts.plot_counts import plot_word_counts


def test_small_book_workflow(tmp_path):
    book_path = tmp_path / "mini_book.txt"
    counts_path = tmp_path / "counts" / "mini_book.tsv"
    figure_path = tmp_path / "figures" / "mini_book.html"
    book_path.write_text("The cat sat. The cat slept.", encoding="utf-8")
    counts_path.parent.mkdir()
    figure_path.parent.mkdir()

    count_word_frequencies(book_path, counts_path)
    plot_word_counts(counts_path, figure_path)

    assert counts_path.exists()
    assert figure_path.exists()
    assert figure_path.with_suffix(".png").exists()

This test follows the same path as the real workflow: book β†’ count file β†’ plot. It calls plot_word_counts(), which uses Altair for the interactive HTML chart and vl-convert-python for PNG export, so it does not require a locally available browser.

Run the tests

Use these commands from the repository root:

# Run every test
uv run pytest

# Run one test file
uv run pytest tests/test_count_words.py

# Run tests with a matching name
uv run pytest -k extract_words

# Show each test name
uv run pytest -v

The important habit is to run the tests whenever you change the code. Tests make the development loop visible and help protect the behavior of the analysis.

When all tests pass, the testing branch is ready to be committed and reviewed.

Wrapping up

Save the work on the testing branch and ask for it to be reviewed before it becomes part of main.

In VS Code:

  1. Open Source Control and review the changes.
  2. Enter a message such as Add pytest examples and select Commit.
  3. Select Publish Branch or Sync Changes to push testing to GitHub.
  4. Open the repository on GitHub and choose Compare & pull request.
  5. Set main as the base branch and testing as the compare branch.
  6. Describe the lesson changes, then create the pull request.

The final shape of the work is:

%%{init: {"theme": "base", "themeVariables": {"git0": "#2563eb", "git1": "#db2777", "gitBranchLabel0": "#ffffff", "gitBranchLabel1": "#ffffff", "commitLabelColor": "#111827", "commitLabelBackground": "#f8fafc"}}}%%
gitGraph LR:
    commit id: "start" tag: "main"
    commit id: "setup"
    branch testing
    commit id: "tests"
    commit id: "fixes" type: HIGHLIGHT
    checkout main
    commit id: "main-work"
    merge testing

The pink testing branch contains the new work. The pull request reviews that work, and the merge commit brings it back into the blue main branch.

Summary

Pytest tests are ordinary Python functions with names beginning with test_. Assertions describe expected behavior, tmp_path provides safe temporary files, and one test can be run with several examples.

For this project, begin by testing individual functions. Then add an integration test to check that count_words.py creates the count-file output consumed by plot_counts.py.