LearnGPT
Text and tokens 01 / 42
01

Visual explanation

A language model starts from a sequence of text. Before tokenization or neural networks exist, the program must locate the dataset, decode its bytes correctly, and verify that it contains readable material.

A text file becomes the learning sequence

Follow the steps in order. Select any card to open its explanation and matching code.

Step 1 of 3
File bytes [84,104,101,32,99,97,116,…]
decode as UTF-8
Python string “The cat sleeps here.”
Step 01

input.txt

Long story short: The text file is the raw dataset.

Its absolute path makes loading reliable from any terminal folder. The visual highlights input.txt at this point. This prepares the next step: read_text().

02

Code

Load a plain-text dataset and inspect the exact characters the model will learn from.

PROJECT_DIR = Path(__file__).resolve().parents[2]
DATASET_PATH = PROJECT_DIR / "data" / "study_sample.txt"
text = DATASET_PATH.read_text(encoding="utf-8")
print("Number of characters:", len(text))
print(text[:500])
From study/lessons/01_read_text.py
03

Code syntax and logic

  1. Step 01
    Code reference 
    PROJECT_DIR = Path(__file__).resolve().parents[2]
    Explanation

    represents the current script; makes its path absolute, and reaches the repository root without depending on the terminal's current directory.

  2. Step 02
    Code reference 
    DATASET_PATH = PROJECT_DIR / "data" / "study_sample.txt"
    Explanation

    The / operator on Path objects joins path components portably on macOS, Linux, and Windows.

  3. Step 03
    Code reference 
    text = DATASET_PATH.read_text(encoding="utf-8")
    Explanation

    opens, decodes, reads, and closes the file. The explicit encoding makes the byte-to-character conversion deterministic.

  4. Step 04
    Code reference 
    print("Number of characters:", len(text))
    Explanation

    counts Python Unicode characters rather than encoded UTF-8 bytes.

  5. Step 05
    Code reference 
    print(text[:500])
    Explanation

    is a non-mutating slice from index 0 up to, but not including, index 500; it provides a short inspection sample without changing text.