Turning my ex girlfriend into an AI because I miss her...

Summary

The creator builds a small character-level language model intended to imitate messages from his ex-girlfriend. He presents the project as a personal experiment rather than a formal tutorial, noting that he is not an expert and that much of the coding process is improvised.

Data and model

• He collected messages from Discord and TikTok, then wrote simple scripts to clean and combine them. The resulting dataset contains about 1,600 lines and 380,000 characters.
• A TikTok data export was missing much of their later conversation, including several months when they exchanged the most messages. He believes this limited the model’s potential.
• The model uses a character-level vocabulary of 131 symbols. The data is split into training and validation portions to monitor loss.
• He implements the model in Python with PyTorch, drawing on Attention Is All You Need and OpenAI’s GPT-3 paper. The architecture includes token and positional embeddings, self-attention and multi-head attention, feed-forward layers, residual connections, layer normalization, and later, dropout.
• He adds checkpoint saving and loading, a time-based training loop, and a separate chatbot script that generates text from the user’s input. He also configures the code to use a GPU when available.

Training and results

• An early, very small model produced mostly gibberish, though it sometimes reproduced names, emojis, and other recognizable patterns.
• After expanding the model and training it overnight on a GPU for roughly eight hours and nearly 12,000 steps, the creator reports a final loss of about 0.0728.
• In recorded tests, the chatbot sometimes produced exchanges that the creator felt closely resembled real conversations. These are informal examples, not an independent evaluation. The model could still respond incoherently, miss conversational context, or generate excessively long replies.
• Longer generations were slow, especially on CPU. The creator also encountered GPU heat and noise during overnight training and used additional fans to cool the computer.

Guides and sources

This is best understood as a long, informal coding walkthrough and project diary, not a step-by-step instructional guide. The creator says he does not explain the implementation systematically.

The project is based primarily on Attention Is All You Need and the GPT-3 paper. The creator notes that links to the papers are provided with the video.

Main speaker and technical sources

• Main speaker: Xander, from the JustZando channel.
• Technical sources referenced: Attention Is All You Need and OpenAI’s GPT-3 paper.