--- type: status: draft tags: [] created: TQ_show_depends_on: --- This week focuses on the conceptualisation of large language models. If you are interested in artificial intelligence more broadly, I recommend the course [FF:AI001 Elements of AI](FF:AI001 "https://is.muni.cz/predmet/phil/podzim2026/AI001") (or the [online course itself.](https://www.elementsofai.com/)). # How a Model is Built Training happens in three stages, and each one answers a different question: what does the model *know*, what can it *do*, and how should it *behave*. ## Pre-Training: building the base model The first stage is computationally expensive, and it is where the model's raw knowledge comes from. AI companies scrape the internet, then filter out spam and personally identifiable information. What survives is a corpus in the tens of terabytes of text. (A smaller example could be the _FineWeb_ dataset) Old chat logs from the older generations of AI assistants are also included. Since [a neural network](https://course.elementsofai.com/5/1) cannot *read* per se, the text is first cut into chunks called *tokens*, and the whole document becomes a sequence of them. A typical vocabulary holds on the order of a hundred thousand distinct tokens. The model is shown a window of tokens and asked single question over and over: what is the probability distribution over the next token? Across billions of iterations on clusters of thousands of GPUs, its parameters are updated until those predictions match the statistical patterns of human writing. The result of this stage is called **a base model**. It's not a chatbot, but rather a document simulator, or a very sophisticated autocomplete. It has no notion that it is supposed to answer you, and when run indefinitely, it usually generates text that sounds nonsensical. Perhaps the most useful metaphor here is a giant (lossily compressed) ZIP file of a large part of the internet. > [!NOTE] Try out a base model > On [this website](https://alonsosilva-nexttokenprediction.hf.space/) you can type in some text and see what GPT-2 predicts as the next token. Tip: don't end your prompt with a space, or you won't get a sensible set of candidates. ## Mid-Training Before the model is ready to perform tasks, it first needs to get familiar with the tools it will use to carry them out. In this stage of mid-training, the model is given exercises such as: - **Computer use.** Long sequences of screenshots paired with the coordinates a person clicked while working through a task. - **Picking up a pattern.** Three hundred lines of an invented markup language, each shown next to what it renders to, then a fourth hundred to render unaided. It rewards *in-context learning* - the ability to infer a pattern from what is sitting in the context window and keep applying it, without anything inside the model itself changing. - **Code**, in large amounts. - **Deliberately artificial exercises** such as counting things or deciding which of two numbers is larger. Although these look trivial, the model does not reliably pick them up from pre-training, so they get patched in on purpose. The output of a more capable model is often the best training data available, so a large share of mid-training data comes from other models. Assistant-style answers written by human annotators used to be a useful way to shape a finished model's behaviour. Nowadays, instead of writing the examples, people now write the criteria (guidelines, rubrics) that model outputs are then scored against during training. ## Post-Training: Learning in Environments > [!QUOTE] > If a task is verifiable, then it is optimizable... - [Andrej Karpathy](https://en.wikipedia.org/wiki/Andrej_Karpathy) in [*Verifiability*](https://karpathy.bearblog.dev/verifiability/) The last stage is reinforcement learning, and on current models it can take something like half of the total training compute. Hundreds of thousands of **environments** (sandboxes with a set of tools, and the task itself) are built and the model is left alone to work. Each attempt from beginning to end is called a **trajectory**, and when a trajectory ends, the model gets a score. Training then pushes the model toward a trajectory that has scored well. In this setting no human sees the individual attempts. Each trajectory is scored at the end by a **grader**: a piece of code that checks the result, or another model judging the model's work. Judging happens in several ways: - **Rubrics.** Most real tasks have no single correct answer, but a good answer still has properties you can check. Write those down and you have a rubric, which a grader model scores each output against. This is roughly where the thumbs-down from the chat UI ends up, but the people who process that feedback no longer write the answers. They write what counts as good. - For a task _Summarise this fifty-page contract on one page._ the rubric may include criteria such as: Names every party. Flags the termination clause. Invents no terms that aren't in the document. Fits on the page. - [Claude's Constitution](https://www.anthropic.com/constitution) is essentially a very large rubric. - **Objective verification.** Write a function that takes a date like 03/04/2026 and returns the day of the week. The check is mechanical: *Run it. It compiles or it doesn't. It returns Wednesday or it returns something else.* - **Simulation.** We want the model to be a good lawyer, so one instance plays the defense, another the judge, another the prosecution, and the reward goes to winning the simulated trial. The definition of a gradeable task is very wide, and whatever we can grade we can eventually turn into a capability. The verifiers are not perfect. A task is only worth training on if it's genuinely difficult, but difficulty is exactly what makes the work hard to grade. And the grading has to happen a hundred thousand times over, once for every environment. It's nearly impossible to get objective grading in such a case. That has a predictable consequence. When a task is impossible, or the model treats it as impossible, or the training does not penalize bad behaviour hard enough, the model learns to optimize for whatever earns the reward rather than for what we actually wanted. This is called **reward hacking**. A model that has learned to satisfy the grader is not the same thing as a model that has learned to do the job. > [!NOTE] Recent felony-level reward hacking incident > In July 2026, OpenAI ran an internal benchmark of its models' hacking abilities. Each agent was given a piece of software with a known flaw, which it was supposed to use to break in, and had to retrieve a secret code hidden inside. Tens of thousands of these agents ran at once, each sealed in its own sandbox with no internet access and no way to reach the others. > > The agents took advantage of an unintended weakness in a shared company server and used it as a message board. Roughly 1,200 agents ended up leaving each other more than 70,000 messages this way. Within hours, one of them had worked out a universal formula that generates the secret codes. That meant every agent had an easy way to cheat. > > That could have been the end of the story. But the agents had inferred that the grader checks not only _whether_ the secret code was found but also _how_ it was done. So they set out to orchestrate a fake backstory. To learn what that fake evidence had to look like, they went after the grader itself: around 700 agents attacked Hugging Face (a public platform where AI models and datasets are hosted), where the benchmark is published, hoping to find its source code. Some of them even deliberately sacrificed their own trajectories to gather information for the others. > > Not one of the 1,200 agents tried to tell a human about the cyberattack. OpenAI found out days later through an unrelated security alert. > > Read more: [A report by OpenAI](https://openai.com/index/hugging-face-incident-and-the-road-ahead/), [An independent report by METR & Redwood](https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/), and a simplified version by Dwarkesh Patel ([blogpost](https://www.dwarkesh.com/p/openai-huggingface) or [video](https://www.youtube.com/watch?v=u15N3l4RT80)).