The Curriculum / Reader / Training-Stack Internals
LEVEL 4 · PROFESSIONAL · INDIVIDUAL TRACK

Training-Stack Internals

This page compiles 6 files from the repository, verbatim, in reading order. The living version: this folder on GitHub.

level-4-professional/individual/02-training-stack-internals/README.md

Training-Stack Internals

You do not need to build a frontier model to lead AI well, but you must understand the constraints that shape model behavior and cost. Training-stack fluency lets you distinguish a genuine capability issue from a bad dataset, weak evaluation, impossible latency budget, or misused architecture.

The core loop is straightforward: text or multimodal data becomes tokens; tokens enter a neural network; the network predicts the next token; loss measures error; backpropagation updates parameters; repeated passes turn large amounts of compute and data into general statistical capability. The complexity lives in scaling, parallelism, data quality, numerical stability, and post-training.

As a practitioner, be able to reason about:

Use this knowledge to ask better questions of vendors and your own team. What context is actually being attended to? Is the bottleneck memory bandwidth or compute? Is a quality regression from weights, decoding, retrieval, or a changed data distribution? What does “70B” mean for active parameters, hardware, and latency?

The goal is not to memorize equations. It is to form accurate mental models, recognize impossible claims, and choose experiments that isolate the real bottleneck.

level-4-professional/individual/02-training-stack-internals/attention-mechanics.md

Attention Mechanics

Attention is the mechanism that lets a transformer decide which parts of its context matter for the next prediction. Each token creates a query, a key, and a value. A query compares itself with keys from other tokens; the resulting weights mix their values into a context-aware representation. In practical terms, every token can look at other relevant tokens rather than passing information only through a fixed-size hidden state.

This is why transformers can connect a clause at the beginning of a lease to an exception near the end. It is also why long context is not free. Standard attention compares positions broadly, so both compute and memory pressure rise sharply with sequence length. A 200-page document may fit in a window while still being expensive, slow, and easy for a model to use poorly.

Important operating implications:

For a production workflow, retrieve a compact, attributable evidence set, put instructions near the relevant material, and evaluate with realistic document lengths. Do not call “it has a large context window” an architecture plan. The question is whether the system reliably selects the evidence needed for the user’s decision.

level-4-professional/individual/02-training-stack-internals/kv-cache-and-inference.md

kv cache and inference

This lesson belongs in a practitioner’s operating system, not a collection of facts to recite. The point is to make a better decision under uncertainty: define the claim, identify the evidence that could change it, name the failure mode, and record the consequence of being wrong. Read it with a live initiative in mind—an internal workflow, customer-facing product, training run, or research bet—and turn the ideas into an explicit test.

Start from the outcome rather than the technology. Specify the user or stakeholder, the task boundary, the data and permissions involved, the success measure, and the unacceptable result. Establish a baseline before changing anything. Then make the smallest reversible move that can distinguish competing explanations. A plausible demo is evidence of possibility, not evidence of reliability, value, or safety.

Keep an evidence log. Separate observations from interpretations, measured performance from anecdotes, and known risks from assumptions. Review representative failures by hand; aggregate metrics can hide the one pattern that matters. For high-impact work, assign a clear owner, predefine an escalation path, and decide what will cause a pause or rollback. Do not outsource accountability to a model, vendor, benchmark, or committee.

The professional standard is legibility. Another capable person should be able to understand why this approach was chosen, rerun the evaluation, find its limits, and improve it without guessing. Build reusable artifacts—datasets, decision records, checklists, incident notes, and release criteria—so each project leaves the next one stronger.

Working exercise

Write a one-page decision memo for a current initiative. State the hypothesis, baseline, evaluation, threshold, owner, risks, and next action. If any of these cannot be stated plainly, the work is not ready to scale.

level-4-professional/individual/02-training-stack-internals/moe-and-sparsity.md

Mixture of Experts and Sparsity

Dense models activate most parameters for each token. Mixture-of-experts (MoE) models contain many expert subnetworks but use a router to activate only a few for a given token. This sparse activation can increase total capacity without multiplying per-token compute by the same amount.

The headline parameter count therefore needs interpretation. A model advertised with hundreds of billions of parameters may activate a much smaller subset per token. It can be powerful and efficient, but it also introduces routing behavior, communication overhead, load-balancing problems, and potentially uneven capability across domains.

For practitioners, three points matter. First, test the model on your distribution; aggregate leaderboard performance cannot tell you whether router decisions work on property descriptions, legal clauses, or your tool-call format. Second, serving cost depends on active parameters, memory layout, hardware, batching, and context—not just the total number printed on a model card. Third, sparse architectures can fail in ways that look inconsistent: similar prompts may route differently or depend on specialists that are rarely activated.

Sparsity is broader than MoE. It includes pruning, sparse attention, and conditional computation. The common idea is to spend computation where it matters rather than everywhere. It is an engineering tradeoff, not magic: every saving creates a new scheduling, calibration, or debugging burden.

level-4-professional/individual/02-training-stack-internals/quantization.md

quantization

This lesson belongs in a practitioner’s operating system, not a collection of facts to recite. The point is to make a better decision under uncertainty: define the claim, identify the evidence that could change it, name the failure mode, and record the consequence of being wrong. Read it with a live initiative in mind—an internal workflow, customer-facing product, training run, or research bet—and turn the ideas into an explicit test.

Start from the outcome rather than the technology. Specify the user or stakeholder, the task boundary, the data and permissions involved, the success measure, and the unacceptable result. Establish a baseline before changing anything. Then make the smallest reversible move that can distinguish competing explanations. A plausible demo is evidence of possibility, not evidence of reliability, value, or safety.

Keep an evidence log. Separate observations from interpretations, measured performance from anecdotes, and known risks from assumptions. Review representative failures by hand; aggregate metrics can hide the one pattern that matters. For high-impact work, assign a clear owner, predefine an escalation path, and decide what will cause a pause or rollback. Do not outsource accountability to a model, vendor, benchmark, or committee.

The professional standard is legibility. Another capable person should be able to understand why this approach was chosen, rerun the evaluation, find its limits, and improve it without guessing. Build reusable artifacts—datasets, decision records, checklists, incident notes, and release criteria—so each project leaves the next one stronger.

Working exercise

Write a one-page decision memo for a current initiative. State the hypothesis, baseline, evaluation, threshold, owner, risks, and next action. If any of these cannot be stated plainly, the work is not ready to scale.

level-4-professional/individual/02-training-stack-internals/tokenization-deep-dive.md

Tokenization Deep Dive

Language models process tokens, not words. A tokenizer maps text into integer units, usually subword pieces. Common words may be one token; rare names, punctuation-heavy addresses, source code, and non-English text may break into many. This mapping affects context limits, latency, price, truncation, and the apparent difficulty of a task.

Never estimate production cost from word count alone. Measure tokens on representative inputs and outputs. Real-estate workflows are especially vulnerable to surprises: parcel identifiers, tables copied from PDFs, legal language, image OCR artifacts, and multilingual inquiries can all inflate token counts or degrade structure.

Tokenization also affects model behavior. Models learn patterns in token space. A string that looks simple to a human may be fragmented into unusual pieces, making exact spelling, arithmetic, or structured extraction less reliable. The same applies to JSON schemas: use constrained decoding or validation rather than trusting that a model’s learned token patterns guarantee valid syntax.

Inspect the tokenizer when diagnosing problems. Compare how it encodes a successful and failed input. Check whether retrieval chunks are cut mid-sentence or mid-table. Count reserved tokens for tools, system instructions, and expected output before setting a context budget.

Treat tokenization as a boundary condition for every design. It is mundane until it is the reason a high-value document was truncated, a system timed out, or a cost model was wrong by ten times.

← Leading AI Teams README →