Course: Course 3 — LLM Fine-Tuning Masterclass Module: FT10 — Full FT vs PEFT: The Decision Duration: 45 minutes Level: Senior Engineer and above Prerequisites: FT08 (LoRA/QLoRA), FT09 (DoRA, rsLoRA, modern PEFT)
After completing this module, you will be able to:
PEFT is not the budget option. It is the correct option for ~95% of the adaptation work you will ever do. Full FT is the exception, and the exception must justify itself.
Default to PEFT. Full fine-tuning must justify a ~30× cost premium and must demonstrate that PEFT provably cannot reach the target.
Read that twice. In most organizations the instinct runs the opposite direction: "full fine-tuning is the real thing; LoRA is the cheap compromise." That instinct is wrong, and it is wrong in two ways at once. It is wrong on cost, and — more subtly — it is wrong on what the two methods actually do.
The cost asymmetry is the easy half. Full fine-tuning a 7B-class model from a usable state typically needs a multi-GPU node — on the order of H100s. A conservative budget for a serious full-FT run on 7B is in the tens of thousands of dollars of compute; scale to 70B and the economics bend sharply. QLoRA on the same 7B runs on a single RTX 4090 — roughly a fifteen-hundred-dollar consumer card — and trains in an evening. That is on the order of a thirty-fold difference in hardware cost for the majority of adaptation tasks where the two methods reach equivalent quality. Any default that begins from the more expensive option must defend the spend.
The structural half is the half most practitioners miss, and it is the half that makes this a decision module rather than a cost module. We covered it in FT00 and now we are going to use it.
Recall the course thesis: fine-tuning steers behavior; it does not teach knowledge. If fine-tuning is steering, then the useful change lives in a low-rank subspace — that is the intrinsic-dimension result (Aghajanyan et al., arXiv:2007.07784), and it is the entire reason adapters work at under one percent of parameters. For a steering task — format, style, instruction-following, preference, refusal removal — a low-rank update is not a compromise. It is the correct representation of the change you are trying to make.
This is the key reframe: when the task is steering, full FT is over-parameterized, not more powerful. It has more degrees of freedom than the task requires, which means it has more room to drift, to forget, and to fit noise. For ~95% of real adaptation work, PEFT is not "almost as good for less money." It is the method whose inductive bias matches the problem.
So when is full FT the right call? Only when the task is not pure steering — when the change you need is genuinely higher-rank. That narrows the field to three conditions.
These are the only honest exits from the PEFT default. Each is rare. Each must be checked, not assumed.
The first condition: the model needs to learn a new way to think, not merely redirect an existing pattern.
Most "reasoning" fine-tuning is not this. When you SFT a model to produce chain-of-thought, or GRPO it on verifiable math rewards, you are activating reasoning pathways the base already has — you are steering the model to use a capability, not installing one. That is steering, and PEFT handles it well.
The genuine case is rarer: you are trying to induce a reasoning behavior the base model cannot produce even with a perfect prompt and a long context. This is the regime where the model must construct new internal computation, and where low-rank updates may be too constrained to express the change. Even here, the evidence is mixed — many "new reasoning" goals turn out to be activation problems in disguise — but this is the most defensible of the three exits. If you have prompt-engineered exhaustively, RAG'd the relevant procedures, and the model still cannot do the kind of reasoning required, full FT is a legitimate try.
The second condition: extreme domain shift where the model needs knowledge it genuinely lacks.
Be suspicious of this one. The course's cardinal error is treating fine-tuning as knowledge injection. For the overwhelming majority of "the model doesn't know X" problems, the answer is RAG — retrieve X and put it in the context. RAG is cheaper, more reliable, auditable, and updateable without retraining. Reach for it first.
Continued pretraining (CPT) or full-FT on a large domain corpus is the rare exception. It is defensible when (a) the domain is so specialized and so voluminous that retrieval becomes intractable — a new programming language with its own semantics, a deep professional vocabulary where surface retrieval misses the structure — and (b) you have the corpus to make it work, which is usually the binding constraint. Medical and legal are the canonical examples people reach for, and even there, RAG usually wins because medical and legal knowledge changes and you cannot retrain on every update. CPT is for when you are building a base for a domain that others will then steer on top of. Almost nobody reading this is doing that at a scale that matters. When you are, full-FT (often via CPT) is the tool, not an adapter.
The third condition is empirical, not conceptual, and it comes from Shuttleworth et al. (arXiv:2410.21228, "An Illusion of Equivalence"). Under certain training regimes — notably larger batch sizes — LoRA degrades faster than full fine-tuning. The mechanism is tied to the structural difference between the two methods (next section): LoRA's low-rank constraint, which is a strength for steering, becomes a liability when the optimization landscape would benefit from higher-rank updates.
The practical implication: if you are training at large batch sizes (for throughput, for stability, or because your infra demands it) and you observe LoRA underperforming, full FT — or a higher-rank / modern PEFT variant like DoRA — is the honest response rather than cranking LoRA's rank until it is no longer PEFT. This is the narrowest of the three exits, but it is real and it is the one most likely to surprise a team that has internalized "LoRA is always enough."
The finding that turns this from a cost decision into a design decision.
Shuttleworth et al. (arXiv:2410.21228) examined the weight matrices produced by LoRA and by full fine-tuning using singular value decomposition. The result: the two methods produce structurally different weight matrices. They are not approximations of each other. LoRA finds a low-rank solution; full FT finds a higher-rank solution. The two can reach similar behavior on a benchmark, but they get there via different geometry.
This matters for two reasons.
First, it confirms the steering thesis (FT00). If fine-tuning were a single well-defined knowledge-injection operation, LoRA and full FT would converge to similar weights. They do not. Multiple solutions exist — which is exactly what you expect for a steering operation (redirect probability mass) and the opposite of what you expect for knowledge injection (one correct answer).
Second — and this is the decision-relevant part — it means the choice between PEFT and full FT is not "cheap approximation vs. expensive ground truth." It is a choice between two structurally distinct solutions with different generalization profiles. LoRA's low-rank solution can be more robust on pure steering tasks (less room to overfit, less catastrophic forgetting) and worse on tasks that need higher-rank changes. Full FT's higher-rank solution can capture changes LoRA cannot, at the cost of more drift and more forgetting.
Shuttleworth et al. also identified a concrete mechanism behind LoRA's degradation in some regimes: "intruder dimensions" — spurious directions introduced during LoRA training that correlate strongly with catastrophic forgetting. These appear especially at low ranks and suboptimal hyperparameters. This is actionable: it is why FT09's guidance (avoid very small ranks; use alpha = 2 * rank; prefer DoRA) exists. The intruder-dimension finding does not invalidate PEFT — it sharpens how you configure it.
The structural non-equivalence finding is the reason this module is called The Decision and not The Cost Trade-off. If the two methods were approximate versions of each other, the only question would be budget. Because they are structurally distinct, the question is: which solution's inductive bias matches your task?
That last bullet is the operational form of the decision rule. Start cheap, measure, escalate with evidence.
Full-FT quality at near-LoRA memory. The method that collapses the cost-quality dichotomy — when you can use it.
The honest tension in the decision is: full FT gives you higher-rank updates (which a minority of tasks need) but at ~30× the memory cost. GaLore (Zhao et al., arXiv:2403.03507, "GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection") is the bridge.
GaLore's idea: instead of constraining the weights to low rank (LoRA), constrain the optimizer's gradient projection to low rank. The weights themselves are full-parameter and update in full; only the optimizer state — which is the bulk of the memory cost in full FT — is projected into a low-rank form via periodic SVD. The result: full-parameter learning (you get the higher-rank solution) at a memory footprint approaching LoRA's. The GaLore paper demonstrates training a 7B model on a single 24GB consumer GPU (RTX 3090/4090) with 8-bit Adam — the same class of hardware QLoRA uses.
Where GaLore sits in the decision: when you want full-FT quality because you have evidence the task is higher-rank, but you are on a memory-limited node, GaLore is the answer. It removes the "I can't afford full FT" excuse without forcing you into LoRA's low-rank constraint when that constraint is the wrong tool. It is not always a drop-in replacement — it has its own tuning (the SVD projection rank, the update frequency) and is slower per step than pure LoRA — but for the narrow case where you need full-FT geometry on PEFT hardware, it is purpose-built.
The decision rule, amended: the escalation ladder is PEFT → DoRA / higher-rank PEFT → GaLore (full-FT quality, PEFT memory) → full FT (when memory is no constraint). GaLore is the rung that makes "I need full-FT quality" and "I have a 4090" compatible.
Six realistic scenarios. The matrix is the deliverable of this module — internalize it, then take the lab.
| Scenario | Recommendation | One-sentence defense |
|---|---|---|
| Medical chatbot formatting (structure outputs as a differential, always cite, flag uncertainty) | PEFT (DoRA / QLoRA) | Pure steering — the model already knows medicine; you are shaping how it presents what it retrieves. |
| Adding a new programming language the base has never seen | Full FT or CPT (after confirming RAG is insufficient) | Genuine knowledge gap — low-rank updates cannot install a language's semantics; this is the rare CPT case. |
| Style transfer (match a brand voice, a specific tone, an author's cadence) | PEFT (LoRA / DoRA) | Archetypal steering — the capability is present; you are redirecting surface generation. |
| Tool-use formatting (reliable JSON function-calling schema) | PEFT (QLoRA) | Steering — the model already understands tools; you are making the format deterministic. |
| Uncensoring (remove refusal for an authorized-use agent) | PEFT (LoRA) or abliteration | Steering away from a direction — an adapter (or abliteration) is the correct low-rank operation; full FT is overkill and risks more forgetting. |
| Domain QA (answer questions over a proprietary knowledge base) | RAG first; PEFT only for the behavior of citing/structuring | Knowledge gap — retrieve the answers; fine-tune at most for how the model uses retrieval, never to memorize the KB. |
Read the matrix as a pattern book, not a lookup table. The shape that recurs: if the base can already do it with a perfect prompt and the right context, it is steering, and PEFT is correct. The two rows that break the pattern (new language, domain QA) are knowledge problems — and note that even there, the first move is RAG, not full FT. Full FT is the answer only when the knowledge is too voluminous or too structural to retrieve and you have the corpus to train it.
The most expensive anti-pattern. Full FT is not uniformly better — for pure steering it is over-parameterized, more prone to forgetting, and ~30× the cost. The structural non-equivalence finding cuts both ways: LoRA's low-rank solution is the correct representation for a steering task. Reach for full FT only when you have evidence the task is higher-rank, not as a default "for quality."
Treating LoRA as "LoRA = full FT minus some quality" — a cheap approximation of the real thing. Shuttleworth et al. showed the two are structurally distinct solutions, not points on a quality continuum. This means you cannot reason about the choice as "how much quality can I afford." You must reason about which solution's geometry fits the task. A team that ignores this will over-spend on full FT for steering tasks and under-spend on it for the rare knowledge tasks that actually need it.
The specific, common instance of the first anti-pattern. A team reaches for full FT to make the model "really learn" the JSON schema, or the brand voice. These are the purest steering tasks in the field — the base already does them unreliably, and a small adapter makes them reliable. Full FT here buys you more drift and more cost for no quality gain. If you take one rule from this module: format and style are always PEFT.
When the task is genuinely a knowledge problem, full FT is one option — but RAG is usually cheaper, more reliable, and updateable. The cardinal error from FT00, restated at the decision layer: do not escalate a knowledge problem straight to full FT. Check whether retrieval solves it first. Full FT (or CPT) is for when retrieval is structurally inadequate, not for when it is merely inconvenient.
Starting at full FT because "we'll probably need it eventually." The correct posture is to start at PEFT, measure, and escalate one rung at a time with evidence. GaLore exists precisely so that the escalation from "PEFT underperforms" does not have to jump straight to a multi-GPU budget.
| Term | Definition |
|---|---|
| Full fine-tuning (full FT) | Updating all (or a large fraction of) the model's parameters; produces a higher-rank weight change; ~30× the memory cost of PEFT |
| PEFT | Parameter-Efficient Fine-Tuning (LoRA, DoRA, rsLoRA, QLoRA); updates a low-rank subset; the default for ~95% of adaptation |
| Structural non-equivalence | The finding (Shuttleworth et al., arXiv:2410.21228) that LoRA and full FT produce structurally different weight matrices — not approximations of each other |
| Intruder dimensions | Spurious directions introduced during LoRA training (especially at low rank) that correlate with catastrophic forgetting |
| GaLore | Gradient Low-Rank Projection (Zhao et al., arXiv:2403.03507); full-parameter learning with the optimizer state projected to low rank — full-FT quality at near-LoRA memory |
| Continued pretraining (CPT) | The rare exception to "steering not teaching" — continues pretraining on a domain corpus to shift the knowledge distribution |
| The decision rule | Default to PEFT; full FT must justify a ~30× cost premium and demonstrate PEFT provably cannot reach the target |
| Escalation ladder | PEFT → DoRA / higher-rank PEFT → GaLore → full FT; escalate one rung at a time with evidence |
See 07-lab-spec.md. "The Decision Matrix" — six realistic adaptation scenarios; for each, state FT-vs-PEFT-and-which-PEFT with a one-sentence defense. No code. This is the judgment module; the deliverable is the reasoning, not a trained model.
# Module FT10 — Full FT vs PEFT: The Decision **Course**: Course 3 — LLM Fine-Tuning Masterclass **Module**: FT10 — Full FT vs PEFT: The Decision **Duration**: 45 minutes **Level**: Senior Engineer and above **Prerequisites**: FT08 (LoRA/QLoRA), FT09 (DoRA, rsLoRA, modern PEFT) --- ## Learning Objectives After completing this module, you will be able to: 1. State the decision rule — **default to PEFT for ~95% of production adaptation** — and justify it with the cost asymmetry and the structural non-equivalence finding. 2. Identify the three conditions that genuinely push a project toward full fine-tuning: new reasoning pathways, extreme domain shift (the rare case RAG cannot solve), and large-batch regimes where LoRA degrades. 3. Explain *structural non-equivalence* (Shuttleworth et al., arXiv:2410.21228): LoRA and full FT are not approximations of each other — they find different solutions, which is why the choice matters rather than being a cost-only decision. 4. Place GaLore on the decision map as the bridge: full-FT quality at near-LoRA memory, for when you want one without the other's trade-off. 5. Apply the decision matrix to a realistic adaptation scenario and produce a defensible recommendation in one sentence. --- # 10.1 — The Default and Why It Exists *PEFT is not the budget option. It is the correct option for ~95% of the adaptation work you will ever do. Full FT is the exception, and the exception must justify itself.* ## The decision rule > **Default to PEFT. Full fine-tuning must justify a ~30× cost premium and must demonstrate that PEFT provably cannot reach the target.** Read that twice. In most organizations the instinct runs the opposite direction: "full fine-tuning is the real thing; LoRA is the cheap compromise." That instinct is wrong, and it is wrong in two ways at once. It is wrong on cost, and — more subtly — it is wrong on what the two methods actually *do*. The cost asymmetry is the easy half. Full fine-tuning a 7B-class model from a usable state typically needs a multi-GPU node — on the order of H100s. A conservative budget for a serious full-FT run on 7B is in the tens of thousands of dollars of compute; scale to 70B and the economics bend sharply. QLoRA on the same 7B runs on a single RTX 4090 — roughly a fifteen-hundred-dollar consumer card — and trains in an evening. That is on the order of a thirty-fold difference in hardware cost for the *majority* of adaptation tasks where the two methods reach equivalent quality. Any default that begins from the more expensive option must defend the spend. The structural half is the half most practitioners miss, and it is the half that makes this a *decision* module rather than a *cost* module. We covered it in FT00 and now we are going to use it. ## Why PEFT wins the default (the steering thesis, applied) Recall the course thesis: fine-tuning steers behavior; it does not teach knowledge. If fine-tuning is steering, then the useful change lives in a low-rank subspace — that is the intrinsic-dimension result (Aghajanyan et al., arXiv:2007.07784), and it is the entire reason adapters work at under one percent of parameters. For a steering task — format, style, instruction-following, preference, refusal removal — a low-rank update is not a compromise. It is the *correct representation* of the change you are trying to make. This is the key reframe: **when the task is steering, full FT is over-parameterized, not more powerful.** It has more degrees of freedom than the task requires, which means it has more room to drift, to forget, and to fit noise. For ~95% of real adaptation work, PEFT is not "almost as good for less money." It is the method whose inductive bias matches the problem. So when is full FT the right call? Only when the task is *not* pure steering — when the change you need is genuinely higher-rank. That narrows the field to three conditions. --- # 10.2 — The Three Conditions for Full FT *These are the only honest exits from the PEFT default. Each is rare. Each must be checked, not assumed.* ## Condition 1 — Genuinely new reasoning pathways The first condition: the model needs to learn a new *way to think*, not merely redirect an existing pattern. Most "reasoning" fine-tuning is not this. When you SFT a model to produce chain-of-thought, or GRPO it on verifiable math rewards, you are activating reasoning pathways the base already has — you are steering the model to *use* a capability, not installing one. That is steering, and PEFT handles it well. The genuine case is rarer: you are trying to induce a reasoning behavior the base model cannot produce even with a perfect prompt and a long context. This is the regime where the model must construct new internal computation, and where low-rank updates may be too constrained to express the change. Even here, the evidence is mixed — many "new reasoning" goals turn out to be activation problems in disguise — but this is the most defensible of the three exits. If you have prompt-engineered exhaustively, RAG'd the relevant procedures, and the model still cannot do the *kind* of reasoning required, full FT is a legitimate try. ## Condition 2 — Extreme domain shift requiring new knowledge The second condition: extreme domain shift where the model needs knowledge it genuinely lacks. Be suspicious of this one. The course's cardinal error is treating fine-tuning as knowledge injection. For the overwhelming majority of "the model doesn't know X" problems, the answer is RAG — retrieve X and put it in the context. RAG is cheaper, more reliable, auditable, and updateable without retraining. Reach for it first. Continued pretraining (CPT) or full-FT on a large domain corpus is the rare exception. It is defensible when (a) the domain is so specialized and so voluminous that retrieval becomes intractable — a new programming language with its own semantics, a deep professional vocabulary where surface retrieval misses the structure — and (b) you have the corpus to make it work, which is usually the binding constraint. Medical and legal are the canonical examples people reach for, and even there, RAG usually wins because medical and legal knowledge *changes* and you cannot retrain on every update. CPT is for when you are building a *base* for a domain that others will then steer on top of. Almost nobody reading this is doing that at a scale that matters. When you are, full-FT (often via CPT) is the tool, not an adapter. ## Condition 3 — Large batch sizes (the LoRA degradation regime) The third condition is empirical, not conceptual, and it comes from Shuttleworth et al. (arXiv:2410.21228, "An Illusion of Equivalence"). Under certain training regimes — notably **larger batch sizes** — LoRA degrades faster than full fine-tuning. The mechanism is tied to the structural difference between the two methods (next section): LoRA's low-rank constraint, which is a strength for steering, becomes a liability when the optimization landscape would benefit from higher-rank updates. The practical implication: if you are training at large batch sizes (for throughput, for stability, or because your infra demands it) and you observe LoRA underperforming, full FT — or a higher-rank / modern PEFT variant like DoRA — is the honest response rather than cranking LoRA's rank until it is no longer PEFT. This is the narrowest of the three exits, but it is real and it is the one most likely to surprise a team that has internalized "LoRA is always enough." --- # 10.3 — Structural Non-Equivalence: Why the Choice Matters *The finding that turns this from a cost decision into a design decision.* ## The finding Shuttleworth et al. (arXiv:2410.21228) examined the weight matrices produced by LoRA and by full fine-tuning using singular value decomposition. The result: the two methods produce **structurally different** weight matrices. They are not approximations of each other. LoRA finds a low-rank solution; full FT finds a higher-rank solution. The two can reach similar *behavior* on a benchmark, but they get there via different *geometry*. This matters for two reasons. First, it confirms the steering thesis (FT00). If fine-tuning were a single well-defined knowledge-injection operation, LoRA and full FT would converge to similar weights. They do not. Multiple solutions exist — which is exactly what you expect for a steering operation (redirect probability mass) and the opposite of what you expect for knowledge injection (one correct answer). Second — and this is the decision-relevant part — it means **the choice between PEFT and full FT is not "cheap approximation vs. expensive ground truth."** It is a choice between two structurally distinct solutions with different generalization profiles. LoRA's low-rank solution can be *more robust* on pure steering tasks (less room to overfit, less catastrophic forgetting) and *worse* on tasks that need higher-rank changes. Full FT's higher-rank solution can capture changes LoRA cannot, at the cost of more drift and more forgetting. ## The "intruder dimensions" Shuttleworth et al. also identified a concrete mechanism behind LoRA's degradation in some regimes: "intruder dimensions" — spurious directions introduced during LoRA training that correlate strongly with catastrophic forgetting. These appear especially at low ranks and suboptimal hyperparameters. This is actionable: it is why FT09's guidance (avoid very small ranks; use `alpha = 2 * rank`; prefer DoRA) exists. The intruder-dimension finding does not invalidate PEFT — it sharpens how you configure it. ## The decision implication The structural non-equivalence finding is the reason this module is called *The Decision* and not *The Cost Trade-off*. If the two methods were approximate versions of each other, the only question would be budget. Because they are structurally distinct, the question is: **which solution's inductive bias matches your task?** - Pure steering (format, style, instruction, preference, refusal): the low-rank solution is the *correct* representation. PEFT. - Higher-rank change (new reasoning, genuine domain knowledge): the low-rank solution may be too constrained. Full FT (or CPT). - Uncertain: start with PEFT (DoRA, reasonable rank). Measure. Only escalate to full FT if PEFT provably fails *and* you have a hypothesis for why the task is higher-rank. That last bullet is the operational form of the decision rule. Start cheap, measure, escalate with evidence. --- # 10.4 — GaLore: The Bridge *Full-FT quality at near-LoRA memory. The method that collapses the cost-quality dichotomy — when you can use it.* The honest tension in the decision is: full FT gives you higher-rank updates (which a minority of tasks need) but at ~30× the memory cost. GaLore (Zhao et al., arXiv:2403.03507, "GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection") is the bridge. GaLore's idea: instead of constraining the *weights* to low rank (LoRA), constrain the *optimizer's gradient projection* to low rank. The weights themselves are full-parameter and update in full; only the optimizer state — which is the bulk of the memory cost in full FT — is projected into a low-rank form via periodic SVD. The result: full-parameter learning (you get the higher-rank solution) at a memory footprint approaching LoRA's. The GaLore paper demonstrates training a 7B model on a single 24GB consumer GPU (RTX 3090/4090) with 8-bit Adam — the same class of hardware QLoRA uses. Where GaLore sits in the decision: **when you want full-FT quality because you have evidence the task is higher-rank, but you are on a memory-limited node, GaLore is the answer.** It removes the "I can't afford full FT" excuse without forcing you into LoRA's low-rank constraint when that constraint is the wrong tool. It is not always a drop-in replacement — it has its own tuning (the SVD projection rank, the update frequency) and is slower per step than pure LoRA — but for the narrow case where you need full-FT geometry on PEFT hardware, it is purpose-built. The decision rule, amended: the escalation ladder is **PEFT → DoRA / higher-rank PEFT → GaLore (full-FT quality, PEFT memory) → full FT (when memory is no constraint).** GaLore is the rung that makes "I need full-FT quality" and "I have a 4090" compatible. --- # 10.5 — The Decision Matrix *Six realistic scenarios. The matrix is the deliverable of this module — internalize it, then take the lab.* | Scenario | Recommendation | One-sentence defense | | --- | --- | --- | | **Medical chatbot formatting** (structure outputs as a differential, always cite, flag uncertainty) | **PEFT (DoRA / QLoRA)** | Pure steering — the model already knows medicine; you are shaping how it presents what it retrieves. | | **Adding a new programming language** the base has never seen | **Full FT or CPT** (after confirming RAG is insufficient) | Genuine knowledge gap — low-rank updates cannot install a language's semantics; this is the rare CPT case. | | **Style transfer** (match a brand voice, a specific tone, an author's cadence) | **PEFT (LoRA / DoRA)** | Archetypal steering — the capability is present; you are redirecting surface generation. | | **Tool-use formatting** (reliable JSON function-calling schema) | **PEFT (QLoRA)** | Steering — the model already understands tools; you are making the format deterministic. | | **Uncensoring** (remove refusal for an authorized-use agent) | **PEFT (LoRA) or abliteration** | Steering away from a direction — an adapter (or abliteration) is the correct low-rank operation; full FT is overkill and risks more forgetting. | | **Domain QA** (answer questions over a proprietary knowledge base) | **RAG first**; PEFT only for the *behavior* of citing/structuring | Knowledge gap — retrieve the answers; fine-tune at most for how the model uses retrieval, never to memorize the KB. | Read the matrix as a pattern book, not a lookup table. The shape that recurs: **if the base can already do it with a perfect prompt and the right context, it is steering, and PEFT is correct.** The two rows that break the pattern (new language, domain QA) are knowledge problems — and note that even there, the first move is RAG, not full FT. Full FT is the answer only when the knowledge is too voluminous or too structural to retrieve *and* you have the corpus to train it. --- # 10.6 — Anti-Patterns ### Defaulting to full FT "because it's better" The most expensive anti-pattern. Full FT is not uniformly better — for pure steering it is over-parameterized, more prone to forgetting, and ~30× the cost. The structural non-equivalence finding cuts both ways: LoRA's low-rank solution is the *correct* representation for a steering task. Reach for full FT only when you have evidence the task is higher-rank, not as a default "for quality." ### Ignoring structural non-equivalence Treating LoRA as "LoRA = full FT minus some quality" — a cheap approximation of the real thing. Shuttleworth et al. showed the two are structurally distinct solutions, not points on a quality continuum. This means you cannot reason about the choice as "how much quality can I afford." You must reason about *which solution's geometry fits the task*. A team that ignores this will over-spend on full FT for steering tasks and under-spend on it for the rare knowledge tasks that actually need it. ### Full FT for format and style tasks The specific, common instance of the first anti-pattern. A team reaches for full FT to make the model "really learn" the JSON schema, or the brand voice. These are the purest steering tasks in the field — the base already does them unreliably, and a small adapter makes them reliable. Full FT here buys you more drift and more cost for no quality gain. If you take one rule from this module: **format and style are always PEFT.** ### Choosing full FT without checking RAG first (the knowledge-task trap) When the task is genuinely a knowledge problem, full FT is one option — but RAG is usually cheaper, more reliable, and updateable. The cardinal error from FT00, restated at the decision layer: do not escalate a knowledge problem straight to full FT. Check whether retrieval solves it first. Full FT (or CPT) is for when retrieval is structurally inadequate, not for when it is merely inconvenient. ### Treating the escalation ladder as one-shot Starting at full FT because "we'll probably need it eventually." The correct posture is to start at PEFT, measure, and escalate one rung at a time with evidence. GaLore exists precisely so that the escalation from "PEFT underperforms" does not have to jump straight to a multi-GPU budget. --- ## Key Terms | Term | Definition | | --- | --- | | **Full fine-tuning (full FT)** | Updating all (or a large fraction of) the model's parameters; produces a higher-rank weight change; ~30× the memory cost of PEFT | | **PEFT** | Parameter-Efficient Fine-Tuning (LoRA, DoRA, rsLoRA, QLoRA); updates a low-rank subset; the default for ~95% of adaptation | | **Structural non-equivalence** | The finding (Shuttleworth et al., arXiv:2410.21228) that LoRA and full FT produce structurally different weight matrices — not approximations of each other | | **Intruder dimensions** | Spurious directions introduced during LoRA training (especially at low rank) that correlate with catastrophic forgetting | | **GaLore** | Gradient Low-Rank Projection (Zhao et al., arXiv:2403.03507); full-parameter learning with the optimizer state projected to low rank — full-FT quality at near-LoRA memory | | **Continued pretraining (CPT)** | The rare exception to "steering not teaching" — continues pretraining on a domain corpus to shift the knowledge distribution | | **The decision rule** | Default to PEFT; full FT must justify a ~30× cost premium and demonstrate PEFT provably cannot reach the target | | **Escalation ladder** | PEFT → DoRA / higher-rank PEFT → GaLore → full FT; escalate one rung at a time with evidence | --- ## Lab Exercise See `07-lab-spec.md`. "The Decision Matrix" — six realistic adaptation scenarios; for each, state FT-vs-PEFT-and-which-PEFT with a one-sentence defense. No code. This is the judgment module; the deliverable is the reasoning, not a trained model. --- ## References 1. **Shuttleworth et al. (2024)** — *LoRA vs Full Fine-tuning: An Illusion of Equivalence*. arXiv:2410.21228, NeurIPS 2025. The structural non-equivalence finding; intruder dimensions; LoRA degradation at large batch sizes. The paper that makes this a *decision* module, not a cost module. 2. **Zhao et al. (2024)** — *GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection*. arXiv:2403.03507, ICML 2024. Full-parameter learning at near-LoRA memory; 7B on a single 24GB GPU. The bridge in the escalation ladder. 3. **Aghajanyan et al. (2020)** — *Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning*. arXiv:2007.07784. The theoretical basis for "fine-tuning is steering" — and thus for PEFT as the default. 4. **Hu et al. (2021)** — *LoRA: Low-Rank Adaptation of Large Language Models*. arXiv:2106.09685. The adapter that made steering cheap. 5. **Dettmers et al. (2023)** — *QLoRA: Efficient Finetuning of Quantized LLMs*. arXiv:2305.14314. 7B on a $1,500 RTX 4090 — the cost-asymmetry data point. 6. **Liu et al. (2024)** — *DoRA: Weight-Decomposed Low-Rank Adaptation*. arXiv:2402.09353. The modern PEFT variant that closes ~half the gap to full FT; the first escalation rung. 7. **FT00 — The Steering Stack** — The thesis (steering not teaching) and the structural-non-equivalence evidence, in foundational form. This module operationalizes that thesis into a decision.