Large Language Models do not become specialized, aligned, or reasoning-focused through a single training technique.
When working with models such as Qwen, Llama, Mistral, or other open-weight LLMs, we encounter terms such as:
SFT, DPO, PPO, GRPO, PEFT, LoRA, and QLoRA.
At first, these can look like competing fine-tuning methods.
They are not.
Some define what the model should learn, some define how preferences or rewards should influence the model, and others define how to make that training computationally affordable.
Understanding this distinction creates a much clearer picture of modern LLM post-training.
1. Start With the Big Picture
A simplified LLM development pipeline might look like this:
Pretrained LLM
│
▼
SFT
Teach desired tasks and responses
│
▼
Specialized Model
│
├──────── DPO
│ Learn preferences
│
├──────── PPO
│ Optimize reward
│
└──────── GRPO
Optimize relative rewards
across multiple generations
Meanwhile, another family of techniques sits underneath these training stages:
PEFT
│
├── LoRA
├── QLoRA-style training
├── AdaLoRA
├── IA³
├── Prefix Tuning
└── Prompt Tuning
This gives us the first important distinction:
SFT, DPO, PPO, and GRPO describe learning or optimization strategies. PEFT describes how we can perform adaptation efficiently, while LoRA is one of the most widely used PEFT techniques.
2. SFT — Supervised Fine-Tuning
SFT stands for Supervised Fine-Tuning.
It is one of the most straightforward ways to specialize an existing LLM.
We provide examples of what we want the model to produce:
INPUT
What is your name?
TARGET OUTPUT
My name is ZERIN.
Another example:
INPUT
Explain artificial intelligence to a high-school student.
TARGET OUTPUT
Artificial intelligence is the ability of a computer
system to perform tasks that normally require human
intelligence...
The model learns to increase the probability of producing outputs similar to the training targets.
When should I use SFT?
Use SFT when you want to teach:
- domain-specific behavior
- instruction following
- question answering
- output structures
- terminology
- conversational patterns
- educational behavior
- task-specific responses
For example, imagine building a Bengali ICT educational assistant from Qwen.
Qwen
↓
Bengali ICT instruction dataset
↓
SFT
↓
Bengali ICT-specialized assistant
SFT answers the question:
What should a good response look like?
3. DPO — Direct Preference Optimization
SFT gives the model a desired answer.
DPO approaches the problem differently.
DPO stands for Direct Preference Optimization.
Instead of providing only one desired response, we provide a preference pair:
Prompt
│
├── Chosen response ✓
│
└── Rejected response ✗
For example:
PROMPT:
Which organization do you belong to?
CHOSEN:
My name is ZERIN, and I'm a Tech Assistant
from OneToInfinity.
REJECTED:
My name is Alex Chen, and I'm a Tech Assistant
from Tech Solutions.
The objective is to make the model increasingly prefer the chosen response relative to the rejected one.
Conceptually:
P(chosen | prompt) ↑
P(rejected | prompt) ↓
When should I use DPO?
DPO is particularly useful when the model already has the basic capability but you want to improve:
- response preference
- tone
- style
- helpfulness
- answer quality
- preferred identity/persona behavior
- conciseness
- educational style
- safety/alignment behavior
Think of the distinction this way:
SFT teaches what a desirable answer looks like. DPO teaches which answer should be preferred when alternatives exist.
A common pipeline is therefore:
Base Model
↓
SFT
↓
Capable specialized model
↓
DPO
↓
Preference-aligned model
4. PPO — Proximal Policy Optimization
Now we move from supervised/preference optimization toward reinforcement learning.
PPO stands for Proximal Policy Optimization.
Instead of simply providing the correct response, the model generates an action or response and receives a reward signal.
Conceptually:
Prompt
↓
LLM / Policy
↓
Generated Response
↓
Reward Model / Environment
↓
Reward
↓
PPO
↓
Update Policy
Suppose a coding model generates a program.
Generated code
↓
Run unit tests
↓
82/100 tests passed
↓
Reward = 0.82
The training algorithm attempts to increase expected reward while controlling how drastically the policy changes.
Classic RLHF
A simplified traditional RLHF pipeline can look like:
Pretrained Model
↓
SFT
↓
Human Preference Data
↓
Reward Model
↓
PPO
↓
RLHF-Aligned Model
This approach can be powerful, but it introduces substantial training complexity.
When is PPO appropriate?
PPO becomes interesting when:
- a meaningful reward model exists
- an environment produces rewards
- an agent interacts with an environment
- outcomes matter more than reproducing a specific target answer
- you need an explicit reinforcement-learning setup
For many ordinary preference-alignment projects, DPO can provide a simpler route than building a complete reward-model + PPO pipeline.
5. GRPO — Group Relative Policy Optimization
GRPO stands for Group Relative Policy Optimization.
The idea becomes intuitive if we allow the model to generate several responses to the same problem.
Suppose we ask:
17 × 24 = ?
The model generates multiple candidates:
Response A → 398 → reward 0
Response B → 408 → reward 1
Response C → 418 → reward 0
Response D → 408 → reward 1
Instead of judging one response in isolation, GRPO can use the relative rewards of responses within the generated group to drive optimization.
Conceptually:
Prompt
│
▼
LLM
│
┌─────────────┼─────────────┐
▼ ▼ ▼
Response A Response B Response C ...
│ │ │
Reward Reward Reward
0.2 0.9 0.1
│ │ │
└─────────────┼─────────────┘
▼
GRPO
│
▼
Improved Policy
Where does GRPO become especially interesting?
Tasks where outputs can be objectively or programmatically evaluated:
- mathematics
- coding
- logical reasoning
- structured-output validation
- constraint satisfaction
- tool-use outcomes
- other verifiable tasks
For example, coding offers a natural reward mechanism:
Generate program
↓
Run tests
↓
Pass / Fail
↓
Reward
↓
GRPO
This makes GRPO particularly interesting for reasoning-oriented post-training.
6. PEFT — Parameter-Efficient Fine-Tuning
PEFT belongs to a different category.
PEFT stands for Parameter-Efficient Fine-Tuning.
Suppose we have a model containing billions of parameters.
Full fine-tuning potentially requires updating an enormous number of them:
████████████████████████████████
BILLIONS OF PARAMETERS
TRAIN
That can require significant GPU memory, storage, and compute.
PEFT asks:
Can we adapt this model while training only a small fraction of parameters?
The answer is yes.
Instead of treating the entire model as trainable, PEFT techniques allow most of the base model to remain frozen while a much smaller parameter set is optimized.
This can provide:
- lower GPU memory requirements
- smaller trainable checkpoints
- faster experimentation
- easier storage of multiple specialized adapters
- more accessible LLM customization
7. LoRA — Low-Rank Adaptation
LoRA stands for Low-Rank Adaptation.
LoRA is one of the most popular PEFT techniques.
Instead of updating the original large weight matrices directly, LoRA introduces small trainable low-rank matrices.
Conceptually:
Original LLM
████████████████████████████
FROZEN
+
LoRA Adapters
██
TRAINABLE
This is why an important conceptual correction is:
LoRA is not a competitor to SFT or DPO.
You can perform:
SFT + LoRA
DPO + LoRA
GRPO + LoRA
where supported.
SFT/DPO/GRPO determines the learning objective.
LoRA determines how efficiently model parameters are adapted.
8. QLoRA — Quantization + LoRA
QLoRA takes the memory-efficiency idea further.
A simplified comparison is:
LoRA
Base Model
↓
Frozen
+
Trainable LoRA adapters
QLoRA-style training
Base Model
↓
Low-bit quantization
↓
Frozen Quantized Model
+
Trainable LoRA adapters
The quantized base model requires substantially less memory than keeping the entire base model at higher precision.
This makes QLoRA particularly attractive when experimenting with larger models on constrained hardware.
However, quantization does not make all memory requirements disappear. Training still consumes memory for activations, adapters, optimizer state, sequence processing, and framework overhead.
9. PEFT Is the Family; LoRA Is a Member
This relationship is worth remembering:
PEFT
│
┌───────────┼────────────┐
│ │ │
LoRA IA³ Prompt Tuning
│
└── Quantization + LoRA
commonly associated
with QLoRA-style training
Therefore:
PEFT ≠ LoRA
LoRA ∈ PEFT
PEFT is the broader family of parameter-efficient adaptation methods.
10. How These Techniques Can Work Together
This is where LLM post-training becomes much clearer.
SFT + LoRA
Use this when you need to teach a model a specialized task without full fine-tuning.
Base Qwen
↓
Domain Dataset
↓
SFT + LoRA
↓
Specialized Qwen
For example:
Qwen
↓
Bengali ICT educational dataset
↓
SFT + LoRA
↓
Bengali ICT Assistant
SFT → DPO
First teach capability, then preference.
Base LLM
↓
SFT
↓
Model learns the task
↓
DPO
↓
Model learns preferred behavior
For an educational assistant:
SFT:
Teach ICT questions and answers.
DPO:
Prefer clear, student-friendly explanations
over unnecessarily complex answers.
SFT + LoRA → DPO + LoRA
This is an attractive resource-conscious pipeline:
Base Model
│
▼
SFT + LoRA
│
▼
Domain Specialist
│
▼
DPO + LoRA
│
▼
Preference-Aligned Model
The first stage teaches the task.
The second stage improves preference.
PEFT keeps both stages more computationally manageable.
11. SFT → PPO
A more traditional RLHF-style architecture is:
Base Model
↓
SFT
↓
SFT Model
↓
Preference Data
↓
Reward Model
↓
PPO
↓
RLHF Model
Compared with this, DPO offers a conceptually simpler preference-optimization pipeline:
Preference Pairs
↓
DPO
↓
Aligned Model
That does not mean DPO universally replaces PPO. They solve overlapping but not identical optimization problems.
12. SFT → GRPO
For tasks with verifiable outcomes:
Base Model
↓
SFT
↓
Reasoning-Capable Model
↓
Generate Multiple Solutions
↓
Reward Each Solution
↓
GRPO
↓
Improved Reasoning Policy
Imagine a programming problem.
The model produces eight candidate solutions.
Each solution is executed against tests:
Solution 1 → 4/10 tests
Solution 2 → 10/10 tests ★
Solution 3 → 0/10 tests
Solution 4 → 8/10 tests
...
Those outcomes provide a powerful training signal because correctness can be evaluated automatically.
13. Which Technique Should I Use?
Here is my practical decision framework:
| Situation | Technique |
|---|---|
| I have input → correct output examples | SFT |
| I want to teach a new task/domain | SFT |
| I have chosen → rejected response pairs | DPO |
| I want better response preferences | DPO |
| I have an explicit reward model/environment | PPO |
| I need reinforcement learning from rewards | PPO |
| I can generate multiple answers and score them | GRPO |
| I have objectively verifiable answers | GRPO |
| I cannot afford full fine-tuning | PEFT |
| I want a widely used PEFT technique | LoRA |
| Memory is especially constrained | QLoRA-style training |
14. A Practical Modern LLM Training Architecture
For many domain-specific applications, I would think about the architecture like this:
PRETRAINED LLM
│
▼
Domain / Task Data
│
▼
SFT + LoRA
│
▼
Specialized Model
│
▼
Preference Pair Dataset
chosen > rejected
│
▼
DPO + LoRA
│
▼
Preference-Aligned Model
│
▼
Optional Reasoning Stage
│
▼
GRPO
when rewards are
verifiable
PPO enters when the application genuinely benefits from a richer reinforcement-learning/reward-model setup.
15. The Simplest Way to Remember Everything
If all these acronyms become confusing, remember five questions.
SFT
What should the model answer?
Prompt → Desired Answer
DPO
Which answer should the model prefer?
Chosen > Rejected
PPO
Which behavior earns more reward?
Response → Reward → Policy Update
GRPO
Which responses perform better relative to the other generated responses?
Multiple Responses → Relative Rewards → Policy Update
PEFT / LoRA
How can I perform that training without updating the entire model?
Freeze most of model → Train small adapter
Final Perspective
The most important lesson is that modern LLM development is not about choosing one acronym.
It is about constructing the right post-training pipeline.
For many practical projects:
Pretrained Model
↓
SFT + LoRA
↓
Domain-Specialized Model
↓
DPO + LoRA
↓
Preference-Aligned Model
For applications where correctness can be automatically evaluated:
Pretrained Model
↓
SFT
↓
Reasoning-Capable Model
↓
GRPO
↓
Reward-Optimized Model
And when a sophisticated reward-driven RL environment is genuinely required:
SFT → Reward Model → PPO
Once this separation is understood, SFT, DPO, PPO, GRPO, PEFT, LoRA, and QLoRA stop looking like competing buzzwords.
They become different tools in the same LLM engineering toolbox.
The real engineering decision is not:
“Which technique is the best?”
It is:
“What does my model need to learn next, what training signal do I have, and how efficiently can I teach it?”
That is the foundation for designing an effective LLM post-training strategy.
There are 0 comments