A LoRA fine-tune that doubles labeling accuracy on a 0.5B model
The task is to read a customer-support message and label three fields, intent, sentiment and urgency, as one strict JSON object. The base Qwen2.5-0.5B-Instruct formats the JSON fine but mislabels, falling back to technical_issue for most messages. After a LoRA fine-tune on 175 examples, field-level accuracy doubled on 8 held-out prompts, from 9/24 correct field values to 18/24, with intent accuracy rising from 3/8 to 7/8.
Field-level accuracy
38%
base
75%
tuned
9/24 to 18/24 field values correct across all three fields.
Intent accuracy
38%
base
88%
tuned
The base defaults to technical_issue; the tuned model names the real intent, 3/8 to 7/8.
Correct values per field, base to tuned, out of 8
Exact schema match, all three fields right at once, went from 1/8 to 4/8. Valid JSON was 8/8 for both models; the base already formats the object, so the lift is in the labels. sentiment and urgency are more subjective than intent.
The task
One instruction, one input message, one strict JSON answer. The schema never changes; only the three field values do.
Instruction
Read the customer-support message and return a single JSON object with exactly these keys: intent, sentiment and urgency. Return JSON only, with no extra text.
Input message
Gold JSON answer
{
"intent": "billing",
"sentiment": "frustrated",
"urgency": "high"
}Pipeline
Four steps, start to finish, all on one laptop without a cloud GPU.
1. Dataset
175 labeled support messages, each paired with its strict JSON answer.
2. LoRA adapters
Rank 16 adapters on the q_proj, k_proj, v_proj and o_proj attention projections.
3. Train on Apple MPS
Gradient checkpointing, batch size 1 with accumulation, all on a laptop GPU.
4. Before and after eval
Greedy generations on 8 held-out prompts the model never saw in training.
LoRA configuration
The full adapter and training setup, read straight from the run config.
Only ~0.44% of the model weights were trained. That is the LoRA story: a tiny set of adapters changes the behavior while the base model stays frozen.
2,162,688 trainable / 496,195,456 total params
Training loss
Loss over training steps. It drops quickly as the adapters learn the schema, then settles.
27 logged points, final loss 0.000
Before and after
The same held-out messages run through the base model and the LoRA-tuned model. Both format the JSON, but the base mislabels, defaulting to technical_issue, while the tuned model picks the right labels.
I've been billed for a plan I downgraded from weeks ago.
{
"intent": "technical_issue",
"sentiment": "negative",
"urgency": "high"
}{"intent": "billing", "sentiment": "frustrated", "urgency": "high"}Gold answer
{
"intent": "billing",
"sentiment": "frustrated",
"urgency": "high"
}The CSV export button does nothing when I click it.
{
"intent": "technical_issue",
"sentiment": "negative",
"urgency": "high"
}{"intent": "feature_request", "sentiment": "negative", "urgency": "medium"}Gold answer
{
"intent": "technical_issue",
"sentiment": "negative",
"urgency": "medium"
}Please refund my last payment, I cancelled before renewal.
{
"intent": "refund_request",
"sentiment": "frustrated",
"urgency": "high"
}{"intent": "refund_request", "sentiment": "neutral", "urgency": "medium"}Gold answer
{
"intent": "refund_request",
"sentiment": "negative",
"urgency": "high"
}I forgot my password and the reset link is expired.
{
"intent": "technical_issue",
"sentiment": "negative",
"urgency": "high"
}{"intent": "account_access", "sentiment": "negative", "urgency": "medium"}Gold answer
{
"intent": "account_access",
"sentiment": "neutral",
"urgency": "medium"
}Could you add a calendar view? Would love that.
{
"intent": "technical_issue",
"sentiment": "neutral",
"urgency": "medium"
}{"intent": "feature_request", "sentiment": "positive", "urgency": "low"}Gold answer
{
"intent": "feature_request",
"sentiment": "positive",
"urgency": "low"
}Three emails and still no reply, very disappointed.
{
"intent": "complaint",
"sentiment": "frustrated",
"urgency": "high"
}{"intent": "complaint", "sentiment": "frustrated", "urgency": "high"}Gold answer
{
"intent": "complaint",
"sentiment": "frustrated",
"urgency": "high"
}I'd like to cancel my account at the end of the month.
{
"intent": "complaint",
"sentiment": "frustrated",
"urgency": "high"
}{"intent": "cancellation", "sentiment": "negative", "urgency": "medium"}Gold answer
{
"intent": "cancellation",
"sentiment": "neutral",
"urgency": "low"
}Do you have a free trial for new users?
{
"intent": "technical_issue",
"sentiment": "neutral",
"urgency": "medium"
}{"intent": "general_question", "sentiment": "neutral", "urgency": "low"}Gold answer
{
"intent": "general_question",
"sentiment": "neutral",
"urgency": "low"
}