Large general-purpose models handled our enterprise tasks, but only with long, carefully engineered prompts and at a cost that grew with every request. I worked on replacing that pattern with smaller IBM Granite models adapted to the task through parameter-efficient fine-tuning.
The work covered the whole lifecycle: curating training data, choosing between tuning methods, training with LoRA and QLoRA, building evaluation loops that decided whether a model shipped, and serving the result efficiently enough to lower the estimated inference cost by about 30%.
The lifecycle at a glance
01 · The task
Why fine-tune at all
A representative example is turning unstructured support tickets into structured JSON: category, priority, affected product, and a short summary. A large model could do it with a long prompt full of rules and examples, but three problems stayed:
- Cost. The instructions and few-shot examples were re-sent on every request, so most input tokens were the same boilerplate.
- Consistency. Output occasionally broke the schema or used labels outside the allowed set.
- Latency. A large model behind a long prompt was slow for an interactive workflow.
Fine-tuning moves those rules from the prompt into the weights. A smaller model can then learn the exact format and domain vocabulary, and the prompt shrinks to the ticket itself.
02 · Data
The dataset matters more than the method
- Collect: real inputs from production logs, paired with answers written or corrected by subject-matter experts. Where examples were scarce, a larger model drafted candidates that humans reviewed before use.
- Clean: scrub PII, remove near-duplicates with MinHash so the model does not memorise repeated tickets, and drop examples where reviewers disagreed.
- Balance: rare categories were upsampled so the model did not learn to always predict the common ones.
- Format: every example is rendered with the model's own chat template through the tokenizer, the same template used at inference time. A mismatched template is a common, silent cause of poor fine-tuning results.
- Split by source, not at random: tickets from the same customer or thread stay on the same side of the train/validation/test split, so the test set measures generalisation rather than recall of near-identical examples.
The test set was frozen before any training and only used at the evaluation gate, never for tuning hyperparameters.
03 · Method
How LoRA works, and why it fits
Full fine-tuning updates every weight in the model. LoRA (Low-Rank Adaptation) freezes the original weights and learns a small update for selected layers instead. For a weight matrix W, it trains two thin matrices A and B whose product BA has the same shape as W but a very low rank r. The layer then computes Wx + BAx, scaled by alpha / r.
For an 8B-parameter model with rank 16 on every linear layer, that is roughly 50 million trainable parameters, about 0.6% of the model. Three things follow:
- Optimizer state only exists for the adapter. That is why training fits on a single GPU instead of a cluster.
- An adapter is a file of tens of megabytes, not a new copy of the model. Many task-specific adapters can share one base model.
- The base weights never change, so general capabilities are better preserved than with full fine-tuning.
QLoRA goes further. It loads the frozen base model in 4-bit NF4 precision and trains the LoRA adapters in bf16 on top of it. Memory drops enough to fine-tune an 8B model on a single 24 to 48 GB GPU, with quality very close to bf16 LoRA.
Comparing approaches on the same data
| Approach | Trainable params | Training memory | What we learned |
|---|---|---|---|
| Base model, few-shot prompt | None | Inference only | Good starting point, but every request carried long instructions and examples. That cost input tokens on every call and still drifted from the output format. |
| Prompt tuning (soft prompts) | <0.01%: ~20 virtual tokens | Low | Cheap to train and to store. It improved tone and format, but plateaued on the harder reasoning cases. |
| LoRA, attention projections only | ~0.2% | One 80 GB GPU (bf16) | A big step up in format reliability. Some domain-specific mistakes remained. |
| LoRA, all linear layers | ~0.6% | One 80 GB GPU (bf16) | Best quality per training dollar. Adapting the MLP layers mattered for domain knowledge, not just style. |
| QLoRA, all linear layers | ~0.6% | Fits on a 24-48 GB GPU | Close to bf16 LoRA quality at a fraction of the memory. It became the default for fast iteration. |
| Full fine-tuning | 100% | Roughly 130 GB+ for an 8B model | Not justified. It needs multiple GPUs, a full model copy per version, and carries more risk of forgetting general skills. |
Parameter shares are approximate for an 8B decoder model with rank 16. The full fine-tuning estimate covers bf16 weights and gradients plus fp32 Adam optimizer states and master weights, before activations.
04 · Training
The configuration
Training used Hugging Face Transformers, PEFT and TRL's supervised fine-tuning trainer, with bitsandbytes for 4-bit loading. Runs were tracked in MLflow so every adapter could be traced back to its data version, config and evaluation scores.
from peft import LoraConfig
from transformers import BitsAndBytesConfig
from trl import SFTConfig
# QLoRA: load the frozen base model in 4-bit NF4, compute in bf16
bnb = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype="bfloat16",
)
lora = LoraConfig(
r=16, # adapter rank
lora_alpha=32, # scale = alpha / r = 2
lora_dropout=0.05,
target_modules=[ # every linear layer in each block
"q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj",
],
task_type="CAUSAL_LM",
)
train_args = SFTConfig(
num_train_epochs=3,
learning_rate=2e-4,
lr_scheduler_type="cosine",
warmup_ratio=0.03,
per_device_train_batch_size=4,
gradient_accumulation_steps=8, # effective batch of 32
gradient_checkpointing=True,
bf16=True,
max_seq_length=4096,
eval_strategy="steps",
eval_steps=100,
save_strategy="steps",
save_steps=100, # checkpoint at every evaluation...
load_best_model_at_end=True, # ...and restore the lowest val loss
)What actually moved the needle
- Loss on the answer only. Masking the prompt tokens so the loss is computed only on the response focuses learning on what the model should produce, not on repeating the input.
- Targeting all linear layers. Adapting only attention fixed formatting. Adding the MLP projections is what improved domain accuracy.
- Fewer epochs than expected. Validation loss usually bottomed out within two to three epochs. Beyond that the model started memorising, so the best checkpoint was kept automatically.
- Learning rate over rank. Raising the rank above 16 rarely helped, while a learning rate that was too high caused instability and forgetting. A short warmup and a cosine decay kept runs stable.
- Cleaner data beat more data. Removing contradictory and duplicate examples improved results more than adding the next batch of raw ones.
05 · Evaluation
An evaluation gate, not a vibe check
A candidate adapter only moved forward if it beat the current model on every axis we cared about:
- Task metrics: JSON schema validity rate, exact match on categorical fields, and field-level F1 on the frozen test set.
- Free-text quality: summaries were scored by an LLM judge with a written rubric and compared pairwise against the current model. Humans reviewed a sample of the judge's decisions to keep it honest.
- Regression checks: a small suite run through EleutherAI's lm-evaluation-harness, plus general instruction-following prompts, to catch any loss of general ability.
- Safety: a fixed set of adversarial and out-of-scope prompts that the model must refuse or deflect.
- Efficiency: input and output tokens per request, p50 and p95 latency, and throughput under load.
Passing offline evaluation earned an A/B test, not a launch. Live traffic was split between the current model and the candidate, and the candidate was promoted only if quality held and cost per request dropped.
06 · Serving
Where the ~30% came from
The cost reduction was not one trick but several that stacked:
- A smaller model: a fine-tuned Granite model replaced a much larger general model for the task.
- Shorter prompts: the rules and few-shot examples now lived in the weights, so each request sent only the actual input.
- Shorter outputs: the model learned to answer in the exact format, without preamble.
- Efficient serving: adapters were either merged into the base weights or served as hot-swappable LoRA adapters on a shared base with vLLM. The merged models were quantized to 8-bit after confirming quality held, and continuous batching raised throughput.
07 · Lifecycle
Keeping a tuned model good over time
A fine-tuned model is a snapshot of the data it saw. Datasets, adapters and evaluation reports were versioned together, so any production model could be traced to exactly what trained it. Failures and low-rated responses were collected into a review queue. Once enough new cases built up, they became the next training round, and the new adapter had to pass the same gate as the last one.
Technology
- Models
- IBM Granite instruct models, larger general models as baselines and data generators, watsonx.ai
- Tuning
- Hugging Face Transformers, PEFT (LoRA, QLoRA, prompt tuning), TRL SFTTrainer, bitsandbytes, Accelerate
- Data
- pandas, Hugging Face Datasets, MinHash de-duplication, PII scrubbing
- Evaluation
- Frozen test sets, schema validation, F1 and exact match, LLM-as-judge with human review, lm-evaluation-harness, A/B testing
- Serving
- vLLM with multi-LoRA, 8-bit quantization, FastAPI, Docker, Kubernetes, MLflow
Client names, data and internal details are confidential. The task and examples on this page are representative of the work rather than copies of client material.