Fine-Tuning an Open Model on a Single GPU Server

Training a model used to mean a cluster and a budget approval. Parameter-efficient methods changed the arithmetic rather than the hardware, and a task-specific tune of an 8B model is now a few days on one rented card. This covers the configuration that works, the parameters worth changing, and the part that decides the result, which is not the hyperparameters.

The reason single-GPU fine-tuning became practical is not bigger cards. It is that you no longer train all the weights. Freeze the base model, train a small number of new parameters alongside it, and a job that needed a cluster fits on one card with memory to spare.

What LoRA Actually Changes

Full fine-tuning updates every weight, so the optimiser has to hold gradients and two momentum terms for each one. That is where the memory goes: the weights themselves are the smaller half of the bill.

LoRA inserts small low-rank matrices into the attention layers and trains only those. The base model stays frozen, which means no gradients and no optimiser state for the vast majority of parameters. QLoRA goes further by holding the frozen base in 4-bit while the trainable adapters stay at higher precision.

The practical consequence is a rough hierarchy of what fits where.

Those are starting points, not guarantees. Sequence length moves them more than model size does, which is the next section. Our guide to GPU memory requirements for local LLMs covers the inference-side arithmetic that the same reasoning rests on.

Sequence Length Is the Real Variable

Activation memory scales with sequence length, and for long-context training it dominates everything else. A job that fits at 1,024 tokens will fail at 4,096 with the same model and the same batch size.

Three levers, in the order to reach for them. Gradient checkpointing recomputes activations during the backward pass instead of storing them, cutting activation memory substantially at maybe 20-30% more compute. Gradient accumulation lets you keep an effective batch size while the per-step batch drops to one. And packing multiple short examples into one sequence stops you paying for padding, which on a dataset of short instructions is most of the tokens.

Turn on checkpointing first. It is the largest single saving and the cost is time, which you have more of than memory.

A Configuration That Works

pip install torch transformers peft trl bitsandbytes accelerate datasets

from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=["q_proj","k_proj","v_proj","o_proj",
                    "gate_proj","up_proj","down_proj"],
)

args = SFTConfig(
    output_dir="out",
    per_device_train_batch_size=1,
    gradient_accumulation_steps=16,
    gradient_checkpointing=True,
    learning_rate=1e-4,
    num_train_epochs=3,
    bf16=True,
    max_seq_length=2048,
    logging_steps=10,
    save_strategy="epoch",
)

Two parameters deserve comment because they are where people go wrong.

Which modules to target. Early LoRA work touched only the query and value projections. Targeting all attention and MLP projections, as above, consistently learns better at a modest memory cost. If you are memory-bound, drop the MLP projections before dropping the rank.

Rank and alpha. Rank 8 to 16 is enough for style, format and tone. Rank 32 to 64 helps when you are teaching genuinely new behaviour. Keep alpha at roughly twice the rank and change one of them at a time. Raising rank because results are poor is usually the wrong response; the data is the more common cause.

The Data Decides the Outcome

This is the part that gets least attention and determines almost everything. A thousand carefully written examples beat fifty thousand scraped ones, reliably.

Three specifics. Every example must use the exact chat template the base model was trained with, including its special tokens, because a mismatch teaches the model a format nobody will use at inference. Mask the loss on the prompt so the model learns to produce the response rather than to reproduce the question. And hold out a genuine validation split before you start, because training loss falling is not evidence of anything useful.

Look at fifty examples by hand before training on ten thousand. Inconsistent formatting, truncated responses and contradictory labels are all visible in a sample and all invisible in a loss curve.

Knowing When to Stop

Small datasets overfit quickly. Two or three epochs is usually right; ten is almost never. Watch validation loss and stop when it turns up, regardless of what the training loss is doing.

Then evaluate on the task rather than on loss. Write twenty prompts that represent what the model will actually be asked, run them against the base model and the tuned one, and compare the outputs side by side. Loss improving while outputs get worse is common, particularly when the model has learned to imitate your dataset's quirks.

Watch for the classic regression too: a model fine-tuned hard on one narrow task often loses general instruction-following. If that matters, mix a small proportion of general instruction data into the training set.

Merge, or Serve the Adapter

Training produces adapter weights of a few dozen megabytes, not a new model. You have two deployment options and the choice is not obvious.

Serve the adapter separately. Inference servers can load a base model once and attach several adapters, so a dozen task-specific tunes share one copy of the weights in memory. This is the right answer when you have more than one.

Merge into the base. One set of weights, no adapter machinery, marginally faster. Note that merging a QLoRA adapter trained against a 4-bit base into a 16-bit base introduces a small mismatch, so evaluate after merging rather than assuming equivalence.

Keep the adapter files and the exact training configuration together in version control. An adapter without the config that produced it cannot be reproduced or explained six months later. Our guide to serving an LLM API with vLLM covers the serving side, including multi-adapter loading.

What This Costs to Run

Fine-tuning is bursty: you want a large card for a few days, then a smaller one for inference, which makes hourly pricing genuinely useful rather than a rounding difference.

MassiveGRID's GPU cloud instances are available on demand or monthly. An A100 40GB with 16 vCPUs, 120 GB of RAM and 512 GB of NVMe is $2.26 per hour or $1,649 a month, which suits LoRA on 7B to 13B models. An A100 80GB with 24 vCPUs and 240 GB of RAM is $3.42 per hour or $2,499. An H100 80GB with 32 vCPUs, 480 GB of DDR5 and 989 TFLOPS of FP16 is $5.48 per hour or $3,999. Instances arrive with CUDA drivers and ML frameworks already configured, which removes the half-day that usually precedes the first training run.

A three-day LoRA run on an A100 40GB is roughly $163 on demand. That is the number to compare against the alternative of prompting a hosted model at volume, and it is usually the argument. Our analysis of renting a GPU server against buying one covers the three-year view, and the GPU infrastructure page covers multi-GPU configurations with NVLink where a single card stops being enough.

GPU instances can be ordered across a partner footprint of more than 700 datacenters in 85 metros, 30 countries and six continents, with auto-provisioning in New York, London, Frankfurt and Singapore.