Fine-Tuning

Researchers fine-tune a pre-trained model on their own labelled data, then run the result over new data as a batch job. Everything executes on SageMaker; the platform owns the catalog, the dataset contract, access control and the budget.

Task types

A task type is the organizing primitive. It determines the dataset contract, the accepted upload formats, which base models are offered, and which Deep Learning Container the job runs in.

Task	Input → output	Dataset upload	Manifest columns
text-classification	text → label	.csv / .jsonl / .json	text, label
image-classification	image → label	.zip	image, label
image-text-classification	image + text → label	.zip	image, text, label
image-text-to-text	image + prompt → free text	.zip	image, prompt, response

The three classification tasks produce the same output: a softmax over the dataset’s own classes, written as a CSV with one probability column per class.

image-text-to-text is generative — it keeps the model’s language head and learns to write the response, so it has no label column and no class list. Its result is still a CSV with one row per input record, carrying an output column instead of probability columns, so the download and the result viewer are unchanged.

Image datasets

An image task takes a single .zip containing a manifest plus the image files it references:

dataset.zip
├── manifest.csv        image,label
│                       images/cat-01.jpg,cat
│                       images/dog-04.jpg,dog
└── images/
    ├── cat-01.jpg
    └── dog-04.jpg

The image column holds each file’s path relative to the archive root. The manifest may be .csv, .jsonl or .json. Archives are treated as untrusted input: entries that use absolute paths or ../ traversal are refused, as are manifest rows pointing outside the archive.

Batch Transform receives the archive as one payload, so inference input is capped at 100 MB per job. Larger inference runs should be split across jobs.

Adding a task type

The registry lives in backend/src/apis/app_api/fine_tuning/task_types.py. Adding a task means registering a TaskSpec and adding a sagemaker_scripts/task_<name>.py module implementing train, model_fn, input_fn and predict_fn. train.py and inference.py are dispatchers and should not need editing.

task_types.py must stay importable without torch, transformers, pandas or PIL — it is read by the app-api container and the unit tests, neither of which has the ML stack.

Deep Learning Containers

Each task declares a DLC family, and the two families move independently:

Family	Training image	Why
text	PyTorch 2.1 / transformers 4.36	Every existing text model was trained and validated here. Bumping it would re-baseline all of them at once.
vision	PyTorch 2.8 / transformers 4.56	Modern vision checkpoints do not load on 4.36.
vlm	PyTorch 2.8 / transformers 4.56	Same image as vision, but its own family so it can install peft and bitsandbytes — and so a future VLM-only image bump cannot re-baseline the image classifiers.

Each family gets its own sourcedir-<family>.tar.gz, because the dependency sets differ. bitsandbytes requires torch >= 2.4, which the text container (torch 2.1) cannot satisfy, so a single shared requirements.txt would break dependency installation for every existing text job. The file each family gets is script_packaging_service.REQUIREMENTS_BY_FAMILY.

If a tag is retired or lags in a region, override it without a code deploy:

FINE_TUNING_TRAINING_IMAGE_VISION=<account>.dkr.ecr.<region>.amazonaws.com/<repo>:<tag>
FINE_TUNING_INFERENCE_IMAGE_VISION=...

Quota

Quota is denominated in US dollars per calendar month, not GPU-hours. Hours stopped describing the budget the moment more than one instance type was offered: ten hours buys about $14 on an ml.g5.xlarge and roughly $450 on an ml.g6e.24xlarge.

Grants written before this change are migrated lazily — read in the new shape and rewritten on the next quota check, converting at the ml.g5.xlarge rate those hours were actually spent at. No backfill run is needed, and a migrated user’s existing spend carries across rather than resetting.

How spend is bounded

A job’s real cost is unknown until it stops, so the budget becomes its stopping condition: MaxRuntimeInSeconds is clamped to the hours the user’s remaining quota affords. SageMaker kills the job at that point, which bounds spend exactly with no reservation bookkeeping.

Rejecting on worst-case cost instead would block essentially everything — the default 24-hour stopping condition is ~$27 even on the cheapest instance, more than an ordinary monthly quota, though such a job usually finishes in minutes. A job is refused outright only when the remaining budget buys less than 30 minutes.

Failed and stopped runs are billed by AWS and are charged against the quota accordingly.

Configuration

Variable	Meaning
FINE_TUNING_DEFAULT_QUOTA_USD	Monthly quota auto-granted to any authenticated user. 0 (default) means whitelist-only.
FINE_TUNING_DEFAULT_QUOTA_HOURS	Legacy fallback, converted to dollars at the ml.g5.xlarge rate.

Instance pricing

Rates live in fine_tuning/pricing.py as literal maps, sourced from the AWS Price List API (not the pricing web page, which rounds and lags). Resync them with:

python backend/scripts/refresh_instance_pricing.py --profile dev-ai

Two things the map gets right that a single flat table could not:

• Training and Batch Transform are priced separately. They agree across the g5 family but diverge on g6e.
• Not every training instance can run Batch Transform. ml.p4d.24xlarge and ml.p5.48xlarge publish a training rate and no transform rate, so offering them would let a researcher fine-tune a model they then cannot run inference on. They are deliberately absent, as is ml.p3.*, which has no on-demand SageMaker rate at all.

Pricing accuracy is load-bearing: a wrong rate does not just misreport spend, it mis-enforces the budget.

Choosing a base model

The catalog offers vetted models per task. Researchers may also supply any HuggingFace model id, which is pre-flighted against the Hub before a GPU is provisioned. A custom model is refused when it does not exist, publishes no loadable weights, or is tagged for a different modality.

The commonest refusal is a GGUF-only repository. GGUF is a llama.cpp inference format and cannot be fine-tuned by transformers at all — look for the original, unquantised repository instead.

Generative vision-language models (LLaVA, Qwen-VL) belong to the image-text-to-text task, not to image-text-classification — the latter needs a dual encoder exposing get_image_features and get_text_features (the CLIP/SigLIP/ALIGN family), and a generative checkpoint has no text tower to pool.

Generative VLMs are LoRA-adapted, not fully fine-tuned

A full fine-tune needs roughly 16 bytes per parameter once gradients and the AdamW moments are resident — about 550 GB for a 34B model, against the 384 GB on the largest instance offered. So image-text-to-text quantises the frozen base to 4-bit (NF4) and trains low-rank adapters on top. Two consequences worth knowing:

• The artifact is an adapter, not a model. It is a few hundred MB rather than tens of GB, and inference reloads the base from the Hub and applies the adapter over it. vlm_adapter.json inside the artifact records which base and whether it was quantised.
• Loss is computed on the response only. The prompt is masked out, so the model does not spend its gradient budget learning to reproduce questions it will always be given.

load_in_4bit, lora_r and lora_alpha are exposed on the create-job form for this task. Models below about 3B are faster in bfloat16 — the 4-bit path pays a dequantisation cost that only earns its keep when the model would not otherwise fit.

Admin surface

Grant access and review spend under Usage & Spend, backed by admin/fine_tuning and the /admin/fine-tuning pages. A grant is a monthly dollar quota against an email address; the AppRole record is not involved.