Every team I’ve talked to this year has the same spreadsheet open: a column for “model quality” and a column for “monthly GPU bill,” and a manager asking why they can’t both go the direction they want. AWS and Unsloth just published a set of deployment patterns that make that trade-off concrete instead of theoretical — and after running one of these patterns against a real fine-tuned 8B model this week, I want to walk through what the numbers actually mean once you’re the one holding the pager.

The core trick: dynamic, not uniform, quantization

Naive quantization takes every layer of a model and drops it to the same bit width — 4-bit everywhere, say — and eats a quality hit across the board because some layers (attention projections, early embedding layers) are far more sensitive to precision loss than others (later FFN blocks). Unsloth Dynamic does a layer-by-layer sensitivity pass first, then allocates precision non-uniformly: sensitive layers stay near 16-bit, everything else compresses hard. Daniel Han’s stated result is a model reduced by roughly 86% in size with only about 14% degradation in eval accuracy — an asymmetry that plain 4-bit quantization doesn’t get you.

Concretely, an 8B parameter model goes from ~16GB in BF16 down to ~5GB in 4-bit GGUF. That’s the number that actually matters for infra planning, because it’s what decides whether you fit on a $1.41/hr g5.xlarge instead of a $7.09/hr g5.12xlarge — roughly a 5x cost delta for a workload that, in our internal test, held up fine on latency for anything that wasn’t a long-context batch job.

Exporting the model

Two export paths, and picking the wrong one is the single most common mistake I saw when reproducing this:

# Single-file GGUF — for llama.cpp-based serving
model.save_pretrained_gguf(
    "gguf_model", tokenizer, quantization_method="q4_k_xl"
)

# Merged safetensors — for vLLM/SGLang/HF-native serving
model.save_pretrained_merged(
    "finetuned_model", tokenizer, save_method="merged_16bit"
)

GGUF and merged safetensors are not interchangeable at deploy time — GGUF only makes sense once you’ve committed to a llama.cpp-family runtime. If your team is standardized on vLLM for batching efficiency, exporting GGUF is a dead end you’ll discover after the container build, not before. Decide the runtime first, then export.

Four patterns, four very different operational postures

#	Infra	Runtime	Who it’s actually for
1	EC2	llama.cpp / llama-server	Small teams, prototypes, anyone who wants SSH access when it breaks at 2am
2	SageMaker custom container	llama.cpp in-container	Teams that want managed autoscaling but aren’t ready to give up llama.cpp’s simplicity
3	SageMaker LMI	vLLM / SGLang	Production traffic, GPU efficiency matters, you have real concurrency
4	EKS / ECS	Any containerized stack	You already run inference next to other services and don’t want a second orchestration layer

Pattern 1 is where I’d start for anything internal-facing — a support-ticket triage model, an internal search reranker. It’s a single binary and a flag:

llama-server --model /models/my-model.gguf --ctx-size 8192 --host 0.0.0.0 --port 8080

and then it’s just an OpenAI-compatible endpoint:

client = OpenAI(base_url="http://<ip>:8080/v1", api_key="not-required")
response = client.chat.completions.create(model="my-model", messages=[...])

Pattern 3 is where I’d push anything customer-facing, because the LMI container gives you rolling-batch scheduling out of the box, which matters the moment concurrency goes above single digits:

HF_MODEL_ID = "Qwen/Qwen3-VL-8B-Instruct"
OPTION_DTYPE = "bf16"
OPTION_ROLLING_BATCH = "vllm"
OPTION_TENSOR_PARALLEL_DEGREE = "4"

What the benchmark numbers don’t tell you

I’ve shipped two quantized model deployments this year, and both times the failure mode wasn’t quality — it was operational drift that only shows up under load:

1. Prompt format drift is the silent killer. If your fine-tuning prompt template and your serving-time chat template diverge by even a whitespace token, quality degrades in a way that looks identical to quantization loss. Pin the template in version control next to the model weights, not in application code where someone will “helpfully” adjust it later.
2. Benchmark the full deployment shape, not the model in isolation. A quantized model that’s fast at ctx-size 2048 and single-request concurrency can fall off a cliff at ctx-size 8192 and 20 concurrent requests — the KV cache, not the weights, becomes the memory bottleneck. Load-test the shape you’ll actually run in production before you commit to an instance size.
3. S3 as the source of truth, always. Don’t let quantized artifacts live only on the instance that produced them. We lost half a day once re-running a quantization job because the only copy of a q4_k_xl export was on an EC2 instance someone terminated during a cost cleanup.
4. Validate the container contract locally before it touches SageMaker. SageMaker’s error messages when a container’s health check or model-loading contract is wrong are notoriously unhelpful — a docker run locally against the same entrypoint catches 90% of what would otherwise be a 20-minute deploy-fail-diagnose loop in the AWS console.

The lead’s actual decision

If you’re a Technical Lead sizing this for your team, the question isn’t “which pattern is best” — it’s “what’s our actual traffic shape and who owns the 2am pager.” Pattern 1 or 2 if you’re pre-product-market-fit and want to move fast without committing to a serving framework. Pattern 3 the moment you have real concurrent traffic and GPU cost is a line item someone asks about in planning. Pattern 4 only if you already have EKS/ECS expertise in-house — don’t adopt a new orchestration layer just for one model.

The 5x cost delta is real and worth chasing. Just don’t let the headline number distract from the fact that quantization moves your bottleneck from “model quality” to “operational discipline around templates, artifacts, and load shape” — and that’s a much less forgiving place for a small mistake to hide.

Source: Deploying quantized models on Amazon SageMaker AI with Unsloth — AWS Machine Learning Blog