Terraform GPU Instance on AWS for LLM Inference (EC2 + vLLM)

Updated 2026-10-26 · 12 min read

A minimal, pinned Terraform config for one GPU EC2 instance that serves an LLM with vLLM or Ollama, reachable through SSM instead of open ports, plus the quota and cost traps to avoid.

To deploy an LLM on AWS EC2 with Terraform, you need one GPU instance (a g6.xlarge or g5.xlarge is the usual starting point), the AWS Deep Learning Base AMI looked up through a data source so drivers and Docker are already installed, an EBS root volume large enough for model weights, and a user_data script that starts vLLM or Ollama in a container. Reach it through SSM Session Manager port forwarding instead of opening SSH or the API port to the internet. Before the first terraform apply, check your GPU vCPU quota. On many accounts it starts at zero.

This guide gives you a minimal Terraform config that does exactly that, explains each choice, and covers the parts that cost money or time when you get them wrong: quotas, spot interruptions, disk sizing and forgetting to stop the instance.

What you get

• One GPU EC2 instance in your default VPC, in an Availability Zone that actually offers the instance type.
• The latest Deep Learning Base OSS Nvidia Driver GPU AMI (Ubuntu 24.04), resolved at plan time.
• An IAM instance profile for SSM, so no SSH key and no port 22.
• A security group with no inbound rules by default, plus an optional rule that opens the API port to one IP.
• vLLM serving an OpenAI-compatible API on port 8000, started by user_data with a pinned image tag.
• Optional spot pricing behind a single variable.

This is deliberately a single box. If you need autoscaling or several models behind one endpoint, you want Kubernetes or a managed service. For a team-internal endpoint, a proof of concept or a nightly batch job, a single instance is cheaper to run and much easier to debug.

Choosing a GPU instance type

The G families are the cost-effective choice for inference. P-family instances (A100, H100 and newer) make sense for large models or training, but they are rarely the first step.

| Family | GPU | GPU memory per GPU | Good for | |---|---|---|---| | g4dn | NVIDIA T4 | 16 GB | Small models (up to roughly 7B at 4-bit), embeddings | | g5 | NVIDIA A10G | 24 GB | 7B to 8B models in 16-bit with moderate context, larger ones quantized | | g6 | NVIDIA L4 | 24 GB | Same model sizes as g5, newer generation | | g6e | NVIDIA L40S | 48 GB | Around 30B models quantized, 8B with long context or high concurrency |

The .xlarge size of g5 and g6 has one GPU, 4 vCPUs and 16 GiB of system memory. Larger sizes in the same family mostly add CPU and RAM, not GPU memory, until you reach the multi-GPU sizes such as g6.12xlarge. Buy the size by GPU memory first, then make sure the host RAM is enough to load the weights. vLLM in particular stages weights through system memory.

For a rough sizing rule: weights take about parameters times bytes per parameter, so an 8B model in 16-bit needs about 16 GB before the KV cache. The VRAM requirements guide walks through the full calculation, including context length and concurrency.

Not every instance type exists in every region or Availability Zone. The config below asks AWS which AZs offer your type and places the instance there, which avoids the classic Unsupported error at apply time.

The GPU quota gotcha

EC2 quotas for GPU instances are counted in vCPUs, not instances, and are separate for on-demand and spot:

| Quota name | Code | Applies to | |---|---|---| | Running On-Demand G and VT instances | L-DB2E81BA | On-demand g4dn, g5, g6, g6e, ... | | All G and VT Spot Instance Requests | L-3819A6DF | Spot requests for the same families |

A g6.xlarge needs 4 vCPUs of quota. A g6.12xlarge needs 48. If your quota is lower than the instance's vCPU count, terraform apply fails with VcpuLimitExceeded after everything else has been created.

Check before you start:

aws service-quotas get-service-quota \
  --service-code ec2 --quota-code L-DB2E81BA \
  --query 'Quota.Value'

aws service-quotas get-service-quota \
  --service-code ec2 --quota-code L-3819A6DF \
  --query 'Quota.Value'

Request an increase if needed:

aws service-quotas request-service-quota-increase \
  --service-code ec2 --quota-code L-DB2E81BA --desired-value 8

Quotas are per region. Increases for GPU families on new accounts are often reviewed manually and can take a while, so request them before you need them, and ask for the vCPU count of the instance you plan to use, not a round number far above it.

Terraform GPU instance on AWS: the configuration

Four files. Commit them together with .terraform.lock.hcl so every run uses the same provider build.

versions.tf

terraform {
  required_version = ">= 1.6.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.70"
    }
  }
}

provider "aws" {
  region = var.region

  default_tags {
    tags = {
      Project = "llm-inference"
      Owner   = var.owner
    }
  }
}

~ 6.70 allows newer 6.x releases from 6.70 on but never 7.0. The lock file pins the exact version until you run terraform init -upgrade. Default tags make the instance easy to find in Cost Explorer later.

variables.tf

variable "region" {
  type    = string
  default = "us-east-1"
}

variable "owner" {
  type = string
}

variable "instance_type" {
  type    = string
  default = "g6.xlarge"
}

variable "root_volume_gb" {
  type    = number
  default = 200
}

variable "model_id" {
  type    = string
  default = "Qwen/Qwen2.5-7B-Instruct"
}

variable "vllm_image" {
  type    = string
  default = "vllm/vllm-openai:v0.32.0"
}

variable "api_key" {
  type      = string
  sensitive = true
}

variable "use_spot" {
  type    = bool
  default = false
}

# Leave null to keep the API private (SSM port forwarding only).
variable "allowed_cidr" {
  type    = string
  default = null
}

main.tf

data "aws_ami" "dlami" {
  most_recent = true
  owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["Deep Learning Base OSS Nvidia Driver GPU AMI (Ubuntu 24.04) *"]
  }

  filter {
    name   = "architecture"
    values = ["x86_64"]
  }

  filter {
    name   = "state"
    values = ["available"]
  }
}

data "aws_vpc" "default" {
  default = true
}

data "aws_ec2_instance_type_offerings" "gpu" {
  location_type = "availability-zone"

  filter {
    name   = "instance-type"
    values = [var.instance_type]
  }
}

data "aws_subnets" "gpu" {
  filter {
    name   = "vpc-id"
    values = [data.aws_vpc.default.id]
  }

  filter {
    name   = "availability-zone"
    values = data.aws_ec2_instance_type_offerings.gpu.locations
  }
}

resource "aws_iam_role" "llm" {
  name = "llm-inference-ssm"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "ec2.amazonaws.com" }
      Action    = "sts:AssumeRole"
    }]
  })
}

resource "aws_iam_role_policy_attachment" "ssm" {
  role       = aws_iam_role.llm.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
}

resource "aws_iam_instance_profile" "llm" {
  name = "llm-inference-ssm"
  role = aws_iam_role.llm.name
}

resource "aws_security_group" "llm" {
  name   = "llm-inference"
  vpc_id = data.aws_vpc.default.id

  dynamic "ingress" {
    for_each = var.allowed_cidr == null ? [] : [var.allowed_cidr]
    content {
      description = "vLLM API from one address"
      from_port   = 8000
      to_port     = 8000
      protocol    = "tcp"
      cidr_blocks = [ingress.value]
    }
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_instance" "llm" {
  ami                    = data.aws_ami.dlami.id
  instance_type          = var.instance_type
  subnet_id              = data.aws_subnets.gpu.ids[0]
  vpc_security_group_ids = [aws_security_group.llm.id]
  iam_instance_profile   = aws_iam_instance_profile.llm.name

  instance_initiated_shutdown_behavior = "stop"

  metadata_options {
    http_tokens = "required"
  }

  root_block_device {
    volume_size           = var.root_volume_gb
    volume_type           = "gp3"
    encrypted             = true
    delete_on_termination = true
  }

  dynamic "instance_market_options" {
    for_each = var.use_spot ? [1] : []
    content {
      market_type = "spot"
      spot_options {
        spot_instance_type             = "persistent"
        instance_interruption_behavior = "stop"
      }
    }
  }

  user_data = templatefile("${path.module}/user_data.sh.tftpl", {
    vllm_image = var.vllm_image
    model_id   = var.model_id
    api_key    = var.api_key
    bind_addr  = var.allowed_cidr == null ? "127.0.0.1" : "0.0.0.0"
  })

  user_data_replace_on_change = true

  tags = {
    Name = "llm-inference"
  }

  lifecycle {
    ignore_changes = [ami]
  }
}

output "instance_id" {
…

A few choices worth explaining:

• AMI lookup. The name filter matches AWS's published naming, Deep Learning Base OSS Nvidia Driver GPU AMI (Ubuntu 24.04) followed by a build date, owned by amazon. The base AMI ships the NVIDIA driver, CUDA, Docker and the NVIDIA Container Toolkit, which is all a containerized inference server needs. The framework-specific DLAMIs (PyTorch and others) add packages you do not need when the model server runs in its own container. AWS also publishes the latest ID as an SSM parameter at /aws/service/deeplearning/ami/x8664/base-oss-nvidia-driver-gpu-ubuntu-24.04/latest/ami-id, which you can read with the awsssm_parameter data source instead.
• **ignore_changes = [ami].** AWS releases new DLAMI builds regularly. Without this, a routine terraform apply weeks later would replace your instance and its disk because a newer AMI appeared. Remove the line when you deliberately want to roll to a new image.
• IMDSv2 required. http_tokens = "required" blocks the older metadata API that SSRF bugs in web apps can abuse to steal instance credentials.
• **instanceinitiatedshutdown_behavior = "stop".** A shutdown from inside the instance stops it rather than terminating it, so an idle-shutdown script cannot delete your disk.

user_data.sh.tftpl

#!/bin/bash
set -euo pipefail

mkdir -p /opt/hf-cache

# Wait until the GPU driver is ready on first boot.
until nvidia-smi > /dev/null 2>&1; do sleep 5; done

docker run -d --name vllm \
  --restart unless-stopped \
  --gpus all \
  --ipc=host \
  -p ${bind_addr}:8000:8000 \
  -v /opt/hf-cache:/root/.cache/huggingface \
  ${vllm_image} \
  --model ${model_id} \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --api-key ${api_key}

The ${...} placeholders are filled by templatefile, not by the shell. If you add shell variables of your own to this file, write them as $${VAR} so Terraform leaves them alone.

Why each flag is there:

• Pinned image tag. vllm/vllm-openai:v0.32.0 instead of latest, because vLLM changes defaults and flags between releases, and a reboot should not silently upgrade your model server. Bump it on purpose and re-test.
• --ipc=host. vLLM uses shared memory between processes, and Docker's default shared memory size is too small. The vLLM docs recommend this flag or a large --shm-size.
• --max-model-len 8192. Caps the context so the KV cache fits next to the weights in 24 GB. Without it, vLLM tries the model's full native context and may refuse to start for lack of memory. Raise it once you know the headroom.
• --restart unless-stopped. The container comes back after a reboot or a spot stop and start, and serves from the cached weights on disk.
• Bind address. With allowed_cidr unset, the port is bound to 127.0.0.1 and is reachable only through SSM port forwarding.

Gated models such as Llama need a Hugging Face token. Pass it with -e HFTOKEN=..., ideally read at boot from SSM Parameter Store rather than baked into userdata. Anyone with ec2:DescribeInstanceAttribute permission can read user data, and the same applies to the API key here. For anything beyond a test, fetch both from Parameter Store or Secrets Manager at boot and grant the instance role read access to just those parameters.

Ollama instead of vLLM

If you want to swap models from the command line, or run GGUF quantizations, replace the docker run block with Ollama:

docker run -d --name ollama \
  --restart unless-stopped \
  --gpus all \
  -p 127.0.0.1:11434:11434 \
  -v ollama:/root/.ollama \
  ollama/ollama:0.40.11

until docker exec ollama ollama list > /dev/null 2>&1; do sleep 2; done
docker exec ollama ollama pull llama3.1:8b

Ollama has no built-in API authentication, so keep it on 127.0.0.1 and change the forwarded port to 11434. Which server fits depends on the workload: vLLM is built for concurrent throughput, Ollama for convenience. Ollama vs vLLM vs llama.cpp compares them in detail.

Sizing the EBS volume for models

The root volume holds the OS, Docker images and model weights. Rough budget:

• AMI base. The root volume cannot be smaller than the AMI's snapshot. Check the snapshot size of the resolved AMI in the console or with aws ec2 describe-images.
• Container images. The vLLM image with its CUDA libraries is several gigabytes.
• Model weights. Parameters times bytes per parameter: about 15 GB for a 7B to 8B model in 16-bit, about 40 GB for a 70B model at 4-bit. Hugging Face downloads can also include duplicate weight formats, so leave headroom.
• Headroom for a second model and for Docker's layer churn when you upgrade images.

200 GB of gp3 is a comfortable starting point for one or two mid-sized models. gp3 lets you raise throughput and IOPS separately from size if model loading from disk is slow. Changing rootvolumegb later resizes the volume in place, but you still need to grow the filesystem inside the instance.

Access through SSM instead of SSH

With the instance profile attached and outbound HTTPS allowed, the instance registers with Systems Manager. Install the Session Manager plugin for the AWS CLI locally, then:

terraform output -raw port_forward | bash

# in a second terminal
curl http://localhost:8000/v1/models \
  -H "Authorization: Bearer $API_KEY"

For a shell on the box, use aws ssm start-session --target . Useful checks once inside:

sudo docker logs -f vllm
nvidia-smi
sudo cat /var/log/cloud-init-output.log

The first start takes a while: the image pull and the model download both happen on first boot. Follow docker logs until vLLM reports that the server is listening.

If the instance never appears in Session Manager, check three things in order: the instance profile is attached, the subnet has a route to the internet (or VPC endpoints for ssm, ssmmessages and ec2messages), and the SSM agent is running on the instance.

Spot vs on-demand

| | On-demand | Spot | |---|---|---| | Price | List price | Discounted, varies by AZ and over time | | Interruptions | None | AWS can reclaim the instance with a two-minute warning | | Capacity | Can still be unavailable for popular GPU types | Often scarcer for GPU types | | Quota | L-DB2E81BA | L-3819A6DF | | Good for | Endpoints people depend on during working hours | Batch jobs, evaluation runs, experiments |

With use_spot = true, the config requests a persistent spot instance that stops instead of terminating when interrupted. The EBS volume and the downloaded weights survive, and the instance restarts when capacity returns, with the container coming back up through Docker's restart policy. What you cannot control is when that happens. For an interactive endpoint, an interruption in the middle of a workday is a real outage. For a job that checkpoints its progress, it is a minor delay.

Note that switching use_spot on an existing instance forces a replacement in Terraform, so decide before the first apply, or accept that the disk and cached models are recreated.

Cost control: stop, destroy, alert

A GPU instance bills per second while it runs, whether or not anyone sends a request. The habits that keep the bill predictable:

• Stop when idle. A stopped instance does not bill compute, but its EBS volume does. aws ec2 stop-instances --instance-ids keeps the disk and the models for the next start. To manage this in Terraform, the awsec2instance_state resource can set the state to stopped or running.
• Destroy when done. terraform destroy removes everything, including the volume. Nothing in this config keeps billing after a destroy.
• Schedule it. For a working-hours endpoint, stop it automatically outside those hours with an EventBridge Scheduler rule or a cron job on a small machine that has the permission.
• Idle shutdown. A cron job on the instance that runs shutdown -h now when GPU utilization has been zero for a while works because of instanceinitiatedshutdown_behavior = "stop".
• Budget alert. Create an AWS Budgets alert (also available as the awsbudgetsbudget resource) on the project tag, so a forgotten instance shows up as an email rather than a surprise invoice.

Next steps

• Put a chat UI in front of the endpoint with the Ollama-based chatbot stack, or point the RAG pipeline stack at the vLLM API, since it speaks the OpenAI format.
• Browse the MLOps stacks for serving and monitoring setups.
• When a single instance is no longer enough, see Deploy Open WebUI and Ollama on Kubernetes with Helm for a GPU node pool setup.