Recipes on Modelplane Docs

All recipes

Every validated recipe in one table: model, size, architecture, precision, and the verified hardware. Select a row for the full recipe.

Model	Size	Arch	Precision	Verified on	Notes
Qwen3-8B qwen	8B	Dense	BF16	EKS L4	An 8.2B dense chat model on one NVIDIA L4.
Qwen2.5-7B qwen	7B	Dense	AWQ INT4	Vultr A16	A 7B dense chat model (AWQ INT4) on one NVIDIA A16 on Vultr.
Qwen3-Coder-480B qwen	480B A35B	MoE	BF16 / FP8	EKS H200	A 480B code MoE, multi-node BF16 over EFA or single-node FP8 on SGLang.
Qwen2.5-72B qwen	72B	Dense	AWQ INT4	AKSNebius A100H100	A 72B dense chat model (AWQ INT4) on one 80 GB GPU, on AKS and Nebius.
Kimi-K2 moonshotai	1T A32B	MoE	INT4	EKS H200	A 1T MoE served prefill/decode disaggregated across two H200 nodes.
Llama-3.1-8B meta-llama	8B	Dense	BF16	EKSGKE L4	An 8B dense chat model on one NVIDIA L4.
GLM-4.5-Air zai-org	106B A12B	MoE	GGUF IQ4_XS	GKE A100	A 106B MoE served from a GGUF checkpoint via llama.cpp on one A100.
Nemotron-3.5-Lightning nvidia	30B A3B	MoE	NVFP4	Nebius H100	An open 30B MoE with 3B active parameters served NVFP4 on one H100 on Nebius.
Laguna-S-2.1 poolside	118B A8B	MoE	FP8	Nebius H100	A 118B code MoE served FP8 on one 8x H100 node on Nebius.

Qwen3-8B

An 8.2B dense chat model on one NVIDIA L4. It’s the smallest recipe, with one Standalone engine, no cache, and weights pulled straight from Hugging Face.

This recipe was run end to end; the InferenceClass and ModelDeployment are the exact manifests from that run. Apply the platform side first, then the ML side.

Validated deployments

Dense 8B 16,384 ctx vLLM

Cloud

Google Cloud Azure Nebius Vultr

GPU

A16 16G L4 24G A100 40/80G H100 80G H200 141G 1× per node

Serving mode

Standalone LeaderWorker PrefillDecode

Precision

Engine

vLLM SGLang llama.cpp

Image

vllm/vllm-openai:v0.23.0

Features

Speculative decoding for low latency and small batch sizes

Manifests

docs/manifests/recipes/qwen3-8b

Platform

# InferenceClass for the L4 shape, validated serving Qwen3-8B on EKS.
#
# One NVIDIA L4 on an EKS g6.xlarge. The single GPU is a claim: DRA device;
# the scheduler matches a ModelDeployment's nodeSelector against its declared
# capacity and DRA binds it to the serving pod.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
 name: eks-l4-1x-g6
spec:
 description: "EKS g6.xlarge, 1x NVIDIA L4"
 provisioning:
 provider: EKS
 eks:
 instanceType: g6.xlarge
 diskSizeGb: 100
 accelerator:
 type: nvidia-l4
 count: 1
 devices:
 - name: gpu
 claim: DRA
 driver: gpu.nvidia.com
 deviceClassName: gpu.nvidia.com
 count: 1
 attributes:
 architecture: { string: Ada Lovelace }
 capacity:
 # The L4's real usable VRAM as the NVIDIA DRA driver reports it, not the
 # nominal 24GB.
 memory: { value: "23034Mi" }

# An EKS InferenceCluster with one L4 node pool, labeled for the
# ModelDeployment's clusterSelector to target.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
 name: eks-l4
 labels:
 modelplane.ai/region: us
spec:
 cluster:
 source: EKS
 eks:
 region: us-west-2
 nodePools:
 - name: gpu-l4
 className: eks-l4-1x-g6
 nodeCount: 1
 minNodeCount: 1
 maxNodeCount: 1
 zones:
 - us-west-2a

Deployment

# Qwen3-8B served on a single NVIDIA L4, validated end to end on EKS.
#
# An 8.2B dense model is a single Standalone engine: one self-contained vLLM
# pod, no ModelCache, weights pulled straight from Hugging Face. The flags carry
# real meaning beyond fit:
#
# --tool-call-parser=hermes the parser for Qwen3 dense (qwen3_xml is
# for Qwen3-Coder, not this model). Qwen3's
# tool-use template ships in the tokenizer,
# so no --chat-template is needed.
# --reasoning-parser=qwen3 with
# --default-chat-template-kwargs turns thinking off. Qwen3 thinks by
# default, burying a one-line answer under a
# <think> block and forbidding greedy decode.
# --max-model-len / --gpu-memory-utilization L4 fit, not correctness.
#
# No --port or --host: Modelplane's routing expects the engine on its default
# :8000 with a /health probe, and passes args through verbatim.
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
 name: qwen3-8b
 namespace: ml-team
spec:
 replicas: 1
 template:
 spec:
 clusterSelector:
 matchLabels:
 modelplane.ai/region: us
 engines:
 - name: qwen3-8b
 members:
 - role: Standalone
 nodeSelector:
 devices:
 - name: gpu
 count: 1
 selectors:
 - cel: |
 device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("20Gi")) >= 0
 template:
 spec:
 containers:
 - name: engine
 image: vllm/vllm-openai:v0.23.0
 args:
 - "--model=Qwen/Qwen3-8B"
 - "--served-model-name=$(MODELPLANE_SERVED_MODEL_NAME)"
 - "--max-model-len=16384"
 - "--gpu-memory-utilization=0.92"
 - "--reasoning-parser=qwen3"
 - "--default-chat-template-kwargs={\"enable_thinking\": false}"
 - "--enable-auto-tool-choice"
 - "--tool-call-parser=hermes"

# Exposes the qwen3-8b deployment as one model a caller can name. Modelplane
# labels each composed ModelEndpoint with the deployment name, so this selector
# reaches every replica. Name the model,
# ml-team/qwen3-8b, in a request to an InferenceGateway that serves it.
apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
 name: qwen3-8b
 namespace: ml-team
spec:
 endpoints:
 - name: qwen3-8b
 selector:
 matchLabels:
 modelplane.ai/deployment: qwen3-8b

Speculative decoding

The same model and platform also serve with n-gram (prompt-lookup) speculative decoding, which proposes tokens by matching the prompt and so doesn’t need a draft model or a second set of weights. On copy-heavy output, editing a pasted code block where most output tokens are copied from the prompt, it roughly doubles decode throughput and halves the time per output token:

Qwen2.5-7B

A 7B dense chat model served from an AWQ INT4 quantization on one NVIDIA A16 on Vultr: one Standalone engine, no cache, weights pulled straight from Hugging Face. The A16 slice on the vcg-a16-6c-64g-16vram plan has 16 GiB of VRAM, so the INT4 weights (~5 GiB) fit with headroom for KV cache; --gpu-memory-utilization=0.85 and --enforce-eager keep the engine inside the small card.

This recipe was run end to end on Vultr (ewr); the InferenceClass, InferenceCluster, and ModelDeployment are the exact manifests from that run. GPU plan availability varies by Vultr region, so check the plan is offered in your region before applying. Apply the platform side first, then the ML side.

Nemotron-3.5-Lightning

NVIDIA’s Nemotron-3.5-Lightning, an open 30B mixture-of-experts model with 3B active parameters built for the execution layer of long-running agents, served NVFP4 as a Standalone vLLM engine on one H100 node on Nebius. Because the NVFP4 checkpoint (~20 GiB) fits one GPU with headroom for the KV and Mamba caches, the engine doesn’t need tensor parallelism, a gang, or prefill/decode disaggregation. Weights stage once to a ModelCache on a Nebius shared filesystem and mount at /mnt/models.

Qwen3-Coder-480B

A 480B code MoE (35B active), validated in two deployments. The BF16 weights span two H200 nodes as a gang over EFA, served from a ModelCache. The FP8 checkpoint fits one node, so it runs as a Standalone engine on SGLang with no cache.

Both deployments were run end to end; the InferenceClass and ModelDeployment are the exact manifests from those runs. Apply the platform side first, then the ML side. Edit the EC2 capacity reservation placeholder in the InferenceCluster before applying it.

GLM-4.5-Air

A 106B MoE served from an Unsloth GGUF checkpoint via llama.cpp instead of vLLM, on one A100 40 GB. Because Modelplane treats llama.cpp like any other OpenAI-compatible container, the only changes from a vLLM deployment are the image and args: the container is still named engine and listens on :8000. vLLM can’t load this Unsloth quantization format. llama.cpp can, and -hf pulls the checkpoint straight from Hugging Face at startup, so a one-time deployment needs no ModelCache.

Kimi-K2

A 1T MoE (1 trillion parameters) served prefill/decode disaggregated across two H200 nodes: two engines, one per phase, with Modelplane composing the llm-d routing layer between them. This recipe serves an INT4 quantization of the model; the native FP8 weights need four such nodes.

This recipe was run end to end; the InferenceClass and ModelDeployment are the exact manifests from that run. Apply the platform side first, then the ML side. Edit the EC2 capacity reservation placeholder in the InferenceCluster before applying it.

Laguna-S-2.1

Poolside’s Laguna-S-2.1 (118B total, 8B active MoE) served FP8 as a Standalone vLLM engine on one 8x H100 node on Nebius. The FP8 weights (~121 GiB) fit one node with headroom for KV cache, so the engine is tensor-parallel across the 8 GPUs over NVLink and doesn’t need a gang or prefill/decode disaggregation. Weights stage once to a ModelCache on a Nebius shared filesystem and mount at /mnt/models.

Qwen2.5-72B

A 72B dense chat model served from an AWQ INT4 quantization on one 80 GB GPU per replica: one Standalone engine fed by a ModelCache. The platform side covers an A100 on AKS and an H100 on Nebius, and the ML side is the same manifest for both. The deployment has two replicas and no clusterSelector. Each pool has exactly one GPU, so with both platforms applied one replica runs on each. The service then splits traffic between the two GPUs by weight, which the last section uses to compare them. To serve on just one platform, apply one tab and drop replicas to 1.

Llama-3.1-8B

An 8B dense chat model on one NVIDIA L4. It’s the entry recipe, with one Standalone engine, no cache, and public weights from a Hugging Face mirror. The deployment has no clusterSelector, so device capacity alone matches it to any compatible L4 in the fleet.

This recipe was run end to end on GKE; the InferenceClass, InferenceCluster, and ModelDeployment are the exact manifests from that run. The EKS platform shape is the standard single-L4 recipe. It passes server validation but was not served in this run. Apply the platform side first, then the ML side. Edit the GCP project placeholder in the GKE InferenceCluster before applying it.