MaaS Inference Capacity Planner v0.27 @Evren BAYCAN Model-as-a-Service planning estimate Calculating

MaaS Inference Sizer v0.27 · GPU, KV & Service Capacity

Bu araç, üretim ortamındaki Model-as-a-Service kurulumlarını boyutlandırır: model ağırlıkları, KV önbelleği, aktivasyonlar, bağımsız DP/TP/PP/EP paralelliği, ağ etkisi, offload maliyeti, VRAM uygunluğu, TTFT, TPOT, token üretim hızı, istek kapasitesi ve SLA için ayrılan pay. Sonuçlar planlama tahminidir; üretici performans garantisi değildir.

Parameter help
1 · Service & Optimization2 · Model & Precision3 · Compute & HA4 · Parallelism5 · Request & Scheduler6 · KV & Memory7 · Production Calibration8 · Offload
Service planning

1) MaaS Service & Optimization

Choose a MaaS service template, adapt its targets, and review the matching scheduler and parallelism proposal. Apply commits the proposed changes.
01
Balanced Production API

Current Topology-
Profile Proposal-
Identifies the engine and its validation context; does not add a speed multiplier.
Average arrivals; peak load includes the configured burst factor.
Calculating adjustmentsAutomatic
Adjustment details

Suggestions preserve model, hardware inventory, prompt/output shape, admitted concurrency, prefix reuse assumptions and service targets.
Model footprint

2) Model & Precision

Select the model, weight format and KV precision. Weight memory uses total parameters; MoE throughput uses active parameters while still validating the full checkpoint footprint.
02
Curated models above 40B total parameters. MoE active parameters may be lower. Custom Model remains available for other checkpoints.
BF16/FP16, FP8/MXFP8, NVFP4/MXFP4, AWQ, GPTQ and INT8. W = weight bits; A = activation bits. Exact checkpoint and kernel support must match.
KV precision materially changes long-context and high-concurrency memory demand.
Total Params-
Active Params-
Layer / Hidden-
KV Architecture Factor-
Precision metadata & custom model overridesAdvanced
Covers scales, metadata and packing overhead beyond raw bit width.
Percentage of backbone parameters retained in 16-bit; separate native lookup tables keep their own format. Ignored for BF16, FP16 and FP32. For MoE, placement changes the active compute mix.
Explicit planning assumption for the entered 16-bit percentage. Shared-first changes active compute and expert traffic; it does not infer the checkpoint's actual retained percentage.
0 = parameter estimate. One complete loaded weight set before offload, DP copies and extra EP replication; includes quantization metadata, excludes KV/runtime buffers.
Infrastructure estate

3) Compute Topology, HA & Fabrics

Define the active node estate, accelerator density, dedicated spare capacity and the local/inter-node fabrics before selecting model-parallel placement.
03
Custom GPU allows manual VRAM, memory bandwidth and relative compute calibration.
Count active physical servers; for NVL72, count complete 72-GPU racks. Cold spares are separate.
Physical GPUs per node; DP/TP/PP/EP controls how many are assigned to the pool.
Platform presets lock the compatible scale-up fabric. NVL72 uses one complete 72-GPU rack per inventory unit.
Checking the selected fabric against the current parallel topology…
High availability
Dedicated replacement capacity; excluded from steady-state throughput.
Enabled only with Custom Spare Count.
Active Accelerator Units8
Active Physical GPUs8
Active Node / Rack Layout1 × 8
Physical GPUs / Active Node or Rack8
Spare Nodes / Racks1
Total Provisioned Nodes / Racks2
Provisioned Accelerator Units16
Provisioned Physical GPUs16
Scale-up Fabric-
Scale-out Fabric-
Custom accelerator overrideAdvanced
vRAM / accelerator-
Memory BW-
Total usable vRAM-
Interconnect effective-
Placement strategy

4) Parallelism Strategy

DP, TP, PP and EP are independent controls. Enable only the methods the deployment will actually use; the sizer validates the resulting GPU equation and fabric boundaries.
04
For MaaS, keep TP/PP/EP only as wide as needed for model fit or latency; use DP for request-level horizontal scale when capacity is available.
0 methods enabled
Data Parallelism (DP)
Independent full-model replicas for request-level scale-out.
Disabled: one logical replica is used.
Tensor Parallelism (TP)
Shards model/tensor work across a low-latency GPU group.
TP communication is modeled only across the selected group.
Pipeline Parallelism (PP)
Splits layers into stages; useful for fit but can introduce pipeline bubbles.
Disabled: one pipeline stage.
Expert Parallelism (EP)
MoE-only expert sharding with explicit physical mapping.
EP is available only for MoE models.
DP Replicas-
TP Size-
PP Size-
EP Size-
Units / Replica-
Configured / Available Units-
Physical GPUs / Replica-
Weight Copies-
Enabled Methods-
Unused Units-
EP Device Group-
Replica vRAM Headroom-
Traffic shape

5) Request Shape & Scheduler

Model the request envelope separately from scheduler limits. Max Active Sequences is configured per DP replica; cluster active capacity scales with the number of replicas.
05
Each DP replica owns a scheduler; this limit determines active sequences and KV memory demand.
Count requests actually in flight at the same instant, not connected customers or sessions. For example, 1,024 requests with 64 active slots means 960 are queued.
Maximum prompt + output context envelope for one request.
Per-replica scheduler token budget; analogous to max batched/scheduled tokens in modern inference engines.
Cluster Active Capacity-
Active Now-
Queued / Waiting-
Active Replicas-
Online serving scheduler featuresScheduler
Recommended for online MaaS serving.
Reduces long-prefill interference with decode work.
Memory policy

6) KV Cache & GPU Memory Guardrails

Separate the resident KV working set, context occupancy and reserve policy from scheduler concurrency. These controls determine whether the selected service envelope can remain resident without excessive eviction or OOM risk.
06
0 = architecture formula. Before TP/EP replication; excludes fixed recurrent state.
0 = preset estimate. Independent of KV quantization; Kimi K3 includes KDA state.
Normally 100% of running sequences need KV state. Reduce only when measured eviction/swap/recompute behavior justifies it.
100% is conservative; lower only with measured context distributions.
Free KV capacity reserved for fragmentation, eviction and preemption headroom.
Usable fraction of physical VRAM before the explicit safety margin.
For production MaaS, calibrate resident ratio, KV occupancy and reserve using engine telemetry such as KV-cache usage, preemptions and real prompt/output distributions.
Capacity policy

7) Production Capacity & Calibration

Convert raw engine estimates into provisionable MaaS capacity by adding operating headroom, failover/autoscaling reserve and measured performance overrides.
07
Plans below the queueing cliff instead of selling 100% of raw engine capacity.
Capacity withheld for failures, rolling maintenance, bursts and scaling lag.
Peak offered load = Target RPS × Burst Factor.
Used only by the heuristic path when measured tok/s overrides are not supplied.
Optional measured calibration Auto-matched

Enter results from your own load test of this configuration. Each edited measurement is matched automatically to the current model, hardware, precision, workload and peak load. Changes make mismatched values inactive; re-enter a value only after measuring the new configuration. Zero uses the estimate. This offline tool does not run GPU benchmarks.

Phase rates are isolated prefill/decode tests at the configured batch. Capacity is the sustainable pool RPS from a load sweep. Endpoint p95 latency must match the configured peak load, including network and queue time.

0 = heuristic. Use the same model, precision, batch shape and topology as production.
0 = heuristic. Enter benchmark data when pricing or committing an SLA.
0 = dense hardware roofline where available, otherwise a generic compute index. Nonzero = achieved per-physical-GPU compute for this workload; no second quantization discount.
Overflow strategy

8) CPU / RAM / NVMe Offload

Use offload as an explicit capacity trade-off, not as a free VRAM extension. It can recover fit but usually increases latency and reduces sustainable throughput.
08

Summary Outputs

Last update: waiting
Configuration & Service TargetsWaiting for calculation.
Total VRAM Used
-
-
Token Generation Speed
-
estimated decode token/s per active sequence
Time to First Token
-
base/prefill + queue estimate
Decode-Phase Throughput
-
decode-only bound; prefill competes for GPU time
TPOT / ITL
-
estimated ms per generated token
Planned Production Request Capacity
-
raw capacity × operating utilization × (1 − operational reserve)
Raw Engine Request Capacity
-
at selected concurrent requests, before production reserve
Peak Target RPS
-
target RPS × burst factor
Queue Delay Estimate
-
mean burst cohort / M/M/c arrival proxy; not p95
Target RPS Load
-
peak target RPS / planned production capacity
MaaS SLO Status
-
-
Mixed-Workload Output Budget
—
planned RPS × average output tokens
Capacity After Node / Rack Loss
—
Surviving Complete Replicas
—
worst case for the displayed placement; cold spares excluded
Offload Host Memory Lower Bound
—
pool total; add OS, runtime, loading and NUMA headroom
VRAM Distribution-
Required Parallel Units-
vRAM Headroom-
Max KV-Resident Sequences-
Offload Penalty-

Parallelism Topology Schema

Node / GPU placement
Active Nodes / Racks-
Spare Nodes / Racks-
Total Provisioned Nodes / Racks-
Units per Node / Rack-
Physical GPUs per Node / Rack-
Replica Placement-
Stage Width-
Pipeline Stages-
DP Replicas-
Unused Active Units-
Scale-up Fabric-
Scale-out Fabric-
Waiting for calculation.

Sizing Recommendations & Warnings

Waiting for calculation
Batch Candidate / Replica (Memory)-
DP Candidate / Memory-Feasible Max-
Concurrency / Cluster Active Capacity-
Replica Memory Fit-
Batch / Active Replica-
Active DP Replicas-
DP Utilization-
Scheduling Waves-
Request Capacity-
Target RPS Load-
TTFT vs Target-
TPOT vs Target-
Queue Wait vs Budget-
Batch and DP candidates are exploratory settings, not a joint SLO guarantee. The optimization preview recalculates the proposed combination and reports its service result.
Recommendations are dynamic MaaS planning heuristics based on model/GPU topology, independently enabled DP/TP/PP/EP methods, scheduler limits, prompt/output distributions, concurrency, KV occupancy, SLA targets, memory headroom and interconnect. Validate production targets with engine-specific benchmarks and traffic traces.

Chart: Memory Components

The chart shows Base Model Weights, Activations, KV Cache, Framework Overhead, Multi-GPU Overhead and Safety Margin.

Detailed Calculation Table

Created by Evren Baycan · Sizer v0.27 · MaaS review · Updated: 2026-10-30