On-Device NPU Bottleneck Calculator
Compare compute-bound and memory-bound throughput for an on-device NPU inference workload. The lower modeled rate is the limiting resource under the assumptions you enter; this is not a benchmark of a specific device.
Result
Estimated limiting resource
Estimated maximum throughput
Compute-bound throughput
Memory-bound throughput
Time per inference at throughput limit
How to use this tool
- Enter the NPU peak TOPS and measured compute efficiency for the model's numeric precision and workload.
- Enter operations per inference in TOP and measured sustained memory bandwidth in GB/s.
- Enter total memory traffic per inference in MB and measured memory efficiency.
- Compare compute-bound and memory-bound rates; the lower rate is the simple throughput limit.
Assumptions
- TOPS, operations per inference, and measured efficiencies must use compatible operation types and numeric precision (for example, INT8 versus FP16).
- Memory traffic is user-estimated and should account for weights, activations, cache reuse and batching as appropriate; the calculator does not infer those from a model file.
- The estimate models a single simplified steady-state bottleneck. It excludes CPU/GPU offload, queueing, batch latency, thermal throttling, kernel launch overhead and device-specific scheduling.
Technical details: what the fields mean
- NPU peak compute
npuPeakTops - Declared or measured peak NPU throughput in TOPS; the operation precision must match operationsPerInferenceTops.
- Measured compute efficiency
computeEfficiencyPercent - Measured fraction of peak compute throughput achieved by this workload, in percent.
- Model operations per inference
operationsPerInferenceTops - Model operations required for one inference, expressed in TOP (10^12 operations).
- Sustained memory bandwidth
memoryBandwidthGbPerSecond - Sustained memory bandwidth available to the NPU in decimal GB/s (1 GB = 1,000 MB).
- Measured memory efficiency
memoryEfficiencyPercent - Measured fraction of sustained memory bandwidth achieved by this workload, in percent.
- Memory traffic per inference
memoryTrafficMbPerInference - Total model weight and activation traffic moved to/from memory per inference, in decimal MB; do not enter model file size unless all those bytes are transferred for each inference.
Methodology
Compute-bound rate = NPU peak TOPS × compute efficiency ÷ model TOP per inference. Memory-bound rate = memory GB/s × 1,000 MB/GB × memory efficiency ÷ MB transferred per inference. Estimated throughput = the lower rate; compute-bound time is 1,000 ms ÷ that throughput.
Worked example
An NPU with 45 TOPS and 150 MB memory traffic per inference
Inputs: NPU peak compute: 45 TOPS, Measured compute efficiency: 60%, Model operations per inference: 0.9 TOP, Sustained memory bandwidth: 68 GB/s, Measured memory efficiency: 70%, Memory traffic per inference: 150 MB
Result: Compute-bound rate is 30.0 inferences/s; memory-bound rate is about 317.3 inferences/s. The simple model is Compute-bound at about 30.0 inferences/s (33.3 ms compute-bound time per inference).
Practical guidance
- Use measured sustained bandwidth and model traffic, not the NPU's theoretical peak alone; cache reuse and tiling can change traffic substantially.
- Validate both rates with a device benchmark at the target batch size, precision, thermal state and latency target.
Common mistakes
- Comparing INT8 NPU TOPS with an FP16 operation count without converting to a compatible basis.
- Using model weight-file size as per-inference memory traffic even when weights are reused from cache.
FAQs
What does Compute-bound mean?
The modeled compute rate is lower than the memory-bound rate for the inputs entered. It does not prove the physical device has no other bottleneck.
Does the result predict real latency?
No. It gives a simple throughput bound and compute-bound time estimate; actual latency depends on batching, kernels, CPU/GPU work, memory reuse, thermals and runtime scheduling.
Last reviewed: 2026-10-18
A two-resource arithmetic model only. It does not measure a device, inspect a model graph, account for CPU/GPU fallback or predict tail latency and power.
This tool provides an indicative estimate, based on the assumptions stated above.
Theme and tool preferences, along with saved and recently used lists, are stored only in this browser.