A neural network accelerator is often introduced by its number of multiply-accumulate units. That is a useful starting point, but it does not explain whether a real model will run efficiently. The more revealing questions are where operands come from, how often they are reused, and which parts of execution remain outside the fast compute array.
Start with a Workload, Not a Peak Number
For a matrix multiplication C = A × B, let A have shape M × K and B have shape K × N. Computing every output requires M × N × K multiply-accumulates. If a multiply and an add count as two operations, the nominal work is 2MNK operations. This convention needs to be stated whenever a throughput number is compared.
M = N = K = 64
macs = M * N * K # 262,144 multiply-accumulates
operations = 2 * macs # 524,288 operations under this convention
Now imagine expanding the array while leaving its input interface unchanged. More multipliers cannot help once they are waiting for data. For a teaching model, an upper bound on sustained throughput is the smaller of compute capacity and memory bandwidth multiplied by arithmetic intensity. Real execution also pays for control, synchronization, and operations that do not fit this simplified model.
Reuse Is an Architectural Choice
Tiling divides a large operation into pieces that fit nearby storage. An output-stationary mapping keeps partial sums near a processing element; a weight-stationary mapping emphasizes reuse of weights. The tradeoff is which other operands must move and how the mapping fits each layer.
Eyeriss made dataflow a central design question. Its row-stationary approach balances reuse of weights, activations, and partial sums rather than optimizing only one kind of movement. This is a useful example of evaluating the memory hierarchy together with the processing array.
The original TPU study is another concrete example: it describes a matrix-oriented inference ASIC with a large software-managed on-chip memory. Read it as a particular architecture evaluated on particular workloads, rather than treating its historical comparisons as a universal ranking of present-day hardware.
Precision Changes More than the Multiplier
Quantization maps real values to a smaller integer representation. A common affine model reconstructs a value as:
x_approx = scale * (q - zero_point)
Jacob et al. describe integer-only inference and a corresponding training procedure. In hardware, the important questions include operand representation, accumulation width, rescaling, and the point where values are rounded or clipped.
Changing an array of FP32 values to one-byte integers reduces that array's raw payload by a factor of four. It does not imply a fourfold reduction in total system memory or a fourfold speedup: scales, intermediate values, wider accumulators, conversions, and unsupported operators still exist. Measure the entire path that matters to the application.
Algorithms Can Reduce Memory Traffic Too
Not every advance requires a new chip. FlashAttention reorganizes exact attention with tiling to reduce transfers between high-bandwidth memory and on-chip SRAM. It illustrates how changing the schedule can improve execution on an existing processor.
This suggests a practical design order: inspect the operator graph, improve data movement and scheduling, and then decide which remaining bottleneck needs specialized hardware. Otherwise, an accelerator can become very good at executing an unnecessarily expensive schedule.
A Useful Evaluation Checklist
- State the workload: model, tensor shapes, batch size, sequence length, and preprocessing.
- State the numeric format: operand and accumulator precision, plus any measured quality change.
- Measure an explicit boundary: a kernel, a complete model, or an end-to-end request including transfers.
- Separate throughput and latency: batching can increase completed work per second while delaying an individual request.
- Explain utilization: include padding, small layers, and time spent outside the compute array.
- Compare the same task: use equivalent inputs, outputs, and quality requirements.
A good small accelerator project can begin with one matrix tile, a clear operand schedule, and a correctness reference. Once that works, vary tile sizes and memory bandwidth to see where performance stops improving. The resulting bottleneck is often more informative than the largest theoretical operations-per-second number.