11 Benchmark contracts and measurement protocol
On this page · Visual guide: The objects and their contracts · Visual guide: Same output before comparing times · Permalink · Source line 1808
Performance data are meaningful only after the mathematical and timing contracts are fixed. The protocol follows standard concerns from numerical linear algebra, floating point analysis, and machine performance modeling [Higham2002], [GolubVanLoan2013], [WilliamsWatermanPatterson2009].
Definition 11.1: Case signature
A case signature records the dimensions, packet layout, rank, right hand side count, random seed, source values, source type, reference computation, and output contract.
Permalink · Source line 1812 · Claim record · Lean coverage · Code · Worked examples
Definition 11.2: Fair match
Two measured rows are a fair match when they share one case signature, return the same declared output, use the same correctness reference, and have identical timing scopes. If one row includes an extra conversion or setup cost, it is a different scope and must be reported separately or added to both rows before comparison.
Permalink · Source line 1816 · Claim record · Lean coverage · Code · Worked examples
Proposition 11.3: Fair matching is an equivalence relation
On rows with complete metadata, equality of the case signature, output contract, correctness reference, and timing scope is an equivalence relation.
Read proof · Read statement and proof side by side · Permalink · Source line 1820 · Claim record · Lean coverage · Code · Worked examples
Proof of Proposition 11.3
Equality of each metadata field is reflexive, symmetric, and transitive. Their conjunction has the same properties.
Back to statement · Read statement and proof side by side · Permalink · Source line 1824 · Claim record · Lean coverage · Code · Worked examples
Definition 11.4: Representation gain
Representation gain compares dense and structured work or storage for the same operator, independently of the processor.
Permalink · Source line 1828 · Claim record · Lean coverage · Code · Worked examples
Definition 11.5: Device gain
Device gain compares processors for the same representation, output, and timing scope.
Permalink · Source line 1832 · Claim record · Lean coverage · Code · Worked examples
Definition 11.6: Kernel gain
Kernel gain compares two implementations on the same device for the same representation, output, and timing scope.
Permalink · Source line 1836 · Claim record · Lean coverage · Code · Worked examples
Definition 11.7: Amortization gain
Amortization gain records the effect of one time discovery, compilation, transfer, capture, or materialization at a declared reuse count.
Permalink · Source line 1840 · Claim record · Lean coverage · Code · Worked examples
A measured total speedup can mix these four effects. A multiplicative decomposition is only exact when matched rows isolate the factors. Otherwise the decomposition is interpretive.
Remark 11.8: Logical work rate
A structured method may report a dense equivalent logical work rate greater than the physical dense throughput because it does not execute the dense operations. This statistic describes avoided work. It is not a claim that the device performed those operations.