Floating Point Errors That Break Sig Fig Calculations

Significant figures (sig figs) are a fundamental tool for expressing measurement precision. They tell us how many digits in a value are meaningful, given the uncertainty of the measurement. However, in the digital age, most calculations are performed using binary floating-point arithmetic, which can introduce subtle errors that corrupt sig fig calculations. This article explores the intersection of floating-point representation and significant figure rules, highlighting common pitfalls and providing standards-based guidance for precision-aware rounding.

Rule Statement

The standard rules for determining significant figures are well known:

• All non-zero digits are significant.
• Zeros between non-zero digits are significant.
• Leading zeros are not significant.
• Trailing zeros after a decimal point are significant.
• Trailing zeros in a number without a decimal point are ambiguous.

These rules assume that the number is expressed exactly in decimal notation. In practice, however, computers store numbers in binary floating-point format (typically IEEE 754 double precision), which cannot represent most decimal fractions exactly. For example, 0.1 in binary is a repeating fraction, just as 1/3 is in decimal. This fundamental limitation means that every floating-point operation is subject to rounding error, which can accumulate and affect the number of significant figures in a result.

Worked Examples

Consider the simple addition 0.1 + 0.2. In decimal arithmetic, the exact sum is 0.3, which has one significant figure (or one decimal place). In IEEE 754 double precision, the result is 0.30000000000000004. If we blindly apply sig fig rules to this binary result, we might count 17 significant figures, which is absurd. The correct approach is to round the result to the appropriate number of decimal places based on the original operands.

Step-by-step:

1. Identify the number of decimal places in each operand: 0.1 has 1 decimal place, 0.2 has 1 decimal place.
2. The result should be rounded to the same number of decimal places (1).
3. So the answer is 0.3, regardless of the binary representation.

Another example: multiply 1.23 × 4.56. Both have 3 significant figures. The exact product is 5.6088. In floating point, you might get 5.608799999999999. Rounding to 3 sig figs gives 5.61, which is correct. However, if you round intermediate steps or use a calculator that displays only a limited number of digits, you might introduce additional errors.

Counter-Examples

A classic counter-example is the rounding of 2.675 to two decimal places. In decimal, 2.675 rounded to two decimal places using half-up rounding is 2.68. But in binary floating-point, 2.675 is represented as 2.6749999999999998, so rounding to two decimal places yields 2.67. This is a direct consequence of the binary representation, and it breaks the expected half-up rule.

Another counter-example: 1.005 rounded to two decimal places. The decimal value 1.005 is represented in binary as 1.0049999999999999, so rounding gives 1.00 instead of 1.01. These errors are not random; they are deterministic and can be reproduced across platforms.

Convention Comparison Table

These methods are defined in various standards, but none of them account for the binary representation error. The only robust solution is to use decimal arithmetic or to apply a correction based on the known precision of the input.

Standards Citation

Several standards address rounding and precision in scientific and technical contexts:

• ISO 80000-1:2009, Section 7.3.4, specifies rules for rounding of numerical values. It recommends that rounding should be performed on the exact value, not on a rounded intermediate result.
• NIST SP 811, Section 7.2, provides guidance on significant figures and rounding. It states that “the last significant figure in a reported result should be consistent with the uncertainty” and that “rounding should be done at the end of the calculation.”
• JCGM 100:2008 (GUM), Section 7.2.6, discusses rounding of measurement results. It recommends that the numerical value of a result be rounded to the same number of significant figures as the uncertainty.
• ASTM E29-13, Section 6, defines standard practice for using significant figures in test data. It emphasizes that the rounding procedure must be specified and applied consistently.

None of these standards explicitly address binary floating-point errors, but they all imply that calculations should be performed with sufficient precision to avoid rounding errors before the final rounding step. This is often achieved by using guard digits or decimal arithmetic.

Common Mistakes

• Assuming floating-point results are exact: Every floating-point operation is rounded to the nearest representable value, introducing an error of up to half a unit in the last place (ULP).
• Rounding intermediate results: Rounding each step of a calculation can compound errors. Always carry extra digits and round only the final result.
• Using binary floating-point for financial or scientific decimal calculations: Many languages offer decimal arithmetic libraries (e.g., Python’s decimal module, Java’s BigDecimal) that should be used when exact decimal representation is required.
• Ignoring the ambiguity of trailing zeros: In floating-point, a number like 1200.0 may have 5 significant figures, but if stored as a double, it’s not clear how many are intended.
• Not using guard digits: When performing a series of operations, keep at least one extra digit to minimize rounding error propagation.

Software Behavior Note

Different software environments handle floating-point and sig figs in varying ways:

• Python: The built-in float uses IEEE 754 double precision. The decimal module provides arbitrary-precision decimal arithmetic, which is ideal for sig fig calculations. The round() function uses bankers’ rounding (half-even) by default.
• JavaScript: All numbers are IEEE 754 doubles. There is no built-in decimal type, but libraries like decimal.js or big.js can be used. The toFixed() method rounds half-up, but it can produce unexpected results due to binary representation.
• MATLAB: Uses double precision by default. The round function uses half-away-from-zero. For exact decimal arithmetic, the Symbolic Math Toolbox can be used.
• Excel: Uses IEEE 754 doubles with a display precision of 15 significant digits. Excel’s rounding functions (ROUND, ROUNDUP, ROUNDDOWN) operate on the binary representation, which can cause the 2.675 problem.

For critical applications, always use a decimal arithmetic library or explicitly format the output to the desired number of significant figures using a known algorithm.

Quick Reference Table

Sources & Further Reading

• ISO 80000-1:2009, Quantities and units – Part 1: General
• NIST SP 811, Guide for the Use of the International System of Units (SI)
• JCGM 100:2008, Evaluation of measurement data – Guide to the expression of uncertainty in measurement (GUM)
• ASTM E29-13, Standard Practice for Using Significant Figures in Test Data
• IEEE 754-2019, Standard for Floating-Point Arithmetic

For more on rounding methods, see our Rounding Methods guide. For related rules, explore Sig Figs in Addition and Subtraction and Sig Figs in Multiplication and Division.