Short Answer
Rule Statement
In measurement, the least count (also called the smallest scale division, resolution, or readability) is the value of one division on an analog scale or the last displayed digit on a digital instrument. A recurring question is how much uncertainty to assign to the act of reading that scale. Two conventions dominate textbooks: half-division and full-division.
Half-division convention: The reading uncertainty is ±0.5 × least count. This assumes the observer can interpolate to about half of the smallest division, and that other error sources are negligible. It is common in introductory physics and chemistry labs.
Full-division convention: The reading uncertainty is ±1 × least count. This is more conservative. It assumes no reliable interpolation, or that the instrument specification already includes a ±1 count error. It appears in many engineering and metrology texts, especially for digital displays.
Half-division is an interpolation convention; full-division is a conservative bounding convention. The GUM treats resolution as a Type B uncertainty component that must be modeled, not chosen by habit.
Neither convention is universally correct. The GUM Guide does not mandate half or full division; it requires you to model the resolution as a probability distribution. For a digital display that rounds to the nearest increment, the maximum quantization error is half the increment, but manufacturer specifications often quote ±1 count. This is why textbooks differ.
Convention Comparison Table
| Convention | Assumed reading model | Typical uncertainty | Distribution | When used |
|---|---|---|---|---|
| Half least count | Observer interpolates to half a division | ±0.5 d | Often treated as rectangular or uniform | Introductory physics, chemistry labs, analog scales |
| Full least count | No interpolation; one division is the reading error | ±1 d | Often treated as a conservative bound | Engineering texts, digital displays, manufacturer specs |
| Digital ±1 count | Last digit may be off by one unit | ±1 LSD | Rectangular or triangular depending on model | Digital multimeters, balances, calipers |
| GUM rectangular half-width | True value lies uniformly within ±d/2 | u = d/(2√3) | Rectangular | Calibration labs, uncertainty budgets |
Here d is the least count or resolution. The symbol u denotes standard uncertainty. Expanded uncertainty U is often k u with k = 2 for approximately 95% coverage.
Worked Examples
Example 1: Analog ruler with 1 mm divisions
A ruler has smallest divisions of 1 mm. You measure a line as 12.3 cm.
- Half-division: uncertainty = ±0.5 × 1 mm = ±0.5 mm. Report 123.0 ±0.5 mm.
- Full-division: uncertainty = ±1 × 1 mm = ±1 mm. Report 123 ±1 mm.
- GUM rectangular model: half-width = 0.5 mm, so u = 0.5/√3 = 0.29 mm. Expanded U = 2 × 0.29 = 0.58 mm. This is close to the half-division expanded value.
The half-division convention is reasonable if you can interpolate and calibration errors are small. The full-division convention is safer if the scale is coarse or parallax is significant.
Example 2: Digital balance with 0.01 g resolution
A balance displays 12.34 g. The last digit is 0.01 g.
- Quantization limit: if the display rounds to nearest, the true value lies within ±0.005 g of 12.34 g. This is the half-division idea.
- Many textbooks use full-division: ±0.01 g, i.e. one count in the last digit.
- GUM rectangular model: u = 0.005/√3 = 0.0029 g; U = 0.0058 g.
- If the manufacturer states accuracy as ±0.02 g, that specification dominates. Do not use resolution alone.
Example 3: Analog voltmeter with 0.1 V divisions
A voltmeter scale has 0.1 V divisions. You read 4.6 V.
- Half-division: ±0.05 V. Report 4.60 ±0.05 V.
- Full-division: ±0.1 V. Report 4.6 ±0.1 V.
- If the meter is rated ±1% of full scale, that may be larger than either resolution term. Always compare contributions.
Counter-Examples
- Using full division when interpolation is possible: A ruler with 1 mm marks can often be read to 0.5 mm. Using ±1 mm overstates uncertainty by a factor of two.
- Using half division for a digital display when the spec says ±1 count: The half-division quantization model may underestimate the manufacturer’s stated error.
- Confusing resolution with accuracy: A balance that reads 0.01 g may have accuracy ±0.02 g. Resolution is not accuracy.
- Double counting: Adding half-division and full-division terms together is wrong. Choose one model or combine independent sources properly.
- Reporting too many digits: 12.345 ±0.5 g should be rounded to 12.3 ±0.5 g. See Rounding Rules and Significant Figures.
Standards Citation
The GUM (JCGM 100:2008, Evaluation of measurement data — Guide to the expression of uncertainty in measurement) is the primary international reference. Clause 4.3 covers Type B evaluation, including information from manufacturer specifications, calibration certificates, and resolution. Clause 4.3.1 specifically addresses manufacturer specifications and calibration certificates. The GUM does not prescribe half or full division; it requires a distribution model.
NIST TN 1297, Guidelines for Evaluating and Expressing the Uncertainty of NIST Measurement Results, Section 4.6 discusses Type B evaluation and notes that scientific judgment is required. Resolution is one such Type B source.
ISO 80000-1:2022, Clause 6.4, gives rounding rules for numbers. It does not set a reading-uncertainty convention, but it governs how you round the final reported value.
ASTM E29-22, Section 6, covers rounding test data for conformance. It is a rounding standard, not an uncertainty standard, but it is often cited alongside uncertainty reporting.
ISO/IEC 17025:2017, Clause 7.6.1, requires testing and calibration laboratories to identify uncertainty contributions, including resolution, and to report uncertainty appropriately.
Discipline Note
Different fields favor different conventions. Physics education often teaches half-division because students are expected to interpolate and because it is simple. Engineering metrology tends to use full-division or GUM-style Type B models, especially when instrument specifications quote ±1 count. Chemistry often emphasizes significant figures and manufacturer accuracy, so the full last digit is common. Astronomy and other fields with noise-dominated measurements may ignore scale resolution entirely because random noise dominates. The correct choice depends on the measurement model, not the textbook tradition.
Common Mistakes
- Assuming half-division is always correct. It is only correct under specific interpolation assumptions.
- Assuming full-division is always conservative enough. It may miss calibration or systematic errors.
- Forgetting that digital resolution error is at most half the last digit when rounding to nearest, but specs often use one full digit.
- Using resolution uncertainty when the instrument accuracy specification is larger.
- Mixing standard uncertainty u with expanded uncertainty U. Always state k.
- Rounding uncertainty to too many digits. One or two significant figures is standard.
- Not matching the decimal place of the measurement and its uncertainty.
Quick Reference Table
| Instrument | Resolution d | Half-division | Full-division | GUM u (rectangular) | GUM U (k=2) |
|---|---|---|---|---|---|
| Ruler | 1 mm | ±0.5 mm | ±1 mm | 0.29 mm | 0.58 mm |
| Digital caliper | 0.01 mm | ±0.005 mm | ±0.01 mm | 0.0029 mm | 0.0058 mm |
| Digital balance | 0.01 g | ±0.005 g | ±0.01 g | 0.0029 g | 0.0058 g |
| Analog voltmeter | 0.1 V | ±0.05 V | ±0.1 V | 0.029 V | 0.058 V |
These values assume resolution is the only uncertainty source. In practice, compare with calibration, accuracy, and repeatability contributions. For propagation, see Error Propagation.
Sources & Further Reading
- JCGM 100:2008, Evaluation of measurement data — Guide to the expression of uncertainty in measurement (GUM).
- NIST Technical Note 1297, Guidelines for Evaluating and Expressing the Uncertainty of NIST Measurement Results.
- ISO 80000-1:2022, Quantities and units — Part 1: General, Clause 6.4.
- ASTM E29-22, Standard Practice for Using Significant Digits in Test Data to Determine Conformance with Specifications, Section 6.
- ISO/IEC 17025:2017, General requirements for the competence of testing and calibration laboratories, Clause 7.6.1.
This site is a precision and rounding reference, not just a calculator. Explore related rules: Measurement Uncertainty, Significant Figures, Rounding Rules, and Error Propagation.
FAQ
Why do some textbooks use full scale division instead of half?
Because they treat the reading error as a conservative bound, assume no interpolation, or follow the digital ±1 count convention. Some also define uncertainty as a maximum possible error rather than a standard uncertainty.
Is half the smallest division always correct?
No. It assumes the observer can interpolate to half a division and that other errors are negligible. If the instrument specification is larger, or if the display is digital with ±1 count accuracy, half-division may underestimate uncertainty.
What does the GUM say about resolution?
The GUM treats resolution as a Type B uncertainty. For a digital display with resolution d, a common rectangular model uses half-width d/2, giving u = d/(2√3). The GUM does not mandate half or full division.
How should I report uncertainty with significant figures?
Round the uncertainty to one or two significant figures, then round the measurement to the same decimal place. Use k = 2 for expanded uncertainty when appropriate. Our significant figures calculator can help apply these rules consistently.
Leave a Reply