Reference library8 entries

Category: Tools & code

Guides and reference for implementing precision and rounding rules in software, spreadsheets, calculators, and markup languages.

How to Enter and Read E-Notation on the Casio fx-991 Calculator

E-notation on Casio fx-991 calculators displays numbers as m E n, meaning m × 10^n. Enter it with the ×10^x or EXP key, read…

Read explanation

SQL and databases

Significant Figures in SQL: Rounding in Queries and Reporting

A comprehensive guide to maintaining metrological precision in SQL databases, contrasting standard rounding functions with the rigorous requirements of significant figures.

Read explanation

Significant Figures in Excel: How to Round to a Given Number of Sig Figs

Excel has no built-in significant-figures function, but you can round to n sig figs with a formula using ROUND, LOG10, and ABS; understanding Excel's…

Read explanation

SQL and databases

Significant Figures in SQL: How to Round and Store Numeric Data in Databases

SQL databases separate exact DECIMAL/NUMERIC storage from approximate FLOAT/REAL storage. Significant-figure rounding is usually a presentation or conformance step, not a storage default.

Read explanation

Significant Figures in Excel, Sheets, Python, R, and Calculators: A Precision Reference

Learn how different software tools handle significant figures and rounding, with standards citations, worked examples, and common pitfalls to avoid.

Read explanation

Why Python’s round() Uses Banker’s Rounding: A Precision Reference

Python's round() implements banker's rounding (round half to even) to reduce cumulative bias in statistical operations. This article explains the rationale, standards, and practical…

Read explanation

Rounding to Significant Figures in Python: Precision, Pitfalls, and Best Practices

Learn how to round numbers to a specified number of significant figures in Python, including the pitfalls of binary floating-point, banker's rounding, and standards-based…

Read explanation

Sig Figs in R, MATLAB, JavaScript, and SQL — Behaviour Compared

Rule Statement Significant figures (sig figs) are the digits in a number that carry meaning contributing to its measurement resolution. The rules for determining…

Read explanation