2016-03-12
Programming with floating point values leads to numerical round off errors due to the nature of binary numbers. For a more detailed discussion see this article or this one. Basically it boils down to the fact that not all real numbers can be represented by a finite binary sequence. Due to this phenomena comparing floating point values directly is strongly discouraged as the results can be unexpected. Normally, the absolute value of the difference is taken and if it is less than some tolerance value it is accepted as a match.
In the work that I have done in the past, the tolerance has always
been a tricky situation. I would take the tolerance equal to some very
small number, say 1e-10. The problem with this approach is
that it seems to be dependent on the machine and operating system you
are using. It is also affected by 32 bit and 64 bit systems. I ended up
approaching it from an actual physical problem where the tolerance has a
physical meaning and not just trying to preserve the maximum amount of
decimal points. Keep in mind that my experimentation in this area was
using visual basic 6 and visual basic.net.
I work with CAD systems and the better ones are able to identify the
units of length used. For the purposes of this article we’ll assume two
length units: feet and meters. This poses a problem for our tolerance
testing. If we choose a generic tolerance of 0.000001 and
our units of measure is in feet. The 0.000001 = 1.2e-5
inches that sounds like a very small number and from experimentation I
found that it works well. If that same value of 0.000001 is
used when the units are meters, it translates to 0.001
millimeters. From experience this tolerance didn’t work very well when
the units were meters. It missed a lot of checks that were just outside
the tolerance limit.
Instead of using a tolerance like 1e-6 I took a look at
what I was trying to do. For example, I was running an intersection
check between to 3D line segments. Typically 3D line segments do not
intersect and are known as skew lines. It is
possible to calculate the distance between two skew lines. If the
distance is 0 then they intersect.
The CAD system was used to model an underground drilling pattern.
Typically the length of the holes could be anywhere from 5 feet to 200
feet. The accuracy is definitely not in the 1E-6 range.
=========== ===============
0.000001 ft 0.000012 inches
0.00001 ft 0.00012 inches
0.0001 ft 0.0012 inches
0.001 ft 0.012 inches
0.01 ft 0.12 inches
0.1 ft 1.2 inches
=========== ===============
========== ==========
0.000001 m 0.001 mm
0.00001 m 0.01 mm
0.0001 m 0.1 mm
0.001 m 1 mm
0.01 m 10 mm
0.1 m 100 mm
========== ==========I ended up choosing 0.001 as my tolerance value. In
meters it represents a tolerance range of 1 mm or so and in feet it is
0.012 inches. These values are more than acceptable given
the accuracy of the physical system that I was modelling. The other
approach to round off error is to actually round the values to a
specific number of decimal places before making the comparison or simply
truncating the value.