Tolerance Testing - Determining an Appropriate Tolerance#

Programming with floating point values leads to numerical round off errors due to the nature of binary numbers. For a more detailed discussion see this article or this one. Basically it boils down to the fact that not all real numbers can be represented by a finite binary sequence. Due to this phenomena comparing floating point values directly is strongly discouraged as the results can be unexpected. Normally, the absolute value of the difference is taken and if it is less than some tolerance value it is accepted as a match.

abs(x - y) <= tolerance

In the work that I have done in the past, the tolerance has always been a tricky situation. I would take the tolerance equal to some very small number, say 1e-10. The problem with this approach is that it seems to be dependent on the machine and operating system you are using. It is also affected by 32 bit and 64 bit systems. I ended up approaching it from an actual physical problem where the tolerance has a physical meaning and not just trying to preserve the maximum amount of decimal points. Keep in mind that my experimentation in this area was using visual basic 6 and visual basic.net.

I work with CAD systems and the better ones are able to identify the units of length used. For the purposes of this article we’ll assume two length units: feet and meters. This poses a problem for our tolerance testing. If we choose a generic tolerance of 0.000001 and our units of measure is in feet. The 0.000001 = 1.2e-5 inches that sounds like a very small number and from experimentation I found that it works well. If that same value of 0.000001 is used when the units are meters, it translates to 0.001 millimeters. From experience this tolerance didn’t work very well when the units were meters. It missed a lot of checks that were just outside the tolerance limit.

Instead of using a tolerance like 1e-6 I took a look at what I was trying to do. For example, I was running an intersection check between to 3D line segments. Typically 3D line segments do not intersect and are known as skew lines. It is possible to calculate the distance between two skew lines. If the distance is 0 then they intersect.

The CAD system was used to model an underground drilling pattern. Typically the length of the holes could be anywhere from 5 feet to 200 feet. The accuracy is definitely not in the 1E-6 range.

=========== ===============
0.000001 ft 0.000012 inches
0.00001  ft 0.00012  inches
0.0001   ft 0.0012   inches
0.001    ft 0.012    inches
0.01     ft 0.12     inches
0.1      ft 1.2      inches
=========== ===============

========== ==========
0.000001 m   0.001 mm
0.00001 m    0.01  mm
0.0001 m     0.1   mm
0.001 m      1     mm
0.01 m      10     mm
0.1 m      100     mm
========== ==========

I ended up choosing 0.001 as my tolerance value. In meters it represents a tolerance range of 1 mm or so and in feet it is 0.012 inches. These values are more than acceptable given the accuracy of the physical system that I was modeling. The other approach to round off error is to actually round the values to a specific number of decimal places before making the comparison or simply truncating the value.