SET PRECISION example
In dBASE PLUS 9the product now supports High Precision Math, this means the total possible PRECISION can be set to 34.
You can temporarily SET PRECISION by using the SET PRECISION TO <nInt>
Or you can permanently SET PRECISION by going to the menu item ... Properties | Desktop Properties 'Programming' tab and setting 'Precision' under 'Command Output'
NOTE: You should SET DECIMAL TO ... to match the new PRECISION support
The following examples demonstrate how numbers are represented and how the precision setting affects data comparisons:
set decimals to 34 // to see as many digits as possible
set precision to 34 // maximum
? 0.5 // 0.500000000000000 - exact
? 0.375 // 0.375000000000000000 - exact
? 0.4 // 0.400000000000000022 - 16 digits precision
? 1/9 // 0.1111111111111111111111111111111111 - 34 digits precision
? 12345.4 // 12345.400000000000000 - 15 digits precision
? 123456789.4 // 123456789.400000010000000 8 digits precision
OLD: ? 12345.4 - 12345 // 0.399999999999636202 11
digits precision
NEW: ? 12345.4 - 12345 // 0.4
OLD: ? 12345.4 - 12345 == 0.4 // False, too much
precision attempted
NEW: ? 12345.4 - 12345 == 0.4 // True?
? 301.840002 - 301.000001 //0.840001
? 123456789.012345 * 87654321.123456789 //10821521028958940.344459595060205
This example shows how SET PRECISION affects comparing two large integers
In dBASE PLUS 8 and dBASE PLUS 9 ...
With SET PRECISION = 10
nIDX
= q.rowset.fields['IDx'].value //Where the current field value is 12345678908.00
?nIDX // returns 12345678908.00
nVAl
= 12345678908
?nVAl // returns 1234567E+4
With
SET PRECISION = 11
nVAl = 12345678908
?nVAl // returns 12345678908