News:

MASM32 SDK Description, downloads and other helpful links
MASM32.com New Forum Link
masmforum WebSite

A glitch in FloatToStr

Started by jj2007, September 22, 2011, 10:15:30 PM

Previous topic - Next topic

jj2007

Under certain conditions, FloatToStr erroneously returns 0.0xx as follows:

.data
r8n Real8 0.00999999
r8o Real8 0.0099999999
r8p Real8 0.009999999999
r8q Real8 0.00999999999999
r8r Real8 0.0099999999999999


0.009999990     r8n MasmBasic
9.999990e-003   r8n FloatToStr

0.010000000     r8o MasmBasic
0.000000e-003   r8o FloatToStr

0.010000000     r8p MasmBasic
0.000000e-003   r8p FloatToStr

0.010000000     r8q MasmBasic
0.000000e-003   r8q FloatToStr

0.010000000     r8r MasmBasic
0.000000e-003   r8r FloatToStr


FloatToStr2 is not affected by this rounding error. Example attached (for use with the MasmBasic library).