Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Strange behavior with double variables
#2
Floating point calculations are not completely precise. For example, 0.03+0.02+0.01 gives result 0.060000000000000005. You can use function Round() to make it 0.06.

Macro Macro1756
Code:
Copy      Help
double Amount

;Amount=0.06
Amount=0.03+0.02+0.01

out
out "Amount: %.5G" Amount
out "Amount: %.20G" Amount ;;0.060000000000000005

if Amount=0.06;; <<<<<<<<<< ERROR <<<<<<<<<<
,out "OK"
else out "ERROR!"

Amount=Round(Amount 2)

if Amount=0.06
,out "OK"
else out "ERROR!"

Same in other languages, eg VBScript.
Macro Macro1758
Code:
Copy      Help
str vbs=
;dim Amount
;Amount=0.03+0.02+0.01
;msgbox Amount
;if Amount=0.06 then
,;msgbox "OK"
;else msgbox "ERROR!"
;end if
VbsExec vbs


Messages In This Thread

Forum Jump:


Users browsing this thread: 1 Guest(s)