Quantcast
Channel: SCN: Message List - PowerBuilder Developer Center
Viewing all articles
Browse latest Browse all 7329

Re: Age Old Floating Point Problem

$
0
0

Your solution gives wrong results if your starting expression is like '0.045'

 

You can try this approach instead:

 

global function decimal f_round (double adb, long al)

     return round(double(string(adb)),al)

end function

 

// test

double ldb, ldb_wrong, ldb_ok

ldb = .01*29*double(1.5)

ldb_wrong = round(ldb,2)

ldb_ok = f_round(ldb,2)

messagebox("results", string(ldb_wrong) + "~r~n" + string(ldb_ok))


Viewing all articles
Browse latest Browse all 7329


<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>