Possibly an old question but…….

In an SP I am looping round a set of unit transactions to calculate the current holding. These transactions are held as multivalues as follows:

UNITS - 3006.58 @VM 2985.80 @VM 20.78

CRDR - DR @VM CR @VM CR

The problem area is this bit of code (not a direct copy, but a summary):

tempTot='

i=1

Loop while crDr[i]

If crDr[i]=CR' then

  tempTot += units[i]

end else

  tempTot -= units[i]

end

i += 1

repeat

As it loops through, tempTot equals the following:

i=1

  tempTot=-3006.58'

i=2

  tempTot=-20.7799999999998'

i=3

  tempTot=2.558E-13'

This causes two problems, when i=2 the basic maths is wrong, when i=3 it errors because 'E' is not numeric.

Any ideas where I've gone wrong, or do I need to perform conversions after every mathematical transaction?

Thank you

Paul,

My suggestion is to Iconv the UNITS variable first and then Oconv the final answer and everything will appear correct. For instance:

/* Changed your @VM to @FM. I'm sure this was a typo. */

UNITS=3006.58":@FM:"2985.80":@FM:"20.78"

UNITS=Iconv(Units, "MD2")

CRDR=DR":@FM:"CR":@FM:"CR"



tempTot='

i=1

Loop while crDr[i]

If crDr[i]=CR' then

tempTot += units[i]

end else

tempTot -= units[i]

end

i += 1

repeat



tempTot=Oconv(tempTot, "MD2")

[email protected]

SRP Computer Solutions

Thanks,

This does work, but why does it go wrong in the first place?

Should I store all figures as integers, or do I Iconv and Oconv them each time? Which is the best method?

Paul

Paul,

What version of OpenInsight are you using? Several problems caused by math rounding and conversion to exponentials have been fixed. Some of these fixes were in 3.0, 3.12, …, all the way up to present.

Cameron Purdy

Revelation Software

Cam,

I tried Paul's example on 3.61 and I get the same weird math.

Ditto.

[email protected]

SRP Computer Solutions

Paul,

The OpenEngine uses "float" operations to do the non-integer math. The C run-time library routine we use have 15 digit precision (64-bit), and the result you are seeing is caused by two similar six digit numbers resulting in a four digit number so the result has a 13 digit precision (which is 15-(6-4)). We try internally to detect rounding errors and precision loss by the floating point unit but in this case the loss is too significant (two orders of magnitude worse than the expected) so it does not get rounded back.

The suggestion to store numbers as masked decimal (i.e. display "10.55" but store 1055 using MD2) is very good if you are deciding before you have tons of work done.

If it is too late, use MD20 conversion to reformat the result, which basically rounds to 2 digits instead of 13 in this case. In your case the last line of code would say:

tempTot=oconv(tempTot, "MD20")

Please let me know if that works well for the problem you saw. Thanks.

-Gene

Back when OI was in beta, I told them just because it's new technology it doesn't mean you have to use the new math. But do they listen….sigh…..

Gary

Not surprising. Paul's on 3.6.1.

[email protected]

Sprezzatura, Inc.

You want to work with the OCONV values. Essentially, you convert all your pence up into pounds and work that way. It's only when you start working with percentages and things that you have to start really thinking. Everything's off by a factor of 100, and with some percentages and things, you have to factor that in, remembering multiplication is associative, so you might have to change these to factors of 10000. It can be pretty gnarly at first to get used to, but once you do, you'll never look at numbers the same way again.

[email protected]

Sprezzatura, Inc.

View this thread on the forum...