HELP rounding in OIWG 3.2 (OpenInsight Specific) [Revelation On-Line Wiki]

OpenInsight Specific

Sign up on the Revelation Software website to have access to the most current content, and to be able to ask questions and get answers from the Revelation community

At 19 JUL 1998 11:33:18PM J JONES wrote:

Cameron if you're there or Donald, please HELP.

Back in OILAN 2.6 we had a terrible problem with addition

and subtraction and had to use the equation X=FMT(INT(X * 10000), "MD4") after each calculation.

Now we've taken our code to OIWG 3.2 with jillions of lines of code.

This is process manufacturing and we're online with several pretty

big customers. Now we find out that if you take 7million and

take the integer of that, you get something like 1280523264.

This is really really badly serious. Do you have any ideas?

Thanks - Jackie Jones

At 20 JUL 1998 01:32AM Don Bakke wrote:

Hi Jackie,

See if this thread gives you some answers.

dbakke@srpcs.com

SRP Computer Solutions

At 20 JUL 1998 07:46AM Cameron Revelation wrote:

Hi Jackie,

Could you give a specific example? I can see if it can be reproduced in 3.2 here as well as our latest version (3.61). Thank you,

Cameron Purdy

Revelation Software

At 22 JUL 1998 02:36PM J JONES wrote:

Hi Cameron, yes, thanks.

X=INT(12345678901) GIVES MINUS 539222987

X=INT(1234567890.1234567) Breaks with error non-numeric data
when numeric required.
Thanks again,

Jackie

At 23 JUL 1998 07:49AM Cameron Revelation wrote:

Jackie,

]] X=INT(12345678901) GIVES MINUS 539222987

This problem has been resolved since 3.2. In 3.61 I get:
<code>
run test "12345678901"

12345678901
Request successfully executed.  1 row(s) processed.
</code>

As an explanation for what you are seeing, in OpenInsight the maximum int value is 2^31-1, or roughly 2.1 billion, because OpenInsight uses 32-bit integers.

We modified the int() function to return a floating point value (ending with .0 so it appears to be an int) when the integer value is out of range of a 32-bit int.

]] X=INT(1234567890.1234567) Breaks with error non-numeric data when numeric required.

This is still a known issue in 3.61. The maximum number of digits in a 64-bit float is 15. The C runtime library therefore believes that string values with additional digits are not numbers.

The number you showed in this case could never be produced in OpenInsight as a result of a mathematical operation. Do you have cases where you assemble numbers from strings that are this large?

Cameron Purdy

Revelation Software

View this thread on the forum...