Handling floating point numbers

The internal expression processor operates on internal floating point fields in their native, hexadecimal floating point format. External floating point fields are converted to double-precision internal format before being processed as if they were internal floating point fields. If two fields in a comparison have different precisions, then the field with the greater precision is rounded to the lesser precision, and the comparison is performed using the lesser precision. Because internal floating point fields are converted to external (decimal) format and passed to REXX as strings, and external floating point fields are also passed as strings, the result of evaluating a comparison can differ between REXX and the internal expression processor.

To ensure consistency, pay attention to the following points when coding expressions involving floating point fields:

• When comparing a floating point field to a constant, make sure you use a constant that matches the external representation used by FMN. For example, -2.2527494E+08 and -2.2527495E+08 are both stored as 'C7D6D6C4'x when entered for a single-precision floating point field, but File Manager always displays the data as -2.2527494E+08. Using either value matches with the internal expression processor, but only -2.2527494E+08 matches with REXX.
• Use hex strings and unformatted field references instead of external format to precisely represent internal floating point values. For example, instead of #2 = -2.2527494E+08 you can use #U2 = x2c(C7D6D6C4). This technique can only be applied if your expression requires REXX, because the x2c() function is not processed internally.
• Comparing floating point fields with different precisions using REXX might not produce the results expected because the external representations of floating point numbers with values that match (with or without rounding) but with different precisions are generally not the same. For example, the external representation of 'C7D6D6C4'x (-2.2527494E+08) is different from that for 'C7D6D6C4 00000000'x (-2.2527494400000000E+08) because the extra precision of the field produces a more precise external representation. This situation, where the internal values match exactly can be handled in REXX by truncating the longer field (assuming #2 refers to a short float and #3 refers to a long float):

  #U2 = Substr(#U3,1,4)

  Simulating the rounding process to achieve the same results as the internal processor, and therefore achieving a more general test of equality between fields of different precision requires the following extraordinary expression:

  #U2 = d2c(c2d(Substr(#U3,1,4))+Substr(x2b(c2x(Substr(#U3,5,1))),1,1),4)

  This works by:

  • Truncating the long float to four bytes and converting it to decimal.
  • Adding the high-order byte of the discarded half of the characteristic.
  • Converting the result back to internal format.

  The addition is valid because the characteristic is in the rightmost bits of the field, and any carry from the addition increments the mantissa as required. The only case it does not handle is that of exponent overflow.