APFloat: x87DoubleExtended pseudo-NaNs (integer_bit==0) not handled as always-signalling.

[eddyb]

Based on https://reviews.llvm.org/D41868 (cc @labath @stephentyrone) and these snippets of APFloat.cpp:

llvm-project/llvm/lib/Support/APFloat.cpp

Lines 3673 to 3679 in ef7d537

	/// Integer bit is explicit in this format. Intel hardware (387 and later)
	/// does not support these bit patterns:
	/// exponent = all 1's, integer bit 0, significand 0 ("pseudoinfinity")
	/// exponent = all 1's, integer bit 0, significand nonzero ("pseudoNaN")
	/// exponent!=0 nor all 1's, integer bit 0 ("unnormal")
	/// exponent = 0, integer bit 1 ("pseudodenormal")
	/// At the moment, the first three are treated as NaNs, the last one as Normal.

llvm-project/llvm/lib/Support/APFloat.cpp

Lines 942 to 946 in ef7d537

	// For x87 extended precision, we want to make a NaN, not a
	// pseudo-NaN. Maybe we should expose the ability to make
	// pseudo-NaNs?
	if (semantics == &semX87DoubleExtended)
	APInt::tcSetBit(significand, QNaNBit + 1);

I would expect the following:

• pseudo-NaNs should always be treated as SNaNs
  • this would presumably also apply to pseudo-infinities and unnormals (all of which have in common integer_bit == 0 w/o being subnormals, AIUI)
  • not the case today: isSignaling ignores the integer bit and only checks the quiet bit (which a pseudo-NaN can set)
• quieting a pseudo-NaN should produce a legal QNaN (while setting the opInvalidOp status)
  • that is, makeQuiet should replicate makeNaN's APInt::tcSetBit(significand, QNaNBit + 1)
  • not the case today: makeQuiet fails to set the integer bit, potentially leaving a pseudo-NaN behind

Or in other words, I'd expect this test to pass (currently it fails to even generate an opInvalidOp status):

TEST(APFloatTest, x87PseudoNaN) {
  APFloat PseudoNaN(APFloat::x87DoubleExtended(), APInt(80, {1ULL << 62, 0x7fff}));
  EXPECT_TRUE(PseudoNaN.isNaN());
  EXPECT_TRUE(PseudoNaN.isSignaling()); // FAIL: pseudo-NaNs aren't detected as SNaNs

  // Any operation (which propagates input SNaNs while quieting them),
  // `roundToIntegral` just happened to have similar tests already.
  APFloat test = PseudoNaN;
  APFloat::opStatus St = test.roundToIntegral(APFloat::rmTowardZero);
  EXPECT_TRUE(test.isNaN());
  EXPECT_FALSE(test.isSignaling()); // would FAIL if pseudo-NaNs were detected as SNaNs,
                                    // and `makeQuiet` wasn't *also* updated
  EXPECT_FALSE(test.isNegative());
  EXPECT_EQ(APFloat::opInvalidOp, St); // FAIL: pseudo-NaNs aren't detected as SNaNs
}