Custom float types

Currently FloatType is tied to LLVM APFloat due to the method llvm::fltSemantics &FloatType::getFloatSemantics() . Therefore it can only support floating point types supported in LLVM.

It seems common in ML to have language/front end with custom floating point type that don’t have any HW support and wouldn’t fit in LLVM.
Right now the solution would be to make a new MyFloatType and use that, however the major downside is that basic Type helpers like getIntOrFloatBitWidth won’t work anymore making the code much less uniform.

It would be nice to be able to make FloatType an interface instead however it seems like a significant change in the design of this type.

Is there a reason why FloatType is tied to the backend/LLVM or did this happen for historical reasons?

Also I’m curious to know how others have solved this so far.

Before we started getting to sub-byte floating point types, the consensus was that it was easy enough to teach APFloat the semantics and inherit “full” support for these that are minor variations on the standard. That has worked well enough up to this point. I implemented a couple of them (E4M3, E5M2) along with the RFC that established the precedent. It wasn’t meant to be the only way to ever do it.

Outside of those bounds, I think we were in a wait and see mode for what comes next. Many of the alternative floating point formats that I have seen actually bear little resemblance to the traditional definition of an element-based floating point value: they tend to be block based in some fashion and logically would require some kind of pack/unpack for general use (or map directly to a hardware intrinsic for supported compute operations). I would apply the same rationale to the sub-byte floats because a way to view them would be as packed sequences of raw values.

I’ve been thinking on/off on it for a bit (but not super seriously), and I think we’re missing a StructuredFloat type of some kind which represents various of these packed formats. Probably a job for a dialect dedicated to that vs a built-in/LLVM-pegged type.

Are you able to say more about what kinds of types you are wanting to support? Would be good to compare notes and figure out a solution.

I think there is definitely a missing interface here. Just not exactly sure what it is.

In what way are we tied to the LLVM backend?
I assumed that APFloat Support is all we need For new floating point types in MLIR?

Interesting, but being in APFloat requires the type is not something completely made up by software I imagine?

Yes I assumed APFloat was limited to formats used by some upstream LLVM backend but maybe this is not correct?

Interesting, yes it sounds like that would address multiple problems I have.

There are multiple cases but some of the simple cases are not even sub-bytes but just supporting custom fp8 formats (pre-existing to what was later added to AMD/NVIDIA HW). You can see some of the related code here. Then yes it would be nice to also support blocked based type as you described if possible.

Right, I was wondering if we should have an interface IntOrFloatLikeType for this kind of helpers. I haven’t dug enough to know whether it would help enough to be justified.

It’s not limited, see the answers in the thread posted above, like here for example. APFloat addition are considered isolated / non-intrusive, and an easier bar to pass than adding a LLVM type.

1 Like

So far (going on memory of discussions), indeed yes, we have been ok adding types to APFloat that exist in some form in hardware, but we have not had a discussion about completely arbitrary things. The general principle is that if it is (going to be) in common use, it is better to have the compiler know the details. So far, landing in some commodity hardware roadmap is a pretty good indication of it being in common use, and implementing in APFloat helps such cases.

There have been various points raised over the years calling for an OpaqueFloat of some kind. I don’t have an opinion on that except that every case I’ve looked at fell into the “Implement it in APFloat” category or was actually not a FloatType at all from the compiler’s viewpoint (i.e. more of the StructuredFloat thing I mentioned).

I’m not any kind of authority on what all is possible wrt floating point. I just named a handful of them and have seen a sampling of the weird and tried to sort use cases.

1 Like

I think Stella summarized it right upthread - “support for these that are minor variations on the standard.”

My understanding of block float formats is that the exponent is stored separately from the mantissa - which means that a tensor of these floating point values share an exponent. If this is the case, then this isn’t a variant of the IEEE representation, this is a completely different species.

Let’s explore what adding this to APFloat would mean. If you did that, then you could model each element as an APFloat/ConstantFP, and then try to use constant abstractions on top of that. However, while this may seem attractive in that it allows reusing some infrastructure, it won’t end well - such a thing will be widely inefficient use of MLIR attributes, but moreover it isn’t a correct way to model the tensor: each element would redundantly represent the exponent, so inserting an element would allow changing the shared exponent, not just the mantissa.

If this is the sort of thing we’re talking about, then I think you should use MLIRs ability to model the domain precisely, and come up with a new thing to represent this problem space.

-Chris

1 Like

Thanks Chris. To clarify, I’m the one who brought up the block floating point and asked Thomas if that’s what he was talking about (since I know it is on a lot of folk’s minds). He said he was talking primarily about something else prompting this question but was also interested in the block fp topic.

I think Thomas is looking for advice on three things:

  1. What about “close to the standard” types that have been left behind in the FP8 category: For these, I think the easiest/best thing to do is just give them a name and add them to APFloat (unless if on further inspection they are pretty far afield).
  2. What about extensibility for a kind of opaque or bring your own floating point type in MLIR. I haven’t seen one of these yet in practical use that wouldn’t fit into APFloat, but I think the question was also along the lines of “can I just have an easy way to define such a thing for myself but still be float-like from MLIR’s perspective”.
  3. Block floating point or other arrangements that start to look more like structs of multiple elements than a single scalar.

I don’t have an opinion on #2. For #3, we’re going to need to define a new class of things, like you say.

1 Like

Oh thank you for the clarification - my opinion is that APFloat is fine for representing scalar floating point values, things that have mantissa/exponent in some format. I’m not aware of “how weird” fp8 is, but it is probably fine.

One test on how far we should stretch this: should APFloat support fixed point representations like “24.8” etc?

Another random comment: the P is wrong in the APFloat name, it should probably be called something like AFloat, “AbitraryFloat” or “AbstractFloat” or GenericFloat or something like that, since (unlike APInt) it really isn’t about arbitrary precision - it is about precise represention of the weird zoo of floating point scalar types llvm cares about.

-Chris

1 Like

Yes, thanks for summarizing it. I’m interested in #1 first. I’m happy to discuss the #2 and #3 but that’s not the first problem I’m trying to solve.

From all the answers (thanks) it sounds like upstreaming this into APFloat is the right thing to do. I’ll look at sending a patch for this solution.

2 Likes

I’ve got two points to chuck in here.

The first is that I raised the block-float question over in [RFC] Should the OCP microscaling float scalars be added to APFloat and FloatType? - #3 by krzysz00, where I observed that the sub-byte scalars of a blocked float are almost misleading as floats in that you’re not intended to arithmetic on them without an accompanying group scale, raising philosophical questions about whether they make sense as APFloat members. (And, though I didn’t mention it there, there’re also fixed-point types).

One observation I came to after posting that thread, but haven’t written up until now, is that my “do we want to extend APFloat?” vs. “a custom type is hard to work with” problem could be solved by opening up getIntOrFloatBitWidth(), the types usable in vector, and so on.

My preliminary thoughts on that are that we want to allow people to stick arbitrary semantic wrappers around N-bit integers and that any such wrapper would be allowable as a vector element, have a bitwidth, and so on.

We could potentially use MemRefElementTypeInterface for this - give it a std::optional<unsigned> getFixedBitWidth() … or, more realistically, a new interface that’d go something like

FixedWidthDataInterface {
  unsigned getBitWidth();
  APInt bitcastConstant(TypedAttribute);
}

to let you simply declare all these custom float types into existence.

(The lowering to LLVM here would be, I’m pretty sure, iN, discarding the semantic context, because I’d expect a lot of these types to be passed to platform-specific intrinsics or library functions)

The one problem with that approach is that these newtype wrappers are necessarily opaque to dialects like arith, so perhaps what we want is instead something along the lines of NumericTypeInterface either instead of or in addition to the ability to mark things as data with a width.

I’m not sure quite what the right solutions are here, but given how ML has a lot of custom numeric formats (… heck, it’s not just ML, if the set of data-like or numeric-like types was open, !posit.posit<N> could be a thing in a third-party library)

I hope these are somewhat useful thoughts.

I agree with you. I’m thinking getIntOrFloatBitWidth should be relaxed to support any type with fixed length and use an interface to allow opaque types to work with it. This helper feels like the main one preventing from writing generic code in my experience.

Do we even need arith to work on those types since in general no arithmetic would be supported?

This works on fixed vectors, it’ll just be getElementType().getBitWidth() * getNumElements(), but how would it look like with scalable vectors? It would also not always work on memref or tensors (ex. sparse). I’d keep this to scalar/element types only.

We’re trying to simplify what arith supports (to only scalar and vector types), so I’d avoid tying this discussion in there.

On the subject of block-floats I think at least one point that might be interesting here is that APFloat/FloatType and their ability to model scalar floating-point values is really a special case of what block-float would require (i.e. with a block size of 1), so it seems apt for block-float to not be built on top of APFloat/FloatType but rather maybe APFloat/FloatType could be re-imagined to be built on top of whatever is envisioned for block-float. I’m not sure if practically this would be useful to do rather than just having them exist entirely separate from each other, but it’s another way of looking at the problem.

While the OCP document specifically mentions block-sizes of 32, practically speaking that size is entirely arbitrary (though may end up being the most commonly adopted size due to hardware constraints) and I’ve seen it span from 1-128 (or beyond) in prior projects. I think the variability in block size + the esoteric nature of the data-type itself will likely lead to vendor-specific layout considerations, but that shouldn’t hold back the ability to ingest the formats and make vendor-specific decisions further in the stack.

+1. @antiagainst and I had a proposal for packed formats last year but it seemed like a too heavy of a hammer: [RFC] Packing for sub-byte types. Having something like a ScalarLikeType interface useable as vector/tensor/memref element type would be enable to implement packed formats downstream.

1 Like

That makes sense to me. Scalar should be enough as far as I can tell.

I’m curious, is this being discussed somewhere? This is unrelated to this thread but I would like to join this conversion if this is happening.

You were participating in it, this was a long time ago:

1 Like

got it, thanks. Yes this is quite orthogonal so let’s stay away from this as you suggested.