[Backend] Improve dot support to target FMA

[binarman]

This PR:

• Refactors FMA dot implementation
• Supports dot3d in FMA path
• Fixes several issues in operand offset computation
• Enables small dot operands

This PR is a part of PR series. Final goal is to improve efficiency of small dot operations and bypass as much shared memory accesses as possible.

Rough list of PRs:

• Basic FMA dot fixes, dot 3d support and relaxing small dimensions for dot (this PR) [Backend] Improve dot support to target FMA #4516
• Blocked->dotOp shared memory bypassing [Backend] Bypass conversion for suitable blocked to dotOperand layout #4538
• Accelerate AMD Matmul + emit dot operations [WIP] [AMD] Emit AMD specific intrinsics for dot  #4594
• Layout optimization, so operand B is loaded in proper mfma layout and do not need to go through LDS [WIP] Optimize fma dot #4581
• Vectorization optimization of dot operands/results (in case llvm can not do this internally)
• Reduction operation hoisting out of the K loop (reduction operation is a byproduct of layout optimization step) Hoist reduction outside a loop #4559

[antiagainst]

First batch of comments; I still need to review SharedToDotOperandFMA.cpp more carefully.

[antiagainst]

When addressing comments, please make sure to add new commits and not squashing into existing ones. Otherwise it's hard to re-review again.

[binarman]

When addressing comments, please make sure to add new commits and not squashing into existing ones. Otherwise it's hard to re-review again.

hmm, let me rework this with merge commits, I did not notice this comment in time

Typically I rebase changes on top of main branch, when I have conflicts. Because this way history look clean and it is easier to review, but I see that you prefer merge updates, so I'll continue doing it that way.

[antiagainst]

Thanks for improving the implementation! I still have a few nits.

The change is quite intensive and I'm not super familiar with shared layout indexing and such. @zhanglx13 please take another look.

[antiagainst]

This now looks good to me. But please wait for @zhanglx13 to double check the logic there given it's fairly involved. I'll approve after Lixun approves.