Bugfix for SimplifiedLayerNormalization #12975
Conversation
| // If it's not GPU EP, since the CPU impl for SimplifiedLayerNormalization doesn't support input and scale | ||
| // having different types for now, and it may also have conflict to InsertCastTransformer, | ||
| // so the sub-graph will not be fused if it contains Cast Op. | ||
| bool is_gpu_ep = pow_node.GetExecutionProviderType() == kCudaExecutionProvider || |
There was a problem hiding this comment.
Can you explain this change a bit more? Based on the comment I would have expected the code to look for a Cast and exit if the CPU EP was involved.
I don't quite understand why we change the first branch which seems to be about setting has_leading_cast and a second location which has nothing to do with has_leading_cast.
There was a problem hiding this comment.
There are only 4 possible cases (x=Pow->ReduceMean->Add->Sqrt->Div, and y=Mul):
(1) cast(to:float)->x->cast(to:fp16)->y : SimplifiedLayerNorm(T:fp16,V:fp16)
(2) cast(to:float)->x->y : SimplifiedLayerNorm(T:fp16,V:float)
(3) x->cast(to:fp16)->y : SimplifiedLayerNorm(T:float,V:fp16)
(4) x->y : SimplifiedlayerNorm(T:float,V:float)
They all work for CUDA EP.
For CPU EP, we have only SimplifiedlayerNorm(T:float,V:float), so only (4) works. But if for (1) and (2), if we just treat the entry cast as a normal node, means has_leading_cast is always false, then for (2), we can still fuse it to "cast(to:float)->SimplifiedlayerNorm(T:float,V:float)" (just like applying (4) to the x->y after cast), so the condition for CPU EP to fuse or not is always set has_leading_cast to false and check if there is a cast between x and y. Having cast between means cannot fuse.
There was a problem hiding this comment.
Would be great to put this excellent explanation in the comment so it's captured for the next person who works on the code.
This PR is to fix #12930 and #12579. In detail: - For CPU EP, since current impl of SimplifiedLayerNormalization doesn't support input and scale having different data types, so if the sub-graph contains Cast Op, the sub-graph will not fused, this guarantee that both inputs and output data type will be same - For CUDA EP, add (fp16, float) support to (T,V) type constraints all combinations of fp16 and float can be supported in the impl With the fix, the original model can be run with SimplifiedLayerNormalization, which also helps to improve the perf.
This PR is to fix #12930 and #12579.
In detail:
With the fix, the original model can be run with SimplifiedLayerNormalization, which also helps to improve the perf.