Swish can be expressed as a combination of ONNX operators and can be easily fused by the backend. Was there a motivation for including it in the spec?

Hello @justinchuby
One big advantage of adding this Op as a standalone Function is that we can then treat it as an Activation node directly. This is quite helpfule for model structure analysis (e.g. We can view it in netron etc.), and can also help us in Quantization design: If we need to quantize all the separate small Ops, this usually cause extra rounding errors and accuracy loss.

I suppose the benefits of implementing such a Spec is smilar to all the existing Activation Function in ONNX like Gelu, HardSwish, HardSigmoid, Mish etc.

@gramalingam @xadupre for more thoughts. I suggest creating a pull request if the op is desired. Additionally, it is possible to implement it as a model local function. You may then be able to use it like you mentioned above.

Additionally, if you would like to see it as a single unit in the exported PyTorch models by torch.onnx.dynamo_export, you are welcome to contribute to https://github.com/microsoft/onnxscript/blob/bec23adc815406e6103dff8463e3386a1be155e7/onnxscript/function_libs/torch_lib/ops/nn.py#L2014

Swish can be expressed as a combination of ONNX operators and can be easily fused by the backend. Was there a motivation for including it in the spec?

@justinchuby, the same is true for other functions like Clip, HardSigmoid, HardSwish, etc

What are the criteria to add these as a function to the spec? According to AddNewOp.md the operator needs to be implemented by at-least one (well-known) framework which is the case for Swish/SiLU. Is this not sufficient?

I understand ONNX is trying to keep the set of operators tight. Although this should be less of a concern when an op can be expressed as a function like in this case. I would bring this up in operators-sig and allow members of the sig to chime in.

My personal opinion is that adding this as a function-op to ONNX standard is reasonable: it can be easily supported by inline-expansion (if no fused implementation is available) or using a specialized kernel. (We can discuss this in the operator SIG meeting.)