You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This appears to be a straightforward plumbing through of DML's interface, which does not require the BiasTensor to be a constant. Neither does the corresponding operator for TFLite. From what I can tell, this seems to be because these frameworks don't have a way to express that some input tensor must be const. The options are either to pass the parameter as the framework's generic representation of a Tensor - which would in practice always(?) be created from a constant() - or to pass the parameter as a 1D array directly. If the parameters may be large (and perhaps unbounded), the former is the more appropriate choice.
To get a sense for whether this is a reasonable hypothesis, I've inspected of all† uses of the affected operators in the WebNN Samples repo repo:
†This list only includes WebNN operators which trivially map to CoreML operators. WebNN operators which need to be in terms of other CoreML operators will be subject to the restrictions of those respective CoreML operators. For example, CoreML doesn't have operators for gruCell or lstmCell, so these operators will need to be implemented in terms of gru and lstm, respectively. These operators will in turn need many of their parameters to be const, as well
††One caller of passes the result of a reshape... but that's only because the sample was written before constant() took an MLOperandDescriptor. The reshape is just assigning dimensions to a constant(). Nowadays we'd just pass the constant() directly
Remarkably, every single instance where one of these params is used in the WebNN Samples, it was created from a constant(). Cool!
Of course, this is not close to a comprehensive list of all models hope to run with WebNN. That being said, if there are no significant known use cases for passing any of these parameters as non-constant tensors - if their non-constness is simply a limitation in the framework and there are no useful reasons to pass non-const tensors - I think there's a reasonable argument that WebNN should require these parameters to be constants. @fwdr could you perhaps provide some more color here? :)
It seems that we have the following options to support each of these operators on CoreML:
Require that operator.param must be a constant MLOperand (my tentative preference)
Note that it's much easier to relax these restrictions than to try to impose them later
This can be done two ways:
Create an MLConstantOperand interface which extends MLOperand, and specify that param takes an MLConstantOperand
The
constant()
operator is special. Constants are effectively inputs that are known at the time of graph compilation (i.e.build()
).You may ask, why do we need a separate method when we could just pass this constant data as an
input()
?Most discussions I've seen so far about
constant()
(#614 (comment), this thread about the since-abandoned "fill sequence" overload, etc) have been concerned with optimizing performance. There are a number of compile-time optimizations a backend may perform if it knows that some data is constant.Our experience with CoreML has given us another reason:
The backend requires that a parameter to an operator must be constant
Take
conv2d()
, for example. It's defined for CoreML here. Thebias
parameter is defined as:bias: const tensor<[C_out],T>
Meanwhile, WebNN allows this to be any
MLOperand
, as long as it's of the appropriate shape and dtype:This appears to be a straightforward plumbing through of DML's interface, which does not require the
BiasTensor
to be a constant. Neither does the corresponding operator for TFLite. From what I can tell, this seems to be because these frameworks don't have a way to express that some input tensor must be const. The options are either to pass the parameter as the framework's generic representation of a Tensor - which would in practice always(?) be created from aconstant()
- or to pass the parameter as a 1D array directly. If the parameters may be large (and perhaps unbounded), the former is the more appropriate choice.To get a sense for whether this is a reasonable hypothesis, I've inspected of all† uses of the affected operators in the WebNN Samples repo repo:
operator.param
batchNormalization.mean
batchNormalization.variance
batchNormalization.scale
batchNormalization.bias
conv2d.bias
convTranspose2d.bias
gru.weight
gru.recurrentWeight
gru.bias
gru.recurrentBias
instanceNormalization.scale
instanceNormalization.bias
layerNormalization.scale
layerNormalization.bias
lstm.weight
lstm.recurrentWeight
lstm.bias
lstm.recurrentBias
lstm.peepholeWeight
prelu.slope
†This list only includes WebNN operators which trivially map to CoreML operators. WebNN operators which need to be in terms of other CoreML operators will be subject to the restrictions of those respective CoreML operators. For example, CoreML doesn't have operators for
gruCell
orlstmCell
, so these operators will need to be implemented in terms ofgru
andlstm
, respectively. These operators will in turn need many of their parameters to be const, as well††One caller of passes the result of a
reshape
... but that's only because the sample was written beforeconstant()
took anMLOperandDescriptor
. Thereshape
is just assigning dimensions to aconstant()
. Nowadays we'd just pass theconstant()
directlyRemarkably, every single instance where one of these params is used in the WebNN Samples, it was created from a
constant()
. Cool!Of course, this is not close to a comprehensive list of all models hope to run with WebNN. That being said, if there are no significant known use cases for passing any of these parameters as non-constant tensors - if their non-constness is simply a limitation in the framework and there are no useful reasons to pass non-const tensors - I think there's a reasonable argument that WebNN should require these parameters to be constants. @fwdr could you perhaps provide some more color here? :)
It seems that we have the following options to support each of these operators on CoreML:
operator.param
must be a constantMLOperand
(my tentative preference)MLConstantOperand
interface which extendsMLOperand
, and specify thatparam
takes anMLConstantOperand
MLOperand
has a "kind", as the Chromium implementation already does, and throw aTypeError
if not a "constant" kind. This may be confusing to developersoperator.param
asequence<MLNumber>
operator.param
is not a constant only on CoreMLbuild()
on Chromium, though we could conceivably make this a synchronous check on the respective builder method, especially if we have defined procedures for querying for backend-specific support (see Allow checking whether operators/types are supported for a backend before creating a graph #463)Thoughts?
The text was updated successfully, but these errors were encountered: