IR Roadmap

[justinchuby]

• Data model https://github.com/microsoft/onnx-rewriter/pull/232 https://github.com/microsoft/onnx-rewriter/pull/257
  • Support tensor protos cleanly https://github.com/microsoft/onnx-rewriter/pull/254 https://github.com/microsoft/onnx-rewriter/pull/264
  • [IR] Support dtypes that are not in numpy #1424
  • [IR] User defined data types #1423
  • Typing system: Unify Sequence types and Optional types https://github.com/microsoft/onnx-rewriter/pull/254 https://github.com/microsoft/onnx-rewriter/pull/264
  • Tensor: implement nbytes, size and itemsize https://github.com/microsoft/onnx-rewriter/pull/254
  • Use slots https://github.com/microsoft/onnx-rewriter/pull/254
  • Shared AttrRef in functions
  • Make Function callable. __call__(self, *values, *attrs) -> Sequence[Value] (but still need to get the nodes); we can also just call a function to inline it? Or we can "expand" it
  • Make Function a subclass of Graph? #1360
  • [IR] Safetensors support #1425
  • [IR] Support empty outputs #1416
  • [IR] Support Map type #1430
  • [IR] Support symbolic values? #1438
  • [IR] Point graph input and initializers back to the Graph #1440
    • [IR] Values that are initialized but not graph inputs do not have their ValueInfo preserved #1501 @justinchuby "I am here"
  • [IR] Data structure invariance #1450
  • [IR] Tensor todos #1499
  • [IR] Figure out how we can share tensors #1554
  • [IR] Create a DEBUG flag to enable addtional checks #1545
• Use weakref for some bidirectional pointers / dictionaries
  • For example, pointer in DiagnosticMessage?
• Serialization to proto https://github.com/microsoft/onnx-rewriter/pull/267
  • Be careful to avoid copying
• Deserialization from proto https://github.com/microsoft/onnx-rewriter/pull/254
  • Test serde with the ONNX test suite https://github.com/microsoft/onnx-rewriter/pull/267
  • User defined external data format in deserializer? They should handle them in a pass
  • [IR] Handle unsorted nodes #1426
• Mutation methods for graph; sorting
  • https://github.com/microsoft/onnx-rewriter/pull/273
  • [IR] Mutable IR implementation #1344
  • Mutable protocols
  • FX graph
    • Immutable collection: https://github.com/pytorch/pytorch/blob/main/torch/fx/immutable_collections.py#L33
  • [IR] Implement topological sorting on Graph #1427
  • Insertion
  • Iterator
  • Naming authority
  • [IR] Enhanced name authority #1535
  • Graph input/output/initializers mutation
  • Validation scenarios
    • Replace subgraph with new node
    • Build new graph from a list of nodes
    • Drop nodes
    • Insert nodes
    • Double iteration
• Initialization / mutation hooks
• [IR] Support sparse tensor #1428
• Function ref in nodes (Just create new constants)
• Use rich to pretty print objects https://github.com/microsoft/onnx-rewriter/pull/282 https://github.com/microsoft/onnx-rewriter/pull/278
• Beautiful (and easy) diagnostics!
  • __repr__ for (nested) graph and model
  • .doctor() method for displaying beautiful and actionable analysis in terminal. Use rich.
  • Diagnosable: define doctor(), diagnose() -> DiagnosticMessage, add_diagnoser, clear_diagnosers() for all entities in the graph
  • Extensible diagnostics with a Diagnoser (<-naming?) class https://github.com/justinchuby/onnx-doctor
  • Things we can produce diagnostics for:
    • Data structure coherency: loops, nodes that do not have an owning graph, etc.
    • Spec violation: Mismatched types, unknown shape/type inputs/outputs, features unsupported by the IR version, mixed opsets etc. Create error codes and reference pages for checker errors onnx/onnx#6082
    • Optimization opportunities: We can make all optimize and rewrite rules diagnosers: constant foldable, detangling nodes, redundant nodes etc.
    • The onnx.checker can be adapted as a diagnoser
  • Tools can use diagnose() to get messages programmatically and produce reports if they want
    • For example, the interested can build a linter out of this. But that's too much work for now.
• Use the metastore for symbolic shape inference
  • [IR] Integrate symbolic shape inference #1383
• Analysis interface? onnx shape inference as an analyzer?
• Creative use of meta
  • Profiling results
  • Node grouping
• Integration IR Integration plan #1384
• Version converters
  • Opset version converter
  • IR version converter / checker
• [IR] Flatbuffer support #1429