Error in CPU optimization

[isgursoy]

21/11/2022 07:29:41 PM [ INFO ] Running Nebullvm optimization on CPU
21/11/2022 07:29:43 PM [ WARNING ] Missing Frameworks: tensorflow.
 Please install them to include them in the optimization pipeline.
21/11/2022 07:29:43 PM [ WARNING ] Missing Compilers: openvino.
 Please install them to include them in the optimization pipeline.
21/11/2022 07:29:43 PM [ WARNING ] Not enough data for splitting the DataManager. You should provide at least 100 data samples to allow a good split between train and test sets. Compression, calibration and precision checks will use the same data.
21/11/2022 07:29:44 PM [ INFO ] Converting the torch model to onnx
21/11/2022 07:30:29 PM [ INFO ] Benchmark performance of original model
21/11/2022 07:31:00 PM [ INFO ] Original model latency: 0.27246932983398436 sec/iter
21/11/2022 07:31:00 PM [ INFO ] Running pipeline: pytorch_pipeline
21/11/2022 07:31:00 PM [ INFO ] Running step: no_compression
21/11/2022 07:31:00 PM [ INFO ] Running step: optimizer_step
21/11/2022 07:31:00 PM [ INFO ] Optimizations: ('torchscript', 'deepsparse', 'intel_neural_compressor')
21/11/2022 07:31:00 PM [ INFO ] Optimizing output of compressor 1/1: no_compression
21/11/2022 07:31:00 PM [ INFO ] [ OPTIMIZER 1/3 ]
21/11/2022 07:31:00 PM [ INFO ] Optimizing with PytorchBackendOptimizer and q_type: None.
21/11/2022 07:31:29 PM [ INFO ] Optimized model latency: 0.23684096336364746 sec/iter
21/11/2022 07:31:29 PM [ INFO ] [ OPTIMIZER 2/3 ]
21/11/2022 07:31:29 PM [ INFO ] Optimizing with DeepSparseOptimizer and q_type: None.
DeepSparse Engine, Copyright 2021-present / Neuralmagic, Inc. version: 1.2.0 COMMUNITY EDITION | (45d54d49) (release) (optimized) (system=avx2, binary=avx2)
[7fcfbcaa8000 >ERROR< validate src/lib/engine/compute/primitive/stacked_gemm.cpp:104] Bias width does not match the number of output columns. Bias: 1,9604,1,1,1, B_matrix: 64,4,1,32,49
DeepSparse Engine, Copyright 2021-present / Neuralmagic, Inc. version: 1.2.0 (45d54d49) (release) (optimized) (system=avx2, binary=avx2)
Date: 11-21-2022 @ 19:32:22 +03
OS: Linux isgursoy-VORKE 5.15.0-54-generic #60-Ubuntu SMP Mon Nov 14 13:48:30 UTC 2022
Arch: x86_64
ISA: x86_64-avx2-32-64-16-64-xbyak
CPU: GenuineIntel
Vendor: Intel
Cores/sockets/threads: [4, 1, 8]
Available cores/sockets/threads: [4, 1, 8]
L1 cache size data/instruction: 32k/32k
L2 cache size: 0.25Mb
L3 cache size: 6Mb
Total memory: 17.5404G
Free memory: 1.51531G

Assertion at ./src/include/wand/engine/debug.hpp:233

Backtrace:
 0# wand::detail::abort_prefix(std::ostream&, char const*, char const*, int, bool, bool, unsigned long) in /opt/projects/akil/classification-engine/venv/lib/python3.10/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
 1# 0x00007FCE7C9A05D4 in /opt/projects/akil/classification-engine/venv/lib/python3.10/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
 2# 0x00007FCE7C9A1092 in /opt/projects/akil/classification-engine/venv/lib/python3.10/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
 3# 0x00007FCE7C643993 in /opt/projects/akil/classification-engine/venv/lib/python3.10/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
 4# 0x00007FCE7C643CA2 in /opt/projects/akil/classification-engine/venv/lib/python3.10/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
 5# 0x00007FCE7C64A64A in /opt/projects/akil/classification-engine/venv/lib/python3.10/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
 6# 0x00007FCE7C64A6C0 in /opt/projects/akil/classification-engine/venv/lib/python3.10/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
 7# 0x00007FCE7BA7A0F7 in /opt/projects/akil/classification-engine/venv/lib/python3.10/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
 8# 0x00007FCE7BA7A5EF in /opt/projects/akil/classification-engine/venv/lib/python3.10/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
 9# 0x00007FCE7B9EEB35 in /opt/projects/akil/classification-engine/venv/lib/python3.10/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
10# 0x00007FCE7B9DE609 in /opt/projects/akil/classification-engine/venv/lib/python3.10/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
11# 0x00007FCE7B9DF8E1 in /opt/projects/akil/classification-engine/venv/lib/python3.10/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
12# 0x00007FCE7C144590 in /opt/projects/akil/classification-engine/venv/lib/python3.10/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
13# 0x00007FCE7C1499CE in /opt/projects/akil/classification-engine/venv/lib/python3.10/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
14# 0x00007FCE7C14C527 in /opt/projects/akil/classification-engine/venv/lib/python3.10/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
15# 0x00007FCE7C14C954 in /opt/projects/akil/classification-engine/venv/lib/python3.10/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
16# 0x00007FCE7B9BC75E in /opt/projects/akil/classification-engine/venv/lib/python3.10/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
17# 0x00007FCE7B9C434D in /opt/projects/akil/classification-engine/venv/lib/python3.10/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
18# 0x00007FCE7B95C6ED in /opt/projects/akil/classification-engine/venv/lib/python3.10/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
19# 0x00007FCE7B968E15 in /opt/projects/akil/classification-engine/venv/lib/python3.10/site-packages/deepsparse/avx2/libonnxruntime.so.1.12.0
20# deepsparse::ort_engine::init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::shared_ptr<wand::parallel::scheduler_factory_t>) in /opt/projects/akil/classification-engine/venv/lib/python3.10/site-packages/deepsparse/avx2/libdeepsparse.so
21# 0x00007FCE7E6B4433 in /opt/projects/akil/classification-engine/venv/lib/python3.10/site-packages/deepsparse/avx2/deepsparse_engine.so
22# 0x00007FCE7E6F06B9 in /opt/projects/akil/classification-engine/venv/lib/python3.10/site-packages/deepsparse/avx2/deepsparse_engine.so
23# 0x00007FCE7E6C1FFE in /opt/projects/akil/classification-engine/venv/lib/python3.10/site-packages/deepsparse/avx2/deepsparse_engine.so

Please email a copy of this stack trace and any additional information to: support@neuralmagic.com

Process finished with exit code 134 (interrupted by signal 6: SIGABRT)

pip git installation. Ubuntu 22.04.

[valeriosofi]

This is strange, I think that also here the cause is python 3.10, but it shouldn't crash like that. I'll investigate as soon as I can, for now you can set ignore_compilers=["deepsparse"] to avoid its usage.

[isgursoy]

Yes, crash is gone now. Lets have a cross reference under issue #103.