Shape Error (reshape wildcard -1 has infinite number of solutions or no solution. Reshaping [0,16] to [0,-1].)

[yunus-kalkan]

Description

I'm trying to convert the faster-rcnn.pt file to the onnx and use it with onnxruntime-gpu and tensorrt provider. When I check the converted onnx with trtexec, I get an Shape Error. One of the node has [0,16] input and that node tries to reshape it [0,-1]. Is there any way to assign the shape of the node manually ? It is because of the tools versions that I used ?

Environment

TensorRT Version: 8.6
NVIDIA GPU: Quadro RTX 4000
NVIDIA Driver Version: 530.30.02
CUDA Version: 12.1
CUDNN Version:
Operating System: Ubuntu 18.04
Python Version (if applicable): 3.9
Tensorflow Version (if applicable):
PyTorch Version (if applicable): 1.7.1
Baremetal or Container (if so, version):

Steps To Reproduce

1.Exported onnx from PyTorch weight file (faster-rcnn model)
dynamic_axes = {"input":{0:"batch_size"},
"boxes":{0:"batch_size"},
"labels":{0:"batch_size"},
"scores":{0:"batch_size"}}

torch.onnx.export(
model.to("cuda"),
dummy_input.to("cuda"),
onnx_filename,
verbose = True,
opset_version = 12,
input_names = input_names,
output_names = output_names,
dynamic_axes = dynamic_axes,
do_constant_folding=True
)
2.Applied polygraphy surgeon
polygraphy surgeon sanitize n.onnx --fold-constants --output n2.onnx

3. Run trtexec on new onnx file
   trtexec --onnx=n2.onnx --shapes=input:1x2x2024x2024 --verbose

ERROR: Reshape node, infinite solution for Reshaping [0,16] to [0,-1].
...
[03/22/2023-10:30:08] [E] [TRT] ModelImporter.cpp:770: input: "2881"
input: "2897"
output: "2898"
name: "Reshape_1605"
op_type: "Reshape"

[03/22/2023-10:30:08] [E] [TRT] ModelImporter.cpp:771: --- End node ---
[03/22/2023-10:30:08] [E] [TRT] ModelImporter.cpp:774: ERROR: ModelImporter.cpp:195 In function parseGraph:
[6] Invalid Node - Reshape_1605
[shapeContext.cpp::operator()::3546] Error Code 4: Shape Error (reshape wildcard -1 has infinite number of solutions or no solution. Reshaping [0,16] to [0,-1].)
[03/22/2023-10:30:08] [E] Failed to parse onnx file
[03/22/2023-10:30:08] [I] Finished parsing network model. Parse time: 0.284967
[03/22/2023-10:30:08] [E] Parsing model failed
[03/22/2023-10:30:08] [E] Failed to create engine from model or file.
[03/22/2023-10:30:08] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8600] # /usr/src/tensorrt/bin/trtexec --onnx=n2.onnx --shapes=input:1x2x2024x2024 --verbose

[zerollzeng]

I would try fix the input shape first, see https://github.com/NVIDIA/TensorRT/tree/main/tools/Polygraphy/examples/cli/surgeon/03_modifying_input_shapes

[zerollzeng]

If it still doesn't work then could you share the onnx here? Thanks!

[yunus-kalkan]

I've tried to fix the input shape with polygraphy, it didn't work. I'm using the Faster-RCNN.onnx model. You can find the model in here: https://github.com/onnx/models/blob/main/vision/object_detection_segmentation/faster-rcnn/model/FasterRCNN-10.onnx
Thanks.

[zerollzeng]

Looks like this model is not supported by TRT yet...

[03/25/2023-02:04:28] [E] [TRT] ModelImporter.cpp:771: While parsing node number 2395 [Cast -> "2408"]:
[03/25/2023-02:04:28] [E] [TRT] ModelImporter.cpp:772: --- Begin node ---
[03/25/2023-02:04:28] [E] [TRT] ModelImporter.cpp:773: input: "2407"
output: "2408"
name: "2401"
op_type: "Cast"
attribute {
  name: "to"
  i: 2
  type: INT
}

[03/25/2023-02:04:28] [E] [TRT] ModelImporter.cpp:774: --- End node ---
[03/25/2023-02:04:28] [E] [TRT] ModelImporter.cpp:777: ERROR: ModelImporter.cpp:253 In function parseGraph:
[8] Assertion failed: legalUINT8 && "TensorRT does not support UINT8 types for intermediate tensors!"

FYI the input should be [3, 1024, 1024]?

$ polygraphy inspect model FasterRCNN-10.onnx
[I] Loading model: FasterRCNN-10.onnx
[I] ==== ONNX Model ====
    Name: torch-jit-export | ONNX Opset: 10

    ---- 1 Graph Input(s) ----
    {image [dtype=float32, shape=(3, 'height', 'width')]}

    ---- 3 Graph Output(s) ----
    {6379 [dtype=float32, shape=('nbox', 4)],
     6381 [dtype=int64, shape=('nbox',)],
     6383 [dtype=float32, shape=('nbox',)]}

    ---- 0 Initializer(s) ----

    ---- 6370 Node(s) ----

[ttyio]

Sorry This is known limitation that TRT not support UINT8 as intermediate tensors. see https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#types-precision

[HyeonseopLim]

I got same error.
I use fasterrcnn_resnet50_fpn in Pytorch and I convert it to ONNX.
but when I convert the ONNX to TensorRT Using trtexec after polygraphy, I got this error.

It has no solution yet?
Should I change model to another one?

[iamamiramine]

Any updates on this issue?
I am facing the same problem after sanitizing my fasterrcnn_resnet50_fpn_v2.

[yunus-kalkan]

Unfortunately I couldn't use the Faster RCNN onnx model with onnxruntime + TensorRT provider. I had to use different model for my project.

[predondo99]

Any updates on this issue? I am facing the same problem after sanitizing my fasterrcnn_resnet50_fpn_v2.

I finally got it working using this code:

   import onnx
   import onnx_graphsurgeon as gs
   import numpy as np

  const_else = gs.Constant("const_else", np.array([0]).astype(np.bool_))

  g = gs.import_onnx(onnx.load(onnx_path))

  # Overwrite condition to a constant.
  for n in g.nodes:
      if n.op == "If":
          n.inputs = [const_else]
          
  # Export the modified graph to a new ONNX model.
  onnx.save(gs.export_onnx(g), onnx_path.replace('.onnx', '_fixed.onnx'))
  onnx_path = onnx_path.replace('.onnx', '_fixed.onnx')

Source: https://forums.developer.nvidia.com/t/conversion-of-fasterrcnn-onnx-model-to-tensorrt-fails/313065