ONNX Model and Tensorrt Engine gives different output for parseq model

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Description

I converted parseq ocr model from pytorch to onnx and tested it on onnx model and every thing is ok, but when I convert onnx to fp32 or fp16 tensorrt engine, output of the model is very different from onnx model.
I use onnxsim to simplify onnx. if i dont use onnxsim all results are nan.

model repo : GitHub - baudm/parseq: Scene Text Recognition with Permuted Autoregressive Sequence Models (ECCV 2022)

Environment

TensorRT Version: TensorRT-8.6.1.6
GPU Type: RTX 3060
Nvidia Driver Version: 531.79
CUDA Version: cuda-12.0
CUDNN Version: cudnn-8.9.1.23_cuda12
Operating System + Version: win 10
Python Version: 3.8
Onnx opset: 14

Relevant Files

onnx model: test.onnx - Google Drive

trtexec.exe --onnx=parseq/test.onnx --workspace=10000 --saveEngine=parseq/test_fp32.trs --verbose
trt engine fp32: test_fp32.trt - Google Drive
trt engine fp32 log: test_fp32_log.txt - Google Drive

trtexec.exe --onnx=parseq/test.onnx --fp16 --workspace=10000 --saveEngine=parseq/test_fp16.trs --verbose
trt engine fp16: test_fp16.trt - Google Drive
trt engine fp16 log: test_fp16_log.txt - Google Drive

Steps To Reproduce

I wrote a sample code to compare similarity of onnx and trt inference result. when I use real data, mean of similarity is 0.3 and when I use random number it is near 0.85

sample code:

sample real data:

2

I have same problem with VitStr based on timm vision transformer

VitStr:

Vision transformer:

It seems that the problem is vision transformer.

3

Hi,
Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:

  1. validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).
2) Try running your model with trtexec command.

In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging
Thanks!

5

It is a bug in 8.6.1 version of tensorrt. and it will be fixed in the next release.

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.