高性能推理概率出现段错误

[flysssss]

Checklist:

• 查找历史相关issue寻求解答
• 翻阅FAQ
• 翻阅PaddleX 文档
• 确认bug是否在新版本里还未修复

描述问题

mask-RT-DETR-L 模型训练完成，部署推理时，启用高性能推理，测试集部分图片推理过程中出现段错误。
将高性能推理关闭后，推理正常。
其中有部分图片高性能推理正常。

复现

1. 高性能推理

   • 您是否完全按照高性能推理文档教程跑通了流程？

   • 是

   • 您使用的是离线激活方式还是在线激活方式？

   • 在线激活

2. 您使用的模型和数据集是？

mask-RT-DETR-L

4. 请提供您出现的报错信息及相关log

正常推理图片

λ ys-ai-GPU-7 /paddle python main.py -c paddlex/configs/instance_segmentation/Mask-RT-DETR-L.yaml -o Global.mode=predict -o Predict.model_dir="./output/best_model/inference" -o Predict.input="./test1/1.jpg"
grep: warning: GREP_OPTIONS is deprecated; please use an alias or script
/usr/local/lib/python3.10/dist-packages/setuptools-68.2.2-py3.10.egg/_distutils_hack/init.py:18: UserWarning: Distutils was imported before Setuptools, but importing Setuptools also replaces the distutils module in sys.modules. This may lead to undesirable behaviors or errors. To avoid these issues, avoid using distutils directly, ensure that setuptools is installed in the traditional way (e.g. not an editable install), and/or make sure that setuptools is always imported before distutils.
warnings.warn(
/usr/local/lib/python3.10/dist-packages/setuptools-68.2.2-py3.10.egg/_distutils_hack/init.py:33: UserWarning: Setuptools is replacing distutils.
warnings.warn("Setuptools is replacing distutils.")
{'input_path': './test1/1.jpg', 'boxes': [{'cls_id': 0, 'label': 'sweetpotato', 'score': 0.9781391024589539, 'coordinate': [670.19, 330.00598, 877.6621, 924.6505]}], 'masks': '...'}
The result has been saved in output/1.json.
The result has been saved in output/1.jpg.

相同图片高性能推理报错：

λ ys-ai-GPU-7 /paddle python test.py
[INFO] fastdeploy/vision/common/processors/transform.cc(45)::FuseNormalizeCast Normalize and Cast are fused to Normalize in preprocessing pipeline.
[INFO] fastdeploy/vision/common/processors/transform.cc(93)::FuseNormalizeHWC2CHW Normalize and HWC2CHW are fused to NormalizeAndPermute in preprocessing pipeline.
[INFO] fastdeploy/vision/common/processors/transform.cc(159)::FuseNormalizeColorConvert BGR2RGB and NormalizeAndPermute are fused to NormalizeAndPermute with swap_rb=1
2025-03-10 07:31:59,514 DEBUG [PADDLEX] 139792047109952 2 licenses loaded
2025-03-10 07:31:59,514 DEBUG [PADDLEX] 139792047109952 /root/.baidu/paddlex/licenses/license.6FAB.6550.02.data
2025-03-10 07:31:59,514 DEBUG [PADDLEX] 139792047109952 /root/.baidu/paddlex/licenses/license.ED44.3725.02.data
2025-03-10 07:31:59,514 DEBUG [PADDLEX] 139792047109952 License [6FAB-D532-...] is permanently valid, skip check local time
2025-03-10 07:31:59,531 DEBUG [PADDLEX] 139792047109952 License [6FAB-D532-...] is permanently valid, skip check local time
[INFO] fastdeploy/runtime/runtime.cc(185)::PaddleXAuth Basic authentication succeeded.
2025-03-10 07:31:59,547 DEBUG [PADDLEX] 139792047109952 License [6FAB-D532-...] is permanently valid, skip check local time
2025-03-10 07:31:59,564 DEBUG [PADDLEX] 139792047109952 License [6FAB-D532-...] is permanently valid, skip check local time
[INFO] fastdeploy/runtime/runtime.cc(263)::PaddleXAuth Task authentication succeeded.
[INFO] fastdeploy/runtime/runtime.cc(327)::Init The unencrypted model will be used for inference.
[INFO] fastdeploy/runtime/backends/paddle/paddle_backend.cc(28)::BuildOption Will inference_precision float32
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0310 07:32:06.301239 22819 gpu_resources.cc:119] Please NOTE: device: 1, GPU Compute Capability: 7.0, Driver API Version: 11.8, Runtime API Version: 11.8
W0310 07:32:06.302947 22819 gpu_resources.cc:164] device: 1, cuDNN Version: 8.9.
[INFO] fastdeploy/runtime/runtime.cc(474)::CreatePaddleBackend Runtime initialized with Backend::PDINFER in Device::GPU.
Batch size updated to 1
1.jpg
Segmentation fault (core dumped)

调试信息：

(gdb) core-file core.99193
[New LWP 99193]
[New LWP 99225]
[New LWP 99219]
[New LWP 99229]
[New LWP 99204]
[New LWP 99209]
[New LWP 99216]
[New LWP 99200]
[New LWP 99206]
[New LWP 99218]
[New LWP 99223]
[New LWP 99208]
[New LWP 99217]
[New LWP 99194]
[New LWP 99238]
[New LWP 99572]
[New LWP 99226]
[New LWP 99231]
[New LWP 99612]
[New LWP 99212]
[New LWP 99603]
[New LWP 99230]
[New LWP 99601]
[New LWP 99203]
[New LWP 99210]
[New LWP 99227]
[New LWP 99239]
[New LWP 99198]
[New LWP 1733]
[New LWP 99590]
[New LWP 99195]
[New LWP 99221]
[New LWP 99960]
[New LWP 99197]
[New LWP 99220]
[New LWP 99215]
[New LWP 99199]
[New LWP 99614]
[New LWP 99213]
[New LWP 99205]
[New LWP 99207]
[New LWP 99605]
[New LWP 1708]
[New LWP 99214]
[New LWP 99201]
[New LWP 99582]
[New LWP 99600]
[New LWP 99233]
[New LWP 99234]
[New LWP 99240]
[New LWP 99581]
[New LWP 99610]
[New LWP 99591]
[New LWP 99235]
[New LWP 99202]
[New LWP 99617]
[New LWP 99222]
[New LWP 99594]
[New LWP 99228]
[New LWP 99585]
[New LWP 1712]
[New LWP 99578]
[New LWP 99211]
[New LWP 99236]
[New LWP 1752]
[New LWP 99577]
[New LWP 99575]
[New LWP 99602]
[New LWP 1718]
[New LWP 99232]
[New LWP 99606]
[New LWP 1739]
[New LWP 1744]
[New LWP 99607]
[New LWP 99588]
[New LWP 1725]
[New LWP 1728]
[New LWP 1750]
[New LWP 99608]
[New LWP 99439]
[New LWP 1719]
[New LWP 99615]
[New LWP 99579]
[New LWP 99576]
[New LWP 1751]
[New LWP 1716]
[New LWP 99611]
[New LWP 1731]
[New LWP 1724]
[New LWP 99599]
[New LWP 99573]
[New LWP 99598]
[New LWP 99616]
[New LWP 99609]
[New LWP 99224]
[New LWP 1713]
[New LWP 1745]
[New LWP 99618]
[New LWP 1723]
[New LWP 99595]
[New LWP 99586]
[New LWP 1706]
[New LWP 99613]
[New LWP 1709]
[New LWP 99583]
[New LWP 1727]
[New LWP 1717]
[New LWP 99959]
[New LWP 1737]
[New LWP 99596]
[New LWP 1720]
[New LWP 1741]
[New LWP 99196]
[New LWP 1747]
[New LWP 99574]
[New LWP 1734]
[New LWP 99584]
[New LWP 1732]
[New LWP 99580]
[New LWP 1746]
[New LWP 1735]
[New LWP 99597]
[New LWP 1715]
[New LWP 1749]
[New LWP 99237]
[New LWP 1740]
[New LWP 99589]
[New LWP 1721]
[New LWP 1722]
[New LWP 1714]
[New LWP 99593]
[New LWP 99587]
[New LWP 1736]
[New LWP 1748]
[New LWP 1742]
[New LWP 1730]
[New LWP 1710]
[New LWP 1743]
[New LWP 1738]
[New LWP 1711]
[New LWP 1707]
[New LWP 1726]
[New LWP 99604]
[New LWP 99592]
[New LWP 1729]
--Type for more, q to quit, c to continue without paging--bt
Core was generated by `python test.py'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007fa34274ba84 in ?? ()
[Current thread is 1 (LWP 99193)]
(gdb) bt
#0 0x00007fa34274ba84 in ?? ()
#1 0x00007fa227962731 in ?? ()
#2 0x00007fff5116d420 in ?? ()
#3 0x0000000000000000 in ?? ()

环境

1. 请提供您使用的PaddlePaddle、PaddleX版本号、Python版本号

paddlex3.0.0b1-paddlepaddle3.0.0b1-gpu-cuda11.8-cudnn8.6-trt8.5

curl -s https://paddle-model-ecology.bj.bcebos.com/paddlex/PaddleX3.0/deploy/paddlex_hpi/install_script/latest/install_paddlex_hpi.py | python3.10 - --arch x86_64 --os linux --device gpu_cuda118_cudnn86 --py 310

[flysssss]

推理代码：
from paddlex import create_pipeline

pipeline = create_pipeline(
pipeline="./instance_segmentation.yaml",
use_hpip=True,
device="gpu:1",
hpi_params={"serial_number": "6FAB-xxxx-4xxx-863B"},
)
img_path= "./test/1.jpg"

output = pipeline.predict(img_path)
for res in output:
res.print() # 打印预测的结构化输出
#res.save_to_img("./res/") # 保存结果可视化图像
#res.save_to_json("./res/") # 保存预测的结构化输出

1.jpg是挑出来报错的图片，use_hpip=True, use_hpip=False, 启用高性能推理时，该图片必现段错误

[Bobholamovic]

建议尝试最新的paddlex 3.0rc版本～

[flysssss]

@Bobholamovic 已切换最新版本，仍然出现该问题

λ ys-ai-gpu03 /paddle/03test/test paddlex --pipeline instance_segmentation.yaml --input ./test2/LD20250306142040_0307_sweetpotato_shrimp_yiwu.jpg --device gpu:0 --use_hpip
Only Paddle model is detected. Paddle model will be used by default.
Backend: paddle_infer
Backend config: cpu_num_threads=8 enable_mkldnn=True enable_trt=False trt_dynamic_shapes={'im_shape': [[1, 2], [1, 2], [8, 2]], 'image': [[1, 3, 640, 640], [1, 3, 640, 640], [8, 3, 640, 640]], 'scale_factor': [[1, 2], [1, 2], [8, 2]]} trt_dynamic_shape_input_data={'im_shape': [[640.0, 640.0], [640.0, 640.0], [640.0, 640.0, 640.0, 640.0, 640.0, 640.0, 640.0, 640.0, 640.0, 640.0, 640.0, 640.0, 640.0, 640.0, 640.0, 640.0]], 'scale_factor': [[2.0, 2.0], [1.0, 1.0], [0.67, 0.67, 0.67, 0.67, 0.67, 0.67, 0.67, 0.67, 0.67, 0.67, 0.67, 0.67, 0.67, 0.67, 0.67, 0.67]]} trt_precision='FP32' enable_log_info=True
[INFO] ultra_infer/vision/common/processors/transform.cc(44)::FuseNormalizeCast Normalize and Cast are fused to Normalize in preprocessing pipeline.
[INFO] ultra_infer/vision/common/processors/transform.cc(91)::FuseNormalizeHWC2CHW Normalize and HWC2CHW are fused to NormalizeAndPermute in preprocessing pipeline.
[INFO] ultra_infer/vision/common/processors/transform.cc(157)::FuseNormalizeColorConvert BGR2RGB and NormalizeAndPermute are fused to NormalizeAndPermute with swap_rb=1
[INFO] ultra_infer/runtime/backends/paddle/paddle_backend.cc(28)::BuildOption Will inference_precision float32
[INFO] ultra_infer/runtime/backends/paddle/paddle_backend.cc(338)::InitFromPaddle Finish paddle inference config with summary as:
[INFO]
+--------------------------+------------------------------------------+
| Option | Value |
+--------------------------+------------------------------------------+
| model_file | best_model/inference/inference.pdmodel |
| params_file | best_model/inference/inference.pdiparams |
+--------------------------+------------------------------------------+
| cpu_math_thread | 8 |
| enable_mkldnn | true |
| mkldnn_cache_capacity | 10 |
+--------------------------+------------------------------------------+
| use_gpu | true |
| use_cutlass | false |
| gpu_device_id | 0 |
| enable_gpu_mixed | 0 |
| mixed_precision_mode | fp32 |
| memory_pool_init_size | 100MB |
| use_external_stream | false |
| thread_local_stream | false |
| use_tensorrt | false |
+--------------------------+------------------------------------------+
| use_xpu | false |
+--------------------------+------------------------------------------+
| use_cinn_compiler | false |
| save_optimized_model | false |
| ir_optim | true |
| ir_debug | false |
| use_optimized_model | false |
| memory_optim | true |
| enable_profile | false |
| enable_log | true |
| collect_shape_range_info | false |
+--------------------------+------------------------------------------+

WARNING: Logging before InitGoogleLogging() is written to STDERR
I0311 10:53:54.710409 42149 analysis_predictor.cc:2057] Ir optimization is turned off, no ir pass will be executed.
--- Running analysis [ir_graph_build_pass]
I0311 10:53:54.741750 42149 executor.cc:183] Old Executor is Running.
--- Running analysis [ir_analysis_pass]
--- Running analysis [ir_params_sync_among_devices_pass]
--- Running analysis [adjust_cudnn_workspace_size_pass]
--- Running analysis [inference_op_replace_pass]
--- Running analysis [save_optimized_model_pass]
--- Running analysis [ir_graph_to_program_pass]
I0311 10:53:55.211637 42149 analysis_predictor.cc:2146] ======= ir optimization completed =======
--- Running PIR pass [add_shadow_output_after_dead_parameter_pass]
--- Running PIR pass [delete_quant_dequant_linear_op_pass]
--- Running PIR pass [delete_weight_dequant_linear_op_pass]
--- Running PIR pass [map_op_to_another_pass]
I0311 10:53:55.312294 42149 print_statistics.cc:50] --- detected [27] subgraphs!
--- Running PIR pass [identity_op_clean_pass]
I0311 10:53:55.323707 42149 print_statistics.cc:50] --- detected [12] subgraphs!
--- Running PIR pass [silu_fuse_pass]
I0311 10:53:55.337136 42149 print_statistics.cc:50] --- detected [19] subgraphs!
--- Running PIR pass [conv2d_bn_fuse_pass]
--- Running PIR pass [conv2d_add_act_fuse_pass]
--- Running PIR pass [conv2d_add_fuse_pass]
I0311 10:53:55.353192 42149 print_statistics.cc:50] --- detected [13] subgraphs!
--- Running PIR pass [embedding_eltwise_layernorm_fuse_pass]
--- Running PIR pass [fused_rotary_position_embedding_pass]
--- Running PIR pass [multihead_matmul_fuse_pass]
--- Running PIR pass [matmul_add_act_fuse_pass]
I0311 10:53:55.446043 42149 print_statistics.cc:50] --- detected [133] subgraphs!
--- Running PIR pass [fc_elementwise_layernorm_fuse_pass]
--- Running PIR pass [add_norm_fuse_pass]
I0311 10:53:55.487458 42149 print_statistics.cc:50] --- detected [20] subgraphs!
--- Running PIR pass [group_norm_silu_fuse_pass]
--- Running PIR pass [matmul_scale_fuse_pass]
--- Running PIR pass [matmul_transpose_fuse_pass]
--- Running PIR pass [transpose_flatten_concat_fuse_pass]
--- Running PIR pass [remove_redundant_transpose_pass]
--- Running PIR pass [horizontal_fuse_pass]
I0311 10:53:55.502025 42149 print_statistics.cc:50] --- detected [1] subgraphs!
--- Running PIR pass [common_subexpression_elimination_pass]
I0311 10:53:55.512122 42149 print_statistics.cc:50] --- detected [558] subgraphs!
--- Running PIR pass [params_sync_among_devices_pass]
I0311 10:53:55.575439 42149 print_statistics.cc:50] --- detected [721] subgraphs!
--- Running PIR pass [constant_folding_pass]
I0311 10:53:55.577561 42149 pir_interpreter.cc:1586] New Executor is Running ...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0311 10:53:55.578130 42149 gpu_resources.cc:119] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.8, Runtime API Version: 11.8
W0311 10:53:55.579555 42149 gpu_resources.cc:164] device: 0, cuDNN Version: 8.6.
I0311 10:53:55.580062 42149 pir_interpreter.cc:1610] pir interpreter is running by multi-thread mode ...
I0311 10:53:55.741286 42149 print_statistics.cc:44] --- detected [160, 2088] subgraphs!
--- Running PIR pass [dead_code_elimination_pass]
I0311 10:53:55.743919 42149 print_statistics.cc:50] --- detected [91] subgraphs!
--- Running PIR pass [replace_fetch_with_shadow_output_pass]
I0311 10:53:55.745721 42149 print_statistics.cc:50] --- detected [3] subgraphs!
--- Running PIR pass [remove_shadow_feed_pass]
I0311 10:53:55.782218 42149 print_statistics.cc:50] --- detected [3] subgraphs!
--- Running PIR pass [inplace_pass]
I0311 10:53:56.059289 42149 print_statistics.cc:50] --- detected [325] subgraphs!
I0311 10:53:56.060429 42149 analysis_predictor.cc:1142] ======= pir optimization completed =======
[INFO] ultra_infer/runtime/runtime.cc(265)::CreatePaddleBackend Runtime initialized with Backend::PDINFER in Device::GPU.
I0311 10:53:56.906312 42149 pir_interpreter.cc:1607] pir interpreter is running by trace mode ...
Segmentation fault (core dumped)

换图片可以正常推理 （两张图片格式一样，分辨率一样，通道数一样）

λ ys-ai-gpu03 /paddle/03test/test paddlex --pipeline instance_segmentation.yaml --input ./test2/LD20250306142346_0307_sweetpotato_shrimp_yiwu.jpg --device gpu:0 --use_hpip
Only Paddle model is detected. Paddle model will be used by default.
Backend: paddle_infer
Backend config: cpu_num_threads=8 enable_mkldnn=True enable_trt=False trt_dynamic_shapes={'im_shape': [[1, 2], [1, 2], [8, 2]], 'image': [[1, 3, 640, 640], [1, 3, 640, 640], [8, 3, 640, 640]], 'scale_factor': [[1, 2], [1, 2], [8, 2]]} trt_dynamic_shape_input_data={'im_shape': [[640.0, 640.0], [640.0, 640.0], [640.0, 640.0, 640.0, 640.0, 640.0, 640.0, 640.0, 640.0, 640.0, 640.0, 640.0, 640.0, 640.0, 640.0, 640.0, 640.0]], 'scale_factor': [[2.0, 2.0], [1.0, 1.0], [0.67, 0.67, 0.67, 0.67, 0.67, 0.67, 0.67, 0.67, 0.67, 0.67, 0.67, 0.67, 0.67, 0.67, 0.67, 0.67]]} trt_precision='FP32' enable_log_info=True
[INFO] ultra_infer/vision/common/processors/transform.cc(44)::FuseNormalizeCast Normalize and Cast are fused to Normalize in preprocessing pipeline.
[INFO] ultra_infer/vision/common/processors/transform.cc(91)::FuseNormalizeHWC2CHW Normalize and HWC2CHW are fused to NormalizeAndPermute in preprocessing pipeline.
[INFO] ultra_infer/vision/common/processors/transform.cc(157)::FuseNormalizeColorConvert BGR2RGB and NormalizeAndPermute are fused to NormalizeAndPermute with swap_rb=1
[INFO] ultra_infer/runtime/backends/paddle/paddle_backend.cc(28)::BuildOption Will inference_precision float32
[INFO] ultra_infer/runtime/backends/paddle/paddle_backend.cc(338)::InitFromPaddle Finish paddle inference config with summary as:
[INFO]
+--------------------------+------------------------------------------+
| Option | Value |
+--------------------------+------------------------------------------+
| model_file | best_model/inference/inference.pdmodel |
| params_file | best_model/inference/inference.pdiparams |
+--------------------------+------------------------------------------+
| cpu_math_thread | 8 |
| enable_mkldnn | true |
| mkldnn_cache_capacity | 10 |
+--------------------------+------------------------------------------+
| use_gpu | true |
| use_cutlass | false |
| gpu_device_id | 0 |
| enable_gpu_mixed | 0 |
| mixed_precision_mode | fp32 |
| memory_pool_init_size | 100MB |
| use_external_stream | false |
| thread_local_stream | false |
| use_tensorrt | false |
+--------------------------+------------------------------------------+
| use_xpu | false |
+--------------------------+------------------------------------------+
| use_cinn_compiler | false |
| save_optimized_model | false |
| ir_optim | true |
| ir_debug | false |
| use_optimized_model | false |
| memory_optim | true |
| enable_profile | false |
| enable_log | true |
| collect_shape_range_info | false |
+--------------------------+------------------------------------------+

WARNING: Logging before InitGoogleLogging() is written to STDERR
I0311 10:54:56.316615 43095 analysis_predictor.cc:2057] Ir optimization is turned off, no ir pass will be executed.
--- Running analysis [ir_graph_build_pass]
I0311 10:54:56.347638 43095 executor.cc:183] Old Executor is Running.
--- Running analysis [ir_analysis_pass]
--- Running analysis [ir_params_sync_among_devices_pass]
--- Running analysis [adjust_cudnn_workspace_size_pass]
--- Running analysis [inference_op_replace_pass]
--- Running analysis [save_optimized_model_pass]
--- Running analysis [ir_graph_to_program_pass]
I0311 10:54:56.821062 43095 analysis_predictor.cc:2146] ======= ir optimization completed =======
--- Running PIR pass [add_shadow_output_after_dead_parameter_pass]
--- Running PIR pass [delete_quant_dequant_linear_op_pass]
--- Running PIR pass [delete_weight_dequant_linear_op_pass]
--- Running PIR pass [map_op_to_another_pass]
I0311 10:54:56.921630 43095 print_statistics.cc:50] --- detected [27] subgraphs!
--- Running PIR pass [identity_op_clean_pass]
I0311 10:54:56.933264 43095 print_statistics.cc:50] --- detected [12] subgraphs!
--- Running PIR pass [silu_fuse_pass]
I0311 10:54:56.946813 43095 print_statistics.cc:50] --- detected [19] subgraphs!
--- Running PIR pass [conv2d_bn_fuse_pass]
--- Running PIR pass [conv2d_add_act_fuse_pass]
--- Running PIR pass [conv2d_add_fuse_pass]
I0311 10:54:56.963368 43095 print_statistics.cc:50] --- detected [13] subgraphs!
--- Running PIR pass [embedding_eltwise_layernorm_fuse_pass]
--- Running PIR pass [fused_rotary_position_embedding_pass]
--- Running PIR pass [multihead_matmul_fuse_pass]
--- Running PIR pass [matmul_add_act_fuse_pass]
I0311 10:54:57.055701 43095 print_statistics.cc:50] --- detected [133] subgraphs!
--- Running PIR pass [fc_elementwise_layernorm_fuse_pass]
--- Running PIR pass [add_norm_fuse_pass]
I0311 10:54:57.096724 43095 print_statistics.cc:50] --- detected [20] subgraphs!
--- Running PIR pass [group_norm_silu_fuse_pass]
--- Running PIR pass [matmul_scale_fuse_pass]
--- Running PIR pass [matmul_transpose_fuse_pass]
--- Running PIR pass [transpose_flatten_concat_fuse_pass]
--- Running PIR pass [remove_redundant_transpose_pass]
--- Running PIR pass [horizontal_fuse_pass]
I0311 10:54:57.111555 43095 print_statistics.cc:50] --- detected [1] subgraphs!
--- Running PIR pass [common_subexpression_elimination_pass]
I0311 10:54:57.121692 43095 print_statistics.cc:50] --- detected [558] subgraphs!
--- Running PIR pass [params_sync_among_devices_pass]
I0311 10:54:57.184898 43095 print_statistics.cc:50] --- detected [721] subgraphs!
--- Running PIR pass [constant_folding_pass]
I0311 10:54:57.186995 43095 pir_interpreter.cc:1586] New Executor is Running ...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0311 10:54:57.187570 43095 gpu_resources.cc:119] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.8, Runtime API Version: 11.8
W0311 10:54:57.188938 43095 gpu_resources.cc:164] device: 0, cuDNN Version: 8.6.
I0311 10:54:57.189483 43095 pir_interpreter.cc:1610] pir interpreter is running by multi-thread mode ...
I0311 10:54:57.352090 43095 print_statistics.cc:44] --- detected [160, 2088] subgraphs!
--- Running PIR pass [dead_code_elimination_pass]
I0311 10:54:57.354604 43095 print_statistics.cc:50] --- detected [91] subgraphs!
--- Running PIR pass [replace_fetch_with_shadow_output_pass]
I0311 10:54:57.356446 43095 print_statistics.cc:50] --- detected [3] subgraphs!
--- Running PIR pass [remove_shadow_feed_pass]
I0311 10:54:57.393235 43095 print_statistics.cc:50] --- detected [3] subgraphs!
--- Running PIR pass [inplace_pass]
I0311 10:54:57.671515 43095 print_statistics.cc:50] --- detected [325] subgraphs!
I0311 10:54:57.672621 43095 analysis_predictor.cc:1142] ======= pir optimization completed =======
[INFO] ultra_infer/runtime/runtime.cc(265)::CreatePaddleBackend Runtime initialized with Backend::PDINFER in Device::GPU.
I0311 10:54:58.593742 43095 pir_interpreter.cc:1607] pir interpreter is running by trace mode ...
{'res': {'input_path': './test2/LD20250306142346_0307_sweetpotato_shrimp_yiwu.jpg', 'page_index': None, 'boxes': [{'cls_id': 0, 'label': 'sweetpotato', 'score': 0.9717305898666382, 'coordinate': [350.5212097167969, 783.5634155273438, 616.4833984375, 963.5111083984375]}, {'cls_id': 0, 'label': 'sweetpotato', 'score': 0.9635534882545471, 'coordinate': [1137.1429443359375, 410.95404052734375, 1331.263427734375, 995.310546875]}], 'masks': '...'}}

[Bobholamovic]

考虑有可能是前后处理的问题，辛苦 @zhang-prog 看看～

[zhang-prog]

这个issue可以和 #3548 合并处理，我已在跟进