Does setting setTensorAddress on every inference call impact throughput?

[spacewalk01]

Hello,

I have been using the older execute inference API in TensorRT for several years. Due to a recent GPU upgrade, I am now migrating to a newer version of TensorRT.

I noticed that when using the enqueueV3 or enqueueV2 APIs, the tensor address must be specified for each inference call via setTensorAddress.

Could you clarify if calling setTensorAddress on every inference introduces any overhead that might impact overall performance or throughput?

Thank you.

[Saad-Mallebhari]

Calling setTensorAddress on every inference call introduces minimal overhead in practice. It is a CPU-side pointer registration operation , it simply updates an internal table mapping tensor names to device memory addresses. There is no GPU work triggered by setTensorAddress itself.

The actual performance-critical path is enqueueV3, which launches the CUDA kernels.

If your tensor addresses remain unchanged between calls (e.g. you're reusing the same device buffers), you can set them once after context creation and skip the repeated calls , though this depends on your specific setup:

// Set once after engine/context creation if buffers are stable
context->setTensorAddress("input", d_input);
context->setTensorAddress("output", d_output);

// Then enqueue repeatedly
for (int i = 0; i < num_inferences; i++) {
    context->enqueueV3(stream);
}

You would need to call setTensorAddress again if your buffer pointers change , for example when using dynamic shapes, ping-pong buffers, or switching execution contexts.

For your migration from the older API, the overhead of calling it every inference is unlikely to be measurable in practice, but avoiding redundant calls when buffers are stable is a cleaner pattern.