Fully utilize the max.in.flight.requests.per.connection parameter on the idempotent producer

[marcin-krystianc]

Currently, the librdkafka implementation sends only one produce request at a time, regardless of the configured value for the max.in.flight.requests.per.connection parameter. This limitation prevents the producer from fully leveraging the potential of concurrent requests, which can impact throughput and performance.

With the proposed changes, the max.in.flight.requests.per.connection parameter will be respected, allowing the number of concurrent requests to be aligned with the specified value. This enhancement aims to optimize the idempotent producer's performance by enabling multiple produce requests to be sent simultaneously, thereby improving overall latency.

To validate my findings, I conducted tests on a network with a simulated delay of 1000 ms, configured using the tc command (tc qdisc add dev eth0 root netem delay 1000ms). I have also attached pcap files for your review.

Relevant producer parameters:

--config enable.idempotence=true
--config max.in.flight.requests.per.connection=5
--config linger.ms=200
--config socket.nagle.disable=true

• librdkafka 2.8.0 (p95 latency ~=1950ms), capture_2.8.0.pcap:

418	13:01:07.506672	172.18.0.4	172.18.0.3	59686,19092	Kafka Produce v10 Request
432	13:01:08.508093	172.18.0.3	172.18.0.4	19092,59686	Kafka Produce v10 Response
454	13:01:08.508619	172.18.0.4	172.18.0.3	59686,19092	Kafka Produce v10 Request
459	13:01:09.509790	172.18.0.3	172.18.0.4	19092,59686	Kafka Produce v10 Response
467	13:01:09.510013	172.18.0.4	172.18.0.3	59686,19092	Kafka Produce v10 Request
474	13:01:10.511038	172.18.0.3	172.18.0.4	19092,59686	Kafka Produce v10 Response
482	13:01:10.511223	172.18.0.4	172.18.0.3	59686,19092	Kafka Produce v10 Request
487	13:01:11.512288	172.18.0.3	172.18.0.4	19092,59686	Kafka Produce v10 Response
495	13:01:11.512462	172.18.0.4	172.18.0.3	59686,19092	Kafka Produce v10 Request
500	13:01:12.513293	172.18.0.3	172.18.0.4	19092,59686	Kafka Produce v10 Response
508	13:01:12.513462	172.18.0.4	172.18.0.3	59686,19092	Kafka Produce v10 Request

• librdkafka 2.8.0+patch (p95 latency ~= 1190ms), capture_2.8.0_patched.pcap:

237	12:58:32.288766	172.18.0.4	172.18.0.3	60190,19092	Kafka Produce v10 Request
239	12:58:32.288771	172.18.0.4	172.18.0.3	60190,19092	Kafka Produce v10 Request
246	12:58:32.329058	172.18.0.4	172.18.0.3	60190,19092	Kafka Produce v10 Request
247	12:58:32.329061	172.18.0.4	172.18.0.3	60190,19092	Kafka Produce v10 Request
250	12:58:32.788187	172.18.0.4	172.18.0.3	60190,19092	Kafka Produce v10 Request
260	12:58:33.301184	172.18.0.3	172.18.0.4	19092,60190	Kafka Produce v10 Response
271	12:58:33.301766	172.18.0.4	172.18.0.3	60190,19092	Kafka Produce v10 Request
272	12:58:33.302767	172.18.0.3	172.18.0.4	19092,60190	Kafka Produce v10 Response
275	12:58:33.330077	172.18.0.3	172.18.0.4	19092,60190	Kafka Produce v10 Response
277	12:58:33.330703	172.18.0.3	172.18.0.4	19092,60190	Kafka Produce v10 Response

[confluent-cla-assistant]

🎉 All Contributor License Agreements have been signed. Ready to merge.
✅ marcin-krystianc
_{Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.}

[marcin-krystianc]

The inflight variable holds the number of inflight messages (rd_atomic32_get(&rktp->rktp_msgs_inflight)) instead of the number of inflight requests.

[emasab]

Thanks @marcin-krystianc for spotting this! I'll check if other changes are needed but it doesn't seem at the moment.