Performance test improvements

[emasab]

measure eachBatch rate, time and lag, avg and max memory usage

[Copilot]

Pull Request Overview

This PR enhances performance testing capabilities by adding detailed metrics collection including eachBatch processing, memory usage tracking, timing measurements, and offset lag monitoring. The changes consolidate common consumer functionality and expand test coverage to include both eachMessage and eachBatch consumption patterns.

Key changes:

• Extracted common consumer logic into a shared module supporting both eachMessage and eachBatch patterns
• Added comprehensive memory tracking with RSS measurements during test execution
• Enhanced metrics collection including offset lag, processing time, and batch statistics
• Migrated performance test script from Bash to Node.js for better cross-platform compatibility

Reviewed Changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
examples/performance/performance-primitives-common.js	New shared module implementing consumer logic for both eachMessage and eachBatch patterns with enhanced metrics
examples/performance/performance-primitives.js	Refactored to use common consumer module with compatibility wrapper
examples/performance/performance-primitives-kafkajs.js	Updated to use common consumer module and added duration logging
examples/performance/performance-consolidated.js	Added memory tracking, expanded test coverage for both consumption patterns, and enhanced logging
ci/tests/run_perf_test.sh	Removed Bash-based performance test script
ci/tests/run_perf_test.js	New Node.js-based performance test script with enhanced metric extraction
.semaphore/semaphore.yml	Updated to use Node.js performance test script

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

The message count has been reduced from 500000 to 50000 compared to the original bash script. This significant reduction (90%) may affect the reliability of performance measurements and should be documented or reconsidered.

Suggested change

	const outputConfluentProducerConsumer = runCommand('MODE=confluent MESSAGE_COUNT=50000 node performance-consolidated.js --create-topics --consumer --producer');
	console.log('Running KafkaJS Producer/Consumer test...');
	const outputKjsProducerConsumer = runCommand('MODE=kafkajs MESSAGE_COUNT=50000 node performance-consolidated.js --create-topics --consumer --producer');
	const outputConfluentProducerConsumer = runCommand('MODE=confluent MESSAGE_COUNT=500000 node performance-consolidated.js --create-topics --consumer --producer');
	console.log('Running KafkaJS Producer/Consumer test...');
	const outputKjsProducerConsumer = runCommand('MODE=kafkajs MESSAGE_COUNT=500000 node performance-consolidated.js --create-topics --consumer --producer');

[emasab]

it makes the test faster with bigger messages sizes and doesn't seem to affect reliability of results.

[milindl]

Thanks for adding the new features to the test (the rss) and eachBatch. The run on semaphore also looks fine to me, as expected in terms of consume throughput and rss.

[milindl]

Minor changes suggested

[milindl]

Instead of doing it here, at a later point, do

if (parameters.securityProtocol.toLowerCase().includes('sasl')) {
 ret['sasl'] = { ... };
}

So we can have just ssl. And above && condition can be changed also

[milindl]

Same for the confluent performance primitives.

[sonarqube-confluent]

Analysis Details

6 Issues

• 0 Bugs
• 4 Vulnerabilities
• 2 Code Smells

Coverage and Duplications

• No coverage information (70.80% Estimated after merge)
• No duplication information (2.00% Estimated after merge)

Project ID: confluent-kafka-javascript

View in SonarQube