rotor: migrate to confluent kafka library WIP

[absorbb]

Summary by cubic

Migrate rotor from kafkajs to Confluent’s Kafka JavaScript client to improve stability and add native compression support. This updates consumer/producer/admin APIs, simplifies config, and enables LZ4 in addition to gzip/snappy/zstd.

• Dependencies

  • Added @confluentinc/kafka-javascript; removed kafkajs, kafkajs-snappy, and @sensejs/kafkajs-zstd-support.
  • Requires Node 18+.

• Refactors

  • Switched to new client wiring in kafka-config; SSL toggled via boolean; SASL passed through unchanged.
  • Consumer/producer/admin options now nested under kafkaJS; fromBeginning/autoCommit configured on the consumer.
  • Producer sets global acks=1 and compression from KAFKA_TOPIC_COMPRESSION; per-send compression removed.
  • getCompressionType now supports GZIP, Snappy, LZ4, ZSTD, and None.
  • Admin fetchTopicOffsets uses timeout and READ_COMMITTED isolation.
  • Consumer uses optional group.instance.id from ROTOR_INSTANCE_ID.
  • Updated tests to new API; minor comment typo fix in webapp.

[Copilot]

Pull Request Overview

This PR migrates the rotor service from kafkajs to Confluent's Kafka JavaScript client to improve stability and add native compression support. Key changes include updating the producer/consumer/admin APIs, enabling LZ4 compression, and simplifying configuration while maintaining backward compatibility.

• Replace kafkajs dependency with @confluentinc/kafka-javascript
• Update API configuration to use nested kafkaJS config structure
• Enable native LZ4 compression support in addition to existing compression types

Reviewed Changes

Copilot reviewed 10 out of 11 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
webapps/console/lib/server/sync.ts	Minor typo fix changing "to low" to "too low"
services/rotor/webpack.config.js	Add external mapping for new Confluent Kafka library
services/rotor/src/lib/rotor.ts	Core migration to new Kafka client with updated consumer/producer configuration
services/rotor/src/lib/metrics.ts	Update type definitions and remove per-send compression
services/rotor/src/lib/kafka-config.ts	Complete rewrite of connection logic for new Kafka client
services/rotor/package.json	Replace kafkajs dependencies with Confluent client
services/rotor/dist_package.json	Update distribution dependencies
services/rotor/tests/kafka.test.ts	Update test configuration for new API
services/profiles/package.json	Bump Confluent client version to 1.4.1
services/profiles/dist_package.json	Update distribution dependency version

Files not reviewed (1)

• pnpm-lock.yaml: Language not supported

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[cubic-dev-ai]

2 issues found across 11 files

Prompt for AI agents (all 2 issues)

Understand the root cause of the following 2 issues and fix them.


<file name="services/rotor/src/lib/rotor.ts">

<violation number="1" location="services/rotor/src/lib/rotor.ts:96">
admin.connect() is not awaited, risking race conditions and unhandled rejections.</violation>

<violation number="2" location="services/rotor/src/lib/rotor.ts:123">
DoS via unbounded parallelism from attacker-controlled CONNECTION_IDS_HEADER (CWE-400)</violation>
</file>

_{React with 👍 or 👎 to teach cubic. Mention @cubic-dev-ai to give feedback, ask questions, or re-run the review.}