Skip to content

Adds mirrormaker example for GKE clusters #175

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 5 commits into from
Closed

Adds mirrormaker example for GKE clusters #175

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
d384382
Just a pod with the Kafka distribution
solsson May 1, 2018
9b66ed0
Specific to my current test clusters
solsson May 1, 2018
abbb2c3
Challenge: auto.offset.reset earliest makes it easy
solsson May 1, 2018
7b789be
Adapts to WARNING: The default partition assignment strategy of
solsson May 1, 2018
247a1c9
More cluster specific stuff, but I'd recommend
solsson May 1, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Specific to my current test clusters
  • Loading branch information
@solsson
solsson committed May 1, 2018
commit 9b66ed077cbc0e8bc2cdfe67b49450beeaace000
30 changes: 27 additions & 3 deletions mirrormaker/mirrormaker.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,31 @@ spec:
containers:
- name: kafka
image: solsson/kafka:1.1@sha256:ba863ca7dc28563930584e37f93d57c2cbf3f46b1c1fa104fe8af7bcc0c31df4
resources:
requests:
cpu: 0m
memory: 80Mi
limits:
cpu: 20m
memory: 200Mi
env:
- name: TARGET_BOOTSTRAP
value: bootstrap.kafka:9092
- name: SOURCE_BOOTSTRAP
value: mirror-from.kafka:32400,mirror-from.kafka:32401,mirror-from.kafka:32402
- name: WHITELIST
value: .*yolean.*yolean.*operations.*
- name: GROUP_ID
value: mirror-to-4
command:
- tail
- -f
- /dev/null
- bash
- -cex
- >
echo "bootstrap.servers=$SOURCE_BOOTSTRAP" > ./config/consumer.properties;
echo "group.id=$GROUP_ID" >> ./config/consumer.properties;
echo "auto.offset.reset=earliest" >> ./config/consumer.properties;
echo "bootstrap.servers=$TARGET_BOOTSTRAP" > ./config/producer.properties;
echo "log4j.logger.org.apache.kafka.clients.Metadata=DEBUG" >> ./config/tools-log4j.properties;
echo "log4j.logger.org.apache.kafka.clients.consumer.internals.AbstractCoordinator=INFO" >> ./config/tools-log4j.properties;
echo "log4j.logger.org.apache.kafka.clients.NetworkClient=INFO" >> ./config/tools-log4j.properties;
./bin/kafka-mirror-maker.sh --consumer.config ./config/consumer.properties --producer.config ./config/producer.properties --whitelist "$WHITELIST"
32 changes: 32 additions & 0 deletions mirrormaker/source-endpoints.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
kind: Service
apiVersion: v1
metadata:
name: mirror-from
namespace: kafka
spec:
ports:
- name: outside-0
port: 32400
- name: outside-1
port: 32401
- name: outside-2
port: 32402
---
kind: Endpoints
apiVersion: v1
metadata:
name: mirror-from
namespace: kafka
subsets:
- addresses:
# source cluster: kubectl -n kafka get pods -l app=kafka -o yaml | grep outside
- ip: 10.132.0.13
- ip: 10.132.0.9
- ip: 10.132.0.10
ports:
- name: outside-0
port: 32400
- name: outside-1
port: 32401
- name: outside-2
port: 32402