Add SQL Server + Debezium v2.x example

integrations/debezium/README.md

@@ -10,7 +10,7 @@ This is a collection of demos that show how to use the [Debezium](https://materi
 | ---------------------------------- | --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
 | [Postgres](postgres)               | Connect to a Postgres database and stream changes to Kafka/Redpanda         | [Postgres](https://materialize.com/docs/ingest-data/cdc-postgres-kafka-debezium/)    |
 | TODO: [MySQL](mysql)               | Connect to a MySQL database and stream changes to Kafka/Redpanda            | [MySQL](https://materialize.com/docs/ingest-data/cdc-mysql/)                         |
-| TODO: [SQL server](sqlserver)      | Connect to a SQL server database and stream changes to Kafka/Redpanda       | TODO                                                                                 |
+| [SQL server](sqlserver)            | Connect to a SQL server database and stream changes to Kafka/Redpanda       | TODO                                                                                 |
 <!-- | TODO: [MongoDB](mongodb)           | Connect to a MongoDB database and stream changes to Kafka/Redpanda     | TODO                                                                                 | -->
 
 ## Prerequisites
@@ -22,3 +22,21 @@ This is a collection of demos that show how to use the [Debezium](https://materi
 ## Running the demos
 
 For each demo, follow the instructions in the demo's README. All demos assume that you have `psql`, `docker` and a publicly accessible Linux environment.
+
+## Notes
+
+Beginning with Debezium 2.0.0, Confluent Schema Registry support is not included in the Debezium containers. To enable the Confluent Schema Registry for a Debezium container, install the following Confluent Avro converter JAR files into the Connect plugin directory:
+
+-   `kafka-connect-avro-converter`
+
+-   `kafka-connect-avro-data`
+
+-   `kafka-avro-serializer`
+
+-   `kafka-schema-serializer`
+
+-   `kafka-schema-registry-client`
+
+-   `common-config`
+
+-   `common-utils`

integrations/debezium/postgres/README.md

@@ -1,4 +1,4 @@
-# 
+# Debezium + PostgreSQL + Materialize
 
 Before trying this out, you will need the following:
 
@@ -163,24 +163,6 @@ To stop the services and remove the containers, run:
 docker compose down
 ```
 
-## Notes
-
-Beginning with Debezium 2.0.0, Confluent Schema Registry support is not included in the Debezium containers. To enable the Confluent Schema Registry for a Debezium container, install the following Confluent Avro converter JAR files into the Connect plugin directory:
-
--   `kafka-connect-avro-converter`
-
--   `kafka-connect-avro-data`
-
--   `kafka-avro-serializer`
-
--   `kafka-schema-serializer`
-
--   `kafka-schema-registry-client`
-
--   `common-config`
-
--   `common-utils`
-
 ## Helpful resources:
 
 * [`CREATE SOURCE: PostgreSQL`](https://materialize.com/docs/sql/create-source/postgres)

integrations/debezium/sqlserver/README.md

@@ -0,0 +1,132 @@
+# Debezium + SQL Server + Materialize
+
+Before trying this out, you will need the following:
+
+- [Materialize account](https://materialize.com/register/).
+- A publicly accessible Linux server with [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/) installed.
+
+## Running the demo
+
+If you want to try it right now, follow these steps:
+
+1. Clone the project on your Linux server and run:
+
+    ```shell session
+    git clone https://github.com/MaterializeInc/demos.git
+    cd demos/integrations/debezium/sqlserver
+    ```
+
+1. Start all containers:
+
+    ```shell
+    export EXTERNAL_IP=$(hostname -I | awk '{print $1}')
+    docker compose up -d --build
+    ```
+
+1. Check the status of the containers:
+
+    ```shell
+    docker compose ps
+    ```
+
+
+1. Exec in to the redpanda container to look around using redpanda's amazing [rpk](https://docs.redpanda.com/docs/reference/rpk/) CLI.
+
+   ```shell session
+   docker compose exec redpanda /bin/bash
+
+   rpk debug info
+
+   rpk topic list
+   ```
+
+1. Connect to Materialize
+
+    If you already have `psql` installed on your machine, use the provided connection string to connect:
+
+    Example:
+
+    ```shell session
+    psql "postgres://user%40domain.com@materialize_host:6875/materialize"
+    ```
+
+    Otherwise, you can find the steps to install and use your CLI of choice under [Supported tools](https://materialize.com/docs/integrations/sql-clients/#supported-tools).
+
+1. Now that you're in the Materialize CLI, define the connection to the Redpanda broker and the schema registry:
+
+    ```sql
+    -- Create Redpanda connection
+    CREATE CONNECTION redpanda_connection
+      TO KAFKA (
+      BROKER '<your_server_ip:9092>');
+
+    -- Create Registry connection
+    CREATE CONNECTION schema_registry
+      TO CONFLUENT SCHEMA REGISTRY (
+      URL 'http://<your_server_ip:8081>');
+    ```
+
+1. Next, define all of the topics as sources:
+
+    ```sql
+    CREATE SOURCE customers
+      FROM KAFKA CONNECTION redpanda_connection (TOPIC 'server1.testDB.dbo.customers')
+      FORMAT AVRO USING CONFLUENT SCHEMA REGISTRY CONNECTION schema_registry
+      ENVELOPE DEBEZIUM
+      WITH (SIZE = '3xsmall');
+
+    CREATE SOURCE orders
+      FROM KAFKA CONNECTION redpanda_connection (TOPIC 'server1.testDB.dbo.orders')
+      FORMAT AVRO USING CONFLUENT SCHEMA REGISTRY CONNECTION schema_registry
+      ENVELOPE DEBEZIUM
+      WITH (SIZE = '3xsmall');
+
+    CREATE SOURCE products
+      FROM KAFKA CONNECTION redpanda_connection (TOPIC 'server1.testDB.dbo.products')
+      FORMAT AVRO USING CONFLUENT SCHEMA REGISTRY CONNECTION schema_registry
+      ENVELOPE DEBEZIUM
+      WITH (SIZE = '3xsmall');
+
+    CREATE SOURCE products_on_hand
+      FROM KAFKA CONNECTION redpanda_connection (TOPIC 'server1.testDB.dbo.products_on_hand')
+      FORMAT AVRO USING CONFLUENT SCHEMA REGISTRY CONNECTION schema_registry
+      ENVELOPE DEBEZIUM
+      WITH (SIZE = '3xsmall');
+    ```
+
+1. Subscribe to the `orders` topic:
+
+    ```sql
+    COPY ( SUBSCRIBE TO orders ) TO STDOUT;
+    ```
+
+1. Next generate some orders:
+
+    ```sh
+    cat sqlserver/orders.sql | docker compose -f docker compose.yaml exec -T sqlserver bash -c '/opt/mssql-tools/bin/sqlcmd -U sa -P $SA_PASSWORD'
+    ```
+
+1. Modify records in the database via SQL Server client (do not forget to add `GO` command to execute the statement)
+
+    ```sh
+    docker compose -f docker compose.yaml exec sqlserver bash -c '/opt/mssql-tools/bin/sqlcmd -U sa -P $SA_PASSWORD -d testDB'
+    ```
+
+
+## Cleanup
+
+To stop the services and remove the containers, run:
+
+```shell session
+docker compose down
+```
+
+## Helpful resources:
+
+* [`CREATE CONNECTION`](https://materialize.com/docs/sql/create-connection/)
+* [`CREATE SOURCE`](https://materialize.com/docs/sql/create-source)
+* [`CREATE MATERIALIZED VIEW`](https://materialize.com/docs/sql/create-materialized-view)
+
+## Community
+
+If you have any questions or comments, please join the [Materialize Slack Community](https://materialize.com/s/chat)!

integrations/debezium/sqlserver/connect/Dockerfile

@@ -0,0 +1,49 @@
+FROM debezium/connect-base:2.4
+
+#
+# Set up the plugins directory ...
+#
+ENV CONFLUENT_VERSION=7.0.1 \
+    AVRO_VERSION=1.10.1 \
+    GUAVA_VERSION=31.0.1-jre
+
+RUN docker-maven-download confluent kafka-connect-avro-converter "$CONFLUENT_VERSION" fd03a1436f29d39e1807e2fb6f8e415a && \
+    docker-maven-download confluent kafka-connect-avro-data "$CONFLUENT_VERSION" d27f30e9eca4ef1129289c626e9ce1f1 && \
+    docker-maven-download confluent kafka-avro-serializer "$CONFLUENT_VERSION" c72420603422ef54d61f493ca338187c && \
+    docker-maven-download confluent kafka-schema-serializer "$CONFLUENT_VERSION" 9c510db58119ef66d692ae172d5b1204 && \
+    docker-maven-download confluent kafka-schema-registry-client "$CONFLUENT_VERSION" 7449df1f5c9a51c3e82e776eb7814bf1 && \
+    docker-maven-download confluent common-config "$CONFLUENT_VERSION" aab5670de446af5b6f10710e2eb86894 && \
+    docker-maven-download confluent common-utils "$CONFLUENT_VERSION" 74bf5cc6de2748148f5770bccd83a37c && \
+    docker-maven-download central org/apache/avro avro "$AVRO_VERSION" 35469fee6d74ecbadce4773bfe3a204c && \
+    docker-maven-download central com/google/guava guava "$GUAVA_VERSION" bb811ca86cba6506cca5d415cd5559a7
+
+# https://github.com/debezium/container-images/blob/main/connect/2.4/Dockerfile
+LABEL maintainer="Debezium Community"
+
+ENV DEBEZIUM_VERSION="2.4.0.Final" \
+    MAVEN_REPO_CENTRAL="" \
+    MAVEN_REPOS_ADDITIONAL="" \
+    MAVEN_DEP_DESTINATION=$KAFKA_CONNECT_PLUGINS_DIR \
+    MONGODB_MD5=a22784387e0ec8a6abb1606c2c365cb2 \
+    MYSQL_MD5=4bff262afc9678f5cbc3be6315b8e71e \
+    POSTGRES_MD5=b42c9e208410f39ad1ad09778b1e3f03 \
+    SQLSERVER_MD5=9b8bf3c62a7c22c465a32fa27b3cffb5 \
+    ORACLE_MD5=21699814400860457dc2334b165882e6 \
+    DB2_MD5=0727d7f2d1deeacef39e230acac835a8 \
+    SPANNER_MD5=186b07595e914e9139941889fd675044 \
+    VITESS_MD5=3b4d24c8c9898df060c408a13fd3429f \
+    JDBC_MD5=77c5cb9adf932ab17c041544f4ade357 \
+    KCRESTEXT_MD5=25c0353f5a7304b3c4780a20f0f5d0af \
+    SCRIPTING_MD5=53a3661e7a9877744f4a30d6483d7957
+
+RUN docker-maven-download debezium mongodb "$DEBEZIUM_VERSION" "$MONGODB_MD5" && \
+    docker-maven-download debezium mysql "$DEBEZIUM_VERSION" "$MYSQL_MD5" && \
+    docker-maven-download debezium postgres "$DEBEZIUM_VERSION" "$POSTGRES_MD5" && \
+    docker-maven-download debezium sqlserver "$DEBEZIUM_VERSION" "$SQLSERVER_MD5" && \
+    docker-maven-download debezium oracle "$DEBEZIUM_VERSION" "$ORACLE_MD5" && \
+    docker-maven-download debezium-additional db2 db2 "$DEBEZIUM_VERSION" "$DB2_MD5" && \
+    docker-maven-download debezium-additional jdbc jdbc "$DEBEZIUM_VERSION" "$JDBC_MD5" && \
+    docker-maven-download debezium-additional spanner spanner "$DEBEZIUM_VERSION" "$SPANNER_MD5" && \
+    docker-maven-download debezium-additional vitess vitess "$DEBEZIUM_VERSION" "$VITESS_MD5" && \
+    docker-maven-download debezium-optional connect-rest-extension "$DEBEZIUM_VERSION" "$KCRESTEXT_MD5" && \
+    docker-maven-download debezium-optional scripting "$DEBEZIUM_VERSION" "$SCRIPTING_MD5"

integrations/debezium/sqlserver/deploy/Dockerfile

@@ -0,0 +1,10 @@
+FROM ubuntu:latest
+
+RUN apt-get update && apt-get -qy install curl
+
+COPY . /deploy
+
+COPY docker-entrypoint.sh /usr/local/bin
+RUN chmod 777 /usr/local/bin/docker-entrypoint.sh
+
+ENTRYPOINT ["docker-entrypoint.sh"]

integrations/debezium/sqlserver/deploy/docker-entrypoint.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+
+set -euo pipefail
+
+cd /deploy
+
+bash sqlserver_dbz.sh

integrations/debezium/sqlserver/deploy/register-sqlserver.json

@@ -0,0 +1,20 @@
+{
+    "name": "inventory-connector",
+    "config": {
+        "connector.class" : "io.debezium.connector.sqlserver.SqlServerConnector",
+        "tasks.max" : "1",
+        "topic.prefix" : "server1",
+        "database.hostname" : "sqlserver",
+        "database.port" : "1433",
+        "database.user" : "sa",
+        "database.password" : "Password!",
+        "database.names" : "testDB",
+        "schema.history.internal.kafka.bootstrap.servers" : "EXTERNAL_IP:9092",
+        "schema.history.internal.kafka.topic": "schema-changes.inventory",
+        "key.converter": "io.confluent.connect.avro.AvroConverter",
+        "value.converter": "io.confluent.connect.avro.AvroConverter",
+        "key.converter.schema.registry.url": "http://redpanda:8081",
+        "value.converter.schema.registry.url": "http://redpanda:8081",
+        "database.encrypt": "false"
+    }
+}

integrations/debezium/sqlserver/deploy/sqlserver_dbz.sh

@@ -0,0 +1,25 @@
+#!/bin/bash
+
+#Initialize Debezium (Kafka Connect Component)
+
+while true; do
+    echo "Waiting for Debezium to be ready"
+    sleep 0.1
+    curl -s -o /dev/null -w "%{http_code}" http://debezium:8083/connectors/ | grep 200
+    if [ $? -eq 0 ]; then
+        echo "Debezium is ready"
+        break
+    fi
+done
+
+# Read the JSON file and register the connector and change the ${EXTERNAL_IP} with the external IP environment variable
+sed -i "s/EXTERNAL_IP/${EXTERNAL_IP}/g" /deploy/register-sqlserver.json
+
+curl -i -X POST -H "Accept:application/json" -H  "Content-Type:application/json" http://debezium:8083/connectors/ -d @/deploy/register-sqlserver.json
+
+if [ $? -eq 0 ]; then
+    echo "Debezium connector registered"
+else
+    echo "Debezium connector registration failed"
+    exit 1
+fi

integrations/debezium/sqlserver/docker-compose.yaml

@@ -0,0 +1,63 @@
+version: '2'
+services:
+  redpanda:
+    image: docker.vectorized.io/vectorized/redpanda:v21.10.1
+    container_name: redpanda
+    command:
+     - redpanda start
+     - --overprovisioned
+     - --smp 1
+     - --memory 1G
+     - --reserve-memory 0M
+     - --node-id 0
+     - --check=false
+     - --kafka-addr 0.0.0.0:9092
+     - --advertise-kafka-addr ${EXTERNAL_IP:-redpanda}:9092
+     - --pandaproxy-addr 0.0.0.0:8082
+     - --advertise-pandaproxy-addr ${EXTERNAL_IP:-redpanda}:9092
+     - --set redpanda.enable_transactions=true
+     - --set redpanda.enable_idempotence=true
+    ports:
+     - 9092:9092
+     - 8081:8081
+     - 8082:8082
+    healthcheck: {test: curl -f localhost:9644/v1/status/ready, interval: 1s, start_period: 30s}
+  sqlserver:
+    # image: mcr.microsoft.com/mssql/server:2019-latest
+    build: ./sqlserver
+    ports:
+     - 1433:1433
+    environment:
+     - ACCEPT_EULA=Y
+     - MSSQL_PID=Standard
+     - SA_PASSWORD=Password!
+     - MSSQL_AGENT_ENABLED=true
+    healthcheck:
+        test: /opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P "$$SA_PASSWORD" -Q "SELECT 1" || exit 1
+        interval: 10s
+        timeout: 3s
+        retries: 10
+        start_period: 10s
+  debezium:
+    build: ./connect
+    container_name: debezium
+    ports:
+     - 8083:8083
+    environment:
+     - BOOTSTRAP_SERVERS=${EXTERNAL_IP:-redpanda}:9092
+     - GROUP_ID=1
+     - CONFIG_STORAGE_TOPIC=my_connect_configs
+     - OFFSET_STORAGE_TOPIC=my_connect_offsets
+     - STATUS_STORAGE_TOPIC=my_connect_statuses
+    healthcheck: {test: curl -f localhost:8083, interval: 1s, start_period: 120s}
+    depends_on:
+      redpanda: {condition: service_healthy}
+      sqlserver: {condition: service_healthy}
+  debezium_deploy:
+    build: ./deploy
+    depends_on:
+      redpanda: {condition: service_healthy}
+      sqlserver: {condition: service_healthy}
+      debezium: {condition: service_healthy}
+    environment:
+      - EXTERNAL_IP=${EXTERNAL_IP:-redpanda}

integrations/debezium/sqlserver/sqlserver/Dockerfile

@@ -0,0 +1,10 @@
+FROM mcr.microsoft.com/mssql/server:2019-latest
+
+# Copy your SQL script into the image
+COPY ./inventory.sql /
+
+# Create an entrypoint script
+COPY ./entrypoint.sh /
+
+# Set the entrypoint script to run on container startup
+ENTRYPOINT ["/entrypoint.sh"]