|
1 | | -# NestJS Microservices with Kafka / Grafana / Kong |
| 1 | +# NestJS Microservices with gRPC / Grafana / Kong (WIP) |
2 | 2 |
|
3 | | -A fully managed microservices starter pack using NestJS and Kafka with SQL databases, Kong API Gateway, and the Grafana logging stack. |
| 3 | +A fully managed microservices starter pack using NestJS and gRPC with SQL databases, Kong API Gateway, and the Grafana logging stack. |
4 | 4 |
|
5 | 5 | For Linux, consider using [Lazydocker](https://github.com/jesseduffield/lazydocker) for Docker container inspection. |
6 | 6 |
|
7 | | ---- |
8 | 7 |
|
9 | | -## Dependencies & Services |
10 | | -- **Kafka** - [Apache Kafka](https://kafka.apache.org) |
11 | | -- **Redis** - [Redis](https://redis.io) |
12 | | -- **Grafana** - [Grafana](https://grafana.com) |
13 | | -- **PostgreSQL** - [PostgreSQL](https://www.postgresql.org) |
14 | | -- **Kong API Gateway** - [Kong](https://konghq.com) |
15 | | -- **Loki** - [Grafana Loki](https://grafana.com/oss/loki) |
16 | | -- **Prisma** - [Prisma ORM](https://www.prisma.io) |
17 | | -- **Fluent-Bit** - [Fluent Bit](https://grafana.com/docs/loki/latest/clients/fluentbit) |
18 | | - |
19 | | ---- |
20 | | - |
21 | | -## Getting Started |
22 | | -### Environment Setup |
23 | | -- Use `git submodule update --init --recursive` to update/fetch submodules. |
24 | | -- Use `.env` for local development and `.env.docker` for Docker Compose environments. |
25 | | -- The Kong API Gateway configuration is in `kong/config.yml`. |
26 | | -- **Kong API Gateway** starts on port `8000`. |
27 | | -- **Health check endpoint**: `host:port/health` |
28 | | -- **Swagger documentation**: `host:port/docs` |
29 | | - |
30 | | -### Running Locally |
31 | | -Start dependency services first (PostgreSQL, Kafka, Redis): |
32 | | - |
33 | | -```bash |
34 | | -docker compose -f docker-compose.local.yml up -d |
35 | | -``` |
36 | | - |
37 | | -Start core services: |
38 | | -```bash |
39 | | -docker compose up |
40 | | -``` |
41 | | - |
42 | | -To stop services: |
43 | | -```bash |
44 | | -docker compose -f docker-compose.local.yml down |
45 | | -docker compose down |
46 | | -``` |
47 | | - |
48 | | -### Running Grafana Stack |
49 | | -To start the Grafana stack locally: |
50 | | -```bash |
51 | | -docker compose -f docker-compose.grafana.yml up -d |
52 | | -``` |
53 | | -To stop the Grafana stack: |
54 | | -```bash |
55 | | -docker compose -f docker-compose.grafana.yml down |
56 | | -``` |
57 | | - |
58 | | -> **Note:** Ensure logging configuration is enabled in `docker-compose.yml` to attach Fluent Bit with service containers. |
59 | | -
|
60 | | ---- |
61 | | - |
62 | | -## Setting up Grafana Dashboard Locally |
63 | | -To view logs in Grafana: |
64 | | - |
65 | | -1. Open **http://localhost:3000** (default credentials: `admin` / `admin`). |
66 | | -2. Go to **http://localhost:3000/datasources** and add **Loki** from the Logging & Document Databases section. |
67 | | -3. Set the **URL** as `http://loki:3100` (or use the host IP if running separately). |
68 | | -4. Click **Save & Test**. |
69 | | -5. Open **Explore** from the sidebar or visit **http://localhost:3000/explore**. |
70 | | -6. Select containers and run queries. |
71 | | - |
72 | | -## Docker Configuration (Services Overview) |
73 | | - |
74 | | -### Kong API Gateway |
75 | | -```yaml |
76 | | -services: |
77 | | - kong: |
78 | | - image: kong:latest |
79 | | - environment: |
80 | | - KONG_DATABASE: "off" |
81 | | - KONG_DECLARATIVE_CONFIG: /usr/local/kong/declarative/kong.yml |
82 | | - KONG_PROXY_LISTEN: 0.0.0.0:8000 |
83 | | - KONG_PROXY_LISTEN_SSL: 0.0.0.0:8443 |
84 | | - KONG_ADMIN_LISTEN: 0.0.0.0:8001 |
85 | | - volumes: |
86 | | - - ./kong/config.yml:/usr/local/kong/declarative/kong.yml |
87 | | - ports: |
88 | | - - "8000:8000" |
89 | | - - "8443:8443" |
90 | | - - "8001:8001" |
91 | | - restart: unless-stopped |
92 | | - networks: |
93 | | - - backendworks |
94 | | - logging: |
95 | | - driver: fluentd |
96 | | - options: |
97 | | - fluentd-address: localhost:24224 |
98 | | - tag: kong-gateway |
99 | | -``` |
100 | | -
|
101 | | -### Auth Service |
102 | | -```yaml |
103 | | - auth-service: |
104 | | - build: |
105 | | - context: ./auth |
106 | | - dockerfile: Dockerfile |
107 | | - environment: |
108 | | - - SERVICE_NAME=auth-service |
109 | | - ports: |
110 | | - - "9001:9001" |
111 | | - networks: |
112 | | - - backendworks |
113 | | - restart: on-failure |
114 | | - logging: |
115 | | - driver: fluentd |
116 | | - options: |
117 | | - fluentd-address: localhost:24224 |
118 | | - tag: auth-service |
119 | | -``` |
120 | | -
|
121 | | -### Post Service |
122 | | -```yaml |
123 | | - post-service: |
124 | | - build: |
125 | | - context: ./post |
126 | | - dockerfile: Dockerfile |
127 | | - environment: |
128 | | - - SERVICE_NAME=post-service |
129 | | - ports: |
130 | | - - "9002:9002" |
131 | | - networks: |
132 | | - - backendworks |
133 | | - restart: on-failure |
134 | | - logging: |
135 | | - driver: fluentd |
136 | | - options: |
137 | | - fluentd-address: localhost:24224 |
138 | | - tag: post-service |
139 | | -``` |
140 | | -
|
141 | | -### PostgreSQL |
142 | | -```yaml |
143 | | - postgres: |
144 | | - image: postgres:latest |
145 | | - restart: unless-stopped |
146 | | - ports: |
147 | | - - "5432:5432" |
148 | | - environment: |
149 | | - POSTGRES_USER: postgres |
150 | | - POSTGRES_PASSWORD: master123 |
151 | | - POSTGRES_DB: postgres |
152 | | - networks: |
153 | | - - backendworks |
154 | | -``` |
155 | | -
|
156 | | -### Kafka |
157 | | -```yaml |
158 | | - kafka: |
159 | | - image: confluentinc/cp-kafka:latest |
160 | | - restart: unless-stopped |
161 | | - ports: |
162 | | - - "9093:9093" |
163 | | - environment: |
164 | | - KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 |
165 | | - KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,EXTERNAL://localhost:9093 |
166 | | - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT |
167 | | - KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true" |
168 | | - networks: |
169 | | - - backendworks |
170 | | -``` |
171 | | -
|
172 | | -### Redis |
173 | | -```yaml |
174 | | - redis: |
175 | | - image: redis:latest |
176 | | - restart: always |
177 | | - ports: |
178 | | - - "6379:6379" |
179 | | - command: redis-server --save 20 1 --loglevel warning |
180 | | - networks: |
181 | | - - backendworks |
182 | | -``` |
183 | | -
|
184 | | -### Fluent-Bit Logging Service |
185 | | -```yaml |
186 | | - fluent-bit: |
187 | | - build: |
188 | | - context: ./fluent-bit |
189 | | - dockerfile: Dockerfile |
190 | | - ports: |
191 | | - - "24224:24224" |
192 | | - environment: |
193 | | - - LOG_LEVEL=debug |
194 | | - - LOKI_URL=http://loki:3100/loki/api/v1/push |
195 | | - restart: unless-stopped |
196 | | - networks: |
197 | | - - backendworks |
198 | | -``` |
199 | | -
|
200 | | ---- |
201 | | -
|
202 | | -## Networks & Volumes |
203 | | -```yaml |
204 | | -networks: |
205 | | - backendworks: |
206 | | - name: backendworks |
207 | | -volumes: |
208 | | - pg_data: |
209 | | - zookeeper_data: |
210 | | - kafka_data: |
211 | | - redis_data: |
212 | | - loki_data: |
213 | | - grafana_data: |
214 | | -``` |
215 | | -
|
216 | | -This documentation provides a clear and structured guide for setting up and managing the NestJS microservices project. Let me know if you need further improvements or additions! |
217 | | -
|
218 | | -## API Access Guide with Kong API Gateway |
219 | | -
|
220 | | -### Overview |
221 | | -This guide explains how to access Swagger documentation and interact with APIs when services are running behind Kong API Gateway. |
222 | | -
|
223 | | -### Prerequisites |
224 | | -- Docker and Docker Compose installed |
225 | | -- Kong API Gateway configured to run on port **8000** |
226 | | -- Services (e.g., `auth-service`, `post-service`) are up and running inside Docker Compose |
227 | | -- Swagger documentation is enabled for each service |
228 | | - |
229 | | -### Accessing Swagger Documentation |
230 | | - |
231 | | -#### 1. Authentication Service |
232 | | -- Open your browser and navigate to: |
233 | | - ``` |
234 | | - http://localhost:8000/auth/docs |
235 | | - ``` |
236 | | -- In the Swagger UI, go to the **top-left server selection dropdown**. |
237 | | -- Select `/auth` as the base path. |
238 | | -- Now, you can explore and test API endpoints for the `auth-service`. |
239 | | - |
240 | | -#### 2. Post Service |
241 | | -- Open your browser and navigate to: |
242 | | - ``` |
243 | | - http://localhost:8000/post/docs |
244 | | - ``` |
245 | | -- In the Swagger UI, go to the **top-left server selection dropdown**. |
246 | | -- Select `/post` as the base path. |
247 | | -- Now, you can explore and test API endpoints for the `post-service`. |
248 | | - |
249 | | -### Running API Requests via Kong |
250 | | -When making API requests via Swagger UI or any API client (e.g., Postman), ensure you use the Kong API Gateway base URL: |
251 | | -- `http://localhost:8000/auth/...` |
252 | | -- `http://localhost:8000/post/...` |
253 | | - |
254 | | -For authentication-protected APIs, add a **Bearer Token** (JWT) in the **Authorization** header. |
255 | | - |
256 | | -### Summary |
257 | | -With Kong API Gateway handling the routing, all services are accessible via port `8000`. Swagger provides an interface to test APIs, ensuring smooth API interactions across services. |
258 | | - |
259 | | -If you encounter any issues, verify that: |
260 | | -- Docker containers are running. |
261 | | -- Kong configuration includes correct routes. |
262 | | -- Swagger is enabled in each service. |
263 | 8 |
|
264 | 9 |
|
0 commit comments