Skip to content

Commit 794c37a

Browse files
hmake98hmake98
committed
feat: updated version of nestjs microservices repo with new feature using kafka broker
1 parent 27d1782 commit 794c37a

28 files changed

+415
-1131
lines changed

‎.gitmodules‎

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,3 @@
44
[submodule "post"]
55
path = post
66
url = https://github.com/BackendWorks/post.git
7-
[submodule "notification"]
8-
path = notification
9-
url = https://github.com/BackendWorks/notification.git
10-
[submodule "files"]
11-
path = files
12-
url = https://github.com/BackendWorks/files.git

‎README.md‎

Lines changed: 223 additions & 99 deletions
Original file line numberDiff line numberDiff line change
@@ -1,140 +1,264 @@
1-
# NestJS Microservices with RabbitMQ / Grafana / Helm Charts
1+
# NestJS Microservices with Kafka / Grafana / Kong
22

3-
Fully managed Microservices starter pack using NestJS / RabbitMQ with SQL and NoSQL databases, Kong API Gateway, Grafana Logging stack, and Helm charts.
3+
A fully managed microservices starter pack using NestJS and Kafka with SQL databases, Kong API Gateway, and the Grafana logging stack.
44

5-
For Linux, consider using this tool for Docker container inspection: Lazydocker.
6-
https://github.com/jesseduffield/lazydocker
5+
For Linux, consider using [Lazydocker](https://github.com/jesseduffield/lazydocker) for Docker container inspection.
76

8-
## Dependencies & Services
9-
10-
- RabbitMQ - https://www.rabbitmq.com
11-
- Redis - https://redis.io
12-
- Grafana - https://grafana.com
13-
- MognoDB - https://www.mongodb.com
14-
- PostgreSQL - https://www.postgresql.org
15-
- Kong - https://konghq.com
16-
- Loki - https://grafana.com/oss/loki
17-
- Prisma - https://www.prisma.io
18-
- Helm - https://helm.sh
19-
- Fluent-Bit - https://grafana.com/docs/loki/latest/clients/fluentbit
20-
21-
## Get started Notes:
22-
23-
- MOST IMPORTANT: for prisma error regarding mongodb transaction [link](https://github.com/prisma/prisma/issues/8266), to workaround exec this in mongodb docker container using `mongosh` and run `rs.initiate({_id: 'rs0', members: [{_id: 0, host: 'localhost:27017'}]});`
24-
- Use `git submodule update --init --recursive` command to update/fetch submodules.
25-
- Use `.env.local` file while working on local environment, use `.env.docker` for docker compose environment and use `.env` for production.
26-
- `kong.yml` from `kong/conf/kong.yml` file is configured for api gateway.
27-
- Kong development server endpoint will start on port `8000`.
28-
- Health endpoint: `host:port/api/health`
29-
- Swagger docs endpoint: `host:port/api/docs`
30-
31-
## Run in local
7+
---
328

33-
Start core services first (Postgres, RabbitMQ, MongoDB, Redis):
9+
## Dependencies & Services
10+
- **Kafka** - [Apache Kafka](https://kafka.apache.org)
11+
- **Redis** - [Redis](https://redis.io)
12+
- **Grafana** - [Grafana](https://grafana.com)
13+
- **PostgreSQL** - [PostgreSQL](https://www.postgresql.org)
14+
- **Kong API Gateway** - [Kong](https://konghq.com)
15+
- **Loki** - [Grafana Loki](https://grafana.com/oss/loki)
16+
- **Prisma** - [Prisma ORM](https://www.prisma.io)
17+
- **Fluent-Bit** - [Fluent Bit](https://grafana.com/docs/loki/latest/clients/fluentbit)
18+
19+
---
20+
21+
## Getting Started
22+
### Environment Setup
23+
- Use `git submodule update --init --recursive` to update/fetch submodules.
24+
- Use `.env` for local development and `.env.docker` for Docker Compose environments.
25+
- The Kong API Gateway configuration is in `kong/config.yml`.
26+
- **Kong API Gateway** starts on port `8000`.
27+
- **Health check endpoint**: `host:port/health`
28+
- **Swagger documentation**: `host:port/docs`
29+
30+
### Running Locally
31+
Start dependency services first (PostgreSQL, Kafka, Redis):
3432

3533
```bash
36-
yarn local:up
34+
docker compose -f docker-compose.local.yml up -d
3735
```
3836

39-
Now go to the service folder:
40-
37+
Start core services:
4138
```bash
42-
yarn dev
39+
docker compose up
4340
```
4441

45-
To stop core services, run:
46-
42+
To stop services:
4743
```bash
48-
yarn local:down
44+
docker compose -f docker-compose.local.yml down
45+
docker compose down
4946
```
5047

51-
Run All Services in Local with Docker Compose
48+
### Running Grafana Stack
49+
To start the Grafana stack locally:
5250
```bash
53-
sh scripts/start.sh
51+
docker compose -f docker-compose.grafana.yml up -d
5452
```
55-
56-
Stop All Services in Local with Docker Compose
53+
To stop the Grafana stack:
5754
```bash
58-
sh scripts/stop.sh
55+
docker compose -f docker-compose.grafana.yml down
5956
```
6057

61-
To run grafana stack in local docker compose
62-
63-
```bash
64-
# up
65-
yarn grafana:up
66-
67-
# down
68-
yarn grafana:down
58+
> **Note:** Ensure logging configuration is enabled in `docker-compose.yml` to attach Fluent Bit with service containers.
59+
60+
---
61+
62+
## Setting up Grafana Dashboard Locally
63+
To view logs in Grafana:
64+
65+
1. Open **http://localhost:3000** (default credentials: `admin` / `admin`).
66+
2. Go to **http://localhost:3000/datasources** and add **Loki** from the Logging & Document Databases section.
67+
3. Set the **URL** as `http://loki:3100` (or use the host IP if running separately).
68+
4. Click **Save & Test**.
69+
5. Open **Explore** from the sidebar or visit **http://localhost:3000/explore**.
70+
6. Select containers and run queries.
71+
72+
## Docker Configuration (Services Overview)
73+
74+
### Kong API Gateway
75+
```yaml
76+
services:
77+
kong:
78+
image: kong:latest
79+
environment:
80+
KONG_DATABASE: "off"
81+
KONG_DECLARATIVE_CONFIG: /usr/local/kong/declarative/kong.yml
82+
KONG_PROXY_LISTEN: 0.0.0.0:8000
83+
KONG_PROXY_LISTEN_SSL: 0.0.0.0:8443
84+
KONG_ADMIN_LISTEN: 0.0.0.0:8001
85+
volumes:
86+
- ./kong/config.yml:/usr/local/kong/declarative/kong.yml
87+
ports:
88+
- "8000:8000"
89+
- "8443:8443"
90+
- "8001:8001"
91+
restart: unless-stopped
92+
networks:
93+
- backendworks
94+
logging:
95+
driver: fluentd
96+
options:
97+
fluentd-address: localhost:24224
98+
tag: kong-gateway
6999
```
70100
71-
Note: To attach Fluent Bit container with the service container, ensure logging configuration is enabled in docker-compose.yml file.
72-
73-
## Setup Grafana Dashboard in Local Environment
74-
75-
To see logs on the Grafana dashboard, follow these steps:
76-
77-
1. Open the browser and go to http://localhost:3000. Use default credentials: username "admin" and password "admin".
78-
79-
2. Go to http://localhost:3000/datasources and select Loki from the Logging and Document Databases section.
101+
### Auth Service
102+
```yaml
103+
auth-service:
104+
build:
105+
context: ./auth
106+
dockerfile: Dockerfile
107+
environment:
108+
- SERVICE_NAME=auth-service
109+
ports:
110+
- "9001:9001"
111+
networks:
112+
- backendworks
113+
restart: on-failure
114+
logging:
115+
driver: fluentd
116+
options:
117+
fluentd-address: localhost:24224
118+
tag: auth-service
119+
```
80120
81-
3. Enter http://loki:3100 in the URL under the HTTP section. This works because Loki and Grafana are running in the same network. Otherwise, use the host IP address and port.
121+
### Post Service
122+
```yaml
123+
post-service:
124+
build:
125+
context: ./post
126+
dockerfile: Dockerfile
127+
environment:
128+
- SERVICE_NAME=post-service
129+
ports:
130+
- "9002:9002"
131+
networks:
132+
- backendworks
133+
restart: on-failure
134+
logging:
135+
driver: fluentd
136+
options:
137+
fluentd-address: localhost:24224
138+
tag: post-service
139+
```
82140
83-
4. Click the "Save and Test" button at the bottom of the page.
141+
### PostgreSQL
142+
```yaml
143+
postgres:
144+
image: postgres:latest
145+
restart: unless-stopped
146+
ports:
147+
- "5432:5432"
148+
environment:
149+
POSTGRES_USER: postgres
150+
POSTGRES_PASSWORD: master123
151+
POSTGRES_DB: postgres
152+
networks:
153+
- backendworks
154+
```
84155
85-
5. Go to the 3rd tab "Explore" from the left sidebar or http://localhost:3000/explore.
156+
### Kafka
157+
```yaml
158+
kafka:
159+
image: confluentinc/cp-kafka:latest
160+
restart: unless-stopped
161+
ports:
162+
- "9093:9093"
163+
environment:
164+
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
165+
KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,EXTERNAL://localhost:9093
166+
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
167+
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
168+
networks:
169+
- backendworks
170+
```
86171
87-
6. Select containers and run the query. You will see a view similar to this:
88-
![image](https://user-images.githubusercontent.com/23061515/217284063-5a548f77-ac0c-42b3-bfdb-963a62f8788a.png)
172+
### Redis
173+
```yaml
174+
redis:
175+
image: redis:latest
176+
restart: always
177+
ports:
178+
- "6379:6379"
179+
command: redis-server --save 20 1 --loglevel warning
180+
networks:
181+
- backendworks
182+
```
89183
90-
## Deployment in ECR
184+
### Fluent-Bit Logging Service
185+
```yaml
186+
fluent-bit:
187+
build:
188+
context: ./fluent-bit
189+
dockerfile: Dockerfile
190+
ports:
191+
- "24224:24224"
192+
environment:
193+
- LOG_LEVEL=debug
194+
- LOKI_URL=http://loki:3100/loki/api/v1/push
195+
restart: unless-stopped
196+
networks:
197+
- backendworks
198+
```
91199
92-
```bash
93-
sh scripts/deploy.sh
200+
---
201+
202+
## Networks & Volumes
203+
```yaml
204+
networks:
205+
backendworks:
206+
name: backendworks
207+
volumes:
208+
pg_data:
209+
zookeeper_data:
210+
kafka_data:
211+
redis_data:
212+
loki_data:
213+
grafana_data:
94214
```
95215
96-
Notes:
216+
This documentation provides a clear and structured guide for setting up and managing the NestJS microservices project. Let me know if you need further improvements or additions!
97217
98-
- Change "aws_region" and "aws_account_id" as per your AWS account.
218+
## API Access Guide with Kong API Gateway
99219
100-
## K8s Deployment with Helm charts:
220+
### Overview
221+
This guide explains how to access Swagger documentation and interact with APIs when services are running behind Kong API Gateway.
101222
102-
Note: I recommend the tool [k9s](https://k9scli.io/) for better understanding of a cluster. To start with Helm deployment on a local machine, ensure that Minikube is started. Service images should be available in your local Docker environment, and core services should be up and running in the local environment.
223+
### Prerequisites
224+
- Docker and Docker Compose installed
225+
- Kong API Gateway configured to run on port **8000**
226+
- Services (e.g., `auth-service`, `post-service`) are up and running inside Docker Compose
227+
- Swagger documentation is enabled for each service
103228

104-
If you're unable to use local images in Minikube cluster, refer to this
105-
[blog](https://medium.com/bb-tutorials-and-thoughts/how-to-use-own-local-doker-images-with-minikube-2c1ed0b0968)
229+
### Accessing Swagger Documentation
106230

107-
```bash
108-
minikube start
109-
```
231+
#### 1. Authentication Service
232+
- Open your browser and navigate to:
233+
```
234+
http://localhost:8000/auth/docs
235+
```
236+
- In the Swagger UI, go to the **top-left server selection dropdown**.
237+
- Select `/auth` as the base path.
238+
- Now, you can explore and test API endpoints for the `auth-service`.
110239

111-
Now, install helm charts
240+
#### 2. Post Service
241+
- Open your browser and navigate to:
242+
```
243+
http://localhost:8000/post/docs
244+
```
245+
- In the Swagger UI, go to the **top-left server selection dropdown**.
246+
- Select `/post` as the base path.
247+
- Now, you can explore and test API endpoints for the `post-service`.
112248

113-
```bash
114-
helm install backendworks helm
115-
```
249+
### Running API Requests via Kong
250+
When making API requests via Swagger UI or any API client (e.g., Postman), ensure you use the Kong API Gateway base URL:
251+
- `http://localhost:8000/auth/...`
252+
- `http://localhost:8000/post/...`
116253

117-
To get services endpoints:
118-
```bash
119-
minikube service list
120-
```
121-
Check Loki stack installed in the current namespace:
254+
For authentication-protected APIs, add a **Bearer Token** (JWT) in the **Authorization** header.
122255

123-
```bash
124-
helm list
125-
```
126-
Get Loki IP from the K8s cluster:
127-
```
128-
kubectl get service backendworks-loki
129-
```
130-
Use this ClusterIP to add the datasource Loki in the Grafana dashboard.
256+
### Summary
257+
With Kong API Gateway handling the routing, all services are accessible via port `8000`. Swagger provides an interface to test APIs, ensuring smooth API interactions across services.
131258

132-
```
133-
minikube service list
134-
```
259+
If you encounter any issues, verify that:
260+
- Docker containers are running.
261+
- Kong configuration includes correct routes.
262+
- Swagger is enabled in each service.
135263

136-
Clean up the local environment:
137264

138-
```bash
139-
helm uninstall backendworks
140-
```

‎auth‎

Submodule auth updated 96 files

0 commit comments

Comments
 (0)