Cloud-Native Observability & Telegram Alerting

An enterprise-grade Observability-as-Code repository demonstrating how to instrument, collect, visualize, and alert on application metrics.

This repository sets up a containerized stack featuring a Node.js Express API instrumented with custom metrics, Prometheus for metrics storage, Grafana for pre-provisioned dashboards, and Alertmanager integrated with a Telegram Bot to deliver ChatOps alert notifications directly to your phone.

---

🏗️ Observability Architecture

graph TD
    subgraph Target Application
        App[Node.js API] -->|Exposes /metrics| Metrics[prom-client Registry]
    end
    
    subgraph Observability Core
        Prom[Prometheus Server] -->|1. Scrapes metrics| App
        Prom -->|2. Evaluates thresholds| Rules[alert.rules.yml]
        Rules -->|3. Fires alerts| AM[Alertmanager]
    end
    
    subgraph Visualization & Alerting
        Grafana[Grafana Dashboard] -->|Queries| Prom
        AM -->|4. Formats & dispatches| TeleBot[Telegram Bot API]
        TeleBot -->|5. Instant Notification| User[Telegram Chat / Phone]
    end

Loading

---

🤖 Telegram Bot & ChatOps Setup Guide

To receive alerts on your phone, you need to create a free Telegram Bot and fetch your Chat ID:

1. Create a Telegram Bot

1. Open Telegram and search for the official @BotFather.
2. Send the command /newbot.
3. Follow the prompts to assign a name and username (e.g. arshad_alerts_bot).
4. Save the returned API Token (formatted like 123456789:ABCdefGhIJKLMNOPqrst...). This is your bot_token.

2. Get Your Telegram Chat ID

1. Search for @userinfobot on Telegram.
2. Send a message to it, and it will immediately return your numeric Id (e.g., 987654321). This is your chat_id.
3. Alternatively, if you want notifications in a group chat, add both your bot and @userinfobot to the group. The bot will print the group's ID (which starts with a negative sign, e.g. -987654321).

3. Update Alertmanager Configuration

Open alertmanager.yml and replace the placeholders:

receivers:
  - name: 'telegram-alerts-receiver'
    telegram_configs:
      - bot_token: 'YOUR_TELEGRAM_BOT_TOKEN_PLACEHOLDER'
        chat_id: YOUR_TELEGRAM_CHAT_ID_PLACEHOLDER

---

📂 Repository Structure

cloud-observability-alerting/
├── .github/
│   └── workflows/
│       └── observability-ci.yml    # CI Pipeline running Promtool and Amtool syntax checks
├── alertmanager/
│   └── alertmanager.yml            # Alert routing & Telegram layout configurations
├── app/
│   ├── Dockerfile                  # Application build specifications
│   ├── package.json
│   └── server.js                   # Node API instrumented with prom-client metrics
├── grafana/
│   └── provisioning/               # Automated dashboards-as-code configurations
│       ├── dashboards/
│       │   ├── dashboards.yaml
│       │   └── node-app-dashboard.json # Pre-built dashboard panels
│       └── datasources/
│           └── datasources.yaml    # Pre-wired Prometheus datasource
├── prometheus/
│   ├── prometheus.yml              # Main scraper intervals and jobs definition
│   └── alert.rules.yml             # PromQL rules for high error rate, latency, and down state
├── docker-compose.yml              # Multi-container orchestration config
├── run-observability.bat           # Utility script to boot Node API locally
└── README.md                       # Comprehensive guide

---

📈 Instrumented Metrics

The Node.js API collects the following metrics exposed on http://localhost:3000/metrics:

• Default System Metrics: CPU load, memory utilization, event loop lag, active handles, and garbage collection statistics.
• http_requests_total (Counter): Tracks requests completed, grouped by HTTP method, route, and status_code.
• http_request_duration_seconds (Histogram): Records execution durations across configurable duration buckets to track performance percentiles.

---

🚀 Running and Testing Locally

1. Spin Up the Stack

Ensure Docker Desktop is running, and launch the environment from the repository root:

docker compose up -d

2. Access the Consoles

• Instrumented Node App: http://localhost:3000 (Visit /metrics to view raw metrics).
• Prometheus Dashboard: http://localhost:9090 (Check the Alerts tab to see active rules).
• Alertmanager UI: http://localhost:9093 (Inspect active alert silences and groups).
• Grafana Panel: http://localhost:3001 (Log in with username admin and password admin. Go to Dashboards -> Observability to view the pre-built Node.js dashboard).

3. Trigger Alerts (Simulate Traffic)

• HTTP 5xx Spikes: Visit http://localhost:3000/error 10-15 times to trigger a 500 error rate spike.
• Latency Spikes: Visit http://localhost:3000/slow 10-15 times to trigger latency alerts.
• Check Alerts: Go back to Prometheus alerts. The rules will switch from Inactive to Pending, then Firing, dispatching a Telegram markdown message directly to your phone.
• Reconciliation: Stop the app service (docker compose stop app). Prometheus will detect the target is missing, fire the InstanceDown alert, and Alertmanager will notify your Telegram.

---

🔍 DevSecOps Config Auditing

The CI workflow uses official Prometheus binaries to audit configuration files:

1. Promtool Config Check: Runs promtool check config to parse YAML structures and detect invalid Prometheus configurations.
2. Promtool Rules Check: Runs promtool check rules to validate PromQL syntax and alerting boundaries.
3. Amtool Validation: Runs amtool check-config to verify Alertmanager routes and receiver integrations.