adding nginx & fluent-bit docs

Signed-off-by: YANGDB <[email protected]>

src/fluent-bit/README.md

@@ -0,0 +1,65 @@
+![](fluentbit.png)
+
+## Fluent-bit 
+
+Fluent-bit is a lightweight and flexible data collector and forwarder, designed to handle a large volume of log data in real-time.
+It is an open-source projectpart of the Cloud Native Computing Foundation (CNCF). and has gained popularity among developers for simplicity and ease of use.
+
+Fluent-bit is designed to be lightweight, which means that it has a small footprint and can be installed on resource-constrained environments like embedded systems or containers. It is written in C language, making it fast and efficient, and it has a low memory footprint, which allows it to consume minimal system resources.
+
+Fluent-bit is a versatile tool that can collect data from various sources, including files, standard input, syslog, and TCP/UDP sockets. It also supports parsing different log formats like JSON, Apache, and Syslog. Fluent-bit provides a flexible configuration system that allows users to tailor their log collection needs, which makes it easy to adapt to different use cases.
+
+One of the main advantages of Fluent-bit is its ability to forward log data to various destinations, including Opensearch, InfluxDB, and Kafka. Fluent-bit provides multiple output plugins that allow users to route their log data to different destinations based on their requirements. This feature makes Fluent-bit ideal for distributed systems where log data needs to be collected and centralized in a central repository.
+
+Fluent-bit also provides a powerful filtering mechanism that allows users to manipulate log data in real-time. It supports various filter plugins, including record modifiers, parsers, and field extraction. With these filters, users can parse and enrich log data, extract fields, and modify records before sending them to their destination.
+
+## Setting Up Fluent-bit agent 
+
+For setting up a fluent-bit agent on Nginx, please follow the next instructions
+
+- Install Fluent-bit on the Nginx server. You can download the latest package from the official Fluent-bit website or use your package manager to install it.
+
+- Once Fluent-bit is installed, create a configuration file named fluent-bit.conf in the /etc/fluent-bit/ directory. Add the following configuration to the file:
+
+```text
+[SERVICE]
+    Flush        1
+    Log_Level    info
+    Parsers_File parsers.conf
+
+[Filter]
+    Name    lua
+    Match   *
+    code    function cb_filter(a,b,c)local d={}local e=os.date("!%Y-%m-%dT%H:%M:%S.000Z")d["observerTime"]=e;d["body"]=c.remote.." "..c.host.." "..c.user.." ["..os.date("%d/%b/%Y:%H:%M:%S %z").."] \""..c.method.." "..c.path.." HTTP/1.1\" "..c.code.." "..c.size.." \""..c.referer.."\" \""..c.agent.."\""d["trace_id"]="102981ABCD2901"d["span_id"]="abcdef1010"d["attributes"]={}d["attributes"]["data_stream"]={}d["attributes"]["data_stream"]["dataset"]="nginx.access"d["attributes"]["data_stream"]["namespace"]="production"d["attributes"]["data_stream"]["type"]="logs"d["event"]={}d["event"]["category"]={"web"}d["event"]["name"]="access"d["event"]["domain"]="nginx.access"d["event"]["kind"]="event"d["event"]["result"]="success"d["event"]["type"]={"access"}d["http"]={}d["http"]["request"]={}d["http"]["request"]["method"]=c.method;d["http"]["response"]={}d["http"]["response"]["bytes"]=tonumber(c.size)d["http"]["response"]["status_code"]=c.code;d["http"]["flavor"]="1.1"d["http"]["url"]=c.path;d["communication"]={}d["communication"]["source"]={}d["communication"]["source"]["address"]="127.0.0.1"d["communication"]["source"]["ip"]=c.remote;return 1,b,d end
+    call    cb_filter
+
+[INPUT]
+    Name              tail
+    Path              /var/log/nginx/access.log
+    Tag               nginx.access
+    DB                /var/log/flb_input.access.db
+    Mem_Buf_Limit     5MB
+    Skip_Long_Lines   On
+
+[OUTPUT]
+    Name        opensearch
+    Match       nginx.*
+    Host        <OSS_HOST>
+    Port        <OSS_PORT>
+    Index       sso_nginx-access-%Y.%m.%d
+```
+Here, we specify the input plugin as tail, set the path to the Nginx access log file, and specify a tag to identify the logs in Fluent-bit. We also set some additional parameters such as memory buffer limit and skipping long lines.
+
+For the output, we use the `opensearch` plugin to send the logs to Opensearch. We specify the Opensearch host, port, and index name.
+
+   - Modify the Opensearch host and port in the configuration file to match your Opensearch installation.
+   - Depending on the system where Fluent Bit is installed:
+     - Start the Fluent-bit service by running the following command:
+
+```text
+sudo systemctl start fluent-bit
+```
+- Verify that Fluent-bit is running by checking its status:
+```text
+sudo systemctl status fluent-bit
+```

src/nginx-otel/README.md

@@ -0,0 +1,33 @@
+![](nginx.png)
+
+# Nginx Integrations 
+
+## What it Nginx ?
+Nginx is a popular open-source web server software used by millions of websites worldwide. It was developed to address the limitations of Apache, which is another popular web server software. Nginx is known for its high performance, scalability, and reliability, and is widely used as a reverse proxy server, load balancer, and HTTP cache.
+
+One of the primary advantages of Nginx is its ability to handle large numbers of concurrent connections and requests. It uses an event-driven architecture that allows it to handle multiple connections with minimal resources, making it an ideal choice for high-traffic websites. In addition, Nginx can also serve static content very efficiently, which further improves its performance.
+
+Another important feature of Nginx is its ability to act as a reverse proxy server. This means that it can sit in front of web servers and route incoming requests to the appropriate server based on various criteria, such as the URL or the type of request. Reverse proxying can help improve website performance and security by caching static content, load balancing incoming traffic, and providing an additional layer of protection against attacks.
+
+Nginx is also widely used as a load balancer. In this role, it distributes incoming traffic across multiple web servers to improve performance and ensure high availability. Nginx can balance traffic using a variety of algorithms, such as round-robin or least connections, and can also perform health checks to ensure that requests are only sent to healthy servers.
+
+Finally, Nginx is also an effective HTTP cache. By caching frequently accessed content, Nginx can reduce the load on backend servers and improve website performance. Nginx can cache content based on a variety of criteria, such as the URL, response headers, or response body.
+
+## What is An Nginx Integration ?
+As described in the [documentation](../../README.md) Nginx integrations is a bundle of resources, assets and documentations. 
+
+An Integration may have multiple ways of ingesting Observability signals, for example nginx logs may arrive via fluent-bit agent or OTEL-logs collector...
+
+## Which are the Nginx Observability providers ?
+Observability Providers are agents which can collect nginx logs, metrics and traces information, convert them to `sso` observability schema and send them to opensearch observability data-streams.
+
+### Fluent-Bit
+Fluent-bit has a dedicated input plugin for Nginx called `in_tail` which can be used to tail the Nginx access logs and send them to a destination of your choice.
+The in_tail plugin reads log files line by line and sends them to Fluent-bit engine to be processed.
+
+See additional details [here](fluet-bit/README.md).
+
+### Dashboards
+The following dashboard preview shows the summarized information collected from the access log index
+
+![nginx-dashboard-preview.png](preview%2Fnginx-dashboard-preview.png)

src/nginx-otel/default.conf

@@ -2,7 +2,7 @@ server {
   listen 90;
 
   location / {
-     proxy_pass http://127.0.0.1:8080/;
+     proxy_pass http://frontend:8080/;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;