Skip to content
Merged
Changes from 1 commit
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
acf1cdc
Initial setup of Observability OTEL demo using Opensearch as store
YANG-DB Apr 26, 2023
c842150
Initial setup of Observability OTEL demo using Opensearch as store
YANG-DB Apr 26, 2023
b83ea99
Merge remote-tracking branch 'origin/opensearch-observability-otel' i…
YANG-DB Apr 26, 2023
9850688
Initial setup of Observability OTEL demo using Opensearch as store
YANG-DB May 1, 2023
0db3223
Merge branch 'main' into opensearch-observability-otel
YANG-DB May 2, 2023
bf41276
add curl to the basic docker O/S
YANG-DB May 3, 2023
4292136
Merge pull request #1 from open-telemetry/main
YANG-DB May 3, 2023
f0600be
add monitoring & opinionated assets for this demo
YANG-DB May 4, 2023
418d196
Merge remote-tracking branch 'origin/opensearch-observability-otel' i…
YANG-DB May 4, 2023
b0db9fc
export F/E & Load-Generator port services
YANG-DB May 8, 2023
9629fb7
Merge pull request #2 from open-telemetry/main
YANG-DB May 8, 2023
ba709f1
fix jaeger service name
YANG-DB May 9, 2023
daedfab
adding nginx as frontend proxy, fluent-bit.conf as log emitter
YANG-DB May 10, 2023
d58ed46
adding nginx & fluent-bit docs
YANG-DB May 10, 2023
a8e3f5d
Merge pull request #3 from open-telemetry/main
YANG-DB May 15, 2023
b6a018a
adding integrationbs docker-compose service for assets loading into o…
YANG-DB May 15, 2023
6031c17
Merge remote-tracking branch 'origin/opensearch-observability-otel' i…
YANG-DB May 15, 2023
92240ba
adding datasource for query prometheus
YANG-DB May 15, 2023
12d52cb
adding architecture.md including images
YANG-DB May 15, 2023
85fefce
add images and service links
YANG-DB May 16, 2023
73e2680
add jaeger-agent container name
YANG-DB May 16, 2023
f6067c6
add data-prepper support for traces ingestion & serviceMap creation
YANG-DB May 16, 2023
24d278c
fix auth issues for data-prepper
YANG-DB May 16, 2023
7410478
add documentation for different ingestion capabilities
YANG-DB May 16, 2023
d8a2abf
Merge pull request #4 from YANG-DB/otel-opensearch-data-prep
YANG-DB May 16, 2023
fe655cc
Initial setup of Observability OTEL demo using Opensearch as store
YANG-DB Apr 26, 2023
0be11cd
Initial setup of Observability OTEL demo using Opensearch as store
YANG-DB May 1, 2023
f54a011
add curl to the basic docker O/S
YANG-DB May 3, 2023
90c9a61
add monitoring & opinionated assets for this demo
YANG-DB May 4, 2023
379372a
[shippingservice] update rust version and dependencies (#865)
julianocosta89 May 3, 2023
6ae5c80
[load generator] Bump loagen dependencies (#869)
julianocosta89 May 3, 2023
a69595a
[grafana] fix demo dashboard to be compatible with spanmetrics connec…
fatsheep9146 May 3, 2023
1c28220
export F/E & Load-Generator port services
YANG-DB May 8, 2023
42537be
[kafka] remove KRaft mode support workarounds (#880)
styblope May 8, 2023
8bd25ad
enabling batch span processor metrics for quoteservice (#878)
brettmc May 8, 2023
2cc7484
Generate random errors in cartservice (#824)
alanwest May 8, 2023
54e3b6f
fix jaeger service name
YANG-DB May 9, 2023
daf0d08
adding nginx as frontend proxy, fluent-bit.conf as log emitter
YANG-DB May 10, 2023
88e0171
adding nginx & fluent-bit docs
YANG-DB May 10, 2023
d292741
adding integrationbs docker-compose service for assets loading into o…
YANG-DB May 15, 2023
6f55bd8
[currencyservice] Fix OTel C++ build and update OTel version (#886)
julianocosta89 May 12, 2023
193552f
adding datasource for query prometheus
YANG-DB May 15, 2023
febdfe1
adding architecture.md including images
YANG-DB May 15, 2023
904ed80
add images and service links
YANG-DB May 16, 2023
9cf4326
add jaeger-agent container name
YANG-DB May 16, 2023
c14aa11
add data-prepper support for traces ingestion & serviceMap creation
YANG-DB May 16, 2023
a55d74f
fix auth issues for data-prepper
YANG-DB May 16, 2023
6032e3e
add documentation for different ingestion capabilities
YANG-DB May 16, 2023
5954bdf
Merge remote-tracking branch 'origin/opensearch-observability-otel' i…
YANG-DB May 18, 2023
4d1d72e
add assets manager flast app for loading the integration assets
YANG-DB May 18, 2023
91f7a0e
remove experimental assets mgr app
YANG-DB May 18, 2023
78b38e6
Opensearch otel demo observability tutorial (#5)
YANG-DB May 23, 2023
fe8a551
Merge pull request #2 from open-telemetry/main
YANG-DB May 26, 2023
b422bc7
Update local-config.md
YANG-DB May 26, 2023
4431d7b
Update README.md
YANG-DB May 26, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Update README.md
update fluent-bit README.md file reference to the config

Signed-off-by: YANGDB <[email protected]>
  • Loading branch information
YANG-DB authored May 26, 2023
commit 4431d7b41b4b88021d80596c21f17fc1a9d4889c
31 changes: 2 additions & 29 deletions src/fluent-bit/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,35 +19,8 @@ For setting up a fluent-bit agent on Nginx, please follow the next instructions

- Install Fluent-bit on the Nginx server. You can download the latest package from the official Fluent-bit website or use your package manager to install it.

- Once Fluent-bit is installed, create a configuration file named fluent-bit.conf in the /etc/fluent-bit/ directory. Add the following configuration to the file:
- Once Fluent-bit is installed, create a configuration file named [fluent-bit.conf](fluent-bit.conf) in the /etc/fluent-bit/ directory. Add the following configuration to the file:

```text
[SERVICE]
Flush 1
Log_Level info
Parsers_File parsers.conf

[Filter]
Name lua
Match *
code function cb_filter(a,b,c)local d={}local e=os.date("!%Y-%m-%dT%H:%M:%S.000Z")d["observerTime"]=e;d["body"]=c.remote.." "..c.host.." "..c.user.." ["..os.date("%d/%b/%Y:%H:%M:%S %z").."] \""..c.method.." "..c.path.." HTTP/1.1\" "..c.code.." "..c.size.." \""..c.referer.."\" \""..c.agent.."\""d["trace_id"]="102981ABCD2901"d["span_id"]="abcdef1010"d["attributes"]={}d["attributes"]["data_stream"]={}d["attributes"]["data_stream"]["dataset"]="nginx.access"d["attributes"]["data_stream"]["namespace"]="production"d["attributes"]["data_stream"]["type"]="logs"d["event"]={}d["event"]["category"]={"web"}d["event"]["name"]="access"d["event"]["domain"]="nginx.access"d["event"]["kind"]="event"d["event"]["result"]="success"d["event"]["type"]={"access"}d["http"]={}d["http"]["request"]={}d["http"]["request"]["method"]=c.method;d["http"]["response"]={}d["http"]["response"]["bytes"]=tonumber(c.size)d["http"]["response"]["status_code"]=c.code;d["http"]["flavor"]="1.1"d["http"]["url"]=c.path;d["communication"]={}d["communication"]["source"]={}d["communication"]["source"]["address"]="127.0.0.1"d["communication"]["source"]["ip"]=c.remote;return 1,b,d end
call cb_filter

[INPUT]
Name tail
Path /var/log/nginx/access.log
Tag nginx.access
DB /var/log/flb_input.access.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On

[OUTPUT]
Name opensearch
Match nginx.*
Host <OSS_HOST>
Port <OSS_PORT>
Index sso_nginx-access-%Y.%m.%d
```
Here, we specify the input plugin as tail, set the path to the Nginx access log file, and specify a tag to identify the logs in Fluent-bit. We also set some additional parameters such as memory buffer limit and skipping long lines.

For the output, we use the `opensearch` plugin to send the logs to Opensearch. We specify the Opensearch host, port, and index name.
Expand All @@ -62,4 +35,4 @@ sudo systemctl start fluent-bit
- Verify that Fluent-bit is running by checking its status:
```text
sudo systemctl status fluent-bit
```
```