Skip to content

Log collection through Filebeat daemonset #88

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 13 commits into from
Closed

Log collection through Filebeat daemonset #88

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
3a0b097
Copies the result of the original filebeat branch ...
solsson Nov 4, 2017
760cff5
filebeat 6.0.0-rc2
solsson Nov 4, 2017
fcc184a
Fixes pod start in GKE, read-only file system
solsson Nov 4, 2017
7f0e564
Adds test (without asserts) for filebeat topic, jq-compatible log
solsson Nov 4, 2017
a401bda
Tentative RBAC; kubectl describe's error messages are helpful
solsson Nov 4, 2017
f0b7fde
ClusterRoleBinding fixes RBAC for filebeat
solsson Nov 4, 2017
52c28e8
Fixes JSON syntax getting broken by "Reached end of topic"
solsson Nov 4, 2017
504911a
Fixes kubernetes metadata inclusion
solsson Nov 4, 2017
8878b5c
Removes kubernetes debug as the result is ...
solsson Nov 4, 2017
e04a403
Dropped the logs- prefix now in the logs-kafka namespace
solsson Nov 4, 2017
d61b341
Memory use little higher than fluent-bit and tail+kafkacat
solsson Nov 4, 2017
a83c6eb
Got OOMKilled
solsson Nov 4, 2017
65d902f
Had lots of restarts still. Memory is now 10x that of tail+kafkacat.
solsson Nov 7, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fixes JSON syntax getting broken by "Reached end of topic"
  • Loading branch information
@solsson
solsson committed Nov 4, 2017
commit 52c28e8057a28270c3197766654ea46a9225976c
1 change: 1 addition & 0 deletions logs-streaming/filebeat-config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ data:
enabled: true
paths:
- /var/log/containers/*.log
# Logs say a lot of https://github.com/elastic/beats/blob/v6.0.0-rc2/filebeat/prospector/log/prospector.go#L250
symlinks: true
# reduce the risk for aggregation recursion: match pod names to exclude own's logs + test logs
exclude_files: ['^(.*\/)?logs-','^(.*\/)?test-']
Expand Down
2 changes: 1 addition & 1 deletion logs-streaming/test/logs-filebeat.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ data:

echo ""
echo "{\"#---\":\"$(date -u --iso-8601='ns')\"}"
kafkacat -b $BOOTSTRAP -C -t $TOPIC -f '{"#topic":"%t","#partition":%p,"#offset":%o,"#key":"%k","=":%s}\n' -o -10 -e
kafkacat -b $BOOTSTRAP -C -t $TOPIC -f '{"#topic":"%t","#partition":%p,"#offset":%o,"#key":"%k","=":%s}\n' -o -10 -e -q

exit 0

Expand Down