EFK (Elasticsearch+Fluentd-(td-agent)+Kibana): Kibana not showing correct logs
I have installed EFK on redhat linux system(version 7.6). Stable distribution of fluentd, that is td-agent is used instead of fluentd. Path of log file(/mnt/Log/Startup.log) is configured in td-agent.conf file. But on kibana dashboard it displays content from td-agent.log instead of log file(Startup.log).
td-agent.log:
2020-09-04 16:02:16 +0530 [info]: parsing config file is succeeded path="/etc/td-agent/td-agent.conf"
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-elasticsearch' version '4.0.9'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-kafka' version '0.13.0'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-prometheus' version '1.8.0'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-prometheus_pushgateway' version '0.0.2'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.3.0'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-s3' version '1.3.2'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-systemd' version '1.0.2'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-td' version '1.1.0'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.4'
2020-09-04 16:02:16 +0530 [info]: gem 'fluent-plugin-webhdfs' version '1.2.5'
2020-09-04 16:02:16 +0530 [info]: gem 'fluentd' version '1.11.1'
2020-09-04 16:02:16 +0530 [info]: 'flush_interval' is configured at out side of <buffer>. 'flush_mode' is set to 'interval' to keep existing behaviour
2020-09-04 16:02:16 +0530 [debug]: 'host localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: localhost' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [debug]: 'index_name fluentd' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: fluentd' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [debug]: 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [debug]: 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [debug]: 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders, but chunk key 'time' is not configured
2020-09-04 16:02:16 +0530 [debug]: 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [debug]: 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [debug]: 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [debug]: 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [debug]: Need substitution: false
2020-09-04 16:02:16 +0530 [debug]: 'host_placeholder localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: localhost' doesn't have tag placeholder
2020-09-04 16:02:16 +0530 [warn]: define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2020-09-04 16:02:16 +0530 [info]: using configuration file: <ROOT>
<system>
log_level debug
</system>
<source>
@type tail
path "/mnt/Log/Startup.log"
pos_file "/mnt/Log/Startup.log.pos"
format multiline
format_firstline /\d{4}-\d{1,2}-\d{1,2}/
format1 /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}) \[(?<thread>.*)\] (?<level>[^\s]+)(?<message>.*)/
tag "log"
<parse>
format_firstline /\d{4}-\d{1,2}-\d{1,2}/
@type multiline
format1 /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}) \[(?<thread>.*)\] (?<level>[^\s]+)(?<message>.*)/
unmatched_lines
</parse>
</source>
<match *.**>
@type elasticsearch
host "localhost"
port 9200
include_tag_key true
tag_key "@log_name"
logstash_format true
flush_interval 10s
<buffer>
flush_interval 10s
</buffer>
</match>
</ROOT>
2020-09-04 16:02:16 +0530 [info]: starting fluentd-1.11.1 pid=67918 ruby="2.4.10"
2020-09-04 16:02:16 +0530 [info]: spawn command to main: cmdline=["/opt/td-agent/embedded/bin/ruby", "-Eascii-8bit:ascii-8bit", "/opt/td-agent/embedded/bin/fluentd", "--log", "/var/log/td-agent/td-agent.log", "--daemon", "/var/run/td-agent/td-agent.pid", "--under-supervisor"]
2020-09-04 16:02:17 +0530 [info]: adding match pattern="*.**" type="elasticsearch"
2020-09-04 16:02:17 +0530 [info]: #0 'flush_interval' is configured at out side of <buffer>. 'flush_mode' is set to 'interval' to keep existing behaviour
2020-09-04 16:02:17 +0530 [debug]: #0 'host localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: localhost' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [debug]: #0 'index_name fluentd' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: fluentd' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [debug]: #0 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [debug]: #0 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [debug]: #0 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders, but chunk key 'time' is not configured
2020-09-04 16:02:17 +0530 [debug]: #0 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [debug]: #0 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [debug]: #0 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [debug]: #0 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [debug]: #0 Need substitution: false
2020-09-04 16:02:17 +0530 [debug]: #0 'host_placeholder localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: localhost' doesn't have tag placeholder
2020-09-04 16:02:17 +0530 [warn]: #0 Detected ES 7.x: `_doc` will be used as the document `_type`.
2020-09-04 16:02:17 +0530 [info]: adding source type="tail"
2020-09-04 16:02:17 +0530 [warn]: #0 define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2020-09-04 16:02:17 +0530 [info]: #0 starting fluentd worker pid=67935 ppid=67930 worker=0
2020-09-04 16:02:17 +0530 [debug]: #0 buffer started instance=70139276565080 stage_size=0 queue_size=0
2020-09-04 16:02:17 +0530 [debug]: #0 enqueue_thread actually running
2020-09-04 16:02:17 +0530 [debug]: #0 tailing paths: target = /mnt/Log/Startup.log | existing =
2020-09-04 16:02:17 +0530 [info]: #0 following tail of /mnt/Log/Startup.log
2020-09-04 16:02:17 +0530 [info]: #0 fluentd worker is now running worker=0
2020-09-04 16:02:17 +0530 [debug]: #0 flush_thread actually running
Though above log shows that is it tailing the Startup.log. Still it shows content from td-agent.log on kibana dashboard and not the content from configured log file.
td-agent.conf :
<system>
log_level debug
</system>
<source>
@type tail
path /mnt/Log/Startup.log
pos_file /mnt/Log/Startup.log.pos
format multiline
format_firstline /\d{4}-\d{1,2}-\d{1,2}/
format1 /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}) \[(?<thread>.*)\] (?<level>[^\s]+)(?<message>.*)/
tag log
</source>
<match *.**>
@type elasticsearch
host localhost
port 9200
include_tag_key true
tag_key @log_name
logstash_format true
flush_interval 10s
</match>
Version details:
Elasticsearch version 7.8.0
Fluentd (td-agent) version 3.8.0
Kibana version 7.8.0
I have installed all above tools using rpms:
elasticsearch-7.8.1-x86_64.rpm
td-agent-3.8.0-0.el7.x86_64.rpm
kibana-7.8.1-x86_64.rpm
curl -X GET "localhost:9200/?pretty" gives:
{
"name" : "ncnsidapp2",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "MnfFXTyZT0ahp9u4eLSW2A",
"version" : {
"number" : "7.8.1",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "b5ca9c58fb664ca8bf9e4057fc229b3396bf3a89",
"build_date" : "2020-07-21T16:40:44.668009Z",
"build_snapshot" : false,
"lucene_version" : "8.5.1",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
What could be the issue here? Please help.