Hi Team,
I have tomcat application running on two servers and sending logs to logstash through filebeat, when checking logs in discover section of kibana, I am not seeing exact sequence of logs in kibana as in server.
filebeat config -
name: app_server1
filebeat.inputs:
- type: log
fields_under_root: true
fields:
log_type: app_server1
app_id: app
multiline.pattern: ^[[:space:]]+(at|\.{3})\b|^Caused by:|^java|^...|^-
multiline.negate: true
multiline.match: after
paths:
- /opt/apache-tomcat/logs/catalina.out
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
setup.dashboards.enabled: true
setup.kibana:
host: "http://kibanaserver1:5601"
username: elastic
password: ${es_pwd}
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
output.logstash:
hosts: ['logstashserver1:5044', 'logstashserver2:5044']
loadbalance: true
logstash config -
input {
beats {
port => 5044
}
}
filter {
if [log_type] == "app_server1" and [app_id] == "app"
{
mutate { gsub => ["message","\|"," "] } grok { patterns_dir => ["/etc/logstash/patterns"] match => { "message" => "%{MY_DATE_PATTERN:timestamp}%{SPACE}%{LOGLEVEL:level}%{SPACE}%{UUID:ConsentID}%{SPACE}%{WORD:TraceID}%{SPACE}%{WORD:TransactionID}%{SPACE}%{GREEDYDATA:messagetext}" } }
mutate {
replace => {
"[type]" => "app_server1"
}
}
}
if [log_type] == "app_server2" and [app_id] == "app"
{
mutate { gsub => ["message","\|"," "] } grok { patterns_dir => ["/etc/logstash/patterns"] match => { "message" => "%{MY_DATE_PATTERN:timestamp}%{SPACE}%{LOGLEVEL:level}%{SPACE}%{UUID:ConsentID}%{SPACE}%{WORD:TraceID}%{SPACE}%{WORD:TransactionID}%{SPACE}%{GREEDYDATA:messagetext}" } }
mutate {
replace => {
"[type]" => "app_server2"
}
}
}
}
output {
if [log_type] == "app_server1" {
elasticsearch {
hosts => ['http://es1:9200', 'http://es2:9200', 'http://es3:9200']
index => "app"
template_name => "app"
template_overwrite => "false"
user => elastic
password => "${es_pwd}"
}
}
if [log_type] == "app_server2" {
elasticsearch {
hosts => ['http://es1:9200', 'http://es2:9200', 'http://es3:9200']
index => "app"
template_name => "app"
template_overwrite => "false"
user => elastic
password => "${es_pwd}"
}
}
elasticsearch {
hosts => ['http://es1:9200', 'http://es2:9200', 'http://es3:9200']
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM}"
user => elastic
password => "${es_pwd}"
}
}
From server, i can see below last logs on App1 server,
On kibana, the last line is different and not same as above server logs. ( I have filtered with tomcat logs and selected App1 server only)
If i try to search last line of server log (i.e consentStatus: accept) in kibana (did ctrl+F to search on page), i found it somewhere in middle and not at the first line (i.e as latest)
Similarly, if i try to search another log line (i.e subject: 1000000....) which is last 4th line in server log but at kibana it is not last 4th line.
Sorry for all the screen shots and highlighted mark.
Why it's not appearing in sequence and how can we fix this?
Thanks,
