I have a setup with Elastic Search, Log Stash, Kibana and Filebeat with versions as shown below.
- Elastic Search: 6.2.4
- LogStash: 6.2.4
- kibana: 6.2.4
My Logstash pipeline is as below:
sudo vim /etc/logstash/conf.d/02-beats-input.conf
input {
beats {
port => "5044"
ssl => true
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
filter {
#If log line contains tab character followed by 'at' then we will tag that entry as stacktrace
if [message] =~ "\tat" {
grok {
match => ["message", "^(\tat)"]
add_tag => ["stacktrace"]
}
}
#Grokking Spring Boot's default log format
grok {
match => { "message" => "(?%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}) %{LOGLEVEL:level} %{NUMBER:pid} --- [(?[A-Za-z0-9-]+)] [A-Za-z0-9.].(?[A-Za-z0-9#_]+)\s:\s+(?.*)"}
}
#grok {
match => { "message" => "(?%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}) %{LOGLEVEL:level} %{NUMBER:pid} --- .+? :\s+(?.*)"}
#}
grok {
match => { "source" => "/var/log/containers/%{DATA:pod_name}%{DATA:namespace}%{GREEDYDATA:container_name}-%{DATA:container_id}.log" }
remove_field => ["source"]
}
sudo vim /etc/logstash/conf.d/30-elasticsearch-output.conf
output {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
Elastic search, Kibana, Logstash are running in one VM where as application and Filebeat are running on a different VM.
Currently there is a discrepancy with date between these two VMs.
The latest log on kibana discover is as below with a specific time stamp.
message:
{"log":"TACACS+: No port assigned for host, "XX.XX.XX.XX". Using default port 49 instead.\n","stream":"stdout","time":"2018-05-17T00:58:09.401752809Z"}
@timestamp:
May 16th 2018, 17:58:09.408
The latest log at the application as below with a specific time stamp.
{"log":"TACACS+: No port assigned for host, "XX.XX.XX.XX". Using default port 49 instead.\n","stream":"stdout","time":"2018-05-17T06:06:44.365607578Z"}
If you see the above two logs, it is clear that kibana is showing up the logs with some delay, particularly in the above case it has delay of around 5 hours. I also see that the delay is keep getting incremented. I see all the logs on the kibana though. Issue is the delay i am seeing.
Could some one help me understand this behavior ? Is this because the time discrepancy between the two VMs ? both are at PDT time zone. The logs size should be small enough and I dont expect any throttling kicks in.
Please let me know if you need any other details on this.