Hi all I need some help.
I set my first elk this morning . I used this tutorial. https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-centos-7
At the end I want my websphere logs to go to logstash but first syslog has to work fine.
Now logstash is receiving input but I have few problems.
First the known problem with changing logstash timestamp with real timestamp from log file. I googled a lot but cant figure this out.
This is my syslog filter
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
Log sample from client
Mar 23 11:12:00 hostname systemd: Starting Session 2 of user root
Locale the same on both
Second problem is that log order is messed up.
This is kibana view.
Mar 23 13:01:01 hostname systemd: Started Session 5 of user root.
Mar 23 13:01:01 hostname systemd: Starting Session 5 of user root.
Mar 23 12:15:12 hostname journal: root[/root] 56092 ipaddress 22: tail -f /var/log/messages
Mar 23 12:15:14 hostname journal: root[/root] 56092 ipaddress 22: locale
Mar 23 12:10:17 hostname journal: root[/root] 56092 ipaddress 22: tail -f /var/log/messages
Mar 23 12:10:23 hostname journal: root[/root] 56092 ipaddress 22: tail -f /var/log/secure
You can see wrong time order, Order in log file on client i as it should be.