Hi,
I am currently trying to associate lines after I encounter the time field in log lines to that time field.
Sample log:
06-26-18 03:58:51 AM
top - 03:58:51 up 84 days, 2:05, 0 users, load average: 0.01, 0.08, 0.12
Tasks: 835 total, 1 running, 833 sleeping, 0 stopped, 1 zombie
Cpu(s): 1.3%us, 0.4%sy, 0.0%ni, 98.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 65695260k total, 54419828k used, 11275432k free, 286076k buffers
Swap: 6143996k total, 0k used, 6143996k free, 39933352k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15213 avendato 20 0 19.9g 670m 19m S 3.6 1.0 2902:06 /usr/java/default/bin/java -Djava.util.logging.config.file=/var/avenda/tomcat/backend/co
6889 appuser 20 0 13.7g 911m 33m S 2.6 1.4 2942:16 /usr/local/avenda/tips/sbin/policy_server
06-26-18 11:57:48 PM
6711 root 20 0 109m 1252 1040 S 0.0 0.0 5:48.79 awk /top - / { print strftime("%m-%d-%y %r")}1
6712 root 20 0 27020 1204 916 S 0.0 0.0 0:24.81 /usr/sbin/rotatelogs /var/avenda/platform/log/system-load-monitor//system-load.%Y-%m-%d.
7183 root 20 0 0 0 0 S 0.0 0.0 0:00.00 [kworker/1:2]
8408 postgres 20 0 16.4g 17m 8160 S 0.0 0.0 0:00.05 postgres: appuser tipsdb [local] idle
06-26-18 11:57:48 PM
top - 23:57:48 up 84 days, 22:04, 0 users, load average: 0.33, 0.23, 0.23
Tasks: 833 total, 1 running, 831 sleeping, 0 stopped, 1 zombie
Cpu(s): 1.4%us, 0.4%sy, 0.0%ni, 98.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 65695260k total, 55910064k used, 9785196k free, 286240k buffers
Swap: 6143996k total, 0k used, 6143996k free, 41379668k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8216 postgres 20 0 16.3g 50m 32m S 4.4 0.1 0:14.68 postgres: appsuperuser tipsdb 127.0.0.1(36820) idle
15213 avendato 20 0 19.9g 671m 19m S 4.0 1.0 2943:49 /usr/java/default/bin/java -
I want each line to be associated with the timestamp I found above that. Could someone please help me out how to make the association?
Below is what I tried out:
file{
sincedb_path => "/dev/null"
path => ["///xyz-2018-12-25.log"]
start_position => "beginning"
type => "log-top"
tags => ["top"]
codec => multiline{
patterns_dir => ["//etc/logstash/conf.d/patterns/patterns.conf"]
pattern => "^%{DATETIME}"
negate => true
what => previous
}
}
if ( "PID USER" in [message] or [message]==""){
drop{}
}
else if([message]=~/( AM)/ or [message]=~ /( PM)/){
grok{
patterns_dir => ["//etc/logstash/conf.d/patterns/patterns.conf"]
match=>["message","%{DATETIME:logtime}"]
}
date{
match=> ["message","MM-dd-yy hh:mm:ss aa"]
target=> "@timestamp"
locale => "en"
}
}
else if([message]=~ /(top -.*)/){
grok{
patterns_dir => ["//etc/logstash/conf.d/patterns/patterns.conf"]
match => ["message","%{NOTSPACE:junk} +%{NOTSPACE:junk} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:seconds} +%{NOTSPACE:state} +%{SINCE:since}, +%{NUMBER:num-of-users:int} +%{NOTSPACE:junk} +%{NOTSPACE:junk} +%{NOTSPACE:junk} +%{NUMBER:load-avg-5min:float}, +%{NUMBER:load-avg-10min:float}, +%{NUMBER:load-avg-15min:float}"]
}
}
else if([message]=~ /(Tasks:.*)/){
grok{
match => ["message","%{NOTSPACE:junk} %{NUMBER:total-task:int} +%{NOTSPACE:junk}, +%{NUMBER:running-task:int} +%{NOTSPACE:junk}, +%{NUMBER:sleeping:int} +%{NOTSPACE:junk}, +%{NUMBER:stopped:int} +%{NOTSPACE:junk}, +%{NUMBER:zombie:int} +%{NOTSPACE:junk}"]
}
}
else if([message]=~ /(Cpu\(s\):.*)/){
grok{
match => ["message","%{NOTSPACE:junk} +%{NUMBER:user-cpu-percent:float}%+%{NOTSPACE:junk}, +%{NUMBER:sys-cpu-percent:float}%+%{NOTSPACE:junk}, +%{NUMBER:nice-cpu-percent:float}%+%{NOTSPACE:junk}, +%{NUMBER:idle-cpu-percent:float}%+%{NOTSPACE:junk}, +%{NUMBER:wait-cpu-percent:float}%+%{NOTSPACE:junk}, +%{NUMBER:hi-cpu-percent:float}%+%{NOTSPACE:junk}, +%{NUMBER:si-cpu-percent:float}%+%{NOTSPACE:junk}, +%{NUMBER:st-cpu-percent:float}%+%{NOTSPACE:junk}"]
}
}
else if([message]=~ /(Mem:.*)/){
grok{
match => ["message","%{NOTSPACE:junk} +%{NUMBER:total-mem:int}+%{NOTSPACE:total-mem-unit} +%{NOTSPACE:junk} +%{NUMBER:used-mem:int}+%{NOTSPACE:used-mem-unit} +%{NOTSPACE:junk} +%{NUMBER:free-mem:int}+%{NOTSPACE:free-mem-unit} +%{NOTSPACE:junk} +%{NUMBER:buf-mem:int}+%{NOTSPACE:buf-mem-unit} +%{NOTSPACE:junk}"]
}
}
else if([message]=~ /(Swap:.*)/){
grok{
match => ["message","%{NOTSPACE:junk} +%{NUMBER:total-swap:int}+%{NOTSPACE:total-swap-unit} +%{NOTSPACE:junk} +%{NUMBER:used-swap:int}+%{NOTSPACE:used-swap-unit} +%{NOTSPACE:junk} +%{NUMBER:free-swap:int}+%{NOTSPACE:free-swap-unit} +%{NOTSPACE:junk} +%{NUMBER:cache-swap:int}+%{NOTSPACE:cache-swap-unit} +%{NOTSPACE:junk}"]
}
}
else {
grok{
match=>["message","%{NUMBER:process-id:int} +%{NOTSPACE:user} +%{NOTSPACE:priority} +%{NUMBER:nice} +%{NOTSPACE:virt-mem-size} +%{NOTSPACE:reside-mem-size} +%{NOTSPACE:shared-mem-size} +%{NOTSPACE:process-status} +%{NUMBER:proc-cpu-use-percent:float} +%{NUMBER:proc-mem-use-percent:float} +%{NOTSPACE:cpu-time} +%{GREEDYDATA:command}"] }
}
mutate {remove_field => ["@version","path","host","junk_perc","junk"]}
}
}
I have defined pattern DATETIME as: DATETIME %{DATE} +%{HOUR}:%{MINUTE}:%{SECOND} +%{WORD} in patterns file.
I am facing issue with associating with the multiline codec. Output still appears with the timestamp of logstash runtime for line without timestamp which are maked as bold in the sample input.
I didn't get any response for the same post mentioned 12 days back @ https://discuss.elastic.co/t/associating-log-lines-with-timestamp-in-log-line/144979
Could someone please help me at the earliest?
Thanks in advance