Logstash Pipeline-Filter

[vincent25]

Hi,
I am new to ELK and struggling to write my first logstash pipeline.
Can anyone help me to write my pipeline with the filter section?Please note that I require the actual log time (e.g 2018-01-23 06:43:48.79851) to create and demonstrate visualization in Kibana.

One line of my log sample is:
[ERROR@[140598203889408]2018-01-23 06:43:48.798511 in src/TMSInfo.cpp(1699)]TMSInfo::GetCorrelationKey: CIN is NULL failed to create corr key based on cin

Thanks in advance.

[tag_v]

for mentioned log line you better choose grok patterns. use https://grokdebug.herokuapp.com/ for writing groks and mention them in filter part of logstash conf file.

sample logstash conf :

input {
stdin { }
}
filter {
grok {
match => { "message" => "%{GREEDYDATA:msg}"}
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
hosts => "localhost:9200"
index => "indexname"
}
}

hope this helps.

[vincent25]

Hi,thanks.I tried that but still not successful.

[tag_v]

as u mentioned only timestamp need to be displayed in kibana this conf works:

input {
stdin { }
}
filter {
grok {
match => { "message" => "%{GREEDYDATA}%{TIMESTAMP_ISO8601:msg} %{GREEDYDATA}"}
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
hosts => "localhost:9200"
index => "indexname"
}
}

ping error if any

[vincent25]

Hi,Index is being created.But I require the msg timestamp as my @timestamp.
Please find below the section from Kibana.

message:[ERROR@[140598195496704]2018-01-30 21:06:23.589005 in src/TMSInfo.cpp(1699)]TMSInfo::GetCorrelationKey: CIN is NULL failed to create corr key based on cin msg:18-01-30 21:06:23.589005 path:C:\x\x\x\x.log @timestamp:March 1st 2018, 16:34:42.768

[vincent25]

Hi,I tried using the date filter to replace my log time with @timestamp,tried changing the Date & Time format in Kibana also.But the problem still exist.I am stuck now.Some help will be much appreciated.
My pipleline and Kibana output is as shown below:

Pipeline:
input {
file {
path => "x:\x\x\x.log"
start_position => beginning
ignore_older => 0
}
}

filter {
grok {
match => { "message" => "%{GREEDYDATA}%{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA}"}
}
date {
match => [ "timestamp" ,"ISO8601" ]
target => "@timestamp"
}
}

output {
elasticsearch {
hosts => [ "localhost:9200" ]
}
}

Kibana Output:

message:[ERROR@[140598203889408]2018-01-23 17:32:16.844510 in src/TMSInfo.cpp(1699)]TMSInfo::GetCorrelationKey: CIN is NULL failed to create corr key based on cin path:x:\x\x\x.log @timestamp:March 3rd 2018, 17:14:25.444 @version:1 host:CT-IL0004 timestamp:18-01-23 17:32:16.844510 tags:_dateparsefailure _id:AWHrrWeQ2YeFYuYd25al _type:logs _index:logstash-2018.03.03 _score: -