# Logstash Pipeline-Filter

**URL:** <https://discuss.elastic.co/t/logstash-pipeline-filter/122034>\
**Category:** Logstash\
**Created:** [March 1, 2018, 10:36am UTC](https://discuss.elastic.co/t/logstash-pipeline-filter/122034 "2018-03-01T10:36:51Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![vincent25](https://avatars.discourse-cdn.com/v4/letter/v/71c47a/32.png) [@vincent25](https://discuss.elastic.co/u/vincent25)\
**Post date:** [March 1, 2018, 10:36am UTC](https://discuss.elastic.co/t/logstash-pipeline-filter/122034/1 "2018-03-01T10:36:51Z")

</div>

Hi,  
I am new to ELK and struggling to write my first logstash pipeline.  
Can anyone help me to write my pipeline with the filter section?Please note that I require the actual log time (e.g 2018-01-23 06:43:48.79851) to create and demonstrate visualization in Kibana.

One line of my log sample is:  
[ERROR@[140598203889408]2018-01-23 06:43:48.798511 in src/TMSInfo.cpp(1699)]TMSInfo::GetCorrelationKey: CIN is NULL failed to create corr key based on cin

Thanks in advance.

---

<div class="post-metadata">

**Author:** ![tag\_v](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tag_v/32/47466_2.png) [@tag\_v](https://discuss.elastic.co/u/tag_v)\
**Post date:** [March 1, 2018, 10:49am UTC](https://discuss.elastic.co/t/logstash-pipeline-filter/122034/2 "2018-03-01T10:49:26Z")

</div>

for mentioned log line you better choose grok patterns. use [https://grokdebug.herokuapp.com/](https://grokdebug.herokuapp.com/) for writing groks and mention them in filter part of logstash conf file.

sample logstash conf :

input {  
stdin { }  
}  
filter {  
grok {  
match =\> { "message" =\> "%{GREEDYDATA:msg}"}  
}  
}  
output {  
stdout { codec =\> rubydebug }  
elasticsearch {  
hosts =\> "localhost:9200"  
index =\> "indexname"  
}  
}

hope this helps.

---

<div class="post-metadata">

**Author:** ![vincent25](https://avatars.discourse-cdn.com/v4/letter/v/71c47a/32.png) [@vincent25](https://discuss.elastic.co/u/vincent25)\
**Post date:** [March 1, 2018, 10:52am UTC](https://discuss.elastic.co/t/logstash-pipeline-filter/122034/3 "2018-03-01T10:52:25Z")

</div>

Hi,thanks.I tried that but still not successful.

---

<div class="post-metadata">

**Author:** ![tag\_v](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tag_v/32/47466_2.png) [@tag\_v](https://discuss.elastic.co/u/tag_v)\
**Post date:** [March 1, 2018, 10:54am UTC](https://discuss.elastic.co/t/logstash-pipeline-filter/122034/4 "2018-03-01T10:54:10Z")

</div>

as u mentioned only timestamp need to be displayed in kibana this conf works:

input {  
stdin { }  
}  
filter {  
grok {  
match =\> { "message" =\> "%{GREEDYDATA}%{TIMESTAMP\_ISO8601:msg} %{GREEDYDATA}"}  
}  
}  
output {  
stdout { codec =\> rubydebug }  
elasticsearch {  
hosts =\> "localhost:9200"  
index =\> "indexname"  
}  
}

ping error if any

---

<div class="post-metadata">

**Author:** ![vincent25](https://avatars.discourse-cdn.com/v4/letter/v/71c47a/32.png) [@vincent25](https://discuss.elastic.co/u/vincent25)\
**Post date:** [March 1, 2018, 11:08am UTC](https://discuss.elastic.co/t/logstash-pipeline-filter/122034/5 "2018-03-01T11:08:37Z")

</div>

Hi,Index is being created.But I require the msg timestamp as my @timestamp.  
Please find below the section from Kibana.

message:[ERROR@[140598195496704]2018-01-30 21:06:23.589005 in src/TMSInfo.cpp(1699)]TMSInfo::GetCorrelationKey: CIN is NULL failed to create corr key based on cin msg:18-01-30 21:06:23.589005 path:C:\x\x\x\x.log @timestamp:March 1st 2018, 16:34:42.768

---

<div class="post-metadata">

**Author:** ![vincent25](https://avatars.discourse-cdn.com/v4/letter/v/71c47a/32.png) [@vincent25](https://discuss.elastic.co/u/vincent25)\
**Post date:** [March 3, 2018, 11:46am UTC](https://discuss.elastic.co/t/logstash-pipeline-filter/122034/6 "2018-03-03T11:46:13Z")

</div>

Hi,I tried using the date filter to replace my log time with @timestamp,tried changing the Date & Time format in Kibana also.But the problem still exist.I am stuck now.Some help will be much appreciated.  
My pipleline and Kibana output is as shown below:

Pipeline:  
input {  
file {  
path =\> "x:\x\x\x.log"  
start\_position =\> beginning  
ignore\_older =\> 0  
}  
}

filter {  
grok {  
match =\> { "message" =\> "%{GREEDYDATA}%{TIMESTAMP\_ISO8601:timestamp} %{GREEDYDATA}"}  
}  
date {  
match =\> ["timestamp" ,"ISO8601"]  
target =\> "@timestamp"  
}  
}

output {  
elasticsearch {  
hosts =\> ["localhost:9200"]  
}  
}

Kibana Output:

message:[ERROR@[140598203889408]2018-01-23 17:32:16.844510 in src/TMSInfo.cpp(1699)]TMSInfo::GetCorrelationKey: CIN is NULL failed to create corr key based on cin path:x:\x\x\x.log @timestamp:March 3rd 2018, 17:14:25.444 @version:1 host:CT-IL0004 timestamp:18-01-23 17:32:16.844510 tags:\_dateparsefailure \_id:AWHrrWeQ2YeFYuYd25al \_type:logs \_index:logstash-2018.03.03 \_score: -

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 31, 2018, 11:46am UTC](https://discuss.elastic.co/t/logstash-pipeline-filter/122034/7 "2018-03-31T11:46:17Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.