# Advise for logstash filter **URL:** \ **Category:** Logstash\ **Created:** [April 18, 2018, 9:17pm UTC](https://discuss.elastic.co/t/advise-for-logstash-filter/128604 "2018-04-18T21:17:55Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Maddy\_M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maddy_m/32/45638_2.png) [@Maddy\_M](https://discuss.elastic.co/u/Maddy_M)\ **Post date:** [April 18, 2018, 9:17pm UTC](https://discuss.elastic.co/t/advise-for-logstash-filter/128604/1 "2018-04-18T21:17:56Z")
Hi, I am trying to parse below line and due to time stamp and varying delimiter I am unable to use it. Could you please advise? 18/09/2016 11:09:12:AYS2345:2 dcs:0 meta-\>info:0 total:6 ym-\>distila:56 I am looking to format as below. { Time: 18/09/2016 11:09:12 user: AYS2345 gear:2 dcs:0 meta-\>info:0 total:6 ym-\>distila:56 } ---
**Author:** ![atira](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/atira/32/28699_2.png) [@atira](https://discuss.elastic.co/u/atira)\ **Post date:** [April 18, 2018, 9:37pm UTC](https://discuss.elastic.co/t/advise-for-logstash-filter/128604/2 "2018-04-18T21:37:15Z")
```auto grok { match => ["message", "%{DATESTAMP:Time}:%{USER:user}:%{INT:gear} dcs:%{INT:dcs} meta->info:%{INT:meta} total:%{INT:total} ym->distila:%{INT:ym}"] } ``` will produce: ```auto { "Time": [ [ "18/09/2016 11:09:12" ] ], "user": [ [ "AYS2345" ] ], "gear": [ [ "2" ] ], "dcs": [ [ "0" ] ], "meta": [ [ "0" ] ], "total": [ [ "6" ] ], "ym": [ [ "56" ] ] } ``` You can't have `->` in a field name, I think. You could also do a combination of grok+dissect filter by filtering out Time with grok and put the rest into a field, then use dissect on the rest since those can be easily specified with delimiters. ---
**Author:** ![Maddy\_M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maddy_m/32/45638_2.png) [@Maddy\_M](https://discuss.elastic.co/u/Maddy_M)\ **Post date:** [April 19, 2018, 1:01pm UTC](https://discuss.elastic.co/t/advise-for-logstash-filter/128604/3 "2018-04-19T13:01:32Z")
> [@atira](#): > > "%{DATESTAMP:Time}:%{USER:user}:%{INT:gear} dcs:%{INT:dcs} meta-\>info:%{INT:meta} total:%{INT:total} ym-\>distila:%{INT:ym}" ] } Hi Atira, Thanks for the response. I am using logstash 5.0 and it seems that spaces between fields is causing an issue. Also, is there a way to test the logstash filter before we apply? ---
**Author:** ![atira](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/atira/32/28699_2.png) [@atira](https://discuss.elastic.co/u/atira)\ **Post date:** [April 19, 2018, 1:06pm UTC](https://discuss.elastic.co/t/advise-for-logstash-filter/128604/4 "2018-04-19T13:06:51Z")
You provided only one example, the filter above parsed it successfully. Do your events have varying whitespaces? It is possible to handle them, but we need more examples to see where these are. > [@Maddy\_M](#): > > Also, is there a way to test the logstash filter before we apply? Well, not with Logstash as far as I know. You can use [Grok Debugger](https://grokdebug.herokuapp.com/) to test your patterns against real examples. ---
**Author:** ![Maddy\_M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maddy_m/32/45638_2.png) [@Maddy\_M](https://discuss.elastic.co/u/Maddy_M)\ **Post date:** [April 19, 2018, 2:27pm UTC](https://discuss.elastic.co/t/advise-for-logstash-filter/128604/5 "2018-04-19T14:27:36Z")
Hi Atira, I am able to figure it out with gork debugger. Thanks for the help. 1. Since the resultant fields are in strings, are they compatible for line chart? or we need to convert it to numbers? 2. I have multiple filebeats configured in different hosts and each has different format. How can include all in one logstash conf? Here I am shipping all to one logstash host. Thanks, Maddy ---
**Author:** ![atira](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/atira/32/28699_2.png) [@atira](https://discuss.elastic.co/u/atira)\ **Post date:** [April 19, 2018, 3:58pm UTC](https://discuss.elastic.co/t/advise-for-logstash-filter/128604/6 "2018-04-19T15:58:52Z")
Actually, there is no string type in Elasticsearch. There are two types that handle strings: 1. text: searchable, but no aggregation possible (that means: no statistics) 2. keyword: not searchable (you'll have to use exact match), but you can aggregate it If you want to work with numbers, you'll have to map them. Multiple Logstash configuration is possible only with conditionals. For example, Filebeat sends the host name in the source field (if I remember correctly), so you can do a conditional like: ```auto if [source] = "hostname" { } ``` If you have a lot of different configs, it might become a conditional hell, but in Logstash 5.x, there is no other way AFAIK. Logstash 6.x accepts multiple config files. ---
**Author:** ![Maddy\_M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maddy_m/32/45638_2.png) [@Maddy\_M](https://discuss.elastic.co/u/Maddy_M)\ **Post date:** [April 19, 2018, 4:00pm UTC](https://discuss.elastic.co/t/advise-for-logstash-filter/128604/7 "2018-04-19T16:00:29Z")
This is the config, I am using. input { beats { port =\> 5044 } } if "totaldist" in [tags] { ``` grok { match => ["message", "%{DATESTAMP:Time}:%{USER:user}:%{INT:gear} dcs:%{INT:dcs} meta->info:%{INT:meta} total:%{INT:total} ym->distila:%{INT:ym}"] } mutate { remove_field => ["beat.hostname","beat.name","beat.version","input_type","type","offset","@version","_type","_id","_score:"] } if "_grokparsefailure" in [tags] { drop { } } } ``` output { if [tags] == "totaldist" { elasticsearch { hosts =\> ["mesta:9200"] index =\> "totaldist-%{+YYYY.MM.dd}" } } } ---
**Author:** ![atira](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/atira/32/28699_2.png) [@atira](https://discuss.elastic.co/u/atira)\ **Post date:** [April 19, 2018, 4:32pm UTC](https://discuss.elastic.co/t/advise-for-logstash-filter/128604/8 "2018-04-19T16:32:46Z")
And your question is...? 🙂 If you're dropping falsely parsed events, you might not realize you don't have all events in ES and even if you realize, you won't know, why the parsing failed. ---
**Author:** ![Maddy\_M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maddy_m/32/45638_2.png) [@Maddy\_M](https://discuss.elastic.co/u/Maddy_M)\ **Post date:** [April 20, 2018, 12:34pm UTC](https://discuss.elastic.co/t/advise-for-logstash-filter/128604/9 "2018-04-20T12:34:55Z")
Hi Atira, The question I have is all the data is going to "message". But I looking to split the fields in "message" section like key value pair. So that I can use the values in line chart. Also, I see the field names are not coming out as expected. { Time: 18/09/2016 11:09:12 user: AYS2345 gear:2 dcs:0 meta-\>info:0 total:6 ym-\>distila:56 } message:05/03/2017 04:13:00:7611:AYS2345 dcs:1 meta-\>infoc:0 total:65 ym-\>distilla:53 The end goal is to draw line chart with values extracted from dcs, info, total and ym Appreciate your help. ---
**Author:** ![Maddy\_M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maddy_m/32/45638_2.png) [@Maddy\_M](https://discuss.elastic.co/u/Maddy_M)\ **Post date:** [April 20, 2018, 7:34pm UTC](https://discuss.elastic.co/t/advise-for-logstash-filter/128604/10 "2018-04-20T19:34:35Z")
I fixed it thanks. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 18, 2018, 7:34pm UTC](https://discuss.elastic.co/t/advise-for-logstash-filter/128604/11 "2018-05-18T19:34:50Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.