# Logstash filter issue remove unwanted fields **URL:** \ **Category:** Logstash\ **Created:** [July 28, 2018, 10:25pm UTC](https://discuss.elastic.co/t/logstash-filter-issue-remove-unwanted-fields/142048 "2018-07-28T22:25:03Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![ramindia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ramindia/32/32459_2.png) [@ramindia](https://discuss.elastic.co/u/ramindia)\ **Post date:** [July 28, 2018, 10:25pm UTC](https://discuss.elastic.co/t/logstash-filter-issue-remove-unwanted-fields/142048/1 "2018-07-28T22:25:03Z")
hi All I have syslog setup and running ok, now try to use logstash and remove some unecessary logs before i send to ES below output show there is 2 date and stamps : like to remove 1 time stamp to stream line the message but not working, can some one help here. Jul 28 23:06:52 CORESW1 2018 Jul 28 22:06:16 UTC: %ETH\_PORT\_CHANNEL-5-FOP\_CHANGED: port-channel100: first operational port changed from Ethernet1/47 to none Jul 28 23:06:52 CORESW1 2018 Jul 28 22:06:16 UTC: %ETHPORT-5-IF\_DOWN\_PORT\_CHANNEL\_MEMBERS\_DOWN: Interface port-channel100 is down (No operational members) Jul 28 23:06:52 CORESW1 2018 Jul 28 22:06:16 UTC: %ETHPORT-5-IF\_DOWN\_INITIALIZING: Interface Ethernet1/47 is down (Initializing) Jul 28 23:06:52 CORESW1 2018 Jul 28 22:06:16 UTC: %ETHPORT-5-IF\_DOWN\_PORT\_CHANNEL\_MEMBERS\_DOWN: Interface port-channel100 is down (No operational members) here is my conf file. input { file { path =\> "/var/syslog-ng/raw/\*.log" start\_position =\> "beginning" type =\> "logstash-syslog" tags =\> ["logstash-syslog"] } } filter { grok { match =\> { "message" =\> "%{SYSLOGTIMESTAMP:syslog\_timestamp} %{DEVICEHOST:device\_src} %{TIMESTAMP\_ISO8601:timestamp1}: %{GREEDYDATA:syslog\_message}" } } mutate { remove\_field =\> [timestamp1] } ruby { code =\> "event.remove('type')" } date { match =\> ["syslog\_timestamp", "yyyy-MM-dd HH:mm:ss,SSS", "ISO8601"] } } output { elasticsearch { hosts =\> ["192.168.1.75:9200"] index =\> "logstash-syslog" } stdout { codec =\> rubydebug } } R! ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 29, 2018, 10:50am UTC](https://discuss.elastic.co/t/logstash-filter-issue-remove-unwanted-fields/142048/2 "2018-07-29T10:50:21Z")
What do you mean by "not working"? What result do you get and what would you like to change in that? BTW, you might do better with dissect rather than grok. ``` dissect { mapping => { "message" => "%{timestamp} %{+timestamp} %{+timestamp} %{device_src} %{} %{} %{} %{} %{} %{syslog_message}" } } ``` ---
**Author:** ![ramindia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ramindia/32/32459_2.png) [@ramindia](https://discuss.elastic.co/u/ramindia)\ **Post date:** [August 12, 2018, 6:19pm UTC](https://discuss.elastic.co/t/logstash-filter-issue-remove-unwanted-fields/142048/3 "2018-08-12T18:19:18Z")
Thank you reply, sorry i was away on holiday. I try to streamline the messages and remove duplicate dates display in the log output of ES. so suggestion is use dissect rather grok ? so grok is not the use case of my issue ? in other words, can this be achieved using GROK ? R! ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 13, 2018, 12:08am UTC](https://discuss.elastic.co/t/logstash-filter-issue-remove-unwanted-fields/142048/4 "2018-08-13T00:08:01Z")
> [@ramindia](#): > > in other words, can this be achieved using GROK ? Absolutely, yes. ---
**Author:** ![ramindia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ramindia/32/32459_2.png) [@ramindia](https://discuss.elastic.co/u/ramindia)\ **Post date:** [August 13, 2018, 5:14am UTC](https://discuss.elastic.co/t/logstash-filter-issue-remove-unwanted-fields/142048/5 "2018-08-13T05:14:50Z")
Thank you. In that case why my syntax not working ? can you suggest here ? R! ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 13, 2018, 1:03pm UTC](https://discuss.elastic.co/t/logstash-filter-issue-remove-unwanted-fields/142048/6 "2018-08-13T13:03:37Z")
Well, I get a syntax error because DEVICEHOST is not defined as a pattern. Plus the second timestamp does not match TIMESTAMP\_ISO8601. You should get a match with ``` "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{HOSTNAME:device_src} %{NUMBER} %{SYSLOGTIMESTAMP} %{WORD}: %{GREEDYDATA:syslog_message}" ``` Note that instead of naming the fields and then removing them, you can simply not name them. Plus the date format does not match. Try ``` date { match => ["syslog_timestamp", "MMM dd HH:mm:ss"] } ``` ---
**Author:** ![ramindia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ramindia/32/32459_2.png) [@ramindia](https://discuss.elastic.co/u/ramindia)\ **Post date:** [August 13, 2018, 6:25pm UTC](https://discuss.elastic.co/t/logstash-filter-issue-remove-unwanted-fields/142048/7 "2018-08-13T18:25:01Z")
Thank you, let me try and send my testing results here soon, appreciated your help and quick response. R! ---
**Author:** ![ramindia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ramindia/32/32459_2.png) [@ramindia](https://discuss.elastic.co/u/ramindia)\ **Post date:** [August 20, 2018, 6:31pm UTC](https://discuss.elastic.co/t/logstash-filter-issue-remove-unwanted-fields/142048/8 "2018-08-20T18:31:06Z")
It worked for your suggest all good. but i have different devices generating different format, how do we normalize them. Working one --- which got 2 dates and time coming. Aug 20 19:20:06 CORESW2 2018 Aug 20 19:20:06 GMT: last message repeated 2 times New one got only 1 time log Aug 20 19:29:54 DHCP-CA-DNS %SYS-5-CONFIG\_I: Configured from console by console any advise, appreciate your help R! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 17, 2018, 6:38pm UTC](https://discuss.elastic.co/t/logstash-filter-issue-remove-unwanted-fields/142048/9 "2018-09-17T18:38:37Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.