Logstash filter issue remove unwanted fields

[ramindia (R!)]

hi All

I have syslog setup and running ok, now try to use logstash and remove some unecessary logs before i send to ES

below output show there is 2 date and stamps : like to remove 1 time stamp to stream line the message but not working, can some one help here.

Jul 28 23:06:52 CORESW1 2018 Jul 28 22:06:16 UTC: %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel100: first operational port changed from Ethernet1/47 to none
Jul 28 23:06:52 CORESW1 2018 Jul 28 22:06:16 UTC: %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel100 is down (No operational members)
Jul 28 23:06:52 CORESW1 2018 Jul 28 22:06:16 UTC: %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/47 is down (Initializing)
Jul 28 23:06:52 CORESW1 2018 Jul 28 22:06:16 UTC: %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel100 is down (No operational members)

here is my conf file.

input {
file {
path => "/var/syslog-ng/raw/*.log"
start_position => "beginning"
type => "logstash-syslog"
tags => [ "logstash-syslog" ]
}
}
filter {
grok {
match => {
"message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{DEVICEHOST:device_src} %{TIMESTAMP_ISO8601:timestamp1}: %{GREEDYDATA:syslog_message}"
}
}
mutate {
remove_field => [timestamp1]
}
ruby {
code => "event.remove('type')"
}
date {
match => ["syslog_timestamp", "yyyy-MM-dd HH:mm:ss,SSS", "ISO8601"]
}
}
output {
elasticsearch {
hosts => ["192.168.1.75:9200"]
index => "logstash-syslog"
}
stdout {
codec => rubydebug
}
}

R!

[Badger]

What do you mean by "not working"? What result do you get and what would you like to change in that?

BTW, you might do better with dissect rather than grok.

dissect { mapping => { "message" => "%{timestamp} %{+timestamp} %{+timestamp} %{device_src} %{} %{} %{} %{} %{} %{syslog_message}" } }

[ramindia (R!)]

Thank you reply, sorry i was away on holiday.

I try to streamline the messages and remove duplicate dates display in the log output of ES.

so suggestion is use dissect rather grok ? so grok is not the use case of my issue ?

in other words, can this be achieved using GROK ?

R!

[Badger]

@ramindia

in other words, can this be achieved using GROK ?

Absolutely, yes.

[ramindia (R!)]

Thank you.

In that case why my syntax not working ? can you suggest here ?

R!

[Badger]

Well, I get a syntax error because DEVICEHOST is not defined as a pattern. Plus the second timestamp does not match TIMESTAMP_ISO8601.

You should get a match with

"message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{HOSTNAME:device_src} %{NUMBER} %{SYSLOGTIMESTAMP} %{WORD}: %{GREEDYDATA:syslog_message}"

Note that instead of naming the fields and then removing them, you can simply not name them.

Plus the date format does not match. Try

date { match => ["syslog_timestamp", "MMM dd HH:mm:ss" ] }

[ramindia (R!)]

Thank you, let me try and send my testing results here soon, appreciated your help and quick response.

R!

[ramindia (R!)]

It worked for your suggest all good.

but i have different devices generating different format, how do we normalize them.

Working one --- which got 2 dates and time coming.

Aug 20 19:20:06 CORESW2 2018 Aug 20 19:20:06 GMT: last message repeated 2 times

New one got only 1 time log
Aug 20 19:29:54 DHCP-CA-DNS %SYS-5-CONFIG_I: Configured from console by console

any advise, appreciate your help

R!