I want to strip out the beginning and ending parts of the log entry

Elastic Stack Logstash
1

Logstash is receiving this, and I want to remove " --------------- Event Log Start Here ---------------\n" and "\n--------------- Event Log End Here ---------------" from it. And, I want to pull Timestamp value and assign it to @timestamp. I've gotten close, but can't get it to do everyting I want. Also, I'm pretty sure that I'm not doing it the best way, either?

--------------- Event Log Start Here ---------------\nEventId : 1, Level : Verbose, Message : Validating Fields to Monitor, if available for notifier for MessageQueueID:0, Payload : [message : Validating Fields to Monitor, if available for notifier for MessageQueueID:0] [applicationName : Not Provided] [hostName : STA-APP-02] [currentPrincipal : ] [executingPrincipal : STA\Framework] [userMessageGuid : 7137b2f9-0882-4d04-bf88-d2101226bb32] , EventName : DebugInfo, Timestamp : 2018-03-24T17:03:40.1346697Z\n--------------- Event Log End Here ---------------

Here's what my Filter script looks like. The @timestamp gets assigned correctly, but am having a hard time replacing the value of "message" with the combo of the new "Message" field and @timestamp. Plus this feels like a kluge, anyway? Any help is greatly appreciated!

        grok {
      patterns_dir => ["/etc/logstash/conf.d/patterns"]
      match => ["message", "[-]{12,18} Event Log Start Here [-]{12,18}\\n%{GREEDYDATA:Message}Timestamp : %{TIMESTAMP_ISO8601:logtime}\\n[-]{12,18} Event Log End Here [-]{12,18}"]
    }
    date {
      match => ["logtime", "ISO8601", "yyyy-MM-dd HH:mm:ss.SSSS", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-dd'T'HH:mm:ss.SSSSSSS'Z'"]
      target => "@timestamp"
    }
    mutate {
      replace => {"message" => "%{[Message]} %{@timestamp}"}
      remove_field => ["logtime"]
      remove_field => ["Message"]
    }

RHEL 7, Logstash 6.2.3

2

Use the gsub function of mutate to remove/replace field values. Something like:

filter {
  mutate {
    gsub => [
      "fieldname", "--------------- Event Log Start Here ---------------", "",
      "fieldname", "--------------- Event Log End Here ---------------", ""
    ]
  }
}

I'm not sure if you can use do both on a single line like "fieldname", "(starthere | endhere)", "" but you could try it out.

3

Thank you, I will try this and let you know.

4

Works, thanks!

5

How did you do it, two separate lines like my example or a single line like I mentioned in my closing paragraph?

6 
mutate {
  gsub => [
	"message", "--------------- Event Log Start Here ---------------\n","",
	"message", "\n--------------- Event Log End Here ---------------", ""
  ]
}
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.