CSV Filter help

Elastic Stack Logstash
1

Hi There,

I am using the following CSV filter after failing to use grok filter for a logfile containing multiple patterns:

filter {
csv {
columns => ["@timestamp", "threadType", "requestType", "requestStatus"]
separator => "|"
}
}

The log file contains the following patterns:

2016-06-03 08:45:03 | INFO | [[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'] | WAKEUP-REQ | SUCCESS | urn:uuid:34567b55-ee31-3f45-859a-899f43fcb39c - 200 - - {"statusCode":"success","statusMessage":"Wakeup request recieved"}

2016-06-03 08:45:04 | INFO | [[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'] | WAKEUP-REQ | RECEIVED | urn:uuid:8251bbf1-6cab-381d-b482-a572834d989c

2016-06-03 08:46:02 | INFO | [jmsContainer-15] | AQ-REQ | PROCESSING | SUCCESS FOR urn:uuid:B9C6DC32-C2D7-4CBE-8268-B5F92CE85B73

However,
I get csvparsefailure on some of the patterns. Please help!

2

The CSV filter can only use a single pattern. You may need to use conditionals when they receive_csvparsefailure to try another pattern in a different CSV filter.

I've also raised this FR - https://github.com/logstash-plugins/logstash-filter-csv/issues/38

3

So, I went back and used the grok filter for the 2 different log patterns I have. The output i am getting is:

Capture.PNG1504×330 9.72 KB

As you can see...I am getting 2 different values for a single field. One is correct and another one is not. My question is..is there anyway I can get rid of the one i do not need? I tried using the remove_field option...but it only removes the field but not with a particular value. If anyone could help me with this i would greatly appreciate it.

Also, Is there a better approach to this multiple pattern log processing than what I have done here?

The logstash config file I have is below

filter {
grok {
tag_on_failure => [ "BROKEN_GROK_SYSLOG", "_grokparsefailure" ]
overwrite => [ "message" ]
match => ["message", "%{NOTSPACE} %{NOTSPACE:threadType} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE:requestType} %{NOTSPACE} %{NOTSPACE:requestStatus} %{NOTSPACE} %{GREEDYDATA:requestDetails}"]

}
grok {
	tag_on_failure => [ "BROKEN_GROK_SYSLOG", "_grokparsefailure" ]
	overwrite => [ "message" ]
    match => ["message", "%{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE:requestType} %{NOTSPACE} %{NOTSPACE:requestStatus} %{NOTSPACE} %{GREEDYDATA:requestDetails}"]

	}
mutate {
    remove => [ "message" ]
}
4

I am still stuck :frowning: