Grok Filter + Expect Blank value into Variable/Parameter

Elastic Stack Logstash
1

Hello All,

Here is grok Filter

grok {
match => ["message", "%{DATESTAMP:timestamp1},%{GREEDYDATA:logmsg1},%{GREEDYDATA:logmsg2},%{INT:auth_method1},%{INT:auth_method2},%{GREEDYDATA:logmsg}"]

				add_field => [ "Timestamp", "%{timestamp1}" ]
				add_field => [ "C1","%{logmsg1}" ]
				add_field => [ "C2", "%{logmsg2}" ]
				add_field => [ "CT1", "%{auth_method1}" ]
				add_field => [ "CT2", "%{auth_method2}" ]
				add_field => [ "Status", "%{logmsg}" ]

}

Here is input

02/06/2016 01:01:07,CL Update,,1046,547,pass

Output

"timestamp1" => "02/06/2016 01:01:07",
   "logmsg1" => "CL_Update_Unit",

"auth_method1" => "1046",
"auth_method2" => "547",
"logmsg" => "fail",
"Timestamp" => "02/06/2016 01:01:07",
"C1" => "CL_Update_Unit",
"C2" => "%{logmsg2}",
"CT1" => "1046",
"CT2" => "547",
"Status" => "fail"

Issue:

"C2" => "%{logmsg2}" it should be blank, but its storing variable "%{logmsg2}"

Expected Output

"C2" => ""

2

I agree, I would've expected it to capture an empty string. But why not use the csv filter instead?

3

Thanks Magnus for your quick response,

Is the a way we can check either %{GREEDYDATA:logmsg2} is blank/null or has a value,
We will check if condition before add_field, so we can avoid this error,

Example,

If (%{GREEDYDATA:logmsg2} is not blank/null)
{
add_field => [ "C2", "%{logmsg2}" ]
}

Because, If we dont add_field then it will be store blank into Elastic search.

4

You could do

if [logmsg2] {
  mutate {
    ...
  }
}

but I don't understand why you don't capture the string straight into C2? What's the point of the intermediate fields? And, again, csv would be even easier.

5

Thanks Magnus for your quick response,

Yes, I agree with your suggestion and i have removed intermediate fields. And it works fine.

I dont know how i can go with CSV filter logic, i did CSV in the past with separate with comma ,
Below are the complicated inputs.

Here is the data defination

  1. we will have timestamp for sure as first
  2. We will have 2 integer values as well (Coz User apply the C1 and C2, T1 and T2 are input time and response time)
  3. If C1 is correct which is next after timestamp then result will be pass/fail
  4. If C1 is incorrct then result will be blank, example 02/10/2016 21:37:11,Error:Cant run any Command,90,200
  5. C2 can be there or not, C2 are rest of the command which user apply

List of Example/ Scenario

02/10/2016 21:37:11,abc1,abc2,abc3,abc4,104,375,pass

Expected Output

CTimeStamp : 02/10/2016 21:37:11
C1: abc1
C2: abc2,abc3,abc4
T1: 104
T2: 375
CStaus: pass

02/10/2016 21:37:11,abc1,abc2,100,375,fail

Expected Output

CTimeStamp : 02/10/2016 21:37:11
C1: abc1
C2: abc2
T1: 100
T2: 375
CStaus: fail

Possible inputs

02/10/2016 22:35:34,FT,,96,421,pass
02/10/2016 22:43:01,FT,FT1,104,447,pass
02/10/2016 22:41:03,ll,ll,ll,ll,162,441,pass

02/10/2016 21:37:11,Error:Cant run any Command,90,200

Expected Output

CTimeStamp : 02/10/2016 21:37:11
C1: Error:Cant Run any command
C2:
T1: 90
T2: 200
CStaus:

6

Hello,

Try adding this way (?:%{GREEDYDATA:logmsg2}|-) this way it should accept blank entries as well.

1 Like
7

thanks peter.