Logstash Pattern not working as expected

Elastic Stack Logstash
1

Log 1 - Request
EventId : 2, Level : Informational, Message : , Payload : [id : 20a3839e-c391-4573-ac1c-99b389f21c46] [request : MyServiceRequest] , EventName : ResponseInfo, Timestamp : 2015-08-04T22:20:17.8222179Z, ProcessId : 10332, ThreadId : 14748

Log 2 - Response
EventId : 2, Level : Informational, Message : , Payload : [id : 20a3839e-c391-4573-ac1c-99b389f21c46] [response : MyServiceResponse] , EventName : ResponseInfo, Timestamp : 2015-08-04T22:20:17.8222179Z, ProcessId : 10332, ThreadId : 14748

GROK - 1
%{WORD:EventId} : %{NUMBER:EventID}, %{WORD:Level} : %{WORD:Level}, %{WORD:Message} : %{DATA:Message}, %{WORD:Payload} : [%{WORD:id} : %{UUID:SessionID}] [%{WORD:request} : %{DATA:SoapRequest}] , %{WORD:EventName} : %{WORD:EventName}, %{WORD:Timestamp} : %{TIMESTAMP_ISO8601:TimeStamp}, %{WORD:ProcessId} : %{NUMBER:ProcessId}, %{WORD:ThreadId} : %{NUMBER:ThreadId}

GROK - 2
%{WORD:EventId} : %{NUMBER:EventID}, %{WORD:Level} : %{WORD:Level}, %{WORD:Message} : %{DATA:Message}, %{WORD:Payload} : [%{WORD:id} : %{UUID:SessionID}] [%{WORD:response} : %{DATA:SoapResponse}] , %{WORD:EventName} : %{WORD:EventName}, %{WORD:Timestamp} : %{TIMESTAMP_ISO8601:TimeStamp}, %{WORD:ProcessId} : %{NUMBER:ProcessId}, %{WORD:ThreadId} : %{NUMBER:ThreadId}

The format is the same. The only difference is the Request and Response tags. Now in the output I get request values properly, but for response the field name is request. I wanted it to be "SoapRequest" and "SoapResponse"

"request": [
[
"response"
]
],
"SoapRequest": [
[
"MyServiceResponse"
]
]

Can anyone please let me know what am I doing wrong here?

2

What does the entire config look like then.

3

%{WORD:EventId} : %{NUMBER:EventID}, %{WORD:Level} : %{WORD:Level}, %{WORD:Message} : %{DATA:Message}, %{WORD:Payload} : [%{WORD:id} : %{UUID:SessionID}] [%{WORD:request} : %{DATA:SoapRequest}] , %{WORD:EventName} : %{WORD:EventName}, %{WORD:Timestamp} : %{TIMESTAMP_ISO8601:TimeStamp}, %{WORD:ProcessId} : %{NUMBER:ProcessId}, %{WORD:ThreadId} : %{NUMBER:ThreadId}

I think you'll find the kv filter useful, but if you insist on using grok this makes better sense:

EventId : %{NUMBER:EventID}, Level : %{WORD:Level}, Message : %{DATA:Message}, Payload : \[id : %{UUID:SessionID}\] \[request : %{DATA:SoapRequest}\] , EventName : %{WORD:EventName}, Timestamp : %{TIMESTAMP_ISO8601:TimeStamp}, ProcessId : %{NUMBER:ProcessId}, ThreadId: %{NUMBER:ThreadId}

The first part of the expression, "EventId" in my example, is a literal string that you want to match. You don't want to match an arbitrary word and put it in the EventId field. Making this change might actually fix your problem too.