Multiline plugin not working as expected

Elastic Stack Logstash
1

I wanted individual field as a Logstash output. i.e. EventID, Level, Session ID, etc. My GROK pattern works fine on "https://grokdebug.herokuapp.com/" where individual values are extracted. But when placed in logstash, my output is just a message

LS Output -

   "message" => "EventId : 1, Level : Informational, Message : Request, Payl

oad : [sessionID : 3e4ad1b0-2d2d-4f77-90b3-6315aacfacb7] [ipAddress : 157.109.26.92] [method : GET] [requestUri : http://server001:8080/WebService
/api/products/48069/location/123] [content : ] , EventName : Requ
estInfo, Timestamp : 2016-04-05T14:35:36.4947170Z, ProcessId : 8924, ThreadId :
7000\r\n\r",

My Actual Log -

EventId : 1, Level : Informational, Message : Request, Payload : [sessionID : 3e4ad1b0-2d2d-4f77-90b3-6315aacfacb7] [ipAddress : 157.109.26.92] [method : GET] [requestUri : http://server001:8080/WebService/api/products/48069/location/123] [content : ] , EventName :
RequestInfo, Timestamp : 2016-04-05T14:35:36.4947170Z, ProcessId : 8924, ThreadId : 6996

Logstash Config File -

input
{
file
{
path => "C:/Logs/*"
codec => multiline
{
pattern => "%{WORD:EventId} : %{NUMBER:EventID}, %{WORD:Level} : %{WORD:EventLevel}, %{WORD:Message} : %{WORD:Operation}, %{WORD:Payload} : [%{WORD:Session_ID} : %{UUID:SessionID}] [%{WORD:IPAddress} : %{IPORHOST:ClientIPAddress}] [%{WORD:Method} : %{WORD:HTTP_Verb}] [%{WORD:requestUri} : %{URI:URL}] [%{WORD:content} : %{DATA:Request}] , %{WORD:EventName} : %{WORD:EventNameValue}, %{WORD:Timestamp} : %{TIMESTAMP_ISO8601:Request_DateTime}, %{WORD:ProcessId} : %{NUMBER:ProcessID}, %{WORD:ThreadID} : %{NUMBER:ThreadID}"
what => next
}
}
}

output
{
elasticsearch
{
hosts => ["myesserver:9200"]
}
stdout
{
codec => rubydebug
}
}

What should I do here ?

2

The multiline codec doesn't extract fields, it only joins lines. You still need a grok filter.

3

Thanks. That worked. Previously, it was not working.

I have one question, How can I have multiple grok patterns in one match => line.

GROK 1 - %{WORD:EventId} : %{NUMBER:EventID}, %{WORD:Level} : %{WORD:EventLevel}
GROK 2 - %{WORD:EventId} : %{NUMBER:EventID}, %{WORD:Level} : %{WORD:EventLevel}, %{WORD:Payload} : [%{WORD:Session_ID} : %{UUID:SessionID}]

4

See the example in the documentation of the match option.

In your example you could do it with a single expression too by making the last parts optional with (...)?:

GROK 2 - %{WORD:EventId} : %{NUMBER:EventID}, %{WORD:Level} : %{WORD:EventLevel}(, %{WORD:Payload} : \[%{WORD:Session_ID} : %{UUID:SessionID}\])?
5

Thanks for the reply Magnus.

I am stuck with 2 more issues. Added a topic but awaiting for reply. Is there any help that you can do here ?

Read logs stored using Enterprise Semantic Logging format -

ssl_certificate_validation not working -