Greedydata not working as expected


I have the following data
2016-02-07 22:45:48,804 [72] FATAL GlobalHttpApplication [(null)] -
[Request url]: /?

[Reported Exception Message]: Server cannot set status after HTTP headers have been sent.

Region: (null)
UserId: (null)
Session: {}

System.Web.HttpException (0x80004005): Server cannot set status after HTTP headers have been sent.
at System.Web.HttpResponse.set_StatusCode(Int32 value)
at System.Web.HttpResponseWrapper.set_StatusCode(Int32 value)
at System.Web.Mvc.HandleErrorAttribute.OnException(ExceptionContext filterContext)
at System.Web.Mvc.ControllerActionInvoker.InvokeExceptionFilters(ControllerContext controllerContext, IList`1 filters, Exception exception)
at System.Web.Mvc.ControllerActionInvoker.InvokeAction(ControllerContext controllerContext, String actionName)
at System.Web.Mvc.Controller.ExecuteCore()
at System.Web.Mvc.ControllerBase.Execute(RequestContext requestContext)
at System.Web.Mvc.ControllerBase.System.Web.Mvc.IController.Execute(RequestContext requestContext)
at AdminWebsite.Controllers.ErrorController.InvokeErrorAction(HttpContextBase httpContext, String action, Exception exception) in d:\jenkins\build\hickory\rc-2.21.0\build-artifacts\workspace\src\App\AdminWebsite\Controllers\ErrorController.cs:line 77
at AdminWebsite.Global.Application_Error(Object sender, EventArgs e) in d:\jenkins\build\hickory\rc-2.21.0\build-artifacts\workspace\src\App\AdminWebsite\Global.asax.cs:line 291

and the conf file is as below:

codec => multiline {
pattern => "^%{TIMESTAMP_ISO8601} "
what => previous
negate => true
grok {
match => [ "message", "%{TIMESTAMP_ISO8601:log_timestamp}[ ][%{GREEDYDATA:debugid}][ ]%{LOGLEVEL:log_level:WORD}[ ]%{NOTSPACE:service}[ ].(null).[ ]-[ ]%{GREEDYDATA:log_details}" ]
date {
match => [ "log_timestamp", "YYYY-MM-dd HH:mm:ss,SSS" ]
target => log_timestamp
locale => en
timezone => UCT
hosts => "localhost:9200"
index => mydata
codec => rubydebug

The output I am getting is as below:

{ "@timestamp" => "2016-04-11T10:01:06.973Z",
"message" => "2016-02-07 22:45:48,804 [72] FATAL GlobalHttpApplication
[(null)] - [Request url]: /?[Reported Exception Message]: Server cannot set sta
tus after HTTP headers have been sent.\r\n Region: (null) \r\n UserId: (null) \r
\n Session: {} \r\n \r\nSystem.Web.HttpException (0x80004005): Server cannot set
status after HTTP headers have been sent.\r\n at System.Web.HttpResponse.set_
StatusCode(Int32 value)\r\n at System.Web.HttpResponseWrapper.set_StatusCode(I
nt32 value)\r\n at System.Web.Mvc.HandleErrorAttribute.OnException(ExceptionCo
ntext filterContext)\r\n at System.Web.Mvc.ControllerActionInvoker.InvokeExcep
tionFilters(ControllerContext controllerContext, IList`1 filters, Exception exce
ption)\r\n at System.Web.Mvc.ControllerActionInvoker.InvokeAction(ControllerCo
ntext controllerContext, String actionName)\r\n at System.Web.Mvc.Controller.E
xecuteCore()\r\n at System.Web.Mvc.ControllerBase.Execute(RequestContext reque
stContext)\r\n at System.Web.Mvc.ControllerBase.System.Web.Mvc.IController.Exe
cute(RequestContext requestContext)\r\n at AdminWebsite.Con
trollers.ErrorController.InvokeErrorAction(HttpContextBase httpContext, String a
ction, Exception exception) in d:\jenkins\build\hickory\rc-2.21.0\build-art
Controller.cs:line 77\r\n at AdminWebsite.Global.Applicatio
n_Error(Object sender, EventArgs e) in d:\jenkins\build\hickory\rc-2.21.0\b
x.cs:line 291\r",
"@version" => "1",
"tags" => [
[0] "multiline"
"host" => "ITT-Surendra",
"log_timestamp" => "2016-02-07T22:45:48.804Z",
"debugid" => "72",
"log_level" => "FATAL",
"service" => "GlobalHttpApplication",
"log_details" => "artifacts\workspace\src\App\AdminWebsite\Global.asax.cs:line 291\r"

If you see the bold text, which is a greedy data, it is not taking all the data, I don't know what is the error. Everything else is working fine other than that. Can someone please help me to fix this?


Grok doesn't work across multiple lines i.e. when using a multiline codec/filter. Add (?m) to the start of your Grok pattern:

grok {
  match => [ "message", "(?m)%{TIMESTAMP_ISO8601:log_timestamp}[ ][%{GREEDYDATA:debugid}][ ]%{LOGLEVEL:log_level:WORD}[ ]%{NOTSPACE:service}[ ].(null).[ ]*-[ ]***%{GREEDYDATA:log_details}**" ]

Does that fix it?

@Alex_6 : No, I am still getting the same output.

@Alex_6 is there any other way in which I can overcome this problem?

@Alex_6 , got it fixed, there was a problem in the grok pattern that's why I was facing the issue.