# Logstash grok pattern working fine in grok debugger, but fails in logstash conf file to parse

**URL:** <https://discuss.elastic.co/t/logstash-grok-pattern-working-fine-in-grok-debugger-but-fails-in-logstash-conf-file-to-parse/198420>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [September 6, 2019, 12:15pm UTC](https://discuss.elastic.co/t/logstash-grok-pattern-working-fine-in-grok-debugger-but-fails-in-logstash-conf-file-to-parse/198420 "2019-09-06T12:15:39Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![sonammarda](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sonammarda/32/52241_2.png) [@sonammarda](https://discuss.elastic.co/u/sonammarda)\
**Post date:** [September 6, 2019, 12:15pm UTC](https://discuss.elastic.co/t/logstash-grok-pattern-working-fine-in-grok-debugger-but-fails-in-logstash-conf-file-to-parse/198420/1 "2019-09-06T12:15:39Z")

</div>

_Logstash version_ - 7.3  
_Filebeat version_ - 7.3  
_OS_ - Windows

**Below is my exception format in application log file-**  
\<log4j:event logger="ESM.EasyPurchaseMarketplace.Infrastructure.Logging.Logger" timestamp="1566964971685" level="ERROR" thread="7388"\>log4j:message

```
    ======================================
     
     MachineName: PRODMAGWEB1 
     TimeStamp: 8/28/2019 12:02:51 AM 
     FullName: ESM.EasyPurchaseMarketplace.Infrastructure.Logging, Version=2.51.7170.6286, Culture=neutral, PublicKeyToken=null 
     AppDomainName: /LM/W3SVC/8/ROOT-1-132113700121002852 
     WindowsIdentity: ESMSOLUTIONS\EPMServiceAppIdentit 
     Message: Exception occured in PurchaseAPI token logic 

     Guid -  
     UserId -  
     EntityId -  
     ImpersonatedBy -  
     ImpersonatingUserId -  
     ImpersonatedUserName -  

     </log4j:message><log4j:properties><log4j:data name="log4net:UserName" value="ESMSOLUTIONS\EPMServiceAppIdentit" /><log4j:data name="log4jmachinename" value="ProdMagWeb1" /><log4j:data name="log4japp" value="/LM/W3SVC/8/ROOT-1-132113700121002852" /><log4j:data name="log4net:HostName" value="ProdMagWeb1" /></log4j:properties><log4j:throwable><![CDATA[System.Web.HttpException (0x80070057): The remote host closed the connection. The error code is 0x80070057.
       at System.Web.Hosting.IIS7WorkerRequest.RaiseCommunicationError(Int32 result, Boolean throwOnDisconnect)
       at System.Web.Hosting.IIS7WorkerRequest.ExplicitFlush()
       at System.Web.HttpResponse.Flush(Boolean finalFlush, Boolean async)
       at System.Web.HttpResponse.Flush()
       at System.Web.HttpWriter.WriteFromStream(Byte[] data, Int32 offset, Int32 size)
       at System.Web.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 count)
       at Microsoft.Owin.Host.SystemWeb.CallStreams.OutputStream.Write(Byte[] buffer, Int32 offset, Int32 count)
       at Microsoft.AspNet.SignalR.Owin.ServerResponse.Write(ArraySegment`1 data)
       at Microsoft.AspNet.SignalR.Hosting.ResponseExtensions.End(IResponse response, String data)
       at Microsoft.AspNet.SignalR.PersistentConnection.SendJsonResponse(HostContext context, String jsonPayload)
       at Microsoft.AspNet.SignalR.PersistentConnection.ProcessNegotiationRequest(HostContext context)
       at Microsoft.AspNet.SignalR.PersistentConnection.ProcessRequest(HostContext context)
       at Microsoft.AspNet.SignalR.Hubs.HubDispatcher.ProcessRequest(HostContext context)
       at Microsoft.AspNet.SignalR.PersistentConnection.ProcessRequest(IDictionary`2 environment)
       at Microsoft.AspNet.SignalR.Owin.Middleware.HubDispatcherMiddleware.Invoke(IOwinContext context)
       at Microsoft.Owin.Infrastructure.OwinMiddlewareTransition.Invoke(IDictionary`2 environment)
       at Microsoft.Owin.Cors.CorsMiddleware.HandleCorsRequestAsync(IOwinContext context, CorsPolicy policy, CorsRequestContext corsRequestContext)
       at Microsoft.Owin.Cors.CorsMiddleware.<Invoke>d__0.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.Owin.Mapping.MapMiddleware.<Invoke>d__0.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at ESM.Purchase.WebApi.Common.GlobalOwinExeptionHandler.<Invoke>d__0.MoveNext()]]></log4j:throwable><log4j:locationInfo class="ESM.EasyPurchaseMarketplace.Infrastructure.Logging.Logger" method="WriteToLog" file="" line="0" /></log4j:event>

```

**Grok Pattern that I have written:**  
`.*?: %{WORD:MachineName} .*?\n .*?: %{DATE:timestamp} %{TIME:time} .*?\n .*?: .*?, Version=%{USER:Version}, Culture=%{USER:Culture}, PublicKeyToken=%{USER:PublicKeyToken} .*?\n .*?: %{PATH:AppDomainName}.*?\n .*?: %{WORD:WindowsIdentity}.*?\n .*?: (?<Exception Message>.?*)\n`

**When I use Grok debugger in Kibana under dev tools, this grok pattern gives me correct results as below-**  
{  
"AppDomainName": "/LM/W3SVC/8/ROOT-1-132113700121002852",  
"WindowsIdentity": "ESMSOLUTIONS",  
"MachineName": "PRODMAGWEB1",  
"Version": "2.51.7170.6286",  
"time": "12:02:51",  
"PublicKeyToken": "null",  
"Exception Message": "Exception occured in PurchaseAPI token logic \r",  
"Culture": "neutral",  
"timestamp": "8/28/2019"  
}

**But when is use same grok pattern in logstash config file, it gives grok parse failure.**  
**Below is my logstash config file-**

```
input { 
	beats{
		port=>5044
	}
}

    filter {
          grok {
            match =>{
    		"message" => ".*?: %{WORD:MachineName} .*?\n .*?: %{DATE:timestamp} %{TIME:time} .*?\n .*?: .*?, Version=%{USER:Version}, Culture=%{USER:Culture}, PublicKeyToken=%{USER:PublicKeyToken} .*?\n .*?: %{PATH:AppDomainName}.*?\n .*?: %{WORD:WindowsIdentity}.*?\n .*?: (?<Exception Message>.?*)\n" 
    			}  
    		}
    }

    output {
    if "esm-purchase-api-log" in [tags] { 
    elasticsearch {
    hosts => ["localhost:9200"]
    index => "esm-purchase-api-log-%{+YYYY.MM.dd}"
    }
    }
    }

```

Can you please tell me where am I going wrong. I am trying this hard since long. Any help would be appreciated!`Preformatted text`

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 6, 2019, 12:34pm UTC](https://discuss.elastic.co/t/logstash-grok-pattern-working-fine-in-grok-debugger-but-fails-in-logstash-conf-file-to-parse/198420/2 "2019-09-06T12:34:00Z")

</div>

Please edit your post, select the configuration, and click on \</\> in the toolbar above the edit pane. You should see the appearance change in the preview pane to the right so that the text

```
looks like this

```

Then do the same for the input.

---

<div class="post-metadata">

**Author:** ![sonammarda](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sonammarda/32/52241_2.png) [@sonammarda](https://discuss.elastic.co/u/sonammarda)\
**Post date:** [September 6, 2019, 1:43pm UTC](https://discuss.elastic.co/t/logstash-grok-pattern-working-fine-in-grok-debugger-but-fails-in-logstash-conf-file-to-parse/198420/3 "2019-09-06T13:43:45Z")

</div>

I have edited my post. Can you please check?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 6, 2019, 2:22pm UTC](https://discuss.elastic.co/t/logstash-grok-pattern-working-fine-in-grok-debugger-but-fails-in-logstash-conf-file-to-parse/198420/4 "2019-09-06T14:22:07Z")

</div>

If I consume that entire message as a single event using a file input then the grok pattern matches

```
  "WindowsIdentity" => "ESMSOLUTIONS",
        "timestamp" => "8/28/2019",
    "AppDomainName" => "/LM/W3SVC/8/ROOT-1-132113700121002852",
          "Version" => "2.51.7170.6286",
      "MachineName" => "PRODMAGWEB1",
          "Culture" => "neutral"

```

etc. So I suspect the problem is the way filebeat is ingesting the event.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 4, 2019, 2:22pm UTC](https://discuss.elastic.co/t/logstash-grok-pattern-working-fine-in-grok-debugger-but-fails-in-logstash-conf-file-to-parse/198420/5 "2019-10-04T14:22:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.