# How to grok complex patterns with logstash 7.1

**URL:** <https://discuss.elastic.co/t/how-to-grok-complex-patterns-with-logstash-7-1/218480>\
**Category:** Logstash\
**Created:** [February 9, 2020, 8:24pm UTC](https://discuss.elastic.co/t/how-to-grok-complex-patterns-with-logstash-7-1/218480 "2020-02-09T20:24:45Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![keshavl](https://avatars.discourse-cdn.com/v4/letter/k/5f8ce5/32.png) [@keshavl](https://discuss.elastic.co/u/keshavl)\
**Post date:** [February 9, 2020, 8:24pm UTC](https://discuss.elastic.co/t/how-to-grok-complex-patterns-with-logstash-7-1/218480/1 "2020-02-09T20:24:45Z")

</div>

HI. I am aware that DATA and GREEDYDATA are expensive patterns but those are most simple to use. I am using following pattern to extract data out of JMS XML . I am pasting both Input as well as Pattern which I have configured . Please Help me to build optimum pattern. Tried all possible patterns to reduce use of DATA and GREEDYDATA but nothing worked.

Input :-  
####\<04-Feb-2020 18:20:07 o'clock GMT\> \<\> \<1580840407665\> \<929588\> \<ID:\<360433.1580840407631.0\>\> \<\> \<com.my.capact.dl.jms.ipub.jmsmodule.NIAB-NPMD-DL-01!com.my.capact.dl.jms.ipub.mli.notification.NIAB-NPMD-DL-01\> \<\> \<\> \<\<?xml version="1.0" encoding="UTF-8"?\>\<mes:WLJMSMessage xmlns:mes="[http://www.bea.com/WLS/JMS/Message](http://www.bea.com/WLS/JMS/Message)"\>\<mes:Header\>\<mes:JMSTimestamp\>1580840407631\</mes:JMSTimestamp\>\<mes:Properties/\>\</mes:Header\>\<mes:Body\>\<mes:Text\>\<soapenv:Envelope xmlns:soapenv="[http://schemas.xmlsoap.org/soap/envelope/](http://schemas.xmlsoap.org/soap/envelope/)" xmlns:soapenc="[http://schemas.xmlsoap.org/soap/encoding/](http://schemas.xmlsoap.org/soap/encoding/)" xmlns:xsi="[http://www.w3.org/2001/XMLSchema-instance](http://www.w3.org/2001/XMLSchema-instance)" xmlns:xsd="[http://www.w3.org/2001/XMLSchema](http://www.w3.org/2001/XMLSchema)" xmlns:m="[http://capabilities.nat.my.com/xsd/ManageEventNotification/2006/12/30](http://capabilities.nat.my.com/xsd/ManageEventNotification/2006/12/30)" xmlns:m0="[http://wsi.nat.my.com/2005/06/StandardHeader/](http://wsi.nat.my.com/2005/06/StandardHeader/)" xmlns:m1="[http://capabilities.nat.my.com/xsd/ManageEventNotification/2006/12/30/CCM/Events](http://capabilities.nat.my.com/xsd/ManageEventNotification/2006/12/30/CCM/Events)"\>\<soapenv:Body\>\<m:notify\>\<m0:standardHeader\>\<m0:e2e\>\<m0:E2EDATA\>E2E.busTxnStage=NOT,E2E.compTxnName=P1,E2E.compTxnID=4p3zpoljmy,E2E.from=IPUB-ROmy,E2E.to=MQREP,E2E.graphID=1.1.1.1,E2E.threadID=4f38vq5djz,E2E.busProcType=notify,E2E.busProcOriginator=NMDB-MLI,E2E.threadID.1=:,E2E.busTxnType=MENNotifications,E2E.busTxnHdr=PCK002069,E2E.busTxnSys=NMDB\_MENNotifica,E2E.busTxnLoc=UNKNOWN,E2E.busTxnUsr=wbrkadm,E2E.busTxnSeq=4f38vq5co9\</m0:E2EDATA\>\</m0:e2e\>\<m0:serviceState\>\<m0:stateCode\>OK\</m0:stateCode\>\<m0:errorCode\>0\</m0:errorCode\>\<m0:errorDesc/\>\<m0:errorText\>0\</m0:errorText\>\</m0:serviceState\>\<m0:serviceAddressing\>\<m0:from\>[http://capabilities.nat.my.com/ManageEventNotification/2006/12/30](http://capabilities.nat.my.com/ManageEventNotification/2006/12/30)\</m0:from\>\<m0:to\>\<m0:address\>java://com.my.capact.dl.jms.ipub.mli.notification.NIAB-NPMD-DL-01@com.my.capact.dl.jms.ipub.cf.NIAB-NPMD-DL-01\</m0:address\>\</m0:to\>\<m0:messageId/\>\<m0:serviceName\>[http://capabilities.nat.my.com/ManageEventNotification/2006/12/30](http://capabilities.nat.my.com/ManageEventNotification/2006/12/30)\</m0:serviceName\>\<m0:action\>[http://capabilities.nat.my.com/ManageEventNotification/2006/12/30#notify](http://capabilities.nat.my.com/ManageEventNotification/2006/12/30#notify)\</m0:action\>\</m0:serviceAddressing\>\<m0:serviceSpecification\>\<m0:payloadFormat\>XML\</m0:payloadFormat\>\<m0:version\>1.0\</m0:version\>\<m0:revision/\>\</m0:serviceSpecification\>\</m0:standardHeader\>\<m:notificationMessage\>\<m1:laxTopicValidation\>false\</m1:laxTopicValidation\>\<m1:subscriptionReference\>\<m1:address\>uuid:f31a19f4-0a9d-11ea-9fc2-0abba92a0000\</m1:address\>\</m1:subscriptionReference\>\<m1:message\>\<mlidata:inventory xmlns:ssp1="[https://collaborate.my.com/svn/edm/ssp/trunk/UnstructuredAddress.xsd](https://collaborate.my.com/svn/edm/ssp/trunk/UnstructuredAddress.xsd)" xmlns:ssp="[https://collaborate.my.com/svn/edm/ssp/trunk/Address.xsd](https://collaborate.my.com/svn/edm/ssp/trunk/Address.xsd)" xmlns:adli="[https://collaborate.my.com/svn/edm/adli/IPAddress](https://collaborate.my.com/svn/edm/adli/IPAddress)" xmlns:xsi="[http://www.w3.org/2001/XMLSchema-instance](http://www.w3.org/2001/XMLSchema-instance)" xmlns:head="[http://wsi.nat.my.com/2005/06/StandardHeader/](http://wsi.nat.my.com/2005/06/StandardHeader/)" xmlns:mlidata="[http://collaborate.my.com/svn/sdkrepo/pbp/MLI/tags/8/ManageLogicalInventoryData/](http://collaborate.my.com/svn/sdkrepo/pbp/MLI/tags/8/ManageLogicalInventoryData/)" xsi:schemaLocation="[http://collaborate.my.com/svn/sdkrepo/pbp/MLI/tags/8/ManageLogicalInventoryData/](http://collaborate.my.com/svn/sdkrepo/pbp/MLI/tags/8/ManageLogicalInventoryData/) ManageLogicalInventoryData.xsd"\>\<head:standardHeader\>\<head:e2e\>\<head:E2EDATA/\>\</head:e2e\>\<head:serviceState\>\<head:stateCode\>OK\</head:stateCode\>\</head:serviceState\>\<head:serviceAddressing\>\<head:from\>[http://capabilities.nat.my.com/ManageLogicalInventory/APP06104](http://capabilities.nat.my.com/ManageLogicalInventory/APP06104)\</head:from\>\<head:to\>\<head:address\>[http://capabilities.nat.my.com/ManageLogicalInventory/NIAB-NPMD-DL-01](http://capabilities.nat.my.com/ManageLogicalInventory/NIAB-NPMD-DL-01)\</head:address\>\</head:to\>\<head:replyTo\>\<head:address\>[http://capabilities.nat.my.com/ManageLogicalInventory/APP06104](http://capabilities.nat.my.com/ManageLogicalInventory/APP06104)\</head:address\>\</head:replyTo\>\<head:messageId\>145498609\</head:messageId\>\<head:serviceName\>[http://capabilities.nat.my.com/ManageLogicalInventory](http://capabilities.nat.my.com/ManageLogicalInventory)\</head:serviceName\>\<head:action\>inventoryNotification\</head:action\>\</head:serviceAddressing\>\<head:serviceSpecification\>\<head:payloadFormat\>XML\</head:payloadFormat\>\<head:version\>8.0\</head:version\>\<head:revision/\>\</head:serviceSpecification\>\</head:standardHeader\>\<mlidata:logicalInventory messageType="Notification"\>\<mlidata:numberOfRecords\>1\</mlidata:numberOfRecords\>\<mlidata:startIndex\>1\</mlidata:startIndex\>\<mlidata:managementDomains\>\<mlidata:managementDomain\>\<mlidata:name\>\<mlidata:rdn\>\<mlidata:type\>AID\</mlidata:type\>\<mlidata:value/\>\</mlidata:rdn\>\</mlidata:name\>\<mlidata:id/\>\<mlidata:managedElements\>\<mlidata:managedElement\>\<mlidata:id\>1064145334\</mlidata:id\>\<mlidata:aliasNames\>\<mlidata:aliasName\>\<mlidata:key\>BFGDeviceId\</mlidata:key\>\<mlidata:value\>19987711\</mlidata:value\>\</mlidata:aliasName\>\<mlidata:aliasName\>\<mlidata:key\>NetworkName\</mlidata:key\>\<mlidata:value\>EU-Switzerland\</mlidata:value\>\</mlidata:aliasName\>\<mlidata:aliasName\>\<mlidata:key\>SysObjectId\</mlidata:key\>\<mlidata:value/\>\</mlidata:aliasName\>\</mlidata:aliasNames\>\<mlidata:userLabel/\>\<mlidata:owner\>NMDB\</mlidata:owner\>\<mlidata:namingOS/\>\<mlidata:source\>NETWORK\_EMS\</mlidata:source\>\<mlidata:resourceState\>INSTALLED\</mlidata:resourceState\>\<mlidata:resourceDates\>\<mlidata:resourceDate\>\<mlidata:dateType\>ACTUAL\_DATE\</mlidata:dateType\>\<mlidata:resourceState\>INSTALLED\</mlidata:resourceState\>\<mlidata:dateTime\>2018-01-23T15:29:48.0Z\</mlidata:dateTime\>\</mlidata:resourceDate\>\<mlidata:resourceDate\>\<mlidata:dateType\>LAST\_MODIFIED\_DATE\</mlidata:dateType\>\<mlidata:resourceState\>INSTALLED\</mlidata:resourceState\>\<mlidata:dateTime\>2020-02-04T18:16:53.0Z\</mlidata:dateTime\>\</mlidata:resourceDate\>\</mlidata:resourceDates\>\<mlidata:operationalState\>Y\</mlidata:operationalState\>\<mlidata:probes/\>\<mlidata:logicalLocation\>\<mlidata:siteId\>3681373\</mlidata:siteId\>\<mlidata:name\>EU-CH-ZUTH\</mlidata:name\>\<mlidata:addressRef\>\<ssp:addressKey/\>\<ssp:street/\>\<ssp:city/\>\<ssp:countyStateProvince/\>\<ssp:country/\>\<ssp:postCode/\>\<ssp:dateTimeCreated/\>lt;/mlidata:additionalInfo\>\<mlidata:additionalInfo\>\<mlidata:key\>REPORT\_SERVICE\_LEVEL\</mlidata:key\>\<mlidata:value/\>\</mlidata:additionalInfo\>\<mlidata:additionalInfo\>\<mlidata:key\>DeviceCategory\</mlidata:key\>\<mlidata:value\>Managed\</mlidata:value\>\</mlidata:additionalInfo\>\<mlidata:additionalInfo\>\<mlidata:key\>DeviceStatus\</mlidata:key\>\<mlidata:value\>In Service\</mlidata:value\>\</mlidata:additionalInfo\>\<mlidata:additionalInfo\>\<mlidata:key\>SerialNumber\</mlidata:key\>mlidata:value/\>\</mlidata:additionalInfo\>\<mlidata:additionalInfo\>\<mlidata:key\>NTN\_SSV\_ID\</mlidata:key\>\<mlidata:value\>2949540\</mlidata:value\>\</mlidata:additionalInfo\>\<mlidata:additionalInfo\>\<mlidata:key\>CEASED\_DATE\</mlidata:key\>\<mlidata:value/\>\</mlidata:additionalInfo\>\<mlidata:additionalInfo\>\<mlidata:key\>CollectedSerialNumber\</mlidata:key\>\<mlidata:value/\>\</mlidata:additionalInfo\>\<mlidata:additionalInfo\>\<mlidata:key\>HsrpIpAddress\</mlidata:key\>\<mlidata:value/\>\</mlidata:additionalInfo\>\<mlidata:additionalInfo\>\<mlidata:key\>CustomerHostName/mlidata:additionalInfo\>\</mlidata:meVendorExtensions\>\<mlidata:managementIPAddresses\>\<mlidata:ipAddress\>\<adli:ipName/\>\<adli:ipValue/\>\<adli:dnsName\>abc-ch-zuth-as01\</adli:dnsName\>\<adli:dnsNameAlias/\>\<adli:subnetworkMask/\>\<adli:ippool/\>\<adli:ipType\>Management\</adli:ipType\>\</mlidata:ipAddress\>\<mlidata:ipAddress\>\<adli:ipName/\>\<adli:ipValue\>10.20.30.40\</adli:ipValue\>\<adli:dnsName/\>\<adli:dnsNameAlias/\>\<adli:subnetworkMask/\>\<adli:ippool/\>\<adli:ipType\>Customer\</adli:ipType\>\</mlidata:ipAddress\>\<mlidata:ipAddress\>\<adli:ipName/\>\<adli:ipValue/\>\<adli:dnsName/\>\<adli:dnsNameAlias/\>\<adli:subnetworkMask/\>\<adli:ippool/\>\<adli:ipType\>CustomerIPv6\</adli:ipType\>\</mlidata:ipAddress\>\</mlidata:managementIPAddresses\>\</mlidata:managedElement\>\</mlidata:managedElements\>\</mlidata:managementDomain\>\</mlidata:managementDomains\>\</mlidata:logicalInventory\>\</mlidata:inventory\>\</m1:message\>\<m1:topic\>\<m1:name\>[http://itprogrammes.intra.my.com/topics:/my\_GS/MLI/Device](http://itprogrammes.intra.my.com/topics:/my_GS/MLI/Device)\</m1:name\>\</m1:topic\>\<m1:defaultPersistence\>true\</m1:defaultPersistence\>\<m1:creationTime\>v\_soap\_date1\</m1:creationTime\>\<m1:timeToLive xsi:nil="true"/\>\<m1:producerReference\>\<m1:address\>NMDB\</m1:address\>\</m1:producerReference\>\</m:notificationMessage\>\</m:notify\>\</soapenv:Body\>\</soapenv:Envelope\>\</mes:Text\>\</mes:Body\>\</mes:WLJMSMessage\>\> \<\>

Pattern I have created is :-  
\<%{DATA:DateTime}\>%{SPACE:thrash}\<%{DATA:thrash}ID:\<%{DATA:jms\_message\_id}\>%{DATA:thrash}!%{DATA:jms\_destination}\> \<%{DATA:jms\_message\_status}\>%{DATA:thrash}soapenv%{DATA:thrash}mlidata:inventory%{DATA:thrash}ManageLogicalInventoryData.xsd%{DATA:thrash}messageId\>%{DATA:mliMessageId}&lt%{DATA:thrash}head:serviceSpecification%{DATA:thrash}mlidata:numberOfRecords%{DATA:thrash}mlidata:managedElements%{DATA:thrash}BFGDeviceId%{DATA:thrash}value\>%{DATA:mliDeviceBFGId}&lt%{DATA:thrash}SysObjectId%{DATA:thrash}NETWORK\_EMS%{DATA:thrash}LAST\_MODIFIED\_DATE%{DATA:thrash}mlidata:logicalLocation%{DATA:thrash}mlidata:meVendorExtensions%{DATA:thrash}BFGProductType%{DATA:thrash}CustomerName%{DATA:thrash}mlidata:value\>%{DATA:mliDeviceCustomerName}&lt%{DATA:thrash}REPORT\_SERVICE\_LEVEL%{DATA:thrash}DeviceStatus%{DATA:thrash}value\>%{DATA:mliDeviceFlowStatus}&lt%{DATA:thrash}NTN\_SSV\_ID%{DATA:thrash}CustomerHostName%{DATA:thrash}mlidata:managementIPAddresses%{DATA:thrash}dnsName\>%{DATA:mliDevice\_Hostname}&lt%{DATA:thrash}ipAddress%{DATA:thrash}ipAddress\>\<adli:ipName%{DATA:thrash}ipValue\>%{DATA:mliDeviceIPAddress}&lt

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 10, 2020, 2:02am UTC](https://discuss.elastic.co/t/how-to-grok-complex-patterns-with-logstash-7-1/218480/2 "2020-02-10T02:02:38Z")

</div>

You really need to learn enough [markdown](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet) to format your posts if you expect folks to read them. Specifically [code quoting](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet#code). A few minutes work that will pay many dividends.

---

<div class="post-metadata">

**Author:** ![keshavl](https://avatars.discourse-cdn.com/v4/letter/k/5f8ce5/32.png) [@keshavl](https://discuss.elastic.co/u/keshavl)\
**Post date:** [February 10, 2020, 8:37pm UTC](https://discuss.elastic.co/t/how-to-grok-complex-patterns-with-logstash-7-1/218480/3 "2020-02-10T20:37:19Z")

</div>

Oh yes .. just wanted to be more elaborative so that real problem understood. Hope some-one helps me on above.

---

<div class="post-metadata">

**Author:** ![keshavl](https://avatars.discourse-cdn.com/v4/letter/k/5f8ce5/32.png) [@keshavl](https://discuss.elastic.co/u/keshavl)\
**Post date:** [February 11, 2020, 7:49am UTC](https://discuss.elastic.co/t/how-to-grok-complex-patterns-with-logstash-7-1/218480/4 "2020-02-11T07:49:39Z")

</div>

HI,  
Will some one please help me on this. I am trying to figure out appropriate grok pattern. Logstash throws below message

Timeout executing grok '\<%{DATA:DateTime}\>%{SPACE:thrash}\<%{DATA:thrash}ID:\<%{DATA:jms\_message\_id}\>%{DATA:thrash}!%{DATA:jms\_destination}\> \<%{DATA:jms\_message\_status}\>%{DATA:thrash}soapenv%{DATA:thrash}mlidata:inventory%{DATA:thrash}ManageLogicalInventoryData.xsd%{DATA:thrash}messageId\>%{DATA:mliMessageId}&lt%{DATA:thrash}head:serviceSpecification%{DATA:thrash}mlidata:numberOfRecords%{DATA:thrash}mlidata:managedElements%{DATA:thrash}BFGDeviceId%{DATA:thrash}value\>%{DATA:mliDeviceBFGId}&lt%{DATA:thrash}SysObjectId%{DATA:thrash}NETWORK\_EMS%{DATA:thrash}LAST\_MODIFIED\_DATE%{DATA:thrash}mlidata:logicalLocation%{DATA:thrash}mlidata:meVendorExtensions%{DATA:thrash}BFGProductType%{DATA:thrash}CustomerName%{DATA:thrash}mlidata:value\>%{DATA:mliDeviceCustomerName}&lt%{DATA:thrash}REPORT\_SERVICE\_LEVEL%{DATA:thrash}DeviceStatus%{DATA:thrash}value\>%{DATA:mliDeviceFlowStatus}&lt%{DATA:thrash}NTN\_SSV\_ID%{DATA:thrash}CustomerHostName%{DATA:thrash}mlidata:managementIPAddresses%{DATA:thrash}dnsName\>%{DATA:mliDevice\_Hostname}&lt%{DATA:thrash}ipAddress%{DATA:thrash}ipAddress\>\<adli:ipName%{DATA:thrash}ipValue\>%{DATA:mliDeviceIPAddress}&lt' **against field 'message' with value 'Value too large to output (13725 bytes)! First 255 chars are**: ####\<11-Feb-2020 07:45:10 o'clock GMT\> \<\> \<1581407110949\> \<562039\> \<ID:\<360433.1581407110917.0\>\> \<\> \<com

---

<div class="post-metadata">

**Author:** ![keshavl](https://avatars.discourse-cdn.com/v4/letter/k/5f8ce5/32.png) [@keshavl](https://discuss.elastic.co/u/keshavl)\
**Post date:** [February 11, 2020, 8:52pm UTC](https://discuss.elastic.co/t/how-to-grok-complex-patterns-with-logstash-7-1/218480/5 "2020-02-11T20:52:33Z")

</div>

HI,  
I replaced **DATA with regex (.\*?)** to reach the next token as fast as Grok can, but logstash gives me error **Value too large to output (15440 bytes)! First 255 chars are**  
Please help me to configure right grok parser.

---

<div class="post-metadata">

**Author:** ![keshavl](https://avatars.discourse-cdn.com/v4/letter/k/5f8ce5/32.png) [@keshavl](https://discuss.elastic.co/u/keshavl)\
**Post date:** [February 24, 2020, 3:54pm UTC](https://discuss.elastic.co/t/how-to-grok-complex-patterns-with-logstash-7-1/218480/6 "2020-02-24T15:54:44Z")

</div>

I tried couple of things to avoid DATA elements.. seems some improvement but problem still persist. Not sure this forum is really helpful  
No point in tracking and expecting good suggestions here !!!!! For new guys who visit this portal .. i think try your own don't linger around this webpages .. they hardly are useful :frowning ☹

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [February 24, 2020, 5:47pm UTC](https://discuss.elastic.co/t/how-to-grok-complex-patterns-with-logstash-7-1/218480/7 "2020-02-24T17:47:34Z")

</div>

have you tried using the xml filter instead of grok?

If that is not an option the dissect filter might be a good fit too as you seem to have structured your grok based on delimiters.

---

<div class="post-metadata">

**Author:** ![keshavl](https://avatars.discourse-cdn.com/v4/letter/k/5f8ce5/32.png) [@keshavl](https://discuss.elastic.co/u/keshavl)\
**Post date:** [February 24, 2020, 8:09pm UTC](https://discuss.elastic.co/t/how-to-grok-complex-patterns-with-logstash-7-1/218480/8 "2020-02-24T20:09:35Z")

</div>

Thanks for suggestion Christian .. I haven't tried xml nor dissect filter. To be honest, I was not aware of it. I will try, if you can provide some link to samples, I will refer to them.

Thanks  
Keshav

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [February 25, 2020, 5:28am UTC](https://discuss.elastic.co/t/how-to-grok-complex-patterns-with-logstash-7-1/218480/9 "2020-02-25T05:28:54Z")

</div>

I would recommend looking at the documentation as I do not have any examples as I almost never have to deal with XML data.

---

<div class="post-metadata">

**Author:** ![keshavl](https://avatars.discourse-cdn.com/v4/letter/k/5f8ce5/32.png) [@keshavl](https://discuss.elastic.co/u/keshavl)\
**Post date:** [February 25, 2020, 11:55pm UTC](https://discuss.elastic.co/t/how-to-grok-complex-patterns-with-logstash-7-1/218480/10 "2020-02-25T23:55:32Z")

</div>

HI Christain,  
After going through documentation I created my own pattern to dissect the message, I have also set "config.support\_escapes" to true, but no luck. PFB screenshot of the dissect message, surely something I am missing but unable to trace as there is no online debugger to verify the pattern of dissect. If you can guide me to right direction I will invetigate it further. I know the screenshot is too big to go through but it holds details of message as well as pattern

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/6/76af60582a4c644855a9199cd2971a211b24ab91.png)

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 26, 2020, 12:25am UTC](https://discuss.elastic.co/t/how-to-grok-complex-patterns-with-logstash-7-1/218480/11 "2020-02-26T00:25:26Z")

</div>

Your dissect pattern does not match the message. You have a very complicated pattern, so I would develop it one field at a time. Start by matching the first field on the line. Do not try to match anything more than that. Start a copy of logstash with

```auto
--config.reload.automatic

```

enabled. That way you only pay the startup cost once, and it will reload the configuration and reinvoke the pipeline each time you modify the configuration. Create a file that contains your log file line then start with

```auto
input { 
    file {
        path => "C:/some/path/log.txt"
        start_position => "beginning"
        sincedb_path => "nul"
    }
}
filter { dissect { mapping => { "message" => "####<%{DateTime}>%{}" } } }
output { stdout { codec => rubydebug } }

```

The trailing %{} in the pattern is needed to consume (and discard) the rest of the message. Once you see a good value for DateTime in the rubydebug output, add the next field to the pattern (or two if you feel lucky). Once you write out the configuration file from your editor logstash will notice a couple of seconds later and restart the pipeline and print out a rubydebug event with (hopefully) some more fields on the event. Repeat this as you perfect the pattern for each field.

It may sound like a lot of effort, but it is actually much quicker than staring at a complicated pattern that doesn't quite match. This way you only ever have to worry about extending the match by one field.

If you end up having to go back to grok then [this post](https://discuss.elastic.co/t/help-needed-in-grok/213827/2) describes a similar technique for grok.

---

<div class="post-metadata">

**Author:** ![keshavl](https://avatars.discourse-cdn.com/v4/letter/k/5f8ce5/32.png) [@keshavl](https://discuss.elastic.co/u/keshavl)\
**Post date:** [February 26, 2020, 9:09pm UTC](https://discuss.elastic.co/t/how-to-grok-complex-patterns-with-logstash-7-1/218480/12 "2020-02-26T21:09:28Z")

</div>

Thanks I will take a look into it and try pattern by pattern by adding one element each time.

---

<div class="post-metadata">

**Author:** ![keshavl](https://avatars.discourse-cdn.com/v4/letter/k/5f8ce5/32.png) [@keshavl](https://discuss.elastic.co/u/keshavl)\
**Post date:** [February 26, 2020, 11:34pm UTC](https://discuss.elastic.co/t/how-to-grok-complex-patterns-with-logstash-7-1/218480/13 "2020-02-26T23:34:52Z")

</div>

The way you have suggested have worked to form the Correct Dissect pattern. I have implemented it in production after testing it, will monitor it for couple of days and verify whether dissect is not throwing up any errors like grok did "Value too large to output"

Below is my final pattern which worked for 3 messages received in production so far.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/1/81bdc1ea68979dc1413fa6b46eaa0faf161dba04.png)

😄 thanks for all your suggestion so far

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 27, 2020, 1:03am UTC](https://discuss.elastic.co/t/how-to-grok-complex-patterns-with-logstash-7-1/218480/14 "2020-02-27T01:03:26Z")

</div>

Note that if you do not want to store a field you do not have to name it. You could replace %{thrash1} with %{} instead of doing the remove\_field.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 26, 2020, 1:03am UTC](https://discuss.elastic.co/t/how-to-grok-complex-patterns-with-logstash-7-1/218480/15 "2020-03-26T01:03:39Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.