# Help with Grok & Regexp: effective reuse of patterns

**URL:** <https://discuss.elastic.co/t/help-with-grok-regexp-effective-reuse-of-patterns/51175>\
**Category:** Logstash\
**Created:** [May 27, 2016, 4:11pm UTC](https://discuss.elastic.co/t/help-with-grok-regexp-effective-reuse-of-patterns/51175 "2016-05-27T16:11:17Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Bruno\_Lavoie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bruno_lavoie/32/8408_2.png) [@Bruno\_Lavoie](https://discuss.elastic.co/u/Bruno_Lavoie)\
**Post date:** [May 27, 2016, 4:11pm UTC](https://discuss.elastic.co/t/help-with-grok-regexp-effective-reuse-of-patterns/51175/1 "2016-05-27T16:11:17Z")

</div>

Hello,

I'm trying to extract fields from messages and making some reuse of information tokens but sometimes my regexps are too greedy and capture outside the end tag.

I have a few different type of messages but some tokens are reappearing (or not) between type of messages and in different order.

Tokens are formatted as : PropertyName={property value}

Here an example of 2 messages:

```
INFO com.l7tech.server.policy.assertion.ServerAuditDetailAssertion: -4: Message={Ouverture session}, MessageType={TypeMessageAuthentification}, Username={etud1}, SessionAuthorization={5af2143d-7368-46de-955e-2a014bd7d39f}, SessionClient={ZxFTldw1Ed242t1L7dML8cIgkM}, RequestId={00000154e84e6393-11147}, ServiceName={auth/oauth/v2/authorize}, ServiceId={7aea4881665af7743edf0dcb0d8ddfef}, ServiceGuid={304e9de3-ba27-4260-b448-e3476530a0c2}, ServiceVersion={54}, ClusterNodeName={Gateway1}
INFO com.l7tech.server.policy.assertion.ServerAuditDetailAssertion: -4: Message={Fermeture session}, MessageType={TypeMessageAuthentification}, SessionClient={p051Q3ZdhoCaNo9ASMp11uEhXHU}, RequestId={00000154e84e6393-1100b}, ServiceName={UL Page Logout}, ServiceId={7aea4881665af7743edf0dcb0d924d36}, ServiceGuid={b931e09d-9eaf-4e43-a14a-120815141b5d}, ServiceVersion={59}, ClusterNodeName={Gateway1}

```

To make it easily readable and fully reusable I have made a pattern for each recurring tokens:

```
################## 
# General pattern, identical beginning part of each messages

UL_GSA_BASE %{LOGLEVEL}%{SPACE}%{JAVACLASS}: %{INT}:
UL_GSA_COMMON %{LOGLEVEL}%{SPACE}%{JAVACLASS}: %{INT}: %{UL_GSA_MESSAGE}, %{UL_GSA_MESSAGE_TYPE}
UL_GSA_COMMON_REMAINING %{UL_GSA_COMMON}, %{GREEDYDATA:remaining}

##################
# Reusable patterns 

UL_GSA_MESSAGE Message={(?<message>.*?)}
UL_GSA_MESSAGE_TYPE MessageType={(?<message_type>.*?)}

UL_GSA_USERNAME Username={(?<username>.*?)}

UL_GSA_SESSION_CLIENT SessionClient={(?<session_client>.*?)}
UL_GSA_SESSION_AUTHORIZATION SessionAuthorization={(?<session_authorization>.*?)}

UL_GSA_SSO SSO={(?<sso>.*?)}
UL_GSA_REQUEST_ID RequestId={(?<request_id>.*?)}
UL_GSA_API_KEY APIKey={(?<api_key>.*?)}
UL_GSA_IP_ADDRESSES IpAddresses={(?<api_adresses>.*?)}

UL_GSA_RESPONSE_CODE ResponseCode={(?<response_code>.*?)}
UL_GSA_RESPONSE_LATENCY ResponseLatency={(?<response_latency>.*?)}
UL_GSA_RESPONSE_ERROR ResponseError={(?<response_error>.*?)}

UL_GSA_SERVICE_ID ServiceId={(?<service_id>.*?)}
UL_GSA_SERVICE_NAME ServiceName={(?<service_name>.*?)}
UL_GSA_SERVICE_GUID ServiceGuid={(?<service_guid>.*?)}
UL_GSA_SERVICE_VERSION ServiceVersion={(?<service_version>.*?)}

UL_GSA_GATEWAY_HOST GatewayHost={(?<gateway_host>.*?)}
UL_GSA_GATEWAY_SERVICE_URL GatewayServiceUrl={(?<gateway_service_url>.*?)}
UL_GSA_CLUSTER_NODE_NAME ClusterNodeName={(?<cluster_node_name>.*?)}

```

Then I construct final patterns by reusing the needed parts in the good order:

```
################## 
# Specific patterns

UL_GSA_OUVERTURE_SESSION %{UL_GSA_COMMON}, %{UL_GSA_USERNAME}, %{UL_GSA_SESSION_AUTHORIZATION}, %{UL_GSA_SESSION_CLIENT}, %{UL_GSA_REQUEST_ID}, %{UL_GSA_SERVICE_NAME}, %{UL_GSA_SERVICE_ID}, %{UL_GSA_SERVICE_GUID}, %{UL_GSA_SERVICE_VERSION}, %{UL_GSA_CLUSTER_NODE_NAME}
UL_GSA_FERMETURE_SESSION %{UL_GSA_COMMON}, %{UL_GSA_SESSION_CLIENT}, %{UL_GSA_REQUEST_ID}, %{UL_GSA_SERVICE_NAME}, %{UL_GSA_SERVICE_ID}, %{UL_GSA_SERVICE_GUID}, %{UL_GSA_SERVICE_VERSION}, %{UL_GSA_CLUSTER_NODE_NAME}

```

Using [http://grokdebug.herokuapp.com/](http://grokdebug.herokuapp.com/) site, testing each message type individually with corresponding pattern works fine.

But when using

`INFO com.l7tech.server.policy.assertion.ServerAuditDetailAssertion: -4: Message={Ouverture session}, MessageType={TypeMessageAuthentification}, Username={etud1}, SessionAuthorization={5af2143d-7368-46de-955e-2a014bd7d39f}, SessionClient={ZxFTldw1Ed242t1L7dML8cIgkM}, RequestId={00000154e84e6393-11147}, ServiceName={auth/oauth/v2/authorize}, ServiceId={7aea4881665af7743edf0dcb0d8ddfef}, ServiceGuid={304e9de3-ba27-4260-b448-e3476530a0c2}, ServiceVersion={54}, ClusterNodeName={Gateway1}`

with the non-corresponding pattern %{UL\_GSA\_FERMETURE\_SESSION} my UL\_GSA\_MESSAGE\_TYPE pattern seems to be too greedy and captures to much, while I would like matching to fail.

Captured message\_type field then looks like this:

```
{
  "message": [
    [
      "Ouverture session"
    ]
  ],
  "message_type": [
    [
      "TypeMessageAuthentification}, Username={etud1}, SessionAuthorization={5af2143d-7368-46de-955e-2a014bd7d39f"
    ]
  ],
...

```

So, I tried a lot of thing... ☹

How to make my capture strictly between one pair of curly braces?

Thanks a lot

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [May 29, 2016, 10:50am UTC](https://discuss.elastic.co/t/help-with-grok-regexp-effective-reuse-of-patterns/51175/2 "2016-05-29T10:50:26Z")

</div>

Try this:

```
UL_GSA_MESSAGE Message={(?<message>[^}]+)}

```

But primarily I'd look into the kv filter. Its specific purpose is to parse data like this.

---

<div class="post-metadata">

**Author:** ![Bruno\_Lavoie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bruno_lavoie/32/8408_2.png) [@Bruno\_Lavoie](https://discuss.elastic.co/u/Bruno_Lavoie)\
**Post date:** [May 30, 2016, 5:07pm UTC](https://discuss.elastic.co/t/help-with-grok-regexp-effective-reuse-of-patterns/51175/3 "2016-05-30T17:07:19Z")

</div>

Shame on me!  
Never thought of kv filter.  
Makes my stuff more resilient to changed and new messages.  
Juste keeping my patterns with your regex suggestion, just in case of.

Thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:55am UTC](https://discuss.elastic.co/t/help-with-grok-regexp-effective-reuse-of-patterns/51175/4 "2017-07-06T04:55:28Z")

</div>