# Grok positional structure problem

**URL:** <https://discuss.elastic.co/t/grok-positional-structure-problem/215475>\
**Category:** Logstash\
**Created:** [January 17, 2020, 2:03pm UTC](https://discuss.elastic.co/t/grok-positional-structure-problem/215475 "2020-01-17T14:03:48Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ahmad\_Moh\_d\_El\_Hasan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ahmad_moh_d_el_hasan/32/45171_2.png) [@Ahmad\_Moh\_d\_El\_Hasan](https://discuss.elastic.co/u/Ahmad_Moh_d_El_Hasan)\
**Post date:** [January 17, 2020, 2:03pm UTC](https://discuss.elastic.co/t/grok-positional-structure-problem/215475/1 "2020-01-17T14:03:48Z")

</div>

Dear sirs

I have a question about the grok plugin:  
I have some logs which are built in the following way and my problem is that the order of my fields can mutate from log to log (in the  
example here I should extract from all the logs both the "user id" and the "document id"). So I am wondering how I should build the grok  
pattern to handle both the logs hopeing there is a better solution of writing two different patterns (or how to write a regex to find  
a specific attribute anywhere in the log without knowing before its exact position)

2019-01-15 INFO myclass - mymethod: user id: 12345 custom message 1 document id: 843572309845  
2019-01-15 WARN myclass - mymethod1: document id: 43543534 custom message 2 user id: 98589348543 custom message 3 agent id: 98435734

the expected result is:

date: 2019-01-15  
level: INFO  
class: myclass  
method: mymethod  
message: user id: 12345 custom message 1 document id: 843572309845  
userId: 12345  
documentId: 843572309845

date: 2019-01-15  
level: WARN  
class: myclass  
method: mymethod1  
message: document id: 43543534 custom message 2 user id: 98589348543 custom message 3 agent id: 98435734  
userId: 98589348543  
documentId: 43543534  
agentId: 98589348543

Looking forward for your answer, thank you.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 17, 2020, 3:06pm UTC](https://discuss.elastic.co/t/grok-positional-structure-problem/215475/2 "2020-01-17T15:06:29Z")

</div>

You could try using

```
grok {
    break_on_match => false
    match => { "message" => ["document id: %{NUMBER:documentId}", "user id: %{NUMBER:userId}"] }
}
```

---

<div class="post-metadata">

**Author:** ![Fabio-sama](https://avatars.discourse-cdn.com/v4/letter/f/b9e5f3/32.png) [@Fabio-sama](https://discuss.elastic.co/u/Fabio-sama)\
**Post date:** [January 17, 2020, 3:18pm UTC](https://discuss.elastic.co/t/grok-positional-structure-problem/215475/3 "2020-01-17T15:18:20Z")

</div>

Are only user\_id and document\_id that can switch order? I mean, is it possible to have something like:

`[timestamp] ... agent_id: XXX ... document_id: YYY ... user_id: ZZZ` ?

So, for example, with agent\_id coming before everything else in the message part of the log?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 17, 2020, 4:04pm UTC](https://discuss.elastic.co/t/grok-positional-structure-problem/215475/4 "2020-01-17T16:04:07Z")

</div>

You could add

```
"agent id: %{NUMBER:agentId}"

```

to the array of patterns that grok matches against.

---

<div class="post-metadata">

**Author:** ![andres-perez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andres-perez/32/136461_2.png) [@andres-perez](https://discuss.elastic.co/u/andres-perez)\
**Post date:** [January 17, 2020, 4:25pm UTC](https://discuss.elastic.co/t/grok-positional-structure-problem/215475/5 "2020-01-17T16:25:18Z")

</div>

The trick here is that, as [break\_on\_match](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#plugins-filters-grok-break_on_match) is set to false, grok won't stop and it will compare the contents of message with every pattern.

Each element of the pattern array will be tested independently against the entire message field, starting from the beginning.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 14, 2020, 4:25pm UTC](https://discuss.elastic.co/t/grok-positional-structure-problem/215475/6 "2020-02-14T16:25:21Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.