# Parsing log message optional fields

**URL:** <https://discuss.elastic.co/t/parsing-log-message-optional-fields/195516>\
**Category:** Logstash\
**Created:** [August 16, 2019, 2:52pm UTC](https://discuss.elastic.co/t/parsing-log-message-optional-fields/195516 "2019-08-16T14:52:37Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![mahmoud\_shiri\_varami](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mahmoud_shiri_varami/32/52446_2.png) [@mahmoud\_shiri\_varami](https://discuss.elastic.co/u/mahmoud_shiri_varami)\
**Post date:** [August 16, 2019, 2:52pm UTC](https://discuss.elastic.co/t/parsing-log-message-optional-fields/195516/1 "2019-08-16T14:52:37Z")

</div>

I'm supporting an application that generates log messages that contain 65 fields. More than 40 fields are optional that means sometimes according to application logic, optional fields exits on log message and sometimes not. I'm using grok filter plugin but the problem is if an optional filed appears on the log message that not match to grok pattern, the log message will not filter and will be bypass by logstash. And also if I write a grok pattern for all of the states, then I should write more 40! grok filter that impossible. What's the best solution for filtering optional fields on log message?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 16, 2019, 3:15pm UTC](https://discuss.elastic.co/t/parsing-log-message-optional-fields/195516/2 "2019-08-16T15:15:09Z")

</div>

It would be a lot easier to answer this if you could give a couple of examples of the log messages.

---

<div class="post-metadata">

**Author:** ![mahmoud\_shiri\_varami](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mahmoud_shiri_varami/32/52446_2.png) [@mahmoud\_shiri\_varami](https://discuss.elastic.co/u/mahmoud_shiri_varami)\
**Post date:** [August 16, 2019, 3:38pm UTC](https://discuss.elastic.co/t/parsing-log-message-optional-fields/195516/3 "2019-08-16T15:38:54Z")

</div>

Here is log message format:

MANDATORY|MANDATORY|MANDATORY|optinal|optional|MANDATORY|optional|MANDATORY|MANDATORY|MANDATORY|optional|MANDATORY|MANDATORY|optional|optional|MANDATORY|optional|optional|optional|optional|optional|optional|optional|MANDATORY|optional|optional|MANDATORY|MANDATORY|MANDATORY|MANDATORY|MANDATORY|optional|optional|optional|MANDATORY|optional|optional|optional|optional|optional|MANDATORY|MANDATORY |optional|optional|optional|optional|optional|optional|optional|MANDATORY|optional|optional|optional|MANDATORY|optional|optional|optional|optional|optional|optional|optional|optional|optional|optional|

And here is log message example:

432757578889|544646998109|1|||FSFSS\_155793420754833334804|222222-1111-3333-a3699cddwaFFFFF0c11099|2019-07-15 20:00:07|654345424|d1ddfnk1a8f36566fefb9|FFFFDSAFESEFSEF|19545442|MSDFDSFSD||438434343|DSFDSF ||||||||hyygyiig|043|4340|DSFDSFWDF44455|de444dfs|15000000|54543340|51434314||33||DE|044|044|440||0|0|success ||35|53||1||eww|eGG|||0|75|1|1|2|86576786654||r444||5555|

The field descriptor is pipeline sign (|), and optional fields show . According to application logic, one or more that one optional filed will appear on log message and other optional filed that does not exist on log message will be shown as blank, for example below string show to optional filed that not exist in log message:

|||

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 16, 2019, 4:27pm UTC](https://discuss.elastic.co/t/parsing-log-message-optional-fields/195516/4 "2019-08-16T16:27:38Z")

</div>

Why not use a csv filter?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 13, 2019, 4:27pm UTC](https://discuss.elastic.co/t/parsing-log-message-optional-fields/195516/5 "2019-09-13T16:27:40Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.