# Text based log format, need help in parsing

**URL:** <https://discuss.elastic.co/t/text-based-log-format-need-help-in-parsing/182149>\
**Category:** Logstash\
**Created:** [May 22, 2019, 7:03am UTC](https://discuss.elastic.co/t/text-based-log-format-need-help-in-parsing/182149 "2019-05-22T07:03:30Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Abhilash\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abhilash_b/32/40270_2.png) [@Abhilash\_B](https://discuss.elastic.co/u/Abhilash_B)\
**Post date:** [May 22, 2019, 7:03am UTC](https://discuss.elastic.co/t/text-based-log-format-need-help-in-parsing/182149/1 "2019-05-22T07:03:30Z")

</div>

Hey All,

I have a log format which looks [like this](https://pastebin.com/nXXft2Fw).

I need to extract all the key value based pairs. My current configuration looks like:

```
filter{
if [message] =~ /^#/ {
  drop {}
} else if [message] =~ /^\s*$/ {
  drop {}
} else if [message] =~ /^[A-Za-z]+:/ {
  mutate { strip => ["message"] }
  ruby {
     init => '
            @@data = {}
        '
        code => '
            msg = event.get("message")
            matches = msg.scan(/^([A-Za-z]+):(.*)/)
            m = matches[0]
            @@data[m[0]] = m[1]
        '
    }
   drop{}
} else if [message] =~ /^[0-9]+/ {
    mutate { strip => ["message"] }
    grok { match => { "message" => '^[0-9]{1,}[.][+](?<key>[^:]+):\s*%{GREEDYDATA:value}' } }
    ruby {
        code => '
            @@data[key] = value
        '
    }
    drop{}
} else if [message] =~ /^Speed/ {
    ruby{
        code => '
            event.set("data", @@data)
        '
    }
} else {
    drop{}
}

```

}

This results in a ruby exception `[2019-05-22T12:11:00,169][ERROR][logstash.filters.ruby] Ruby exception occurred: undefined method`' for nil:NilClass  
`.

Any ideas or better solutions?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 22, 2019, 12:08pm UTC](https://discuss.elastic.co/t/text-based-log-format-need-help-in-parsing/182149/2 "2019-05-22T12:08:15Z")

</div>

> [@Abhilash\_B](#):
>
> This results in a ruby exception `[2019-05-22T12:11:00,169][ERROR][logstash.filters.ruby] Ruby exception occurred: undefined method for nil:NilClass

That will occur if you make an array index reference to a variable that is not an array.

---

<div class="post-metadata">

**Author:** ![Abhilash\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abhilash_b/32/40270_2.png) [@Abhilash\_B](https://discuss.elastic.co/u/Abhilash_B)\
**Post date:** [May 22, 2019, 12:28pm UTC](https://discuss.elastic.co/t/text-based-log-format-need-help-in-parsing/182149/3 "2019-05-22T12:28:11Z")

</div>

> [@Abhilash\_B](#):
>
> msg = event.get("message") matches = msg.scan(/^([A-Za-z]+):(.\*)/) m = matches[0] @@data[m[0]] = m[1]

```
msg = event.get("message")
        matches = msg.scan(/^([A-Za-z]+):(.*)/)
        m = matches[0]
        @@data[m[0]] = m[1]

```

The above code seems to be working fine when I run it separately. Also I removed

```
else if [message] =~ /^[0-9]+/ {
mutate { strip => ["message"] }
grok { match => { "message" => '^[0-9]{1,}[.][+](?<key>[^:]+):\s*%{GREEDYDATA:value}' } }
ruby {
    code => '
        @@data[key] = value
    '
}
drop{}

```

}  
from the filter, still I get the same issue.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 19, 2019, 12:28pm UTC](https://discuss.elastic.co/t/text-based-log-format-need-help-in-parsing/182149/4 "2019-06-19T12:28:13Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.