# I need to parse this log message format

**URL:** <https://discuss.elastic.co/t/i-need-to-parse-this-log-message-format/170765>\
**Category:** Logstash\
**Created:** [March 4, 2019, 4:58pm UTC](https://discuss.elastic.co/t/i-need-to-parse-this-log-message-format/170765 "2019-03-04T16:58:18Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tanya\_Sharma](https://avatars.discourse-cdn.com/v4/letter/t/a88e57/32.png) [@Tanya\_Sharma](https://discuss.elastic.co/u/Tanya_Sharma)\
**Post date:** [March 4, 2019, 4:58pm UTC](https://discuss.elastic.co/t/i-need-to-parse-this-log-message-format/170765/1 "2019-03-04T16:58:18Z")

</div>

Hello everyone,

The message format of the logs is:  
\< [2019-03-04T12:29:49,990][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600} /\>

I have to skip the square brackets and parse this message.  
I tried this  
\< filter {  
grok{  
match =\> { "message" =\> "[%{TIMESTAMP\_ISO8601:logdate}[%{LOGLEVEL:LEVEL}][%{GREEDYDATA:errormsg}]" }  
}  
}  
/\>  
but its not working.  
I debugged it and it is parsing only the date. but [info] is not getting parsed because of the brackets maybe. There's no such familiar patterns in any of the answers.

Please help!

---

<div class="post-metadata">

**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\
**Post date:** [March 4, 2019, 5:09pm UTC](https://discuss.elastic.co/t/i-need-to-parse-this-log-message-format/170765/2 "2019-03-04T17:09:53Z")

</div>

Do you use Filebeat? Then you could use [https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-logstash.html](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-logstash.html)

For the Grok pattern, it looks like there is a space after INFO which I do not see in your pattern...

---

<div class="post-metadata">

**Author:** ![Tanya\_Sharma](https://avatars.discourse-cdn.com/v4/letter/t/a88e57/32.png) [@Tanya\_Sharma](https://discuss.elastic.co/u/Tanya_Sharma)\
**Post date:** [March 4, 2019, 5:18pm UTC](https://discuss.elastic.co/t/i-need-to-parse-this-log-message-format/170765/3 "2019-03-04T17:18:14Z")

</div>

No, i use graylog .. I mean i have to filter some logs and the logs are in the pattern mentioned in my post.

---

<div class="post-metadata">

**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\
**Post date:** [March 4, 2019, 5:19pm UTC](https://discuss.elastic.co/t/i-need-to-parse-this-log-message-format/170765/4 "2019-03-04T17:19:15Z")

</div>

Try something like

```
filter {
  grok{
    match => { "message" => "\<%{SPACE}\[%{TIMESTAMP_ISO8601:logdate}\]\[%{LOGLEVEL:LEVEL}%{SPACE}\]\[%{GREEDYDATA:foo}%{SPACE}\]%{SPACE}%{GREEDYDATA:errormsg} \/\>$" }
  }
}
```

---

<div class="post-metadata">

**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\
**Post date:** [March 5, 2019, 10:02am UTC](https://discuss.elastic.co/t/i-need-to-parse-this-log-message-format/170765/5 "2019-03-05T10:02:35Z")

</div>

> [@Tanya\_Sharma](#):
>
> \< filter {  
> grok{  
> match =\> { "message" =\> "[%{TIMESTAMP\_ISO8601:logdate}[%{LOGLEVEL:LEVEL}][%{GREEDYDATA:errormsg}]" }  
> }  
> }  
> /\>

did you actually get Logstash to start with that config? Logstash is usually very picky about config file syntax (well, which program isn't). The special character `</>` need to be within a filer definition. You can see my example above 🙂

---

<div class="post-metadata">

**Author:** ![Ganesh2303](https://avatars.discourse-cdn.com/v4/letter/g/57b2e6/32.png) [@Ganesh2303](https://discuss.elastic.co/u/Ganesh2303)\
**Post date:** [March 5, 2019, 10:40am UTC](https://discuss.elastic.co/t/i-need-to-parse-this-log-message-format/170765/6 "2019-03-05T10:40:49Z")

</div>

> [@Tanya\_Sharma](#):
>
> grok{  
> match =\> { "message" =\> "[%{TIMESTAMP\_ISO8601:logdate}[%{LOGLEVEL:LEVEL}][%{GREEDYDATA:errormsg}]" }  
> }

Try this filter you will get the match and output

\< [%{TIMESTAMP\_ISO8601:date}][%{LOGLEVEL:LEVEL}]%{GREEDYDATA:errormsg}

---

<div class="post-metadata">

**Author:** ![Tanya\_Sharma](https://avatars.discourse-cdn.com/v4/letter/t/a88e57/32.png) [@Tanya\_Sharma](https://discuss.elastic.co/u/Tanya_Sharma)\
**Post date:** [March 5, 2019, 10:41am UTC](https://discuss.elastic.co/t/i-need-to-parse-this-log-message-format/170765/7 "2019-03-05T10:41:31Z")

</div>

I just started working on this so i haven't had any idea about the logs or how to parse them.  
Can you please tell me how do filter the logs that's already in JSON format because i used `type => json` in my input block of logstash.conf.

```
<{
            "repo" => "feature/3.0.2_DeviceServices",
           "build" => 2,
   "short_message" => "BUILD_FINISHED",
        "duration" => 829193,
        "@version" => "1",
         "message" => [
       [0] "Branch indexing",
       [].................../>

```

This is the format in which the logs appear.

---

<div class="post-metadata">

**Author:** ![Ganesh2303](https://avatars.discourse-cdn.com/v4/letter/g/57b2e6/32.png) [@Ganesh2303](https://discuss.elastic.co/u/Ganesh2303)\
**Post date:** [March 5, 2019, 10:47am UTC](https://discuss.elastic.co/t/i-need-to-parse-this-log-message-format/170765/8 "2019-03-05T10:47:51Z")

</div>

Just provide your full log with what all data you need to parse into ES

---

<div class="post-metadata">

**Author:** ![Tanya\_Sharma](https://avatars.discourse-cdn.com/v4/letter/t/a88e57/32.png) [@Tanya\_Sharma](https://discuss.elastic.co/u/Tanya_Sharma)\
**Post date:** [March 5, 2019, 11:05am UTC](https://discuss.elastic.co/t/i-need-to-parse-this-log-message-format/170765/10 "2019-03-05T11:05:20Z")

</div>

I need to send this data to graylog after filtering the errormsg. How to i do that?

---

<div class="post-metadata">

**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\
**Post date:** [March 5, 2019, 11:20am UTC](https://discuss.elastic.co/t/i-need-to-parse-this-log-message-format/170765/11 "2019-03-05T11:20:33Z")

</div>

I have not done that but sounds like you would have to use [https://www.elastic.co/guide/en/logstash/6.6/plugins-outputs-gelf.html](https://www.elastic.co/guide/en/logstash/6.6/plugins-outputs-gelf.html)

---

<div class="post-metadata">

**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\
**Post date:** [March 5, 2019, 11:29am UTC](https://discuss.elastic.co/t/i-need-to-parse-this-log-message-format/170765/13 "2019-03-05T11:29:40Z")

</div>

> [@Tanya\_Sharma](#):
>
> I just started working on this so i haven't had any idea about the logs or how to parse them.  
> Can you please tell me how do filter the logs that's already in JSON format because i used `type => json` in my input block of logstash.conf.
> 
> ```auto
> <{
> "repo" => "feature/3.0.2_DeviceServices",
> "build" => 2,
> "short_message" => "BUILD_FINISHED",
> "duration" => 829193,
> "@version" => "1",
> "message" => [
> [0] "Branch indexing",
> [].................../>
> 
> ```
> 
> This is the format in which the logs appear.

That does not look like the source is in JSON. Logstash does not convert log formats into JSON by using `type => json` it is just told to expect JOSN on the input so it knows how to tokenise/parse it correctly. Are you feeding Logstash JSON or the original format you posted (which is)?

```
< [2019-03-04T12:29:49,990][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600} />

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 2, 2019, 11:29am UTC](https://discuss.elastic.co/t/i-need-to-parse-this-log-message-format/170765/14 "2019-04-02T11:29:40Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.