I need to parse this log message format

Elastic Stack Logstash
1

Hello everyone,

The message format of the logs is:
< [2019-03-04T12:29:49,990][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} />

I have to skip the square brackets and parse this message.
I tried this
< filter {
grok{
match => { "message" => "[%{TIMESTAMP_ISO8601:logdate}[%{LOGLEVEL:LEVEL}][%{GREEDYDATA:errormsg}]" }
}
}
/>
but its not working.
I debugged it and it is parsing only the date. but [info ] is not getting parsed because of the brackets maybe. There's no such familiar patterns in any of the answers.

Please help!

2

Do you use Filebeat? Then you could use https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-logstash.html

For the Grok pattern, it looks like there is a space after INFO which I do not see in your pattern...

3

No, i use graylog .. I mean i have to filter some logs and the logs are in the pattern mentioned in my post.

4

Try something like

filter {
  grok{
    match => { "message" => "\<%{SPACE}\[%{TIMESTAMP_ISO8601:logdate}\]\[%{LOGLEVEL:LEVEL}%{SPACE}\]\[%{GREEDYDATA:foo}%{SPACE}\]%{SPACE}%{GREEDYDATA:errormsg} \/\>$" }
  }
}
5

did you actually get Logstash to start with that config? Logstash is usually very picky about config file syntax (well, which program isn't). The special character </> need to be within a filer definition. You can see my example above :slight_smile:

6

Try this filter you will get the match and output

< [%{TIMESTAMP_ISO8601:date}][%{LOGLEVEL:LEVEL} ]%{GREEDYDATA:errormsg}

7

I just started working on this so i haven't had any idea about the logs or how to parse them.
Can you please tell me how do filter the logs that's already in JSON format because i used type => json in my input block of logstash.conf.

<{
            "repo" => "feature/3.0.2_DeviceServices",
           "build" => 2,
   "short_message" => "BUILD_FINISHED",
        "duration" => 829193,
        "@version" => "1",
         "message" => [
       [    0] "Branch indexing",
       [    ].................../>

This is the format in which the logs appear.

8

Just provide your full log with what all data you need to parse into ES

10

I need to send this data to graylog after filtering the errormsg. How to i do that?

11

I have not done that but sounds like you would have to use https://www.elastic.co/guide/en/logstash/6.6/plugins-outputs-gelf.html

13

That does not look like the source is in JSON. Logstash does not convert log formats into JSON by using type => json it is just told to expect JOSN on the input so it knows how to tokenise/parse it correctly. Are you feeding Logstash JSON or the original format you posted (which is)?

< [2019-03-04T12:29:49,990][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} />
14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.