Grok Patterns for Application logs

Elastic Stack Logstash
1

I am newbie to ELK and need to integrate my logs as below to elasticsearch from logstash.

Can some help me by providing log patterrns for below log format

#Filelog
2017-06-05 00:03:03 INFO HeartBeatDetailsTimerTask:94 - Live Integration List is empty
2017-06-05 00:03:03 INFO HeartBeatDetailsTimerTask:96 - HeartBeatDetails end
2017-06-05 00:07:03 INFO HeartBeatDetailsTimerTask:94 - Live Integration List is empty
2017-06-05 00:07:03 INFO HeartBeatDetailsTimerTask:96 - HeartBeatDetails end

#webservicelog
2017-06-02 12:36:47 INFO PortalUpload- Upload Medical Record webservice invoked from partner Id medall
2017-06-02 12:36:48 DEBUG PortalUpload- ROR response Json:{"ref_id":351883,"original_file_path":"files/13701_351883_1496387026446.jpg","image_path":"preview/13701_351883_1496387026446.jpg","thumbnail_file_path":"thumbnails/13701_351883_1496387026446_thumbnail.jpg","healthhubId":"LJWA-9923","status":"success"}
2017-06-02 12:36:48 INFO DatabaseConnection- Get Mongo DB Database connection...

#Emaillog
2017-06-03 00:03:03 INFO HeartBeatDetailsTimerTask:94 - Live Integration List is empty
2017-06-03 00:03:03 INFO HeartBeatDetailsTimerTask:96 - HeartBeatDetails end
2017-06-03 00:07:03 INFO HeartBeatDetailsTimerTask:94 - Live Integration List is empty

Thanks :slight_smile:

2

You can use the grok constructor site as a guide to creating grok expressions that match a particular line of input.

3

2017-06-05 00:03:03 INFO HeartBeatDetailsTimerTask:94 - Live Integration List is empty

Is this correct for above input

grok pattern= %{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:logLevel}%{SPACE}%{GREEDYDATA:message}

4

That looks okay, although I'd extract HeartBeatDetailsTimerTask:94 to separate fields too. Don't forget to set the overwrite option so that you can overwrite the existing message value.

5

mean

grok {
match => [ %{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:logLevel}%{SPACE}%{GREEDYDATA:message} ]
overwrite => [ "message" ]
}

I didn't get "extract HeartBeatDetailsTimerTask:94 to separate fields too" Can you please elaborate in details or with examples

Thanks,

6

mean

Yes.

I didn't get "extract HeartBeatDetailsTimerTask:94 to separate fields too" Can you please elaborate in details or with examples

In addition to the timestamp, loglevel, and message fields I'd want to have one field with "HeartBeatDetailsTimerTask" and one field with "94".

7

For

2017-06-02 12:36:47 INFO PortalUpload - Upload Medical Record webservice invoked from partner Id medall

> \A%{TIMESTAMP_ISO8601:timestamp}\s+%{LOGLEVEL:loglevel}\s+(?<logger>(?:-[a-zA-Z0-9-]+\.)*[A-Za-z0-9$]+)\s

Is working fine.

But Original Log is

2017-06-02 12:36:47 INFO PortalUpload- Upload Medical Record webservice invoked from partner Id medall

note: PortalUpload- no space between eiphens and after space message starts, in that case how to correct my above grok filters

8

Use ? to indicate that the preceding token is optional, i.e. foo ?bar matches both "foo bar" and "foobar".

9

Can you please correct my grok filter for expected original logs format

2017-06-02 12:36:47 INFO PortalUpload- Upload Medical Record webservice invoked from partner Id medall

10

Is this below correct

filter{

grok {
  match => [ "message", %{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel} %{GREEDYDATA:message} (?<logger>(?:[a-zA-Z0-9]+\.)*[-A-Za-z0-9$]+)
  overwrite => [ "message" ]

}

}

11

Does it work and give the expected results? Then it's probably correct.

Does the logger really come after the message?

12 
grok {
     match => [ "message", %{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel}  (?<logger>(?:[a-zA-Z0-9]+\.)*[-A-Za-z0-9$]+) %{GREEDYDATA:message}
   overwrite => [ "message" ]

No Message comes last, reordered,

I am using 5.4 version ELK, where throwing error as below

 bin/logstash -f /etc/logstash/conf.d/filelog.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default config which logs to console
19:04:25.883 [LogStash::Runner] FATAL logstash.runner - Logstash could not be started because there is already another instance using the configured data directory.  If you wish to run multiple instances, you must change the "path.data" setting.

I have set values in elasticsearch.yml

 path.data: /var/lib/logstash
 path.config: /etc/logstash/conf.d

Still throwing error also tried

    bin/logstash --path.config /etc/logstash/conf.d/filelog.conf
    WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
    Could not find log4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default config which logs to console
    19:08:53.683 [LogStash::Runner] FATAL logstash.runner - Logstash could not be started because there is already 
another instance using the configured data directory.  If you wish to run multiple instances, you must change the "path.data" setting.

Let me know how to know proceed further

Thanks,
Pandiyan

13

I have set values in elasticsearch.yml

elasticsearch.yml?

You can override the data path setting with the --path.data command line option. If you run Logstash as a service but also want to run it interactively like in this case this is a reasonable workaround.

14

It was logstash.yml, typo error :frowning:

bin/logstash --path.config /etc/logstash/conf.d/filelog.conf
got same error

15

Got Error as below

 bin/logstash --path.config /etc/logstash/conf.d/filelog.conf
    WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
    Could not find log4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default config which logs to console
    12:36:59.314 [LogStash::Runner] ERROR logstash.agent - Cannot create pipeline {:reason=>"Expected one of #, \", ', -, [, { at line 11, column 29 (byte 167) after filter{\n    \n    grok {      \n      match => [ \"message\", "}

Below is the file.conf file

input {
  file {
    path => "/var/log/file.log"
    codec => json
    start_position => "beginning"
   }
}
filter{

    grok {
      match => [ "message", %{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel} (?<logger>(?:[a-zA-Z0-9]+\.)*[-A-Za-z0-9$]+) %{GREEDYDATA:message}
      overwrite => [ "message" ]
  }

}
output {
        elasticsearch {
                codec => rubydebug
                host => ['localhost:9200']
                index => 'filelogs-%{+YYYY.MM.dd}'
        }
}
16

As I think I've said in another thread already you need to surround your grok expression with quotes.

17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.