What is the grok pattern for this?

[Yaswanth]

Hi,

Seems to be a easy question but i am quite new to this grok pattern.I have logs like this

{"@timestamp": "2017-04-24T07:03:07.985Z", "function": "cli", "linenum": 204, "loglevel": "INFO", "message": "Job completed.", "name": "curator.cli"}

i created grok pattern for every field like this and i also checked here

%{TIMESTAMP_ISO8601:TIMESTAMP}, %{GREEDYDATA:function},%{NUMBER:linenum},%{LOGLEVEL:LEVEL}, %{GREEDYDATA:message}, %{GREEDYDATA:name}

But when i am combining and passing it is throwing an error.

Please say whether my pattern is correct or not

Thanks

[magnusbaeck (Magnus Bäck)]

You don't need grok for just. Just use a json codec or filter to deserialize your JSON log.

[Yaswanth]

Thanks @magnusbaeck

I used the below config file

 input {
  file {
    path => "C:\Users\thunder\Desktop\curator.log"
    codec => json
    start_position => "beginning"
   
  }
}
filter{
    json{
        source => "message"
    }
}
output {
stdout{
codec => rubydebug
}
}

I am getting error like this;

←[33mError parsing json {:source=>"message", :raw=>"Preparing Action ID: 1, \"de
lete_indices\"", :exception=>#<LogStash::Json::ParserError: Unrecognized token '
Preparing': was expecting ('true', 'false' or 'null')
 at [Source: [B@aef59c; line: 1, column: 11]>, :level=>:warn}←[0m
←[33mError parsing json {:source=>"message", :raw=>"Trying Action ID: 1, \"delet
e_indices\": Delete indices older than 3 day (based on index name), for logstash
- prefixed indices. Ignore the error if the filter does not result in an actiona
ble list of indices (ignore_empty_list) and exit cleanly.", :exception=>#<LogSta
sh::Json::ParserError: Unrecognized token 'Trying': was expecting ('true', 'fals
e' or 'null')
 at [Source: [B@335638; line: 1, column: 8]>, :level=>:warn}←[0m
←[33mError parsing json {:source=>"message", :raw=>"DRY-RUN MODE.  No changes wi
ll be made.", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'DRY
': was expecting ('true', 'false' or 'null')
 at [Source: [B@2c8ee7; line: 1, column: 5]>, :level=>:warn}←[0m
←[33mError parsing json {:source=>"message", :raw=>"(CLOSED) indices may be show
n that may not be acted on by action \"delete_indices\".", :exception=>#<LogStas
h::Json::ParserError: Unexpected character ('(' (code 40)): expected a valid val
ue (number, String, array, object, 'true', 'false' or 'null')
 at [Source: [B@19e3bba; line: 1, column: 2]>, :level=>:warn}←[0m
←[33mError parsing json {:source=>"message", :raw=>"DRY-RUN: delete_indices: med
iacontent-2017-01-20 with arguments: {}", :exception=>#<LogStash::Json::ParserEr
ror: Unrecognized token 'DRY': was expecting ('true', 'false' or 'null')
 at [Source: [B@1051180; line: 1, column: 5]>, :level=>:warn}←[0m
←[33mError parsing json {:source=>"message", :raw=>"DRY-RUN: delete_indices: med
iacontent-2017-02-20 with arguments: {}", :exception=>#<LogStash::Json::ParserEr
ror: Unrecognized token 'DRY': was expecting ('true', 'false' or 'null')
 at [Source: [B@41a16c; line: 1, column: 5]>, :level=>:warn}←[0m
←[33mError parsing json {:source=>"message", :raw=>"DRY-RUN: delete_indices: med
iacontent-2017-03-20 with arguments: {}", :exception=>#<LogStash::Json::ParserEr
ror: Unrecognized token 'DRY': was expecting ('true', 'false' or 'null')
 at [Source: [B@1269a31; line: 1, column: 5]>, :level=>:warn}←[0m
←[33mError parsing json {:source=>"message", :raw=>"Action ID: 1, \"delete_indic
es\" completed.", :exception=>#<LogStash::Json::ParserError: Unrecognized token
'Action': was expecting ('true', 'false' or 'null')
 at [Source: [B@3ffaef; line: 1, column: 8]>, :level=>:warn}←[0m
←[33mError parsing json {:source=>"message", :raw=>"Job completed.", :exception=
>#<LogStash::Json::ParserError: Unrecognized token 'Job': was expecting ('true',
 'false' or 'null')
 at [Source: [B@1daaaa6; line: 1, column: 5]>, :level=>:warn}←[0m
{
    "@timestamp" => "2017-04-24T07:03:07.892Z",
      "function" => "cli",
       "linenum" => 151,
      "loglevel" => "INFO",
       "message" => "Preparing Action ID: 1, \"delete_indices\"",
          "name" => "curator.cli",
      "@version" => "1",
          "path" => "C:\\Users\\571952\\Desktop\\curator.log",
          "host" => "PC326906",
          "tags" => [
        [0] "_jsonparsefailure"
    ]
}
{
    "@timestamp" => "2017-04-24T07:03:07.907Z",
      "function" => "cli",
       "linenum" => 173,
      "loglevel" => "INFO",
       "message" => "Trying Action ID: 1, \"delete_indices\": Delete indices old
er than 3 day (based on index name), for logstash- prefixed indices. Ignore the
error if the filter does not result in an actionable list of indices (ignore_emp
ty_list) and exit cleanly.",
          "name" => "curator.cli",
      "@version" => "1",
          "path" => "C:\\Users\\571952\\Desktop\\curator.log",
          "host" => "PC326906",
          "tags" => [
        [0] "_jsonparsefailure"
    ]
}
{
    "@timestamp" => "2017-04-24T07:03:07.985Z",
      "function" => "show_dry_run",
       "linenum" => 636,
      "loglevel" => "INFO",
       "message" => "DRY-RUN MODE.  No changes will be made.",
          "name" => "curator.utils",
      "@version" => "1",
          "path" => "C:\\Users\\571952\\Desktop\\curator.log",
          "host" => "PC326906",
          "tags" => [
        [0] "_jsonparsefailure"
    ]
}

My logs are like this:

{"@timestamp": "2017-04-24T07:03:07.892Z", "function": "cli", "linenum": 151, "loglevel": "INFO", "message": "Preparing Action ID: 1, \"delete_indices\"", "name": "curator.cli"}
{"@timestamp": "2017-04-24T07:03:07.907Z", "function": "cli", "linenum": 173, "loglevel": "INFO", "message": "Trying Action ID: 1, \"delete_indices\": Delete indices older than 3 day (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.", "name": "curator.cli"}
{"@timestamp": "2017-04-24T07:03:07.985Z", "function": "show_dry_run", "linenum": 636, "loglevel": "INFO", "message": "DRY-RUN MODE.  No changes will be made.", "name": "curator.utils"}

Thanks

[magnusbaeck (Magnus Bäck)]

Use a json codec or a json filter. Not both.

[Yaswanth]

Thanks it worked