Parsing a Raw JSON file "Invalid Setting for json filter"

Wilks · August 8, 2021, 2:59pm

Hi,
I am trying to parse a simple raw JSON file but I am getting an error about invalid setting for JSON filter. I am wondering if its the way my filter is setup and the fact that its nested json?

[ERROR] 2021-08-08 10:52:34.886 [Converge PipelineAction::Create] json - Invalid setting for json filter plugin:

filter {
json {
# This setting must be a string
# Expected string, got ["TEST", "eventName", "date", "time", "ID"]
source => ["TEST", "eventName", "date", "time", "ID"]
...
}
}

The raw log looks like this:

{
  "TEST" : {
    "dataRec" : {
      "eventName" : "IN",
      "date" : "210721",
      "time" : "06:27:56",
      "ID" : "TEST ",
      "group" : "TEST0001",
      "job" : "QRTSDHL4",
      "terminal" : "ABC21QJ2",
      "Name" : "TESTPREPROD",
      "returnCode" : "00",
      "detailsReasonCode" : "TEST ",
      "userName" : "1234567,LAST,FIRST (ABCDEF)   "
    }
  }
}

Filter looks like this

input {
  file{
  # type => "json"
  # codec => json {  }
   path => "/home/test.txt"
   mode => "tail"
   start_position => "beginning"
  }
}

filter {
  json {
  source => "TEST"
  source => "eventName"
  source => "date"
  source => "time"
  source => "ID"
  source => "group"
     }
}
 
output {
stdout {} 
file { path => "/home/output.log" }
}

leandrojmp · August 8, 2021, 3:05pm

The json filter only supports one field at a time, if your original message is a json document, then you just need to use:

json {
    source => "message"
}

Wilks · August 8, 2021, 3:45pm

Thanks for the response. When I change it to that I get the following error:
so filter:

filter {
  json {
    source => "message"
    
}     
}

Error

[WARN ] 2021-08-08 11:39:16.919 [[main]>worker0] json - Error parsing json {:source=>"message", :raw=>"{", :exception=>#<LogStash::Json::ParserError: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (byte[])"{"; line: 1, column: 1])
 at [Source: (byte[])"{"; line: 1, column: 3]>}
[WARN ] 2021-08-08 11:39:16.920 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>"  \"TEST\" : {", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (byte[])"  "TEST" : {"; line: 1, column: 11]>}
[WARN ] 2021-08-08 11:39:16.923 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>"    \"dataRec\" : {", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (byte[])"    "dataRec" : {"; line: 1, column: 16]>}
[WARN ] 2021-08-08 11:39:16.924 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>"      \"eventName\" : \"IN\",", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (byte[])"      "eventName" : "IN","; line: 1, column: 20]>}
[WARN ] 2021-08-08 11:39:16.925 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>"      \"date\" : \"210721\",", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (byte[])"      "date" : "210721","; line: 1, column: 15]>}
[WARN ] 2021-08-08 11:39:16.926 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>"      \"time\" : \"06:27:56\",", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (byte[])"      "time" : "06:27:56","; line: 1, column: 15]>}
[WARN ] 2021-08-08 11:39:16.927 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>"      \"accountID\" : \"TEST \",", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (byte[])"      "accountID" : "TEST ","; line: 1, column: 20]>}
[WARN ] 2021-08-08 11:39:16.928 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>"      \"department\" : \"TEST0001\",", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (byte[])"      "department" : "TEST0001","; line: 1, column: 21]>}
[WARN ] 2021-08-08 11:39:16.929 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>"      \"jobName\" : \"QRTSDHL4\",", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (byte[])"      "jobName" : "QRTSDHL4","; line: 1, column: 18]>}
[WARN ] 2021-08-08 11:39:16.930 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>"      \"terminalID\" : \"ABC21QJ2\",", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (byte[])"      "terminalID" : "ABC21QJ2","; line: 1, column: 21]>}
[WARN ] 2021-08-08 11:39:16.931 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>"      \"facilityName\" : \"TESTPREPROD\",", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (byte[])"      "facilityName" : "TESTPREPROD","; line: 1, column: 23]>}
[WARN ] 2021-08-08 11:39:16.932 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>"      \"returnCode\" : \"00\",", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (byte[])"      "returnCode" : "00","; line: 1, column: 21]>}
[WARN ] 2021-08-08 11:39:16.933 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>"      \"detailsReasonCode\" : \"TEST \",", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (byte[])"      "detailsReasonCode" : "TEST ","; line: 1, column: 28]>}
[WARN ] 2021-08-08 11:39:16.934 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>"      \"userName\" : \"1234567,LAST,FIRST (ABCDEF)   \"", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (byte[])"      "userName" : "1234567,LAST,FIRST (ABCDEF)   ""; line: 1, column: 19]>}
[WARN ] 2021-08-08 11:39:16.937 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>"    }", :exception=>#<LogStash::Json::ParserError: Unexpected close marker '}': expected ']' (for root starting at [Source: (byte[])"    }"; line: 1, column: 0])
 at [Source: (byte[])"    }"; line: 1, column: 6]>}
[WARN ] 2021-08-08 11:39:16.954 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>"  }", :exception=>#<LogStash::Json::ParserError: Unexpected close marker '}': expected ']' (for root starting at [Source: (byte[])"  }"; line: 1, column: 0])
 at [Source: (byte[])"  }"; line: 1, column: 4]>}
[WARN ] 2021-08-08 11:39:16.957 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>"}", :exception=>#<LogStash::Json::ParserError: Unexpected close marker '}': expected ']' (for root starting at [Source: (byte[])"}"; line: 1, column: 0])
 at [Source: (byte[])"}"; line: 1, column: 2]>}
[WARN ] 2021-08-08 11:39:16.959 [[main]>worker1] json - Parsed JSON object/hash requires a target configuration option {:source=>"message", :raw=>""}
{
    "@timestamp" => 2021-08-08T15:39:16.596Z,
          "tags" => [
        [0] "_jsonparsefailure"
    ],
      "@version" => "1",
          "host" => "testhost.ca",
          "path" => "/home/example/test2.log",
       "message" => "{"
}
[INFO ] 2021-08-08 11:39:17.235 [[main]>worker0] file - Opening file {:path=>"/home/example/output.log"}
{
    "@timestamp" => 2021-08-08T15:39:16.622Z,
          "tags" => [
        [0] "_jsonparsefailure"
    ],
      "@version" => "1",
          "host" => "testhost.ca",
          "path" => "/home/example/test2.log",
       "message" => "  \"TEST\" : {"
}
{
    "@timestamp" => 2021-08-08T15:39:16.623Z,
          "tags" => [
        [0] "_jsonparsefailure"
    ],
      "@version" => "1",
          "host" => "testhost.ca",
          "path" => "/home/example/test2.log",
       "message" => "    \"dataRec\" : {"
}
{
    "@timestamp" => 2021-08-08T15:39:16.623Z,
          "tags" => [
        [0] "_jsonparsefailure"
    ],
      "@version" => "1",
          "host" => "testhost.ca",
          "path" => "/home/example/test2.log",
       "message" => "      \"eventName\" : \"IN\","
}
{
    "@timestamp" => 2021-08-08T15:39:16.624Z,
          "tags" => [
        [0] "_jsonparsefailure"
    ],
      "@version" => "1",
          "host" => "testhost.ca",
          "path" => "/home/example/test2.log",
       "message" => "      \"date\" : \"210721\","
}
{
    "@timestamp" => 2021-08-08T15:39:16.624Z,
          "tags" => [
        [0] "_jsonparsefailure"
    ],
      "@version" => "1",
          "host" => "testhost.ca",
          "path" => "/home/example/test2.log",
       "message" => "      \"time\" : \"06:27:56\","
}
{
    "@timestamp" => 2021-08-08T15:39:16.624Z,
          "tags" => [
        [0] "_jsonparsefailure"
    ],
      "@version" => "1",
          "host" => "testhost.ca",
          "path" => "/home/example/test2.log",
       "message" => "      \"accountID\" : \"TEST \","
}
{
    "@timestamp" => 2021-08-08T15:39:16.625Z,
          "tags" => [
        [0] "_jsonparsefailure"
    ],
      "@version" => "1",
          "host" => "testhost.ca",
          "path" => "/home/example/test2.log",
       "message" => "      \"department\" : \"TEST0001\","
}
{
    "@timestamp" => 2021-08-08T15:39:16.625Z,
          "tags" => [
        [0] "_jsonparsefailure"
    ],
      "@version" => "1",
          "host" => "testhost.ca",
          "path" => "/home/example/test2.log",
       "message" => "      \"jobName\" : \"QRTSDHL4\","
}
{
    "@timestamp" => 2021-08-08T15:39:16.625Z,
          "tags" => [
        [0] "_jsonparsefailure"
    ],
      "@version" => "1",
          "host" => "testhost.ca",
          "path" => "/home/example/test2.log",
       "message" => "      \"terminalID\" : \"ABC21QJ2\","
}
{
    "@timestamp" => 2021-08-08T15:39:16.626Z,
          "tags" => [
        [0] "_jsonparsefailure"
    ],
      "@version" => "1",
          "host" => "testhost.ca",
          "path" => "/home/example/test2.log",
       "message" => "      \"facilityName\" : \"TESTPREPROD\","
}

Badger · August 8, 2021, 3:52pm

Your JSON is pretty-printed across multiple lines. You will need to use a multiline codec to combine all the lines of a single JSON object into one [message] field. See here and here.

Wilks · August 8, 2021, 7:38pm

thanks. that worked!!!

system · September 5, 2021, 7:39pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Unable to parse nested json using logstash json filter Logstash	6	1187	October 16, 2019
Logstash json parse failure when json contains a "message" field Logstash	6	3343	July 4, 2019
Logstash JSON Filter Error Logstash	3	296	September 7, 2023
Parsing filter for nested json Logstash	2	318	July 27, 2020
Logstash json parsing Logstash	2	215	May 19, 2021

Parsing a Raw JSON file "Invalid Setting for json filter"

Related topics