Parsing a Raw JSON file "Invalid Setting for json filter"

Elastic Stack Logstash
1

Hi,
I am trying to parse a simple raw JSON file but I am getting an error about invalid setting for JSON filter. I am wondering if its the way my filter is setup and the fact that its nested json?

[ERROR] 2021-08-08 10:52:34.886 [Converge PipelineAction::Create] json - Invalid setting for json filter plugin:

filter {
json {
# This setting must be a string
# Expected string, got ["TEST", "eventName", "date", "time", "ID"]
source => ["TEST", "eventName", "date", "time", "ID"]
...
}
}

The raw log looks like this:

{
  "TEST" : {
    "dataRec" : {
      "eventName" : "IN",
      "date" : "210721",
      "time" : "06:27:56",
      "ID" : "TEST ",
      "group" : "TEST0001",
      "job" : "QRTSDHL4",
      "terminal" : "ABC21QJ2",
      "Name" : "TESTPREPROD",
      "returnCode" : "00",
      "detailsReasonCode" : "TEST ",
      "userName" : "1234567,LAST,FIRST (ABCDEF)   "
    }
  }
}

Filter looks like this

input {
  file{
  # type => "json"
  # codec => json {  }
   path => "/home/test.txt"
   mode => "tail"
   start_position => "beginning"
  }
}

filter {
  json {
  source => "TEST"
  source => "eventName"
  source => "date"
  source => "time"
  source => "ID"
  source => "group"
     }
}
 
output {
stdout {} 
file { path => "/home/output.log" }
}
2

The json filter only supports one field at a time, if your original message is a json document, then you just need to use:

json {
    source => "message"
}
3

Thanks for the response. When I change it to that I get the following error:
so filter:

filter {
  json {
    source => "message"
    
}     
}

Error

[WARN ] 2021-08-08 11:39:16.919 [[main]>worker0] json - Error parsing json {:source=>"message", :raw=>"{", :exception=>#<LogStash::Json::ParserError: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (byte[])"{"; line: 1, column: 1])
 at [Source: (byte[])"{"; line: 1, column: 3]>}
[WARN ] 2021-08-08 11:39:16.920 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>"  \"TEST\" : {", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (byte[])"  "TEST" : {"; line: 1, column: 11]>}
[WARN ] 2021-08-08 11:39:16.923 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>"    \"dataRec\" : {", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (byte[])"    "dataRec" : {"; line: 1, column: 16]>}
[WARN ] 2021-08-08 11:39:16.924 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>"      \"eventName\" : \"IN\",", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (byte[])"      "eventName" : "IN","; line: 1, column: 20]>}
[WARN ] 2021-08-08 11:39:16.925 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>"      \"date\" : \"210721\",", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (byte[])"      "date" : "210721","; line: 1, column: 15]>}
[WARN ] 2021-08-08 11:39:16.926 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>"      \"time\" : \"06:27:56\",", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (byte[])"      "time" : "06:27:56","; line: 1, column: 15]>}
[WARN ] 2021-08-08 11:39:16.927 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>"      \"accountID\" : \"TEST \",", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (byte[])"      "accountID" : "TEST ","; line: 1, column: 20]>}
[WARN ] 2021-08-08 11:39:16.928 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>"      \"department\" : \"TEST0001\",", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (byte[])"      "department" : "TEST0001","; line: 1, column: 21]>}
[WARN ] 2021-08-08 11:39:16.929 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>"      \"jobName\" : \"QRTSDHL4\",", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (byte[])"      "jobName" : "QRTSDHL4","; line: 1, column: 18]>}
[WARN ] 2021-08-08 11:39:16.930 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>"      \"terminalID\" : \"ABC21QJ2\",", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (byte[])"      "terminalID" : "ABC21QJ2","; line: 1, column: 21]>}
[WARN ] 2021-08-08 11:39:16.931 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>"      \"facilityName\" : \"TESTPREPROD\",", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (byte[])"      "facilityName" : "TESTPREPROD","; line: 1, column: 23]>}
[WARN ] 2021-08-08 11:39:16.932 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>"      \"returnCode\" : \"00\",", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (byte[])"      "returnCode" : "00","; line: 1, column: 21]>}
[WARN ] 2021-08-08 11:39:16.933 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>"      \"detailsReasonCode\" : \"TEST \",", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (byte[])"      "detailsReasonCode" : "TEST ","; line: 1, column: 28]>}
[WARN ] 2021-08-08 11:39:16.934 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>"      \"userName\" : \"1234567,LAST,FIRST (ABCDEF)   \"", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (byte[])"      "userName" : "1234567,LAST,FIRST (ABCDEF)   ""; line: 1, column: 19]>}
[WARN ] 2021-08-08 11:39:16.937 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>"    }", :exception=>#<LogStash::Json::ParserError: Unexpected close marker '}': expected ']' (for root starting at [Source: (byte[])"    }"; line: 1, column: 0])
 at [Source: (byte[])"    }"; line: 1, column: 6]>}
[WARN ] 2021-08-08 11:39:16.954 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>"  }", :exception=>#<LogStash::Json::ParserError: Unexpected close marker '}': expected ']' (for root starting at [Source: (byte[])"  }"; line: 1, column: 0])
 at [Source: (byte[])"  }"; line: 1, column: 4]>}
[WARN ] 2021-08-08 11:39:16.957 [[main]>worker1] json - Error parsing json {:source=>"message", :raw=>"}", :exception=>#<LogStash::Json::ParserError: Unexpected close marker '}': expected ']' (for root starting at [Source: (byte[])"}"; line: 1, column: 0])
 at [Source: (byte[])"}"; line: 1, column: 2]>}
[WARN ] 2021-08-08 11:39:16.959 [[main]>worker1] json - Parsed JSON object/hash requires a target configuration option {:source=>"message", :raw=>""}
{
    "@timestamp" => 2021-08-08T15:39:16.596Z,
          "tags" => [
        [0] "_jsonparsefailure"
    ],
      "@version" => "1",
          "host" => "testhost.ca",
          "path" => "/home/example/test2.log",
       "message" => "{"
}
[INFO ] 2021-08-08 11:39:17.235 [[main]>worker0] file - Opening file {:path=>"/home/example/output.log"}
{
    "@timestamp" => 2021-08-08T15:39:16.622Z,
          "tags" => [
        [0] "_jsonparsefailure"
    ],
      "@version" => "1",
          "host" => "testhost.ca",
          "path" => "/home/example/test2.log",
       "message" => "  \"TEST\" : {"
}
{
    "@timestamp" => 2021-08-08T15:39:16.623Z,
          "tags" => [
        [0] "_jsonparsefailure"
    ],
      "@version" => "1",
          "host" => "testhost.ca",
          "path" => "/home/example/test2.log",
       "message" => "    \"dataRec\" : {"
}
{
    "@timestamp" => 2021-08-08T15:39:16.623Z,
          "tags" => [
        [0] "_jsonparsefailure"
    ],
      "@version" => "1",
          "host" => "testhost.ca",
          "path" => "/home/example/test2.log",
       "message" => "      \"eventName\" : \"IN\","
}
{
    "@timestamp" => 2021-08-08T15:39:16.624Z,
          "tags" => [
        [0] "_jsonparsefailure"
    ],
      "@version" => "1",
          "host" => "testhost.ca",
          "path" => "/home/example/test2.log",
       "message" => "      \"date\" : \"210721\","
}
{
    "@timestamp" => 2021-08-08T15:39:16.624Z,
          "tags" => [
        [0] "_jsonparsefailure"
    ],
      "@version" => "1",
          "host" => "testhost.ca",
          "path" => "/home/example/test2.log",
       "message" => "      \"time\" : \"06:27:56\","
}
{
    "@timestamp" => 2021-08-08T15:39:16.624Z,
          "tags" => [
        [0] "_jsonparsefailure"
    ],
      "@version" => "1",
          "host" => "testhost.ca",
          "path" => "/home/example/test2.log",
       "message" => "      \"accountID\" : \"TEST \","
}
{
    "@timestamp" => 2021-08-08T15:39:16.625Z,
          "tags" => [
        [0] "_jsonparsefailure"
    ],
      "@version" => "1",
          "host" => "testhost.ca",
          "path" => "/home/example/test2.log",
       "message" => "      \"department\" : \"TEST0001\","
}
{
    "@timestamp" => 2021-08-08T15:39:16.625Z,
          "tags" => [
        [0] "_jsonparsefailure"
    ],
      "@version" => "1",
          "host" => "testhost.ca",
          "path" => "/home/example/test2.log",
       "message" => "      \"jobName\" : \"QRTSDHL4\","
}
{
    "@timestamp" => 2021-08-08T15:39:16.625Z,
          "tags" => [
        [0] "_jsonparsefailure"
    ],
      "@version" => "1",
          "host" => "testhost.ca",
          "path" => "/home/example/test2.log",
       "message" => "      \"terminalID\" : \"ABC21QJ2\","
}
{
    "@timestamp" => 2021-08-08T15:39:16.626Z,
          "tags" => [
        [0] "_jsonparsefailure"
    ],
      "@version" => "1",
          "host" => "testhost.ca",
          "path" => "/home/example/test2.log",
       "message" => "      \"facilityName\" : \"TESTPREPROD\","
}
4

Your JSON is pretty-printed across multiple lines. You will need to use a multiline codec to combine all the lines of a single JSON object into one [message] field. See here and here.

5

thanks. that worked!!!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.