exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')

srbhklkrn · July 14, 2020, 9:30pm

I need to parse and have all the JSON object in one message but it is splitting into different messages as you can see in stdout below

JSON:

       [
        {
            "SOURCE": "Source A",
            "Model": "ModelABC",
            "Qty": "3"
        },
        {
            "SOURCE": "Source B",
            "Model": "MoBC",
            "Qty": "31"
        },
        {
            "SOURCE": "Source C",
            "Model": "MoBCSss",
            "Qty": "3qq"
        }
    ]

but getting error:

exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')

here is the full logstash config file:

input {

      file {
        path => "/usr/share/logstash/sample-log/Test-Log-For-Kibana.json"
        start_position => "beginning"
        codec => "plain"
    }
}

filter {
    json { source =>"message"
    target => "log"
   }
}

    output {
      stdout { codec => rubydebug }
        elasticsearch {
            hosts => "elasticsearch:9200"
            user => "elastic"
            password => "changeme"

        }
    }

stacktrace and stdout:

logstash_1       | [2020-07-14T21:22:34,209][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"]{\r", :exception=>#<LogStash::Json::ParserError: Unexpected c"; line: 1, column: 0])ed '}' (for root starting at [Source: (byte[])"]{
"; line: 1, column: 2]>}Source: (byte[])"]{
logstash_1       | [2020-07-14T21:22:34,212][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"    \"time\": \"[30/Jul/2017:17:21:45 +0000]\",\r", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
"; line: 1, column: 12]>}ource: (byte[])"    "time": "[30/Jul/2017:17:21:45 +0000]",
logstash_1       | [2020-07-14T21:22:34,214][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"    \"host\": \"my.host.local\",\r", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
"; line: 1, column: 12]>}ource: (byte[])"    "host": "my.host.local",
logstash_1       | [2020-07-14T21:22:34,217][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"    \"remoteIP\": \"192.168.2.1\",\r", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
"; line: 1, column: 16]>}ource: (byte[])"    "remoteIP": "192.168.2.1",
logstash_1       | [2020-07-14T21:22:34,219][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"    \"query\": \"\",\r", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
"; line: 1, column: 13]>}ource: (byte[])"    "query": "",
logstash_1       | [2020-07-14T21:22:34,223][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"    \"status\": \"200\",\r", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
"; line: 1, column: 14]>}ource: (byte[])"    "status": "200",
logstash_1       | [2020-07-14T21:22:34,225][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"    \"referer\": \"-\"\r", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
"; line: 1, column: 15]>}ource: (byte[])"    "referer": "-"
logstash_1       | [2020-07-14T21:22:34,224][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"    \"request\": \"/index.html\",\r", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
"; line: 1, column: 15]>}ource: (byte[])"    "request": "/index.html",
logstash_1       | [2020-07-14T21:22:34,228][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"    \"method\": \"GET\",\r", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
"; line: 1, column: 14]>}ource: (byte[])"    "method": "GET",
logstash_1       | [2020-07-14T21:22:34,236][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"    \"userAgent\": \"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; YTB720; GTB7.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)\",\r", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
logstash_1       |  at [Source: (byte[])"    "userAgent": "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; YTB720; GTB7.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3"; line: 1, column: 17]>}5.30729)",
logstash_1       | {
logstash_1       |           "host" => "d9e25305bbed",
logstash_1       |        "message" => "    \"time\": \"[30/Jul/2017:17:21:45 +0000]\",\r",
logstash_1       |       "@version" => "1",
logstash_1       |     "@timestamp" => 2020-07-14T21:22:34.088Z,
logstash_1       |           "path" => "/usr/share/logstash/sample-log/Test-Log-For-Kibana.json",
logstash_1       |           "tags" => [
logstash_1       |         [0] "_jsonparsefailure"
logstash_1       |     ]
logstash_1       | }
logstash_1       | {
logstash_1       |           "host" => "d9e25305bbed",
logstash_1       |        "message" => "    \"host\": \"my.host.local\",\r",
logstash_1       |       "@version" => "1",
logstash_1       |     "@timestamp" => 2020-07-14T21:22:34.092Z,
logstash_1       |           "path" => "/usr/share/logstash/sample-log/Test-Log-For-Kibana.json",
logstash_1       |           "tags" => [
logstash_1       |         [0] "_jsonparsefailure"
logstash_1       |     ]
logstash_1       | }
logstash_1       | {
logstash_1       |           "host" => "d9e25305bbed",
logstash_1       |        "message" => "    \"query\": \"\",\r",
logstash_1       |       "@version" => "1",
logstash_1       |     "@timestamp" => 2020-07-14T21:22:34.093Z,
logstash_1       |           "path" => "/usr/share/logstash/sample-log/Test-Log-For-Kibana.json",
logstash_1       |           "tags" => [
logstash_1       |         [0] "_jsonparsefailure"
logstash_1       |     ]
logstash_1       | }
logstash_1       | {
logstash_1       |           "host" => "d9e25305bbed",
logstash_1       |        "message" => "    \"status\": \"200\",\r",
logstash_1       |       "@version" => "1",
logstash_1       |     "@timestamp" => 2020-07-14T21:22:34.099Z,
logstash_1       |           "path" => "/usr/share/logstash/sample-log/Test-Log-For-Kibana.json",
logstash_1       |           "tags" => [
logstash_1       |         [0] "_jsonparsefailure"
logstash_1       |     ]
logstash_1       | }
logstash_1       | {
logstash_1       |           "host" => "d9e25305bbed",
logstash_1       |        "message" => "    \"referer\": \"-\"\r",
logstash_1       |       "@version" => "1",
logstash_1       |     "@timestamp" => 2020-07-14T21:22:34.106Z,
logstash_1       |           "path" => "/usr/share/logstash/sample-log/Test-Log-For-Kibana.json",
logstash_1       |           "tags" => [
logstash_1       |         [0] "_jsonparsefailure"
logstash_1       |     ]
logstash_1       | }
logstash_1       | {
logstash_1       |           "host" => "d9e25305bbed",
logstash_1       |        "message" => "]{\r",
logstash_1       |       "@version" => "1",
logstash_1       |     "@timestamp" => 2020-07-14T21:22:34.085Z,
logstash_1       |           "path" => "/usr/share/logstash/sample-log/Test-Log-For-Kibana.json",
logstash_1       |           "tags" => [
logstash_1       |         [0] "_jsonparsefailure"
logstash_1       |     ]
logstash_1       | }
logstash_1       | {
logstash_1       |           "host" => "d9e25305bbed",
logstash_1       |        "message" => "    \"remoteIP\": \"192.168.2.1\",\r",
logstash_1       |       "@version" => "1",
logstash_1       |     "@timestamp" => 2020-07-14T21:22:34.091Z,
logstash_1       |           "path" => "/usr/share/logstash/sample-log/Test-Log-For-Kibana.json",
logstash_1       |           "tags" => [
logstash_1       |         [0] "_jsonparsefailure"
logstash_1       |     ]
logstash_1       | }
logstash_1       | {
logstash_1       |           "host" => "d9e25305bbed",
logstash_1       |        "message" => "    \"request\": \"/index.html\",\r",
logstash_1       |       "@version" => "1",
logstash_1       |     "@timestamp" => 2020-07-14T21:22:34.092Z,
logstash_1       |           "path" => "/usr/share/logstash/sample-log/Test-Log-For-Kibana.json",
logstash_1       |           "tags" => [
logstash_1       |         [0] "_jsonparsefailure"
logstash_1       |     ]
logstash_1       | }
logstash_1       | {
logstash_1       |           "host" => "d9e25305bbed",
logstash_1       |        "message" => "    \"method\": \"GET\",\r",
logstash_1       |       "@version" => "1",
logstash_1       |     "@timestamp" => 2020-07-14T21:22:34.098Z,
logstash_1       |           "path" => "/usr/share/logstash/sample-log/Test-Log-For-Kibana.json",
logstash_1       |           "tags" => [
logstash_1       |         [0] "_jsonparsefailure"
logstash_1       |     ]
logstash_1       | }
logstash_1       | {
logstash_1       |           "host" => "d9e25305bbed",
logstash_1       |        "message" => "    \"userAgent\": \"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; YTB720; GTB7.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)\",\r",
logstash_1       |       "@version" => "1",
logstash_1       |     "@timestamp" => 2020-07-14T21:22:34.100Z,
logstash_1       |           "path" => "/usr/share/logstash/sample-log/Test-Log-For-Kibana.json",
logstash_1       |           "tags" => [
logstash_1       |         [0] "_jsonparsefailure"
logstash_1       |     ]
logstash_1       | }

Badger · July 14, 2020, 9:51pm

Your JSON appears to be pretty-printed in the source file. You would have to use a multiline codec to combine all the lines of one object into a single event.

If you want to consume the entire file as one event then you could use

codec => multiline { pattern => "^Spalanzani" negate => true what => "previous" auto_flush_interval => 2 }

If you have multiple pretty printed objects then you might be able to "^}" as the pattern. It really depends on your data.

srbhklkrn · July 15, 2020, 1:49am

Thank you for replying Badger, I tried your solution and it is taking entire file in one message, what I need is e.g split each json object into different message
message 1
{
"SOURCE":"Source A",
"Model":"ModelABC",
"Qty":"3"
},

message 2:
{
"SOURCE":"Source B",
"Model":"MoBC",
"Qty":"31"
}

and then mutate them and add_field for the each key so that it can be queryable

Badger · July 15, 2020, 1:51am

As I said ... If you have multiple pretty printed objects then you might be able to use "^}" as the pattern. It really depends on your data.

srbhklkrn · July 15, 2020, 1:53am

You can consider my data is identical to one I mentioned in my question.

srbhklkrn · July 15, 2020, 2:01am

This is my current config:

input {

    file {
        path => "/usr/share/logstash/sample-log/Test-Log-For-Kibana.json"
        start_position => "beginning"
        codec => multiline { pattern => "^}" negate => true what => previous auto_flush_interval => 1 multiline_tag => "" }

    }
}

filter {
        json { source => "message" target => "someField" remove_field => [ "message" ] } 

        mutate {    
            add_field => {
            "SOURCE" => "%{[someField][SOURCE]}"
            "Model" => "%{[someField][Model]}"
            "Qty" => "%{[someField][Qty]}"
        }

            remove_field => [ "json", "message" ]
        }
}

output {

  stdout { codec => rubydebug }

    elasticsearch {
        hosts => "elasticsearch:9200"
        user => "elastic"
        password => "changeme"
    }
}

Badger · July 15, 2020, 2:26am

We are not going to guess what your data looks like. Delete the

remove_field => [ "json", "message" ]

and show us what

stdout { codec => rubydebug }

looks like. You can redact any values you want to, but do not change the structure of [message] or of the JSON.

srbhklkrn · July 15, 2020, 2:37am

here is the stdout:

logstash_1       | [2020-07-15T02:35:46,867][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"[\r\n    {\r\n        \"SOURCE\": \"Source A\",\r\n        \"Model\": \"ModelABC\",\r\n        \"Qty\": \"3\"\r\n    },\r\n    {\r\n        \"SOURCE\": \"Source B\",\r\n        \"Model\": \"MoBC\",\r\n        \"Qty\": \"31\"\r\n    },\r\n    {\r\n        \"SOURCE\": \"Source C\",\r\n        \"Model\": \"MoBCSss\",\r\n        \"Qty\": \"3qq\"\r\n    }\r", :exception=>#<LogStash::Json::ParserError: Unexpected end-of-input: expected close marker for Array (start marker at [Source: (byte[])"[
logstash_1       |     {
logstash_1       |         "SOURCE": "Source A",
logstash_1       |         "Model": "ModelABC",
logstash_1       |         "Qty": "3"
logstash_1       |     },
logstash_1       |     {
logstash_1       |         "SOURCE": "Source B",
logstash_1       |         "Model": "MoBC",
logstash_1       |         "Qty": "31"
logstash_1       |     },
logstash_1       |     {
logstash_1       |         "SOURCE": "Source C",
logstash_1       |         "Model": "MoBCSss",
logstash_1       |         "Qty": "3qq"
"; line: 1, column: 1])}
logstash_1       |  at [Source: (byte[])"[
logstash_1       |     {
logstash_1       |         "SOURCE": "Source A",
logstash_1       |         "Model": "ModelABC",
logstash_1       |         "Qty": "3"
logstash_1       |     },
logstash_1       |     {
logstash_1       |         "SOURCE": "Source B",
logstash_1       |         "Model": "MoBC",
logstash_1       |         "Qty": "31"
logstash_1       |     },
logstash_1       |     {
logstash_1       |         "SOURCE": "Source C",
logstash_1       |         "Model": "MoBCSss",
logstash_1       |         "Qty": "3qq"
"; line: 17, column: 288]>}
logstash_1       | /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
logstash_1       | {
logstash_1       |          "Model" => "%{[someField][Model]}",
logstash_1       |     "@timestamp" => 2020-07-15T02:35:46.123Z,
logstash_1       |           "path" => "/usr/share/logstash/sample-log/Test-Log-For-Kibana.json",
logstash_1       |           "host" => "9575943f44dd",
logstash_1       |       "@version" => "1",
logstash_1       |        "message" => "[\r\n    {\r\n        \"SOURCE\": \"Source A\",\r\n        \"Model\": \"ModelABC\",\r\n        \"Qty\": \"3\"\r\n    },\r\n    {\r\n        \"SOURCE\": \"Source B\",\r\n        \"Model\": \"MoBC\",\r\n        \"Qty\": \"31\"\r\n    },\r\n    {\r\n        \"SOURCE\": \"Source C\",\r\n        \"Model\": \"MoBCSss\",\r\n        \"Qty\": \"3qq\"\r\n    }\r",
logstash_1       |         "SOURCE" => "%{[someField][SOURCE]}",
logstash_1       |           "tags" => [
logstash_1       |         [0] "_jsonparsefailure"
logstash_1       |     ],
logstash_1       |            "Qty" => "%{[someField][Qty]}"
logstash_1       | }

system · August 12, 2020, 2:37am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
JSON parse error, original data now in message field {:message=>"Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')\n at [Source: (String) Logstash	12	5540	February 1, 2023
LogStash::Json::ParserError: Unexpected character ('(' (code 40)) Logstash	10	1517	August 9, 2022
JSON parse error, original data now in message field Logstash	3	1372	March 24, 2022
JSON parser error Logstash	6	667	May 28, 2021
LogStash::Json::ParserError: Unexpected character (':' (code 58)) Elasticsearch	2	210	August 10, 2023

exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')

Related topics