Logstash configuration parsing error

Elastic Stack Logstash
1

Hi,
I'm new with Logstash and getting the error below, I think it's not parsing the json properly, does anyone see the problem in my Logstash config .

test1.json

[
{
    "upsert": {
        "reference": "https://www.abc.ca/en/events/#/details/14618",
        "metadata": {
            "author": [
                "ABC"
            ],
            "body_content": [
                "abc content "
            ]
        },
        "content": "content of abc"
    }
},
{
    "upsert": {
        "reference": "https://www.bbc.ca/en/events/#/details/14418",
        "metadata": {
            "author": [
                "BBC"
            ],
            "body_content": [
                "BBC Content "
            ]
        },
        "content": "content of BBC"
    }
}
]

logstash.conf

input {
  file {
    path => "/datadrive/json/test1.json"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => multiline {
      pattern => "^\{"
      negate => "true"
      what => "previous"
      auto_flush_interval => 1
    }
  }
}

filter {
  json {
    source => "message"
  }

  mutate {
    rename => {
      "[upsert][reference]" => "reference"
      "[upsert][metadata][author]" => "author"
      "[upsert][metadata][body_content]" => "body_content"
      "[upsert][content]" => "content"
    }
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

Error

[2024-07-13T09:35:27,033][WARN ][logstash.filters.json    ][main][9e71b43cd430180easdfedfdfedfe0d6ee81ae53223e64c] Error parsing json {:source=>"message", :raw=>"{\n    \"upsert\": {\n        \"reference\": \"https://www.bbc.ca/en/events/#/details/14418\",\n        \"metadata\": {\n            \"author\": [\n                \"BBC\"\n            ],\n            \"body_content\": [\n                \"BBC Content \"\n            ]\n        },\n        \"content\": \"content of BBC\"\n    }\n}\n]", :exception=>#<LogStash::Json::ParserError: Unexpected close marker ']': expected '}' (for root starting at [Source: (byte[])"{
    "upsert": {
        "reference": "https://www.bbc.ca/en/events/#/details/14418",
        "metadata": {
            "author": [
                "BBC"
            ],
            "body_content": [
                "BBC Content "
            ]
        },
        "content": "content of BBC"
    }
}
]"; line: 1, column: 0])
 at [Source: (byte[])"{
    "upsert": {
        "reference": "https://www.bbc.ca/en/events/#/details/14418",
        "metadata": {
            "author": [
                "BBC"
            ],
            "body_content": [
                "BBC Content "
            ]
        },
        "content": "content of BBC"
    }
}
]"; line: 15, column: 2]>}
{
    "@timestamp" => 2024-07-13T13:35:26.931027712Z,
      "@version" => "1",
         "event" => {
        "original" => "{\n    \"upsert\": {\n        \"reference\": \"https://www.bbc.ca/en/events/#/details/14418\",\n        \"metadata\": {\n            \"author\": [\n                \"BBC\"\n            ],\n            \"body_content\": [\n                \"BBC Content \"\n            ]\n        },\n        \"content\": \"content of BBC\"\n    }\n}\n]"
    },
          "tags" => [
        [0] "multiline",
        [1] "_jsonparsefailure"
    ],
           "log" => {
        "file" => {
            "path" => "/datadrive/json/test1.json"
        }
    },
       "message" => "{\n    \"upsert\": {\n        \"reference\": \"https://www.bbc.ca/en/events/#/details/14418\",\n        \"metadata\": {\n            \"author\": [\n                \"BBC\"\n            ],\n            \"body_content\": [\n                \"BBC Content \"\n            ]\n        },\n        \"content\": \"content of BBC\"\n    }\n}\n]",
          "host" => {
        "name" => "ES-Demo"
    }
}
2

Welcome, @temp_data.

On a StackOverflow post about a similar topic, someone suggested using a JSON linting tool like this one, which could be helpful here.

3

@jessgarson Thank you for the hint, compressing the json and making it one line and using the basic setting work.

1 Like
4

Thanks for following up, @temp_data. Let us know if there anything else I can help you with here.

5

The problem was your multiline codec. It consumes the second array entry (BBC) and the closing ] of the array as a single event, and that is not valid JSON, because the opening [ of the array is in a different event.