JSON data split up (_jsonparseerror) in logstash-7.2.0-1. Works fine in logstash-7.1.1-1

#1

Hello!

I've been in the midst of working with the ELK stack for the first time. I'm currently sending data from Metricbeat -> Logstash -> an external script (output sent over TCP) -> Logstash -> Elasticsearch. However, when I upgrade the logstash version to logstash-7.2.0-1 from logstash-7.1.1-1, My JSON is split up and sent in multiple documents instead of a single document which naturally leads to a _jsonparseerror tag. For example:

An output with logstash-7.2.0-1 where the "message field" is half the message JSON and the "..." indicates the middle of the message. This shows up with a _jsonparsefailure tag:

[2019-07-15T18:17:29,435][ERROR][logstash.codecs.json ] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (String)"{"service": {"type": "system"}, "tags": "... \"ios\": 66"}

The output with logstash-7.1.1-1 is clean and is parsed properly by elasticsearch. The only difference between the two is the logstash version. Any ideas?

#2

Check this thread once.

#3

We are having the same problem.

  • Using LogstashTcpSocketAppender from Java to log to our logstash

7.1.1 works fine
7.2.1 gives the error
7.3.0 also gives the error

  • Discovered when initially wanting to upgrade our old logstash 5.6.0
#4

Possible solution.

switching from codec json => json_lines seems to solve it. (have only tested for 5 minutes so far)
Probably worked before because of different code used to convert tcp stream into messages.

#5

This solution seemed to work for me as well! Thanks.
This would have been nice to document as a breaking change in documentation.

#6

We noticed it is actually mentioned in the documentation of the logstash logback encoder. But aside from that, what made it difficult to find is that the setting "json" almost works :slight_smile: - I suppose "json" probably has a maximum object size it will read up/buffer to before splitting.

1 Like