Logstash to Logstash via TCP is slow/delayed/waiting for something

[cawoodm (Marc)]

We process logfiles with filebeat into logstash (CONF1). If the log entry is marked as an error, we clone it and send it to another instance of logstash (CONF2) via the tcp output plugin.

The problem is that while we see the message go out (CODE1) the 2nd instance does not show the message on it's output (CODE2) UNTIL we shutdown logstash2. Then we see our message (see below).

Why the "delay"? What is logstash waiting for in order to flush?

CONF1:

output {
	if [@metadata][index] == "errors" {
                # CODE1
		stdout {codec => rubydebug { metadata => true } }
		# Reprocess errors with Logstash
		tcp {
			host => "ecom-repository01"
			port => "5062"
		}
	}
}

CONF2:

input {
	tcp {
		port => 5062
		codec => json
		add_field => {
			system => "dev"
		}
	}
}

filter {}

output {
        # CODE2
	stdout {codec => rubydebug { metadata => true } }
}

[cawoodm (Marc)]

I suspect this is a bug. The DEBUG log is showing me that's it's switching me over to the json_lines codec even though I specified the json codec. I suspect it's waiting for a newline which never comes.

Can anyone confirm this bug?

I worked around it by switching from TCP to HTTP input/output plugins.

[cawoodm (Marc)]

Issue: Logstash TCP/JSON should not switch to json_lines and not provide a newline

[guyboertje]

I saw that you created the issue in the LS repo as well as switching to the http input output pair.

I am not sure of the history behind the implicit json_lines substitution in place of the json codec for inputs. It has been this way for many years.

As a matter of interest, did you try the json_lines codec in the tcp output (the default is json and the switch does not occur for outputs)?

[cawoodm (Marc)]

Indeed, if I use the json_lines codec on both ends it works. Thanks!

However, I think I'm gonna stick by my guns and say it's defective as is: many users of LS are gonna try json (because it's a thing) and not json_lines and conclude that it doesn't work because LS is "ignoring" that the output codec is json.

At least if "json_lines" is the de facto default output codec for tcp it should be the default input.

[guyboertje]

I understand and yes it is a quirk.

I asked to try the json_lines codec in the tcp output for future readers. However, it is still our recommendation at present to use the lumberjack output -> beats input for LS to LS communications because the beats input understands the lumberjack protocol. The lumberjack protocol, at the time of writing, is Logstash specific and IIRC can handle back-pressure better.

The quirk is hard to undo as many users will have running Logstash instances that secretly rely on the substitution.

The substitution is done by Logstash plugin support code and not in any individual plugin.