Send one event in one packet

Elastic Stack Logstash
1

Hi all,
I´m trying to use logstash to collect some data from a legacy shipper using tcp connection. Each event is in a tcp packet.

I want to filter some features and output the result event, but I need to put just one event in just one TCP output packet. Is there a way to do that?

I think of pipeline.batch.size to 1, but seems to be only for input, is there something similar for tcp output plugin?

2

What have you tried so far, and what were the results? It be easier to help if you were a bit more specific.

3

My end of event (delimiter) is ";" (not carriage return)
new_message is an "enriched message" by filters plugins.

With this configuration, when two or more events arrive in less than ¿1 second? output send them in just one TCP packet, all together (I can see this with wireshark or tcpdump)

My intention is to send out three packets if three events arrive, four packets for four events, and so on..., just one per event.

input
{
	tcp {
		host => "0.0.0.0"
		mode => "server"
		port => 9998
		codec =>
					line {
						charset => "UTF-8"
						delimiter => ";"
					}
	}
}
filter { ... }

output {
	tcp {
		host => "x.y.z.w"
		port => "9995"
		mode => "client"
		reconnect_interval => "10"
		codec =>
				line {
						format => "%{new_message}"
				}
	}
}
4

TCP is a stream oriented protocol. It does not have any conception of packets. The underlying transport (e.g. Ethernet) does, but TCP does not. You need to provide a way divide the stream up into events, which you do using the codec.

Even if you turn off the batching and constantly push data (and the standard output does not support TCP PSH) any TCP stack that handles the data may recombine packets. You cannot impose packet support on TCP like this.

5

I know it, but my legacy shipper is working that way since years ago, and I am not able to change it right now.

Anyway, thank you.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.