How can I make the Logstash TCP input to separate messages that it listens on a TCP port?

Elastic Stack Logstash
1

I want to send data to Logstash using the TCP protocol. And to send the data I'm using the Node-RED. A simple configuration for doing that looks like this:

In the Logstash folder, I created a file called nodered.conf with the following content:

input {
    tcp {
        port => "3999"
    }   
}

output {
    stdout { codec => rubydebug }
}

Right now, I just want to print on my screen the information that the Logstash is receiving. That's why I used stdout { codec => rubydebug } on my output.

So, inside the Logstash folder, I started the Logstash with the following command:

bin/logstash -f nodered.conf --config.reload.automatic

The problem is that all the messages that I send to Logstash with the Node-RED are aggregated into a single message. For example, if I inject 5 messages with the Node-RED into my TCP port 3999, after redeploying the Node-RED I receive the following content on my Logstash terminal:

user@computer:/home/Dados/ELK/logstash-5.4.0$ bin/logstash -f nodered.conf --config.reload.automatic
Sending Logstash's logs to /home/Dados/ELK/logstash-5.4.0/logs which is now configured via log4j2.properties
[2017-05-29T15:14:52,388][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2017-05-29T15:14:52,417][INFO ][logstash.inputs.tcp      ] Starting tcp input listener {:address=>"0.0.0.0:3999"}
[2017-05-29T15:14:52,430][INFO ][logstash.pipeline        ] Pipeline main started
[2017-05-29T15:14:52,513][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
{
    "@timestamp" => 2017-05-29T18:19:33.277Z,
          "port" => 54316,
      "@version" => "1",
          "host" => "127.0.0.1",
       "message" => "hellohellohellohellohello"
}

And I actually would like to see something like this without having to redeploy:

user@computer:/home/Dados/ELK/logstash-5.4.0$ bin/logstash -f nodered.conf --config.reload.automatic
Sending Logstash's logs to /home/Dados/ELK/logstash-5.4.0/logs which is now configured via log4j2.properties
[2017-05-29T15:27:24,168][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2017-05-29T15:27:24,191][INFO ][logstash.inputs.tcp      ] Starting tcp input listener {:address=>"0.0.0.0:3999"}
[2017-05-29T15:27:24,200][INFO ][logstash.pipeline        ] Pipeline main started
[2017-05-29T15:27:24,260][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
{
    "@timestamp" => 2017-05-29T18:27:48.394Z,
          "port" => 54518,
      "@version" => "1",
          "host" => "127.0.0.1",
       "message" => "hello"
}
{
    "@timestamp" => 2017-05-29T18:27:51.657Z,
          "port" => 54546,
      "@version" => "1",
          "host" => "127.0.0.1",
       "message" => "hello"
}
{
    "@timestamp" => 2017-05-29T18:27:58.691Z,
          "port" => 54600,
      "@version" => "1",
          "host" => "127.0.0.1",
       "message" => "hello"
}
{
    "@timestamp" => 2017-05-29T18:28:06.330Z,
          "port" => 54656,
      "@version" => "1",
          "host" => "127.0.0.1",
       "message" => "hello"
}
{
    "@timestamp" => 2017-05-29T18:28:14.347Z,
          "port" => 54682,
      "@version" => "1",
          "host" => "127.0.0.1",
       "message" => "hello"
}

The conclusion is that I don't know how to make the Logstash interpret each message as a unique one instead of concatenating everything that it receives. I've tried using different codecs in my nodered.conf file but I didn't succeed. How can I make the Logstash see every message that it listens on a TCP port as a single one?

2

Have you tried to simply end each message with a newline character? I don't think the tcp input supports adding an end-of-message marker when connections are closed.

4

Do you mean adding a /n? I've tried that but it didn't work :confused: ... But I'm not sure if what I need to do to solve it is to add a marker when the connection is closed. My problem is actually that the messages don't stop concatenating while there's a connection. If I keep sending messages the Logstash keeps joining them without seeing them as new ones.

5

Do you mean adding a /n?

You mean \n but yes, a newline character. Exactly how are you sending the messages?

6

Yeah, I meant \n... :smile:

I actually solved this issue by changing my Node-RED diagram, it wasn't an issue with the Logstash. I had to add an extra node and concatenate it with \n as follows:

The code of the function node is this one:

msg.payload = msg.payload + "\n";
return msg;

So you were right! :slight_smile: It was just about adding \n to the string. But if I try to concatenate \n to my string directly inside the injection node with the Node-RED it doesn't work. I have to create an extra function node in this case.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.