Need splitting incoming data

Elastic Stack Logstash
1

Hi all,

i try using split filter property to split incoming data from tcp port, this data has the line format of seven values plus time stamp separated by comma "," and terminated with "\n" as following :

[ v1,v2,v3,v4,v5,v6,v7,ts,\n ]

The rate of it is 100 line each 700 millisecond, the problem is that i just receive first line from each sending ( one line each 700 millisecond ) like this

605.0,575.0,506.0,433.0,272.0,318.0,849.0,2015-11-25T02:30:37.555
601.0,529.0,557.0,458.0,228.0,336.0,850.0,2015-11-25T02:30:38.310
600.0,651.0,557.0,398.0,351.0,274.0,850.0,2015-11-25T02:30:39.060
600.0,610.0,551.0,418.0,310.0,295.0,850.0,2015-11-25T02:30:39.813
600.0,594.0,579.0,426.0,294.0,303.0,850.0,2015-11-25T02:30:40.568

the configure file is

input {
tcp {
host => "192.168.0.188"
port => "12397"

}
stdin {}
}
filter {
#split{}

csv {
separator => ","
columns => ["v1", "v2", "v3", "v4", "v5", "v6", "v7", "ts"]
}

mutate {convert => ["v1", "float"]}
mutate {convert => ["v2", "float"]}
mutate {convert => ["v3", "float"]}
mutate {convert => ["v4", "float"]}
mutate {convert => ["v5", "float"]}
mutate {convert => ["v6", "float"]}
mutate {convert => ["v7", "float"]}
mutate {convert => ["ts", "string"]}
}

output {
csv {
path => "/home/ahmed/Downloads/logstash-2.0.0/test2.csv"
fields => ["v1", "v2", "v3", "v4", "v5", "v6", "v7", "ts"]

}
stdout {}
}

Thank you very much for helping me

Regards

2

The default codec of the tcp input is line which emits one event per line received so what you want to accomplish should work out of the box without a split filter. I would do this do debug this further:

  1. Explicitly set codec => line just to be sure.
  2. Use a stdout { codec => rubydebug } output to get the raw event data.
  3. Test with synthetic data that I send to Logstash with e.g. netcat so I know exactly what Logstash should see.
  4. If necessary crank up logging with --verbose or --debug.
1 Like
3

Dear friend,

Kindly, i try with your suggestions, but the problem still exist, the output file just contains 1 line instead of 100 per each data sending, from your notice about tcp codec input, is it mean the codec just catch the first line and omit the other for each transmission ?

Thanks for your guide

Regards

4

As I said, the line codec is supposed to emit one event per line. I can't reproduce what you describe with Logstash 1.5.3. With this simple configuration,

input { tcp { port => 13383 } }
output { stdout { codec => rubydebug } }

I can send multi-line messages over a single TCP connection with

$ echo 'message one
> message two' | netcat -q 0 localhost 13383

and get the following from Logstash:

$ /opt/logstash/bin/logstash -f test.config 
Logstash startup completed
{
       "message" => "message one",
      "@version" => "1",
    "@timestamp" => "2015-11-25T12:21:31.011Z",
          "host" => "0:0:0:0:0:0:0:1"
}
{
       "message" => "message two",
      "@version" => "1",
    "@timestamp" => "2015-11-25T12:21:31.012Z",
          "host" => "0:0:0:0:0:0:0:1"
}
1 Like