Geo_shape: linestring with data from csv

[vincedt09 (Vince DT)]

Hi!
I am working on a project that involves drawing a line in kibana using data from a csv file.

Csv file:

latitude,longitude
44.453538,26.086064
44.455499,26.086069
44.457815,26.086090
44.459435,26.086106

Config file:

filter {

    csv {

        separator => ","

        skip_empty_rows => true

        columns => [ "latitude","longitude" ]

    }


    mutate {

        convert => ["latitude", "float"]

        convert => ["longitude", "float"]

        remove_field => ["message", "@version", "path", "host"]

    }



    mutate {

        add_field => {

            "[link_line][type]" =>  "LineString"            

        }

    }

    ruby {

        code => "event.set('[link_line][coordinates]', [event.get('%longitude').each, event.get('%latitude').each])" 

    }

}

I know that my ruby code is incorrect, but I don't know how to write it, I'm new to this. I want to put the data from csv in link_line and, in kibana, to visualize a line from the first point (44.453538,26.086064) to the second point (44.455499,26.086069), from the second point to the third point and so on.

code => "event.set('[link_line][coordinates]', [[ 26.086064, 44.453538], [ 26.086069, 44.455499], [ 26.086090, 44.457815], [ 26.086106, 44.459435]])

This works properly, but I would like to take the data from the csv file

Result in cmd:

{
        "link_line" => {
        "coordinates" => [
            [0] [
                [0] 26.086064,
                [1] 44.453538
            ],
            [1] [
                [0] 26.086069,
                [1] 44.455499
            ],
            [2] [
                [0] 26.08609,
                [1] 44.457815
            ],
            [3] [
                [0] 26.086106,
                [1] 44.459435
            ],
        ],
               "type" => "LineString"
    },
        "longitude" => 26.086069,
       "@timestamp" => 2021-03-11T15:07:02.815Z,
       "link_point" => "44.455499,26.086069",
         "latitude" => 44.455499
}

I hope I was quite explicit and someone can help me.

Thanks,
Vince

[Badger]

If you want the whole file as a list of coordinates I would consume it using a multiline code with a pattern that never matches...

file {
    path => "/home/user/foo.txt"
    sincedb_path => "/dev/null"
    start_position => beginning
    codec => multiline {
        pattern => "^Spalanzani"
        negate => true
        what => previous
        auto_flush_interval => 1
    }
}

The use a ruby filter to split it etc.

    ruby {
        code => '
            m = event.get("message").split
            m.shift # Discard header
            a = []
            m.each { |x|
                x = x.split(",")
                x[0] = x[0].to_f
                x[1] = x[1].to_f
                a << x
            }
            event.set("[link_line][coordinates]", a)
        '
    }

which will get you

 "link_line" => {
    "coordinates" => [
        [0] [
            [0] 44.453538,
            [1] 26.086064
        ],
        [1] [
            [0] 44.455499,
            [1] 26.086069
        ],
        [2] [
            [0] 44.457815,
            [1] 26.08609
        ],
        [3] [
            [0] 44.459435,
            [1] 26.086106
        ]
    ]
},

[vincedt09 (Vince DT)]

Hi! Thanks for the replay and help!

I'm using Windows 10 and 7.10.0 version. Sorry for not mentioning this.

I tried your method, but logstash get stuck at

[2021-03-13T10:40:38,401][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>1.46}
[2021-03-13T10:40:38,920][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2021-03-13T10:40:38,989][INFO ][filewatch.observingtail ][main][fb84af4c2f69110682fd4133d8d0f439dd4553ea0137fdfd1350591911204daf] START, creating Discoverer, Watch with file and sincedb collections
[2021-03-13T10:40:39,017][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[2021-03-13T10:40:39,480][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

If I remove the codec part the resuld in cmd looks like this:

{
"link_point" => "44.459435,26.086106",
"link_line" => {
"type" => "LineString",
"coordinates" =>
},
"latitude" => 44.459435,
"@timestamp" => 2021-03-13T09:15:31.420Z,
"longitude" => 26.086106,
"message" => "44.459435,26.086106\r"
}

[vincedt09 (Vince DT)]

Thanks @Badger for the replay!

I upgraded ELK to 7.11.2 version, rewrite the config file and it's work perfectly.

Best regards!