Problem to parse geo_shape from .csv trough logstash

[grainsngrubble]

Hi! I´m trying to ingest a .csv with geo_shapes as linestrings. Though I keep getting parse exceptions. Have tries to search and to read the documentation, but no progress so I put my faith in you now

Even though I occasionally succeed to transfer the data to Elasticsearch with Logstash, Kibana won't recognize it as a geo_shape... any suggestions?

Geo_shape in input.csv below: [[lon, lat],[lon, lat]]

[[-74.0446, 40.6899],[-74.0416, 40.6996]]

Logstash.conf

input {
	file {	  path => ["D:/input.csv"]
			  start_position => beginning
			  sincedb_path => "D:/Logstash/logs/last_value_csv.log"		
	}
}
filter {
	csv {     separator => 		","
		     columns => 	[ "lineprojection" ]
		     convert =>         { "[lineprojection]" => "float" }
	}
	mutate {  remove_field =>       [ "message" ]
        }	
}
output {
    elasticsearch {
              hosts => "localhost:9200"
              action => "index"
              manage_template => true
              document_id => "%{timestamp}"
              index => "lines"
              template => "D:/template.json"
              template_name => "template"
    }
    stdout {  codec => rubydebug 
    }
}

Template

{
  "order": 0,
  "index_patterns": [ "lines*" ],
    "settings": {
      "index" : {
        "lifecycle" : {
          "name" : "lines-policy",
          "rollover_alias" : "lines"
        },
        "number_of_shards" : "1",
        "number_of_replicas" : "0"
        }
      },
      "mappings": {
          "properties": {
            "lineprojection": {
              "type": "geo_shape"
            }
       }
    }
}

[grainsngrubble]

I found a solution myself with a lot of help from this post by Badger and thought I share it if others encounter the same issue.

First, the coordinates should not be arranged in an array in the inout.csv file. Of course you can split the array etc in logstash. But for convenience I arranged my coordinates as below:

-74.0446, 40.6899,-74.0416, 40.6996

My logstash.conf for converting the coordinates into a geo_point location and a linestring geo_shape. I have excluded the part not relevant for the geo operations.

filter {
	 csv {     separator =>    ","
	           columns =>      [ "loc_lat","loc_lon","proj_lat","proj_lon" ]
	 }
	 mutate {  add_field =>    ["[location]", "%{loc_lon}"]
                   add_field =>    ["[location]", "%{loc_lat}"]
         }
         mutate {  convert =>      ["[location]", "float"]
                   convert =>      ["[loc_lat]", "float"]
                   convert =>      ["[loc_lon]", "float"]
                   convert =>      ["[proj_lat]", "float"]
                   convert =>      ["[proj_lon]", "float"]
         }
         mutate {  add_field =>    ["[projection][type]", "linestring"]
         }
         ruby {    code =>          "event.set('[projection][coordinates]', [[ event.get('loc_lon'), event.get('loc_lat')], [ event.get('proj_lon'), event.get('proj_lat')]])"
}

Result in cmd with rubedebug:

  {          
        "projection" => {
                "coordinates" => [
                    [0] [
                        [0] -74.0446,
                        [1] 40.6899
                    ],
                    [1] [
                        [0] -74.0416,
                        [1] 40.6996
                    ]
                ],
                       "type" => "linestring"
            },
                "location" => [
                    [0] -74.0446,
                    [1] 40.6899
                ]
  }