How to set geo_point type from csv files using logstash?

Elastic Stack Logstash
1

Hi, I'm trying to set geo_point type from csv files which have latitude and longitude using logstash. But I really don't know how to convert latitude and longitude to location(geo_point type). I'm using logstash for this, but it seems that there is not enough information about it.
I found this way.

input {
  file {
    path => "/Users/kakao/Downloads/coronavirusdataset_20200601/Case.csv"
    start_position => "beginning"
    #sincedb_path => "/dev/null"    
  }
}
filter {
  csv {
      separator => ","
      columns => ["case_id","province","city","group","infection_case","confirmed","latitude","longitude"]
  }
  mutate {convert => ["latitude", "float"]}
  mutate {convert => ["longitude", "float"]}
  mutate {convert => ["confirmed", "float"]}
  mutate {rename => ["latitude", "[location][lat]"]}
  mutate {rename => ["longitude", "[location][lon]"]}
output {  
    elasticsearch {
        template => "/Users/kakao/Downloads/logstash-7.7.1/config/elasticsearch-template.json"
        template_overwrite => true
        action => "index"
        hosts => ["http://localhost:9200"]
        index => "logstash-%{+YYYY.MM.dd}"
        user => "user-name"
        password => "password"
        workers => 1
    }
    stdout {}
}

and this is my elasticsearch-template.json.

{
    "template" : "logstash-*",
    "settings" : {
      "index.refresh_interval" : "5s"
    },
    "mappings" : {
      "_default_" : {
        "_all" : {"enabled" : true, "omit_norms" : true},
        "dynamic_templates" : [ {
          "message_field" : {
            "match" : "message",
            "match_mapping_type" : "string",
            "mapping" : {
              "type" : "string", "index" : "analyzed", "omit_norms" : true,
              "fielddata" : { "format" : "disabled" }
            }
          }
        }, {
          "string_fields" : {
            "match" : "*",
            "match_mapping_type" : "string",
            "mapping" : {
              "type" : "string", "index" : "analyzed", "omit_norms" : true,
              "fielddata" : { "format" : "disabled" },
              "fields" : {
                "raw" : {"type": "string", "index" : "not_analyzed", "ignore_above" : 256}
              }
            }
          }
        } ],
        "properties" : {
          "@timestamp": { "type": "date" },
          "@version": { "type": "string", "index": "not_analyzed" },
          "geoip"  : {
            "dynamic": true,
            "properties" : {
              "ip": { "type": "ip" },
              "location" : { "type" : "geo_point" },
              "latitude" : { "type" : "float" },
              "longitude" : { "type" : "float" }
            }
          },
    "location" : { "type": "geo_point" }
        }
      }
    }
  }

Finally, I got geoip.location which is geo_point type, but I couldn't find any data of geoip.location in discover. I only have location.lat, location.lon which are numbers. It means that I just have field, but no data in that field.

2

You may have to refresh your index pattern from Kibana console for the new fields to appear if the index already existed from before and the mappings was changed.

3

스크린샷 2020-06-09 오전 12.10.52
스크린샷 2020-06-09 오전 12.10.522170×156 5.03 KB
스크린샷 2020-06-09 오전 12.11.03
스크린샷 2020-06-09 오전 12.11.032186×322 19.2 KB
스크린샷 2020-06-09 오전 12.11.32
스크린샷 2020-06-09 오전 12.11.32432×1240 16 KB

The problem is like above. When I look at discover, it only has location.lat and location.lon fields. There is no geoip.location what I want....

4

Did you try refreshing the index pattern?

5

스크린샷 2020-06-09 오전 12.17.10
스크린샷 2020-06-09 오전 12.17.102224×334 42.4 KB

Yes, I just have refreshed the index pattern, but it outputs same result.
Should I have datas related to "geoip" in the csv file I'm using? Cause I only have longitude and latitude datas...

6

change this to

mutate {rename => ["latitude", "[geoip][location][lat]"]}
  mutate {rename => ["longitude", "[geoip][location][lon]"]}
7

Thanks for your reply.. But it doesn't work anyway..
If like that, there is no geoip.location which is geo_point type.

8

geo_point data type accepts latitude longitude pair. in your previous screenshot you already have geoip.location with geo_point data type. which means you only need to add lat-lon pair to that field, which is why i suggested the previous step.
latitude and longitude are stored as numeric float.

more info about geo_point

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.