Add Geopoint based off of parsed value to logstash config

Elastic Stack Logstash
1

Hello,

Part of my matched message returns the fields
%{NUMBER:XCent} %{NUMBER:YCent}
which are lat, long points.

I'm attempting to add a location pin but keep getting a config failure when i use the --debug flag on my configuration file apparently it's expecting a # symbol???

    if [XCent] and [YCent] {
        "location" : {
            "lat" : %{XCent},
            "lon" : %{YCent},
            "type" : "geo_point"
        }
    }

Any suggestions? Much thanks!

2

Is that what's in your configuration file? Because it barely resembles Logstash's configuration file syntax. You need something like this:

filter {
  if [XCent] and [YCent] {
    mutate {
      add_field => {
        "[location][lat]" => "%{XCent}"
        "[location][lon]" => "%{YCent}"
      }
    }
    mutate {
      convert => {
        "[location][lat]" => "float"
        "[location][lon]" => "float"
      }
    }
  }
}
3

Sorry for the confusion, that was only an excerpt from my logstash configuration file.

Your suggestion passes the debug flag, but the location points still don't show up in the tile map. Can I change the second conversion to:

  convert => {
    "[location][lat]" => "float"
    "[location][lon]" => "float"
    "[location]" => "geo_point"
  }

Or is there a different recommended way to drop the pin of the location into a format that the tile map can pick up?

Again thank you for the assistance thus far!

4

The mutate filter can only change between different JSON types and "geo_point" is an ES concept. To have a field mapped as geo_point I think you need to modify the mapping.

(For testing configurations you want the --configtest option, not --debug. Well, the latter will implicitly test the configuration upon startup but so will running Logstash without any such options.)

5

I've looked at the gioip filter rb files, and the elasticsearch.yml file. I'm not sure where I apply the mapping for the location field to cast it as a geo point. I also tried to use the geoip plugin and override the values as a work around, but it's just not coming together easily.

Thanks for the suggestion of the --configtest flag, it's made debugging a lot quicker!

6

Have a look at index templates. Logstash manages the index template for logstash-* indexes by default but you can override it and supply your own template (based on the Logstash default). In that template you can specify that your location field has the geo_point type.

7

My apologies I'm still very confused by this matter.

I've reviewed both of the guides by elastic.
https://www.elastic.co/guide/en/elasticsearch/reference/1.5/indices-templates.html
https://www.elastic.co/guide/en/elasticsearch/guide/current/custom-dynamic-mapping.html

As well as this stack overflow post

I guess I'm just not sure where to save the new template. Again thanks for all the knowledge.

8

Eventually the template is saved in the cluster itself, but you'll typically have a version-controlled source file somewhere (i.e. anywhere).

The simplest way is to make a copy of Logstash's template file (the exact path varies but IIRC the filename is elasticsearch-template.json) and point the template option of the elasticsearch output to it. Then modify the template to your needs, i.e. by adding a geo_point-typed field. When Logstash creates the next index after the configuration change (typically tomorrow's index) that index should have the correct mapping.

Another option is to disable Logstash's index template handling altogether with the manage_template option and post your tempate into the cluster on your own. That's what I prefer, but YMMV.

9

Thank you! That's exactly what I had been missing :slight_smile:

10

I am using elasticsearch 2.3.3 and logstash-2.3.1. What is the best way to create the geo_point

11

@spraveenjd, please start a new thread for your question, and when you do please ask a more specific question that includes information about what information you want to turn into a geo_point (an example event as produced by a stdout { codec => rubydebug } would be great) and your current configuration.

12

Hi,

Also i have difficulty setting the type to geo_point for the geoip.location field, this is my .conf you have tips ??

  if [FRAZ_IP] =~ /.+/ {
    geoip {
      source => "FRAZ_IP"
      target => "geoip"
      fields => ["city_name", "latitude", "longitude", "location","ip"]
      database => "/home/elkadmin/geoip/geo_ip_poste.mmdb"
      }
      mutate {
             add_field => ["[geoip][location]","%{longitude}"]
             add_field => ["[geoip][location]","%{latitude}"]
            }
    mutate {
           convert => [ "[geoip][location]", "float" ]
           }
}

}

this is my pattern index after the import logs

Cattura.PNG1471×131 7.76 KB

13

I have followed this post, and still not able to convert the location to geo_point,

I have created another post details

https://discuss.elastic.co/t/location-to-geo-point-mapping-not-working-with-logtash-and-kibana/160980/3

I am using manage_template and have my own template with location as type geo_point

14

I solved my issue by using this solution

    filter{
    mutate {
      add_field => {
        "[location][type]" => "point"
      }
    }
	ruby{
		code => '
			event.set("[location][coordinates]", [event.get("latitude"), event.get("longitude")])
		'
	}
}