Add a field as a "parent" parameter with add_field (for geo_point)

Elastic Stack Logstash
1

Hi,

I have a problem with the geo_point type. I already applied a template on elasticsearch to transform the data i get into a geo_point. This data is supposed to be the parameter location.
I don't receive it from the log, that's why i want to add a new field named location by using the add_field property.

I receive the logs in this format:
08-07-2016 14:10:22.039 country=Germany,lat=52,lon=12

Thus, i would like to construct a parameter Location, parent of lat and lon to match my template.

My filter in logstash for this line looks like this:

 filter { 
     grok 
         ["message_loc", "%{LOG_DATE:log_date} %{GREEDYDATA:msg_location}"] 
     }
 }
 filter {
     kv {
         field_split => ","
         source => "msg_location"
     }
 }
 filter {
     mutate {
         convert => [ "lat", "float" ]
     }
     mutate {
         convert => [ "lon", "float" ]
     }
     mutate {
         add_field => {
             "[location][lat]" => "lat"
             "[location][lon]" => "lon" 
         }
     }
 }

I tried some solutions that come from this post:

First of all, i don't know if changing the type of lat and lon before is the right way to do it or not.

Then, the second problem is that the add_field value seems to block the data. I'm also supposed to receive other logs through the same filter. However, no data are received in ES and Kibana at all.
I made a test by removing this field and it seemed to work. But i need this field :sweat_smile:
If you have some ideas.

If you need more informations don't hesitate.
Thank you for your attention.

2

Have you run this on the data and seen the output using stdout with rubydebug? I don't think it does what you think it does.

3

Hi, thank you for your answer.

I just used stdout, and i replaced message_loc by message.
Also, LOG_DATE is something i defined in my filter and i didn't mention in this topic.

After using the output stdout function i got these results:

{
            "message" => "13-07-2016 14:35:39.000 country=Germany,lat=52,lon=12",
           "@version" => "1",
         "@timestamp" => "2016-07-13T12:35:39.000Z",
         "input_type" => "log",
             "fields" => nil,
               "type" => "log",
              "count" => 1,
               "beat" => {
           "hostname" => "lls-geoloc-elk-gs-environment-compute-gsa-83ge8",
               "name" => "lls-geoloc-elk-gs-environment-compute-gsa-83ge8"
},
             "source" => "/tmp/geekshop.log",
             "offset" => 207856325,
               "host" => "louis-server-geoloc-elk-gs-environment-compute-gsa-83ge8",
               "tags" => [
            [0] "beats_input_codec_plain_applied"
        ],
           "log_date" => "13-07-2016 14:35:39.000",
            "country" => "Germany",
                "lat" => 52,
                "lon" => 12
           "location" => {
                "lat" => "lat",
                "lon" => "lon"
     }

}

Actually, it's what i expected.
But i still don't get my geolocation.lat and geolocation.lon parameter in Kibana

This is my template for ES:

curl -XPUT localhost:9200/_template/template1 -d '
{
     "template": "logstash-*", 
     "settings":  {
           "number_of_shards": 1 
     }, 
     "mappings": { 
           "log": { 
                  "properties" : { 
                         "location": { 
                                "type": "geo_point"
                          } 
                   } 
            } 
      }
}'