Adding a GPS entry in custom field in config file

[lnac98]

Hello, everyone. New to Beats, and to the entire ELK stack so forgive me if this is an easy one (although nothing has shown up on my searches). I need to add some static data into Elasticsearch from Filebeats and I am doing that via the custom fields in the filebeat.yml config file. One of these pieces of information, however, needs to be treated as a GPS location. I can't seem to get Elasticsearch/Kibana to look at this field as a geo-point/geo-locatino/geo-hash. I've tried different formats (see below), but nothing seems to work. Can this be accomplished, or am I barking up the wrong tree? Looking forward to hearing back from you guys. Thanks. Lance.

fields:
camera_loc: 9q8yyk8yuv5k (as geo-hash)
camera_city: San Francisco
camera_nbr: 11527

fields:
camera_loc: [37.774929, -122.419416] (as coordinates)
camera_city: San Francisco
camera_nbr: 11527

[andrewkroh]

What version of Elasticsearch are you using? What version of Filebeat?

[lnac98]

Latest and greatest of both (production releases, that is)

[andrewkroh]

You should just need an index template that marks camera_loc as a geo_point. The index template will only affect new indices. So if you already have a daily index for today then this will take affect tomorrow.

curl -XPUT "http://elasticsearch:9200/_template/filebeat-geo?pretty" -d'
{
  "order": 1,
  "template": "filebeat-*",
  "mappings": {
    "_default_": {
      "properties": {
        "camera_loc": {
          "type": "geo_point"
        }
      }
    }
  }
}'

This solution works because Elasticsearch will merge all the templates that apply to an index. So the default index template from Filebeat will be applied first then this one will be merged in next (order 1). I believe that template merge is either deprecated or removed in ES 6 so for 6 you need to add this field to the main template installed by Filebeat.

[lnac98]

Sweet! I'll give this a try!

[lnac98]

So I applied the changes, but you are saying that there is no way to see if it worked until tomorrow? Am I understanding that correctly?

[andrewkroh]

You can delete today's index, but you will lose today's data.

curl -X DELETE http://elasticsearch:9200/filebeat-2017.10.24?pretty

After it is deleted, when the next event is written the index will be recreated using the new index template.

You can use curl http://elasticsearch:9200/filebeat-*/_mapping/field/current_loc?pretty to check the mapping for that field.

Then in Kibana you will need to reload the index pattern.

[lnac98]

Aha. Great explanation. I will need to wait until tomorrow then. Thank you so much for your assistance!

[lnac98]

Hey, so I did finally get this to work today...but I had to actually delete my index from within Kibana in order for it to pick the field up as a geopoint. I tried all of the things you suggested first, but this is what it took in my case. Probably a more elegant solution out there, but that's what worked for me. Thanks for all of your help. Lance.

[andrewkroh]

It's solved now, but FYI it is possible to reindex the data through Elasticsearch's API

For example:

POST _reindex
{
  "source": {
    "index": "filebeat-2017.01.01"
  },
  "dest": {
    "index": "filebeat-2017.01.01a"
  }
}

Then once reindexing completes you would delete the filebeat-2017.01.01 index.

DELETE filebeat-2017.01.01