Geo_point Unindexed

Elastic Stack Elasticsearch
1

Hi,

I am not able to get the geo_point indexed in the records
The field gets converted with correct latitude and longitude fields
storing it to location field also gets added successfully
Also create a mapping template as below:

PUT _template/map
{
  "template": "scada-*",
  "settings": {},
  "mappings": {
    "_default_": {
      "properties": {
        "location": {
          "type": "geo_point",
            "ignore_malformed":true
          }
        }
      }
  }
}

Logstash Configuration

filter
{
mutate {
convert => {"latitude" => "float"}
convert => {"longitude" => "float"}
add_field => ["location","%{latitude},%{longitude}"]
convert => {"location" => "float"}
         }
}

Should i be using the default logstash template or make changes to the same?

2

Still facing the issue.
Now i am getting the location field as "string".
The problem is there are documents which are not parsed if the field do not have the latitude or longitude
Event though mentioning the "location" field as float in the filter config applied.
Also tried placing the ignore_malformed in places in the mapping.
Still no luck.

3

Whats the output section look like?

4

Here is the output

output { 
elasticsearch
{ 
hosts => ["array of hosts"]
document_id => "%{field1}%{field2}"
index => "events-%{+YYYY.MM.dd}"
}
}
5

Does not match;

The template directive above refers to the index pattern it should be matching.

6

I believe i tried that already , well that might be the issue, i have to check when i am back to work.
Also will that index the geo_point location field? Since i did get the geo_point field but couldn't get that indexed.

During this struggle i removed the template above and tried to use the default logstash template named "elasticsearch-template_es5x.json"

Will that also not bring the correct mapping because of the reason you mentioned about "directive not matching" have an effect on the data we index?

7

Most likely, the default template matches logstash-*.

8

I believe i tried that already , well that might be the issue, i have to check that again
Also will that index the geo_point location field? Since i did get the geo_point field but couldn't get that indexed.

And what needs to be added in the configuration to make sure the records with no Lat/lon details are not dropped from the index?

9

You'll need to zero them out so it holds a value, because a null value geopoint is invalid.

10

The value for such records is

latitude: -
longitude: -

currently the fields bring the empty data in the index like the way it is added
location : %{latitude},%{longitude}

How to zero them out? in Logstash config or through ES mapping?

11

You can do it in Logstash, with a conditional, or you can use something like https://www.elastic.co/guide/en/elasticsearch/reference/5.6/null-value.html in Elasticsearch.

1 Like
12

This also tried but no luck , I am getting all records but the location field still parsed as string

13

After trying multiple mappings and configuration changes The location field gets to geo_point but remains unindexed.(missing events which do not have latitude or longitude)
Please help, I believe this is the last hurdle

14

latlon

Different results , yet not getting indexed
Have been after it since a long time. No luck

15

What do you mean? It looks like it's a valid geopoint.

16

But i am not able to use it in my visualizations as this is unindexed

17

The geopoint? What sort of visualisations are you looking at doing, because the main point of that is for maps.

18

That is correct, i need this for the Maps only,

19

Ok Now i am able to see the location field working!!!
WOW!!
Thanks for all the help Mark @warkolm
Also last thing, since this is an unindexed field, will i be able to apply filters to it?

map
map.PNG664×762 78 KB

20

Yes you can apply filters to it.