Geo_point problems with multiple indexes and Logstash

Elastic Stack Elasticsearch
1

Hi,

We have a problem loading geo_point data, the index template, after some tests, now is configured as:
location..........conflict
location.lat.....number
location.lon....number

So, when I try to create a map visualization there is no data.

We load the data from Logstash:
mutate {
add_field => [ "[geolocation][lat]", "%{lat}" ]
add_field => [ "[geolocation][lon]", "%{lng}" ]
convert => {
"[geolocation][lat]" => "float"
"[geolocation][lon]" => "float"
}
remove_field => [ "lat", "lng" ]
}

Until update to 7.2 it was working fine.

Something has changed or something we've started doing wrong, any ideas will be welcome.

Thank you.

2

as this seems to involve an index template with, an index template, and sample data, having a fully reproducible example would be great.

Also error messages or exceptions would help a lot.

Thanks!

3

Hi @jjfbcn

Couple things I see you should just create a type geo_point in your mapping you don't need to define the lat long separately as shown here

PUT my_index
{
  "mappings": {
    "properties": {
      "location": {
        "type": "geo_point"
      }
    }
  }
}

Also make sure the field name is the same looks like it may be location in the mapping and in logstash may be geolocation.

4

Hi @stephenb,

That was working but just for the existent data, the new data is agreggated in another index name and so it fails again. We have indexes in this format:

statsxxx-20190801
statsyyy-20190801

statsxxx-20190802
statsyyy-20190802
...

the problem is that when we try to put:

PUT stats*
{
  "mappings": {
    "properties": {
      "location": {
        "type": "geo_point"
      }
    }
  }
}

we receive the next error:

{
  "error": {
    "root_cause": [
      {
        "type": "security_exception",
        "reason": "action [indices:admin/create] is unauthorized for user [elastic]"
      }
    ],
    "type": "security_exception",
    "reason": "action [indices:admin/create] is unauthorized for user [elastic]",
    "caused_by": {
      "type": "illegal_state_exception",
      "reason": "There are no external requests known to support wildcards that don't support replacing their indices"
    }
  },
  "status": 403
}

we have activated the x-pack and configured the security, we're logged with user "elastic" that has, at least we think so, all the privileges, has superuser role.

Thank you,
5

Please try without the wildcard * on a single index and let us know what happens.

What version are you on?

6

Hi,

If we delete all data and indexes, and create the mapping without wildcard:

PUT stats
{
  "mappings": {
    "properties": {
      "location": {
        "type": "geo_point"
      }
    }
  }
}

{
"acknowledged" : true,
"shards_acknowledged" : true,
"index" : "stats"
}

Then we have the index:
stats with 0 documents

Then we load data and the index generated now are:
statsxxx-20190830 with 1 document
statsyyy-20190830 with 1 document
stats with 0 documents

In the index pattern we create a pattern named "stats*" and once created the mapping shows a conflict:

location conflict

location.lat number
location.lon numberk

Is it possible to create a default mapping for all indexes?

We have version 7.2.

Thank you

7

Yes use an index template see here it is exactly for this case.

It will be applied to all new indices that fit the pattern apologies I should have showed you this at first I just wasn't sure what you were trying to accomplish.

it would look something like this and every new index created with the pattern will apply this mapping

PUT _template/stats
{
  "index_patterns": ["stats*"],
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "properties": {
      "location": {
        "type": "geo_point"
      }
    }
  }
}
8

Thank you Stephen, the template did the trick, everything is working fine.

Now, I think I'm going to open a new thread, I'd like that each point drawed in the map show more information when you get the mouse over a single point, right now is displayed the latitude and longitude information, but I'd like to show other fields with more valuable information, I'm not sure if this is possible...

Regards,

9

Hi @stephenb I have the same issue and I tried to put index template but getting below error

PUT _template/file
{
"index_patterns": ["file*"],
"settings": {
"number_of_shards": 1
},
"mappings": {
"properties": {
"location": {
"type": "geo_point"
}
}
}
}

{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "Root mapping definition has unsupported parameters: [location : {type=geo_point}]"
}
],
"type": "mapper_parsing_exception",
"reason": "Failed to parse mapping [properties]: Root mapping definition has unsupported parameters: [location : {type=geo_point}]",
"caused_by": {
"type": "mapper_parsing_exception",
"reason": "Root mapping definition has unsupported parameters: [location : {type=geo_point}]"
}
},
"status": 400
}

10

@ashok9177 What version of Elasticsearch are you using?

11

6.5.1

12

this template is worked for me https://www.elastic.co/guide/en/elasticsearch/reference/6.5/indices-templates.html

but I am still looking conflict error

image
image.png1213×566 26.3 KB

13

That probably means there is more than 1 type of data for that field name for the indexes that the pattern applies too, meaning most likelly the pattern points to 1 or indexes that has the right data type geo_point and 1 or more that points to an index (probably before you create the mapping) that has the wrong data type.

14

I have deleted all existing indices, getting below error

[2019-09-18T05:40:54,063][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"file-2019.09.18", :_type=>"doc", :_routing=>nil}, #LogStash::Event:0x2203d9e7], :response=>{"index"=>{"_index"=>"file-2019.09.18", "_type"=>"doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [file-2019.09.18] as the final mapping would have more than 1 type: [_doc, doc]"}}}}

15

For 6.5.1
Probably should be "_type"=>"_doc"

16

below is my logstash configuration

filter {
if "iddiapi" in [fields][component] {
  grok {

  match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} (\[%{WORD:loglevel}\]) %{DATA} - SegmentName : %{WORD:segment} Longitude : %{BACULA_DEVICE:longitude} Latitude : %{BACULA_DEVICE:latitude}" }

  }



mutate {
    add_field => [ "[geolocation][lat]", "%{latitude}" ]
    add_field => [ "[geolocation][lon]", "%{longitude}" ]
}

 mutate {
            convert => {
          "[geolocation][lat]" => "float"
          "[geolocation][lon]" => "float"
}
}


}


output {
  elasticsearch {
    hosts => ["localhost"]
    user => 'elastic'
    password => 'password'
    manage_template => false
    index => "file-%{+YYYY.MM.dd}"
}

My sample log message

2019-09-18 12:57:16.398 [INFO] from application in pool-1-thread-21 - SegmentName : IDVerified Longitude : 17.467793 Latitude : 78.388074 for transactionId : 56ou9P3JQ-gDo6

is it correct tor not? can you please suggest me any changes ?

17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.