# Geo\_point problems with multiple indexes and Logstash

**URL:** <https://discuss.elastic.co/t/geo-point-problems-with-multiple-indexes-and-logstash/197121>\
**Category:** Elasticsearch\
**Created:** [August 28, 2019, 1:24pm UTC](https://discuss.elastic.co/t/geo-point-problems-with-multiple-indexes-and-logstash/197121 "2019-08-28T13:24:14Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![jjfbcn](https://avatars.discourse-cdn.com/v4/letter/j/7cd45c/32.png) [@jjfbcn](https://discuss.elastic.co/u/jjfbcn)\
**Post date:** [August 28, 2019, 1:24pm UTC](https://discuss.elastic.co/t/geo-point-problems-with-multiple-indexes-and-logstash/197121/1 "2019-08-28T13:24:14Z")

</div>

Hi,

We have a problem loading geo\_point data, the index template, after some tests, now is configured as:  
location..........conflict  
location.lat.....number  
location.lon....number

So, when I try to create a map visualization there is no data.

We load the data from Logstash:  
mutate {  
add\_field =\> ["[geolocation][lat]", "%{lat}" ]  
add\_field =\> ["[geolocation][lon]", "%{lng}" ]  
convert =\> {  
"[geolocation][lat]" =\> "float"  
"[geolocation][lon]" =\> "float"  
}  
remove\_field =\> ["lat", "lng"]  
}

Until update to 7.2 it was working fine.

Something has changed or something we've started doing wrong, any ideas will be welcome.

Thank you.

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [August 28, 2019, 2:27pm UTC](https://discuss.elastic.co/t/geo-point-problems-with-multiple-indexes-and-logstash/197121/2 "2019-08-28T14:27:12Z")

</div>

as this seems to involve an index template with, an index template, and sample data, having a fully reproducible example would be great.

Also error messages or exceptions would help a lot.

Thanks!

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 28, 2019, 2:28pm UTC](https://discuss.elastic.co/t/geo-point-problems-with-multiple-indexes-and-logstash/197121/3 "2019-08-28T14:28:48Z")

</div>

Hi @jjfbcn

Couple things I see you should just create a type `geo_point` in your mapping you don't need to define the lat long separately as shown [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html)

```
PUT my_index
{
  "mappings": {
    "properties": {
      "location": {
        "type": "geo_point"
      }
    }
  }
}

```

Also make sure the field name is the same looks like it may be `location` in the mapping and in logstash may be `geolocation`.

---

<div class="post-metadata">

**Author:** ![jjfbcn](https://avatars.discourse-cdn.com/v4/letter/j/7cd45c/32.png) [@jjfbcn](https://discuss.elastic.co/u/jjfbcn)\
**Post date:** [August 29, 2019, 6:44am UTC](https://discuss.elastic.co/t/geo-point-problems-with-multiple-indexes-and-logstash/197121/4 "2019-08-29T06:44:36Z")

</div>

Hi @stephenb,

That was working but just for the existent data, the new data is agreggated in another index name and so it fails again. We have indexes in this format:

statsxxx-20190801  
statsyyy-20190801

statsxxx-20190802  
statsyyy-20190802  
...

the problem is that when we try to put:

```auto
PUT stats*
{
  "mappings": {
    "properties": {
      "location": {
        "type": "geo_point"
      }
    }
  }
}

we receive the next error:

{
  "error": {
    "root_cause": [
      {
        "type": "security_exception",
        "reason": "action [indices:admin/create] is unauthorized for user [elastic]"
      }
    ],
    "type": "security_exception",
    "reason": "action [indices:admin/create] is unauthorized for user [elastic]",
    "caused_by": {
      "type": "illegal_state_exception",
      "reason": "There are no external requests known to support wildcards that don't support replacing their indices"
    }
  },
  "status": 403
}

we have activated the x-pack and configured the security, we're logged with user "elastic" that has, at least we think so, all the privileges, has superuser role.

Thank you,
```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 29, 2019, 2:07pm UTC](https://discuss.elastic.co/t/geo-point-problems-with-multiple-indexes-and-logstash/197121/5 "2019-08-29T14:07:29Z")

</div>

Please try without the wildcard `*` on a single index and let us know what happens.

What version are you on?

---

<div class="post-metadata">

**Author:** ![jjfbcn](https://avatars.discourse-cdn.com/v4/letter/j/7cd45c/32.png) [@jjfbcn](https://discuss.elastic.co/u/jjfbcn)\
**Post date:** [August 30, 2019, 6:54am UTC](https://discuss.elastic.co/t/geo-point-problems-with-multiple-indexes-and-logstash/197121/6 "2019-08-30T06:54:25Z")

</div>

Hi,

If we delete all data and indexes, and create the mapping without wildcard:

```auto
PUT stats
{
  "mappings": {
    "properties": {
      "location": {
        "type": "geo_point"
      }
    }
  }
}

```

{  
"acknowledged" : true,  
"shards\_acknowledged" : true,  
"index" : "stats"  
}

Then we have the index:  
stats with 0 documents

Then we load data and the index generated now are:  
statsxxx-20190830 with 1 document  
statsyyy-20190830 with 1 document  
stats with 0 documents

In the index pattern we create a pattern named "stats\*" and once created the mapping shows a conflict:

location conflict

location.lat number   
location.lon numberk

Is it possible to create a default mapping for all indexes?

We have version 7.2.

Thank you

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 30, 2019, 1:46pm UTC](https://discuss.elastic.co/t/geo-point-problems-with-multiple-indexes-and-logstash/197121/7 "2019-08-30T13:46:27Z")

</div>

Yes use an index template see [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html) it is exactly for this case.

It will be applied to all new indices that fit the pattern apologies I should have showed you this at first I just wasn't sure what you were trying to accomplish.

it would look something like this and every new index created with the pattern will apply this mapping

```
PUT _template/stats
{
  "index_patterns": ["stats*"],
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "properties": {
      "location": {
        "type": "geo_point"
      }
    }
  }
}
```

---

<div class="post-metadata">

**Author:** ![jjfbcn](https://avatars.discourse-cdn.com/v4/letter/j/7cd45c/32.png) [@jjfbcn](https://discuss.elastic.co/u/jjfbcn)\
**Post date:** [September 2, 2019, 12:08pm UTC](https://discuss.elastic.co/t/geo-point-problems-with-multiple-indexes-and-logstash/197121/8 "2019-09-02T12:08:30Z")

</div>

Thank you Stephen, the template did the trick, everything is working fine.

Now, I think I'm going to open a new thread, I'd like that each point drawed in the map show more information when you get the mouse over a single point, right now is displayed the latitude and longitude information, but I'd like to show other fields with more valuable information, I'm not sure if this is possible...

Regards,

---

<div class="post-metadata">

**Author:** ![ashok9177](https://avatars.discourse-cdn.com/v4/letter/a/edb3f5/32.png) [@ashok9177](https://discuss.elastic.co/u/ashok9177)\
**Post date:** [September 17, 2019, 12:29pm UTC](https://discuss.elastic.co/t/geo-point-problems-with-multiple-indexes-and-logstash/197121/9 "2019-09-17T12:29:36Z")

</div>

Hi @stephenb I have the same issue and I tried to put index template but getting below error

> PUT \_template/file  
> {  
> "index\_patterns": ["file\*"],  
> "settings": {  
> "number\_of\_shards": 1  
> },  
> "mappings": {  
> "properties": {  
> "location": {  
> "type": "geo\_point"  
> }  
> }  
> }  
> }

> {  
> "error": {  
> "root\_cause": [  
> {  
> "type": "mapper\_parsing\_exception",  
> "reason": "Root mapping definition has unsupported parameters: [location : {type=geo\_point}]"  
> }  
> ],  
> "type": "mapper\_parsing\_exception",  
> "reason": "Failed to parse mapping [properties]: Root mapping definition has unsupported parameters: [location : {type=geo\_point}]",  
> "caused\_by": {  
> "type": "mapper\_parsing\_exception",  
> "reason": "Root mapping definition has unsupported parameters: [location : {type=geo\_point}]"  
> }  
> },  
> "status": 400  
> }

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [September 17, 2019, 1:41pm UTC](https://discuss.elastic.co/t/geo-point-problems-with-multiple-indexes-and-logstash/197121/10 "2019-09-17T13:41:52Z")

</div>

@ashok9177 What version of Elasticsearch are you using?

---

<div class="post-metadata">

**Author:** ![ashok9177](https://avatars.discourse-cdn.com/v4/letter/a/edb3f5/32.png) [@ashok9177](https://discuss.elastic.co/u/ashok9177)\
**Post date:** [September 17, 2019, 1:51pm UTC](https://discuss.elastic.co/t/geo-point-problems-with-multiple-indexes-and-logstash/197121/11 "2019-09-17T13:51:17Z")

</div>

6.5.1

---

<div class="post-metadata">

**Author:** ![ashok9177](https://avatars.discourse-cdn.com/v4/letter/a/edb3f5/32.png) [@ashok9177](https://discuss.elastic.co/u/ashok9177)\
**Post date:** [September 17, 2019, 2:00pm UTC](https://discuss.elastic.co/t/geo-point-problems-with-multiple-indexes-and-logstash/197121/12 "2019-09-17T14:00:14Z")

</div>

this template is worked for me [https://www.elastic.co/guide/en/elasticsearch/reference/6.5/indices-templates.html](https://www.elastic.co/guide/en/elasticsearch/reference/6.5/indices-templates.html)

but I am still looking conflict error

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/0/f0adca7b6d5cc037dbe8a770b2bfc9548c63f1af.png)

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [September 17, 2019, 2:09pm UTC](https://discuss.elastic.co/t/geo-point-problems-with-multiple-indexes-and-logstash/197121/13 "2019-09-17T14:09:57Z")

</div>

That probably means there is more than 1 type of data for that field name for the indexes that the pattern applies too, meaning most likelly the pattern points to 1 or indexes that has the right data type `geo_point` and 1 or more that points to an index (probably before you create the mapping) that has the wrong data type.

---

<div class="post-metadata">

**Author:** ![ashok9177](https://avatars.discourse-cdn.com/v4/letter/a/edb3f5/32.png) [@ashok9177](https://discuss.elastic.co/u/ashok9177)\
**Post date:** [September 18, 2019, 5:43am UTC](https://discuss.elastic.co/t/geo-point-problems-with-multiple-indexes-and-logstash/197121/14 "2019-09-18T05:43:53Z")

</div>

I have deleted all existing indices, getting below error

> [2019-09-18T05:40:54,063][WARN][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=\>400, :action=\>["index", {:\_id=\>nil, :\_index=\>"file-2019.09.18", :\_type=\>"doc", :\_routing=\>nil}, #LogStash::Event:0x2203d9e7], :response=\>{"index"=\>{"\_index"=\>"file-2019.09.18", "\_type"=\>"doc", "\_id"=\>nil, "status"=\>400, "error"=\>{"type"=\>"illegal\_argument\_exception", "reason"=\>"Rejecting mapping update to [file-2019.09.18] as the final mapping would have more than 1 type: [\_doc, doc]"}}}}

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [September 18, 2019, 5:59am UTC](https://discuss.elastic.co/t/geo-point-problems-with-multiple-indexes-and-logstash/197121/15 "2019-09-18T05:59:31Z")

</div>

> [@ashok9177](#):
>
> "\_type"=\>"doc"

For 6.5.1  
Probably should be `"_type"=>"_doc"`

---

<div class="post-metadata">

**Author:** ![ashok9177](https://avatars.discourse-cdn.com/v4/letter/a/edb3f5/32.png) [@ashok9177](https://discuss.elastic.co/u/ashok9177)\
**Post date:** [September 18, 2019, 6:01am UTC](https://discuss.elastic.co/t/geo-point-problems-with-multiple-indexes-and-logstash/197121/16 "2019-09-18T06:01:18Z")

</div>

below is my logstash configuration

```
filter {
if "iddiapi" in [fields][component] {
  grok {

  match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} (\[%{WORD:loglevel}\]) %{DATA} - SegmentName : %{WORD:segment} Longitude : %{BACULA_DEVICE:longitude} Latitude : %{BACULA_DEVICE:latitude}" }

  }

mutate {
    add_field => ["[geolocation][lat]", "%{latitude}" ]
    add_field => ["[geolocation][lon]", "%{longitude}" ]
}

 mutate {
            convert => {
          "[geolocation][lat]" => "float"
          "[geolocation][lon]" => "float"
}
}

}

output {
  elasticsearch {
    hosts => ["localhost"]
    user => 'elastic'
    password => 'password'
    manage_template => false
    index => "file-%{+YYYY.MM.dd}"
}

```

My sample log message

> 2019-09-18 12:57:16.398 [INFO] from application in pool-1-thread-21 - SegmentName : IDVerified Longitude : 17.467793 Latitude : 78.388074 for transactionId : 56ou9P3JQ-gDo6

is it correct tor not? can you please suggest me any changes ?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 16, 2019, 6:04am UTC](https://discuss.elastic.co/t/geo-point-problems-with-multiple-indexes-and-logstash/197121/17 "2019-10-16T06:04:35Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.