Geo_point problems with multiple indexes and Logstash

jjfbcn · August 28, 2019, 1:24pm

Hi,

We have a problem loading geo_point data, the index template, after some tests, now is configured as:
location..........conflict
location.lat.....number
location.lon....number

So, when I try to create a map visualization there is no data.

We load the data from Logstash:
mutate {
add_field => [ "[geolocation][lat]", "%{lat}" ]
add_field => [ "[geolocation][lon]", "%{lng}" ]
convert => {
"[geolocation][lat]" => "float"
"[geolocation][lon]" => "float"
}
remove_field => [ "lat", "lng" ]
}

Until update to 7.2 it was working fine.

Something has changed or something we've started doing wrong, any ideas will be welcome.

Thank you.

spinscale · August 28, 2019, 2:27pm

as this seems to involve an index template with, an index template, and sample data, having a fully reproducible example would be great.

Also error messages or exceptions would help a lot.

Thanks!

stephenb · August 28, 2019, 2:28pm

Hi @jjfbcn

Couple things I see you should just create a type geo_point in your mapping you don't need to define the lat long separately as shown here

PUT my_index
{
  "mappings": {
    "properties": {
      "location": {
        "type": "geo_point"
      }
    }
  }
}

Also make sure the field name is the same looks like it may be location in the mapping and in logstash may be geolocation.

jjfbcn · August 29, 2019, 6:44am

Hi @stephenb,

That was working but just for the existent data, the new data is agreggated in another index name and so it fails again. We have indexes in this format:

statsxxx-20190801
statsyyy-20190801

statsxxx-20190802
statsyyy-20190802
...

the problem is that when we try to put:

PUT stats*
{
  "mappings": {
    "properties": {
      "location": {
        "type": "geo_point"
      }
    }
  }
}

we receive the next error:

{
  "error": {
    "root_cause": [
      {
        "type": "security_exception",
        "reason": "action [indices:admin/create] is unauthorized for user [elastic]"
      }
    ],
    "type": "security_exception",
    "reason": "action [indices:admin/create] is unauthorized for user [elastic]",
    "caused_by": {
      "type": "illegal_state_exception",
      "reason": "There are no external requests known to support wildcards that don't support replacing their indices"
    }
  },
  "status": 403
}

we have activated the x-pack and configured the security, we're logged with user "elastic" that has, at least we think so, all the privileges, has superuser role.

Thank you,

stephenb · August 29, 2019, 2:07pm

Please try without the wildcard * on a single index and let us know what happens.

What version are you on?

jjfbcn · August 30, 2019, 6:54am

Hi,

If we delete all data and indexes, and create the mapping without wildcard:

PUT stats
{
  "mappings": {
    "properties": {
      "location": {
        "type": "geo_point"
      }
    }
  }
}

{
"acknowledged" : true,
"shards_acknowledged" : true,
"index" : "stats"
}

Then we have the index:
stats with 0 documents

Then we load data and the index generated now are:
statsxxx-20190830 with 1 document
statsyyy-20190830 with 1 document
stats with 0 documents

In the index pattern we create a pattern named "stats*" and once created the mapping shows a conflict:

location conflict

location.lat number
location.lon numberk

Is it possible to create a default mapping for all indexes?

We have version 7.2.

Thank you

stephenb · August 30, 2019, 1:46pm

Yes use an index template see here it is exactly for this case.

It will be applied to all new indices that fit the pattern apologies I should have showed you this at first I just wasn't sure what you were trying to accomplish.

it would look something like this and every new index created with the pattern will apply this mapping

PUT _template/stats
{
  "index_patterns": ["stats*"],
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "properties": {
      "location": {
        "type": "geo_point"
      }
    }
  }
}

jjfbcn · September 2, 2019, 12:08pm

Thank you Stephen, the template did the trick, everything is working fine.

Now, I think I'm going to open a new thread, I'd like that each point drawed in the map show more information when you get the mouse over a single point, right now is displayed the latitude and longitude information, but I'd like to show other fields with more valuable information, I'm not sure if this is possible...

Regards,

ashok9177 · September 17, 2019, 12:29pm

Hi @stephenb I have the same issue and I tried to put index template but getting below error

PUT _template/file
{
"index_patterns": ["file*"],
"settings": {
"number_of_shards": 1
},
"mappings": {
"properties": {
"location": {
"type": "geo_point"
}
}
}
}

{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "Root mapping definition has unsupported parameters: [location : {type=geo_point}]"
}
],
"type": "mapper_parsing_exception",
"reason": "Failed to parse mapping [properties]: Root mapping definition has unsupported parameters: [location : {type=geo_point}]",
"caused_by": {
"type": "mapper_parsing_exception",
"reason": "Root mapping definition has unsupported parameters: [location : {type=geo_point}]"
}
},
"status": 400
}

stephenb · September 17, 2019, 1:41pm

@ashok9177 What version of Elasticsearch are you using?

ashok9177 · September 17, 2019, 1:51pm

6.5.1

ashok9177 · September 17, 2019, 2:00pm

this template is worked for me https://www.elastic.co/guide/en/elasticsearch/reference/6.5/indices-templates.html

but I am still looking conflict error

stephenb · September 17, 2019, 2:09pm

That probably means there is more than 1 type of data for that field name for the indexes that the pattern applies too, meaning most likelly the pattern points to 1 or indexes that has the right data type geo_point and 1 or more that points to an index (probably before you create the mapping) that has the wrong data type.

ashok9177 · September 18, 2019, 5:43am

I have deleted all existing indices, getting below error

[2019-09-18T05:40:54,063][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"file-2019.09.18", :_type=>"doc", :_routing=>nil}, #LogStash::Event:0x2203d9e7], :response=>{"index"=>{"_index"=>"file-2019.09.18", "_type"=>"doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [file-2019.09.18] as the final mapping would have more than 1 type: [_doc, doc]"}}}}

stephenb · September 18, 2019, 5:59am

For 6.5.1
Probably should be "_type"=>"_doc"

ashok9177 · September 18, 2019, 6:01am

below is my logstash configuration

filter {
if "iddiapi" in [fields][component] {
  grok {

  match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} (\[%{WORD:loglevel}\]) %{DATA} - SegmentName : %{WORD:segment} Longitude : %{BACULA_DEVICE:longitude} Latitude : %{BACULA_DEVICE:latitude}" }

  }



mutate {
    add_field => [ "[geolocation][lat]", "%{latitude}" ]
    add_field => [ "[geolocation][lon]", "%{longitude}" ]
}

 mutate {
            convert => {
          "[geolocation][lat]" => "float"
          "[geolocation][lon]" => "float"
}
}


}


output {
  elasticsearch {
    hosts => ["localhost"]
    user => 'elastic'
    password => 'password'
    manage_template => false
    index => "file-%{+YYYY.MM.dd}"
}

My sample log message

2019-09-18 12:57:16.398 [INFO] from application in pool-1-thread-21 - SegmentName : IDVerified Longitude : 17.467793 Latitude : 78.388074 for transactionId : 56ou9P3JQ-gDo6

is it correct tor not? can you please suggest me any changes ?

system · October 16, 2019, 6:04am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.