Elasticseach 2 geoip problem

Elastic Stack Elasticsearch
1

Hi,
I just upgraded a 1.7.1 instance of elasticsearch+logstash, and after upgrading and bringing up the services I'm getting this output from the elasticsearch log:
2015-10-30 18:34:14,041][DEBUG][action.admin.indices.create] [Charcoal] [logstash-2015.10.31] failed to create
MapperParsingException[mapping [default]]; nested: MapperParsingException[Mapping definition for [geoip] has unsupported parameters: [path : full]];
at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$2.execute(MetaDataCreateIndexService.java:359)
at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:388)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:225)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:188)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Is this a known issue for 2.0 ?
Is there any workaround for it ?

Thanks
Francisco

1 Like
2

It's probably this https://www.elastic.co/guide/en/elasticsearch/reference/current/_mapping_changes.html#migration-meta-fields

3

Hi!
I have the same issue; it's really frustrating!
Since my migration to the 2.0.0 my ELK stack isn't writing any indexes anymore.
On the one hand there's not really some kind of migration guide and on the other hand it seems that my ELK stack is falling apart.

4

This is what I did to fix it on the logstash box[es]:

  1. In this file: ./vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.1.2-java/lib/logstash/outputs/elasticsearch/elasticsearch-template.json
    remove this line:
    "path": "full",
    to leave it like:
    "properties" : {
    "@version": { "type": "string", "index": "not_analyzed" },
    "geoip" : {
    "type" : "object",
    "dynamic": true,
    "properties" : {
    "location" : { "type" : "geo_point" }
    }
    }

2.Added this to the ES output
output {
elasticsearch {
...
template_overwrite => true
...
}
}

Hope this helps anyone having this issue also.

Francisco

3 Likes
5

You'd be better off copying the template somewhere else and then using that, you run the risk over having that one overwritten during upgrades.

6

Thank you very much for the soluton. It really helped.

Just to understand...
Could you explain the "2" point?
What "template_overwrite => true" do ?

In my case (Ubuntu 14.04, ES installed three days ago from ES ubuntu repo) the elasticsearch-template.json has been already corrected. But without "template_overwrite => true" it still didn't work.

7

This is what I found in the Logstash docs:

https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-template_overwrite

My interpretation of it, is that with this setting = true, it makes the change done in step 1 work.

1 Like
8

Hey @flopez, @paulz, @mkh1973,

FIrst, I want to thank you all for raising this! You caught something the logstash testing framework didn't catch regarding upgrades. This is an artifact of this issue, which was changed in 2.0. The logstash folks are going to update the upgrade documentation to reflect this manual step that may need to be done and look into updates to the testing framework to cover things like this in the future. And credit to @flopez for pointing everybody in the correct direction.

Thanks again for helping!

1 Like
9

Thanks again!

Users may have custom template changes, so by default a Logstash upgrade will leave the template as is. Even if you don’t have a custom template, Logstash will not overwrite an existing template by default. We removed GeoIP mapping issue in 1.5, but folks who have migrated from 1.4 and before would still have this issue and advised to overwrite their templates. Please be sure to save your existing template if you have custom changes.

Docs updated here: https://www.elastic.co/guide/en/logstash/current/_upgrading_logstash_and_elasticsearch_to_2_0.html

2 Likes
10

Thanks ..... template_overwrite => "true" fixed my problem. After a 2.1 upgrade I was getting no indices. Adding that flag to my elasticsearch {template_overwrite => "true"} logstash->output settings fixed my problem.

11

Yep，I want to report a issue about packetbeat-1.2.3 and elasticsearch-2.3.3, there is a similar problem about [packetbeat-2016.06.23] failed to create.
But，in the packetbeat.yml ，I have done just what you have said .like this template_overwrite=”true"
Thanks .

12

Hi,

I am on Logstash 2.4 as indicated below,

# /opt/logstash/bin/logstash version
logstash 2.4.0

Also, when cross-checked the file, /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/elasticsearch-template.json, the geo-ip section is as below which is intact with no mention of "path":"full" as per the above discussions.

"properties" : {
"@timestamp": { "type": "date" },
"@version": { "type": "string", "index": "not_analyzed" },
"geoip" : {
"dynamic": true,
"properties" : {
"ip": { "type": "ip" },
"location" : { "type" : "geo_point" },
"latitude" : { "type" : "float" },
"longitude" : { "type" : "float" }
}
}
}

Also, the logstash output has the below config,

output {
elasticsearch {
hosts => localhost
index => "proxyfinal-%{+YYYY.MM.dd}"
template_overwrite => true
}
}

However, I still encounter the same issue of "mapper parsing exception" as others have experienced.

Logstash log as below,

"status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"Failed to parse mapping [default]: Mapping definition for [geoip] has unsupported parameters: [path : full]", "caused_by"=>{"type"=>"mapper_parsing_exception", "reason"=>"Mapping definition for [geoip] has unsupported parameters: [path : full]"}}}}, :level=>:warn}

Could you please help !!

13

Same problem here. How can I find the mapping which is used?

14

Got it! After listing all my templates(!) I found some that collided with the template I wanted.

curl -sv localhost:9200/_template | jq .

leads to a formatted (thanks to jq!) list of all templates known to ES.

Deleting all I don't need did it for me. There were several which used the same pattern for the index name but had the old mapping...