Logstash cannot index to Elasticsearch

[haseebsabir]

I am very new to ELK. I am getting error message that logstash Could not index event to Elasticsearch.

[2018-04-18T16:30:06,128][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2018.04.18", :_type=>"doc", :_routing=>nil}, #LogStash::Event:0x19f861b6], :response=>{"index"=>{"_index"=>"logstash-2018.04.18", "_type"=>"doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"Failed to parse mapping [default]: [include_in_all] is not allowed for indices created on or after version 6.0.0 as [_all] is deprecated. As a replacement, you can use an [copy_to] on mapping fields to create your own catch all field.", "caused_by"=>{"type"=>"mapper_parsing_exception", "reason"=>"[include_in_all] is not allowed for indices created on or after version 6.0.0 as [_all] is deprecated. As a replacement, you can use an [copy_to] on mapping fields to create your own catch all field."}}}}}

Can someone help to resolve this issue.

[JKhondhu (Jymit Singh Khondhu)]

Hi,
The resolution begins first by understanding the reason for the WARN logging.
In particular "reason"=>"[include_in_all] is not allowed for indices created on or after version 6.0.0 as [_all] is deprecated. As a replacement, you can use an [copy_to] on mapping fields to create your own catch all field."}}}}}

You may need to review your mapping with this in mind: https://www.elastic.co/guide/en/elasticsearch/reference/current/copy-to.html

[haseebsabir]

I am new to logstash. Can you guide me how to review mapping.

Thanks in advance.

[Jenni]

I guess you are using a wrong template. Did you specify a template one in your pipeline configuration in the logstash ElasticSearch output?
If not, you could execute GET /_cat/templates in the console in Kibana to list all templates in ES and GET /_template/<template name> to have a look at one.
If you did not create any index template in ElasticSearch, it's probably using a default logstash one and you can look for it in your logstash installation directory at /vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch. (I don't have any experience with those)

[JKhondhu (Jymit Singh Khondhu)]

@haseebsabir
What version of logstash are you using?

In Elasticsearch 6.0 the _all field is disabled/deprecated as outlined in the docs about breaking mapping changes: https://www.elastic.co/guide/en/elasticsearch/reference/6.0/breaking_60_mappings_changes.html
It seems you are using include_in_all in your mapping template, which is causing this.

@Jenni => https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/master/lib/logstash/outputs/elasticsearch/elasticsearch-template-es6x.json

[haseebsabir]

I am using logstash 6.2.3 but when i ran Get /_template/logstash it shows "version": 50001.

"logstash": {
"order": 0,
"version": 50001,
"index_patterns": [
"logstash-*"
],
"settings": {
"index": {
"refresh_interval": "5s"
}
},
"mappings": {
"default": {
"_all": {
"enabled": true,
"norms": false
},

I had logstash 5 previously installed. I removed it using "rpm -e logstash" command. It seems it is trying to use data created by previous version.

This is config of output section currently running on logstash 6.2.3.

output {
elasticsearch {
hosts => "localhost:9200"
index => "logstash-%{+YYYY.MM.dd}"
}

How i can remove config of older version.

[haseebsabir]

issue resolved.

First list all templates as

GET /_cat/templates

then delete all templates

delete /_template/logstash

Restart ELK services