Cannot push to elasticsearch since migration of ES from 5.6 to 6.1

[jdefaver (Jerome De Favereau)]

Hi,

I have done a rolling migration from ES 5.6 to 6.1. I realized logstash could not push my events to ES anymore. I first updated logstash, then I realized I had the

"[include_in_all] is not allowed for indices created on or after version 6.0.0 as [_all] is deprecated. As a replacement, you can use an [copy_to] on mapping fields to create your own catch all field."

error and updated my template to

{ "logstash" : { "order" : 0, "version" : 60001, "index_patterns" : [ "logstash-*" ], "settings" : { "index" : { "refresh_interval" : "5s" } }, "mappings" : { "_default_" : { "dynamic_templates" : [ { "message_field" : { "path_match" : "message", "match_mapping_type" : "string", "mapping" : { "type" : "text", "norms" : false } } }, { "string_fields" : { "match" : "*", "match_mapping_type" : "string", "mapping" : { "type" : "text", "norms" : false, "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } } ], "properties" : { "@timestamp" : { "type" : "date" }, "@version" : { "type" : "keyword" }, "geoip" : { "dynamic" : true, "properties" : { "ip" : { "type" : "ip" }, "location" : { "type" : "geo_point" }, "latitude" : { "type" : "half_float" }, "longitude" : { "type" : "half_float" } } } } } }, "aliases" : { } } }

Which fixed it. I then deleted the latest index (logstash-2017.12.19) and ran logstash again but now I get this error:

[logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2017.12.19", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x49a57436>], :response=>{"index"=>{"_index"=>"logstash-2017.12.19", "_type"=>"doc", "_id"=>"qtMSb2AB7h3H07pPaT8i", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Could not convert [fqans.index] to boolean", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Failed to parse value [analyzed] as only [true] or [false] are allowed."}}}}}}

I think I understood that the type of .index was changed, but I don't know how to fix this. Could someone help me ?

Thanks !

Jerome

[sdinh]

I'm running into the same problem. Just recently upgraded ES to 6.1.0-1 and Logstash is still on 5.6.5-1.

One strange thing is that Logstash was pushing data to ES for like 30 minutes post-upgrade and then it just stopped outputting data to ES all of a sudden without any errors.

[****@ops-logstash1 logstash]# systemctl status logstash.service ● logstash.service - logstash Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2017-12-19 20:51:17 PST; 49min ago Main PID: 6925 (java) CGroup: /system.slice/logstash.service └─6925 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplic...

[@ops-logstash1 logstash]# netstat -antp | grep LISTEN | grep java
tcp6 0 0 :::5044 ::: LISTEN 6925/java
tcp6 0 0 127.0.0.1:9600 ::: LISTEN 6925/java
tcp6 0 0 :::5000 ::: LISTEN 6925/java
tcp6 0 0 :::5001 ::: LISTEN 6925/java

I've been running iftop to monitor the network traffic and there is zero traffic to the ES cluster even though Logstash is running fine.

Any help here would be greatly appreciated as this is impacting my production environment!

Thanks!
-Steven

[jdefaver (Jerome De Favereau)]

Hi,

I finally found the issue. I had failed to delete the previous template, and it was used instead of the new one. Once I removed the old one, things worked again.

I don't think my problem is linked to @sdinh : you should probably open an independent issue.