Bumping this up to the top for extra eyes. I believe I understand the root cause of the issue being the "string" type has been deprecated according to the Elastic blog post linked below. I used the mapping API to retrieve the mapping of our Logstash index using GET /logstash-2019.09/_mapping. The only two places where the "string" type is used are posted below.
Digging through the rest of the index mapping for the other values I'm getting mapper_parsing_exception errors on seems to show me that the values causing errors are mapped properly using the "keyword" type.
Is there any way to up the character limit on these posts? I'd like to post the full index mapping. Unfortunately though, the full map is over 7,000 lines long. Which I suspect is a problem in of itself.
Also for what it is worth I started our ES migration by uploading existing data from our on-prem deployment (5.3) to our new Amazon ES deployment (6.8) by using the Snapshot/Restore API. I am now trying to index live events using Logstash 6.8. Could this be part of the problem? Is it possible I need to re-index the existing data in some way?
GET /logstash-2019.09/_mapping
Mapping Snippet with "string"
"string_fields" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : {
"fielddata" : {
"format" : "disabled"
},
"fields" : {
"raw" : {
"ignore_above" : 256,
"index" : "not_analyzed",
"type" : "string"
Sample Mapping Snippets:
There are 19 occurrences of this same mapping for "syslog_text" all of them are defined the same.
[2019-09-23T19:53:02,394][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2019.09", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x3fa39351>], :response=>{"index"=>{"_index"=>"logstash-2019.09", "_type"=>"doc", "_id"=>"AW1gweMirQfrb52qfo0M", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to find type parsed [string] for [syslog_text]"}}}}
},
"syslog_text" : {
"type" : "text",
"norms" : false,
"fields" : {
"raw" : {
"type" : "keyword",
"ignore_above" : 256
}
There are 18 occurrences of this same mapping for "path" all of them are defined the same.
[2019-09-23T19:53:02,395][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2019.09", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x701eb47d>], :response=>{"index"=>{"_index"=>"logstash-2019.09", "_type"=>"doc", "_id"=>"AW1gweMirQfrb52qfo0O", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to find type parsed [string] for [path]"}}}}
"path" : {
"type" : "text",
"norms" : false,
"fields" : {
"raw" : {
"type" : "keyword",
"ignore_above" : 256