Hi,
I have updated my ElasticSearch cluster from 6.6 to 7.6.2, as part of this upgrade the _all field and the _default_ fields were removed.
I updated the cluster, checked the logs, and sure enough, one of my templates was complaining those fields existed. All my other templates were still correctly collecting and pushing data to ElasticSearch/Kibana.
So, using the API console, I ran a PUT to /_template/logstash-qa01-stats to update the template from:
{
"template" : "logstash-prod-stats-*",
"settings" : {
"index.refresh_interval" : "10s"
},
"mappings" : {
"_default_" : {
"_all" : { "enabled" : false, "norms" : false },
"dynamic_templates" : [ {
"string_fields": {
"match": "*",
"match_mapping_type": "string",
"mapping": { "type": "text", "index": true }
}
}, {
"byte_fields" : {
"match" : "*",
"match_mapping_type" : "long",
"mapping" : { "type" : "double", "doc_values": true }
}
}, {
"short_fields" : {
"match" : "*",
"match_mapping_type" : "long",
"mapping" : { "type" : "double", "doc_values": true }
}
}, {
"integer_fields" : {
"match" : "*",
"match_mapping_type" : "long",
"mapping" : { "type" : "double", "doc_values": true }
}
}, {
"long_fields" : {
"match" : "*",
"match_mapping_type" : "long",
"mapping" : { "type" : "double", "doc_values": true }
}
}, {
"float_fields": {
"match": "*",
"match_mapping_type": "double",
"mapping": { "type": "double", "doc_values": true }
}
}],
"properties" : {
"@timestamp": { "type" : "date", "format": "date_optional_time" },
"@version" : { "type" : "keyword", "index": true },
"type_instance" : { "type" : "text", "index" : true, "norms" : false,
"fields" : {
"raw" : { "type": "text", "index" : true }
}
}
}
}
}
}
To:
{
"index_pattern": "logstash-qa01-stats-*",
"settings": {
"index.refresh_interval": "10s"
},
"mappings": {
"dynamic_templates": [{
"string_fields": {
"match": "*",
"match_mapping_type": "string",
"mapping": {
"type": "string",
"index": true,
"doc_values": true
}
}
}, {
"byte_fields": {
"match": "*",
"match_mapping_type": "long",
"mapping": {
"type": "double",
"doc_values": true
}
}
}, {
"short_fields": {
"match": "*",
"match_mapping_type": "long",
"mapping": {
"type": "double",
"doc_values": true
}
}
}, {
"integer_fields": {
"match": "*",
"match_mapping_type": "long",
"mapping": {
"type": "double",
"doc_values": true
}
}
}, {
"long_fields": {
"match": "*",
"match_mapping_type": "long",
"mapping": {
"type": "double",
"doc_values": true
}
}
}, {
"float_fields": {
"match": "*",
"match_mapping_type": "double",
"mapping": {
"type": "double",
"doc_values": true
}
}
}],
"properties": {
"@timestamp": {
"type": "date",
"format": "date_optional_time"
},
"@version": {
"type": "keyword",
"index": true
},
"type_instance": {
"type": "text",
"index": true,
"norms": false,
"fields": {
"raw": {
"type": "string",
"index": true,
"doc_values": true,
"ignore_above": 256
}
}
}
}
}
}
The console accepted this and replied with 200 OK.
I went back to my logs to make sure the data was being processed and now all I get is:
[2020-04-15T20:28:52,111][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [https://logstash-qa01:xxxxxx@ea33a63cd7ed235230b7aedcd62e4c37.eu-west-1.aws.found.io:9243/][Manticore::SocketTimeout] Read timed out", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>64}
[2020-04-15T20:28:52,663][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://logstash-qa01:xxxxxx@ea33a63cd7ed235230b7aedcd62e4c37.eu-west-1.aws.found.io:9243/][Manticore::SocketTimeout] Read timed out {:url=>https://logstash-qa01:xxxxxx@ea33a63cd7ed235230b7aedcd62e4c37.eu-west-1.aws.found.io:9243/, :error_message=>"Elasticsearch Unreachable: [https://logstash-qa01:xxxxxx@ea33a63cd7ed235230b7aedcd62e4c37.eu-west-1.aws.found.io:9243/][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2020-04-15T20:28:52,664][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [https://logstash-qa01:xxxxxx@ea33a63cd7ed235230b7aedcd62e4c37.eu-west-1.aws.found.io:9243/][Manticore::SocketTimeout] Read timed out", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>64}
[2020-04-15T20:28:56,245][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://logstash-qa01:xxxxxx@ea33a63cd7ed235230b7aedcd62e4c37.eu-west-1.aws.found.io:9243/, :path=>"/"}
[2020-04-15T20:28:56,346][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"https://logstash-qa01:xxxxxx@ea33a63cd7ed235230b7aedcd62e4c37.eu-west-1.aws.found.io:9243/"}
Over and over again. No data is being logged to ElasticSearch/Kibana unless I restart logstash, then the missing data is pushed but then it stops sending new data.
This cluster has been fine for years, and I don't understand how a template change could have broken it this badly.
I am able to access my Elastic domain and get the following:
{
"name" : "instance-0000000105",
"cluster_name" : "ea33a63cd7ed235230b7aedcd62e4c37",
"cluster_uuid" : "Tl84gLdZRI26oPe9FRWVuw",
"version" : {
"number" : "7.6.2",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "ef48eb35cf30adf4db14086e8aabd07ef6fb113f",
"build_date" : "2020-03-26T06:34:37.794943Z",
"build_snapshot" : false,
"lucene_version" : "8.4.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
My other environments, using the same ElasticSearch cluster are only impacted when I update the same template that is reporting issues in their environments.
Does anyone have any suggestions as to what might have happened here?