Remove `_all` mapping from older indexes

[willdennis]

Upgraded to v6.8 recently; trying to ensure I can now upgrade to v7.x Having an issue when trying to reindex my old logstash indexes created on v5.6.x - getting this message when the system goes to create the new index (using the "7.0 Upgrade Assistant" thru Kibana):

[mapper_parsing_exception] [include_in_all] is not allowed for indices created on or after version 6.0.0 as [_all] is deprecated. As a replacement, you can use an [copy_to] on mapping fields to create your own catch all field.

Is there a way to get rid of the _all mapping on the current indexes? (Sorry, have been running ELK stack for a while, but never dug deep into Elasticsearch... getting my education now!) Or, how else could I handle this error?

[willdennis]

Looks like I also have a similar issue with Logstash writing to ES as well --

[2022-01-24T13:35:37,059][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2022.01.24", :_type=>"doc", :routing=>nil}, #<LogStash::Event:0x5d051aa4>], :response=>{"index"=>{"_index"=>"logstash-2022.01.24", "_type"=>"doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"Failed to parse mapping [_default_]: [include_in_all] is not allowed for indices created on or after version 6.0.0 as [_all] is deprecated. As a replacement, you can use an [copy_to] on mapping fields to create your own catch all field.", "caused_by"=>{"type"=>"mapper_parsing_exception", "reason"=>"[include_in_all] is not allowed for indices created on or after version 6.0.0 as [_all] is deprecated. As a replacement, you can use an [copy_to] on mapping fields to create your own catch all field."}}}}}

Is this some kind of template issue?

[warkolm (Mark Walkom)]

Welcome to our community!

What's your index template look like for the Logstash indices?

[willdennis]

I listed the templates I have on my system, and I see this:

root@logstash01:~# curl -XGET http://localhost:9200/_cat/templates?v
name                          index_patterns             order      version
.monitoring-alerts            [.monitoring-alerts-6]     0          6070299
.ml-notifications             [.ml-notifications]        0          6082299
.ml-anomalies-                [.ml-anomalies-*]          0          6082299
security_audit_log            [.security_audit_log*]     1000
.watch-history-9              [.watcher-history-9*]      2147483647
kibana_index_template:.kibana [.kibana]                  0
filebeat                      [filebeat-*]               0
.watches                      [.watches*]                2147483647
.triggered_watches            [.triggered_watches*]      2147483647
.ml-config                    [.ml-config]               0          6082299
.monitoring-beats             [.monitoring-beats-6-*]    0          6070299
.monitoring-kibana            [.monitoring-kibana-6-*]   0          6070299
logstash                      [logstash-*]               0          50001
.monitoring-es                [.monitoring-es-6-*]       0          6070299
logstash-index-template       [.logstash]                0
.ml-state                     [.ml-state*]               0          6082299
security-index-template       [.security-*]              1000
.management-beats             [.management-beats]        0          67000
.monitoring-logstash          [.monitoring-logstash-6-*] 0          6070299
.kibana_task_manager          [.kibana_task_manager]     0          6082399
.ml-meta                      [.ml-meta]                 0          6082299

So I see the "logstash" one has a 'version' from v5.0; If I list this template, I see it does indeed have the _all field, included from the (now-deprecated) _default_ mapping

root@logstash01:~# curl -X GET -u undefined:$ESPASS "localhost:9200/_template/logstash?pretty"
{
  "logstash" : {
    "order" : 0,
    "version" : 50001,
    "index_patterns" : [
      "logstash-*"
    ],
    "settings" : {
      "index" : {
        "refresh_interval" : "5s"
      }
    },
    "mappings" : {
      "_default_" : {
        "_all" : {
          "enabled" : true,
          "norms" : false
        },
        "dynamic_templates" : [
          {
            "message_field" : {
              "path_match" : "message",
              "match_mapping_type" : "string",
              "mapping" : {
                "type" : "text",
                "norms" : false
              }
            }
          },
          {
            "string_fields" : {
              "match" : "*",
              "match_mapping_type" : "string",
              "mapping" : {
                "type" : "text",
                "norms" : false,
                "fields" : {
                  "keyword" : {
                    "type" : "keyword"
                  }
                }
              }
            }
          }
        ],
        "properties" : {
          "@timestamp" : {
            "type" : "date",
            "include_in_all" : false
          },
          "@version" : {
            "type" : "keyword",
            "include_in_all" : false
          },
          "geoip" : {
            "dynamic" : true,
            "properties" : {
              "ip" : {
                "type" : "ip"
              },
              "location" : {
                "type" : "geo_point"
              },
              "latitude" : {
                "type" : "half_float"
              },
              "longitude" : {
                "type" : "half_float"
              }
            }
          }
        }
      }
    },
    "aliases" : { }
  }
}

[warkolm (Mark Walkom)]

Ok, try stopping Logstash, deleting that template and then restarting Logstash.

[willdennis]

This seems to have done the trick; saw this in Logstash startup messages:

[2022-01-24T20:01:50,431][INFO ][logstash.outputs.elasticsearch] Using default mapping template
[2022-01-24T20:01:50,487][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2022-01-24T20:01:50,601][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/logstash

Now when I list the templates, the logstash one looks better:

root@logstash01:~# curl -XGET http://localhost:9200/_cat/templates?v
name                          index_patterns             order      version
--{snip}--
logstash                      [logstash-*]               0          60001

Thanks!