Where is the "doc" mapping defined?

[cawoodm (Marc)]

Since 6.0 we have been using the mapping/_type of doc in all our ES Indices. As we have now upgraded to 6.5 we want to migrate to the (almost official) _doc mapping type.

We tried this by creating a new index with the _doc mapping, reindexing (specifying doc -> _doc) and deleting the old index (alias etc.).

The problem is now that we are unable to index any documents. The error is:

Rejecting mapping update to [myindex-dev2] as the final mapping would have more than 1 type: [_doc, doc]

We have checked all our settings (filebeat, logstash and elasticsearch) and nowhere do we specify the _type.

If we add document_type => "_doc" to our elasticsearch output config in logstash it solves the problem.

Why does Logstash assume "doc" unless we specify "_doc"?? We are on ELK 6.5

PS: When we upgrade we always maintain our config/ and data/ folders, deleting the ones delivered with Elasticsearch, Logstash and FileBeat.

[cawoodm (Marc)]

I'm also getting a deprecation warning since I added document_type to the ES output.

You are using a deprecated config setting "document_type" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Document types are being deprecated in Elasticsearch 6.0, and removed entirely in 7.0. You should avoid this feature If you have any questions about this, please visit the logstash channel on freenode irc.

[jakelandis]

@cawoodm It's a bit of a web of versions and plugins...

It was not until 6.2 of Elasticsearch that "_doc" became a valid type name. To avoid the "Rejecting mapping update ..." error the default type was not changed to "_doc" for 6.x. Had it been changed, assuming daily indexes, would result in errors ("Rejecting mapping update ...") for the rest of that day after the upgrade. In addition it is often recommended (for a variety of reasons) to upgrade the Elasticsearch output plugin independent of the Logstash version and we wanted the plugin to be compatible with pre 6.2 versions of Elasticsearch.

As for the deprecation, that is because explicitly setting the type is something that is getting removed. https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html#_schedule_for_removal_of_mapping_types. It's less then ideal to have to set this to use _doc, but is currently the only way I am aware.

[cawoodm (Marc)]

OK, so I surmise that "doc" is indeed the default type and that it would have been better for me to stick to that than to switch to "_doc". For now, I'll live with "_doc" and the deprecation warning.