Override _default_ mapping properties

Elastic Stack Elasticsearch
1

Hi,

I started a while ago using the logstash default elasticsearch template "https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/master/lib/logstash/outputs/elasticsearch/elasticsearch-template.json" on an existing cluster (1.7.3), so that I could use doc_value on every field.

Thing is, some "already existing fields" haven't changed. They are still using fielddata and putting some memory pressure on my cluster.

I use the logstash (1.5.2) elasticsearch output plugin with 'template_overwrite => true' , but still, the default mapping properties are not updated, the dynamic template part is OK.

Can I override this default mapping ? Or maybe remove the properties which are not using doc_value ?

2

What's your config look like, have you set template_overwrite?

3

Hi,

I'm using the elasticsearch output plugin

 elasticsearch {
                host => [<%= @elasticsearch_datanodes %>]
                cluster => "<%= @elasticsearchcluster %>"
                protocol => "http"
                port => 9200
                workers => 8
                flush_size => 500
                **template => "/opt/application/logstash/template/elasticsearch_mapping.json"**
**                template_overwrite => true**
            }

What I do not understand, is that when a new index gets created @ midnight, its mappings contains a default configuration for some fields, that I'd like to be different.

For instance those 2 fields are using fielddata (extract from the default part of today's mapping) :

    "level": {
                            "type": "string",
                            "fields": {
                                "raw": {
                                    "type": "string",
                                    "index": "not_analyzed",
                                    "ignore_above": 256
                                }
                            }
                        },
                        "message": {
                            "type": "string",
                            "fields": {
                                "raw": {
                                    "type": "string",
                                    "index": "not_analyzed",
                                    "ignore_above": 256
                                }
                            }
                        }

Meanwhile, some other fields are Ok :

"operation": {
                        "type": "string",
                        "norms": {
                            "enabled": false
                        },
                        **"fielddata": {**
**                            "format": "disabled"**
**                        },**
                        "fields": {
                            "raw": {
                                "type": "string",
                                "index": "not_analyzed",
                                "doc_values": true,
                                "ignore_above": 256
                            }
                        }
                    },

Why do some field gets a "default" configuration?. I do not understand where those properties come from. It is like I have, somewhere in elasticsearch, a default which is applied to every newly created index.

I tried to remove the default from every mapping yesterday, (curl -XDELETE elastic:9200/_template/default) and I was not seeing this default in my old index mapping. But this morning logstash-2016.06.03 got this default with some field missing the fielddata part

My biggest problem is that those concerned fields are the most requested from Kibana (message, level, host..), resulting in some memory pressure on my cluster.

4

Hi,

Anyone got an idea on this ?

I will try to hard code fielddata missing properties to disable it on fields that are getting an incorrect mapping, but I do not feel this is the right way to deal with my problem