# Settings in Logstash output config file for "template =\>" not applied

**URL:** <https://discuss.elastic.co/t/settings-in-logstash-output-config-file-for-template-not-applied/89732>\
**Category:** Logstash\
**Created:** [June 16, 2017, 2:35pm UTC](https://discuss.elastic.co/t/settings-in-logstash-output-config-file-for-template-not-applied/89732 "2017-06-16T14:35:56Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\
**Post date:** [June 16, 2017, 2:35pm UTC](https://discuss.elastic.co/t/settings-in-logstash-output-config-file-for-template-not-applied/89732/1 "2017-06-16T14:35:56Z")

</div>

Hi,  
I have struggled with this now for a bit over a week and I just can't seem to figure it out.

I'm running 3 Elastic Stack setups for logging. In one setup I can't get "ignore\_malformed": "true" set for some reason. All three setups have been upgraded to 5.4.1 for Filebeat, Logstash, Elasticsearch and Kibana. All running on Debian Jessie (probably slightly different version). All three setups are deployed through Puppet using the same codebase.

Logstash output

> ```
> cat /etc/logstash/conf.d/output
> output {
> if [log_id] == "gcp" {
> elasticsearch {
> hosts => ["10.0.255.35:9202", "10.0.255.36:9202", "10.0.255.37:9202", "10.0.255.38:9202"]
> index => "logstash-gcp-%{+YYYY.MM.dd}"
> flush_size => 200
> template => "/etc/logstash/gcp_template_es_index.json"
> }
> }
> else {
> elasticsearch {
> hosts => ["10.0.255.35:9202", "10.0.255.36:9202", "10.0.255.37:9202", "10.0.255.38:9202"]
> index => "logstash-%{+YYYY.MM.dd}"
> flush_size => 200
> template => "/etc/logstash/default_template_es_index.json"
> }
> }
> }
> 
> ```

Template

> ```
> cat /etc/logstash/gcp_template_es_index.json
> {
> "template" : "logstash-gcp-*",
> "version" : 50001,
> "settings" : {
> "index.refresh_interval" : "5s",
> "index.mapping.ignore_malformed": true
> },
> "mappings" : {
> "_default_" : {
> "_all" : {"enabled" : true, "norms" : false},
> "dynamic_templates" : [ {
> "message_field" : {
> "path_match" : "message",
> "match_mapping_type" : "string",
> "mapping" : {
> "type" : "text",
> "norms" : false
> }
> }
> }, {
> "string_fields" : {
> "match" : "*",
> "match_mapping_type" : "string",
> "mapping" : {
> "type" : "text", "norms" : false,
> "fields" : {
> "keyword" : { "type": "keyword" }
> }
> }
> }
> } ],
> "properties" : {
> "@timestamp": { "type": "date", "include_in_all": false },
> "@version": { "type": "keyword", "include_in_all": false },
> "geoip" : {
> "dynamic": true,
> "properties" : {
> "ip": { "type": "ip" },
> "location" : { "type" : "geo_point" },
> "latitude" : { "type" : "half_float" },
> "longitude" : { "type" : "half_float" }
> }
> }
> }
> }
> }
> }
> 
> ```

With `GET logstash-2017.06.16/_settings` I see this on the working setup

> ```
> {
> "logstash-2017.06.16": {
> "settings": {
> "index": {
> "mapping": {
> "ignore_malformed": "true"
> },
> "refresh_interval": "5s",
> "number_of_shards": "5",
> "provided_name": "logstash-2017.06.16",
> "creation_date": "1497571200021",
> "number_of_replicas": "1",
> "uuid": "ahGtYCnuQEC_Zs77rrmuqQ",
> "version": {
> "created": "5040099"
> }
> }
> }
> }
> }
> 
> ```

And on the one that I'm struggling with I see this

> ```
> {
> "logstash-gcp-2017.06.16": {
> "settings": {
> "index": {
> "refresh_interval": "5s",
> "number_of_shards": "5",
> "provided_name": "logstash-gcp-2017.06.16",
> "creation_date": "1497571204225",
> "number_of_replicas": "1",
> "uuid": "6O9-PEmSS3adnf1pXnbsDQ",
> "version": {
> "created": "5040199"
> }
> }
> }
> }
> }
> 
> ```

So, the `mapping` part doesn't seem to get applied for some reason. Logstash has definitely been restarted since my last change to the files. Where should I start looking for the solution? Any help would be greatly appreciated.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 19, 2017, 9:06pm UTC](https://discuss.elastic.co/t/settings-in-logstash-output-config-file-for-template-not-applied/89732/2 "2017-06-19T21:06:45Z")

</div>

Was logstash-gcp-2017.06.16 created after the template was updated? What if you create a brand new index yourself, say logstash-gcp-2017.06.30, and inspects its mappings?

Wait. The /\_settings API retrieves the index settings. I don't think the index mappings are ever included in that response. Try the /\_mappings API instead.

---

<div class="post-metadata">

**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\
**Post date:** [June 20, 2017, 6:12am UTC](https://discuss.elastic.co/t/settings-in-logstash-output-config-file-for-template-not-applied/89732/3 "2017-06-20T06:12:44Z")

</div>

Hi Magnus,  
thanks for the reply, much appreciated 🙂

Yes, the indices I show the \_settings for both where created after the last change to the Logstash output config and the JSON template files.

I had the same idea of creating an index to see what happens so I did this

> ```
> PUT logstash-2017.06.20
> {
> "settings": {
> "index.mapping.ignore_malformed": true 
> }
> }
> 
> ```

which means there will be an index ready for todays logs with `"index.mapping.ignore_malformed": true`already set. Now I have to wait one more day to see if the next one will have this setting without intervention...

Because I'm setting `"index.mapping.ignore_malformed": true` for the whole index it shows up under \_settings and not \_mappings, as far as I can tell based on this [ignore\_malformed | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/ignore-malformed.html#ignore-malformed-setting)

Cheers,  
AB

---

<div class="post-metadata">

**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\
**Post date:** [June 21, 2017, 8:35am UTC](https://discuss.elastic.co/t/settings-in-logstash-output-config-file-for-template-not-applied/89732/4 "2017-06-21T08:35:56Z")

</div>

Quick update: The next daily index that Logstash created is again without `"index.mapping.ignore_malformed": true`

So, creating the index and applying the setting from Kibana works but using a JSON template in Logstash does not, in this one case. On two other setups that should be identical (as they use the same version of the whole stack and same puppet code base) this works...

Any suggestions on what I could try next are much appreciated...

---

<div class="post-metadata">

**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\
**Post date:** [July 5, 2017, 1:32pm UTC](https://discuss.elastic.co/t/settings-in-logstash-output-config-file-for-template-not-applied/89732/5 "2017-07-05T13:32:50Z")

</div>

I solved this by creating a cron job that creates the daily indices before logstash does it.

```
#!/bin/bash

DATE=$(date --date='1 day' +%Y.%m.%d)

curl -XPUT "http://logs.foo.bar:9200/logstash-$DATE" -d'
{
  "settings": {
    "index.mapping.ignore_malformed": true
  }
}'

```

In case that helps someone. Not the most elegant solution but looks like it's doing what I need...

AB

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 2, 2017, 1:45pm UTC](https://discuss.elastic.co/t/settings-in-logstash-output-config-file-for-template-not-applied/89732/6 "2017-08-02T13:45:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.