# Logstash output custom template

**URL:** <https://discuss.elastic.co/t/logstash-output-custom-template/251664>\
**Category:** Logstash\
**Created:** [October 11, 2020, 4:15am UTC](https://discuss.elastic.co/t/logstash-output-custom-template/251664 "2020-10-11T04:15:18Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![venku](https://avatars.discourse-cdn.com/v4/letter/v/a698b9/32.png) [@venku](https://discuss.elastic.co/u/venku)\
**Post date:** [October 11, 2020, 4:15am UTC](https://discuss.elastic.co/t/logstash-output-custom-template/251664/1 "2020-10-11T04:15:19Z")

</div>

Hi, I am new to ELK stack. This is my set so far for sending application logs : filebeat -\> redis(dockers) -\> logstash(dockers) -\> ES.  
_My Question is_ what is the the correct way to setup custom index in logstash output?  
According to my understanding I need to create a index template directly in ES and use the same name in the output of the logstash. Is this not the right approach? Because I don't see the fields are index in the way I defined in the template. Below are steps I followed  
**Step:**

1. Create a index template in ES using this - \_index\_template/index2\_template and passed JSON with fields types.
2. Used the same name in the output of the logstash.

**JSON for creating index template:**

```auto
        {
    "index_patterns": ["index2-*"],
      "template": {
      "settings": {
          "number_of_shards": 1
        },
        "mappings": {
          "properties": {
            "timestamp": {
              "type": "date"
            },
    		"level": {
              "type": "keyword"
            },
            "mdc.audit_id": {
              "type": "text",
              "index": false
            },
            "mdc.status": {
              "type": "keyword"
            },
            "mdc.instance_id": {
              "type": "text",
              "index": false
            },
            "mdc.url": {
              "type": "text"
            },
            "mdc.executionTime": {
              "type": "integer"
            },
    		"thread": {
              "type": "keyword",
              "index": false
            },
    		"logger": {
              "type": "text",
              "index": false
            },
    		"message": {
              "type": "keyword"
            },
    		"exception": {
              "type": "text"
            }
          }
        }
      },
      "priority": 51,
      "version": 1,
      "_meta": {
        "description": "my custom"
      }
    }

```

**In the logstash I have the following:**

```auto
    input {
      redis {
        host => "XXX.1X.X.X"
        key => "app_logs"
        data_type => "list"
      }
    }

    output {
      elasticsearch {
        hosts => ["eslocalhost:9200"]
        index => "index2_template"
        #manage_template => true
        template_name => "index2_template"
    	template => "index2_template"
      }
    }

```

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [October 11, 2020, 5:44pm UTC](https://discuss.elastic.co/t/logstash-output-custom-template/251664/2 "2020-10-11T17:44:53Z")

</div>

Your index pattern in your template does not match your index name.

In your template you have:

```auto
"index_patterns": ["index2-*"]

```

But in your logstash output to elasticsearch you have:

```auto
index => "index2_template"

```

The index `index2_template` does not match the pattern `index2-*`, your index name should be something like `index2-template` to match your template. Or you need to change the patter in your template, it does matter what you will change, but the index name needs to match the index pattern.

---

<div class="post-metadata">

**Author:** ![venku](https://avatars.discourse-cdn.com/v4/letter/v/a698b9/32.png) [@venku](https://discuss.elastic.co/u/venku)\
**Post date:** [October 12, 2020, 5:11pm UTC](https://discuss.elastic.co/t/logstash-output-custom-template/251664/3 "2020-10-12T17:11:58Z")

</div>

Sorry for the delayed reply. I applied the changes you suggested but I am getting an error in logstash. Can you please suggest what else is going wrong here?

Below is the error I see in logstash logs:  
`[2020-10-12T17:07:45,964][ERROR][logstash.outputs.elasticsearch][main][fc94776fa3282fe5e77f25f9ff0aa9c3626017d4d0bd9ab347a9e4e7847d1e71] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"index3-*", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x5c8380c>], :response=>{"index"=>{"_index"=>"index3-*", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"invalid_index_name_exception", "reason"=>"Invalid index name [index3-*], must not contain the following characters [, \", *, \\, <, |, ,, >, /, ?]", "index_uuid"=>"_na_", "index"=>"index3-*"}}}}`

my template:

```
{
"index_patterns": ["index3-*"],
  "template": {
  "settings": {
      "number_of_shards": 1,
      "index.lifecycle.name": "ecom_ilm_policy",
      "index.lifecycle.rollover_alias":"ecom"
    },
    "mappings": {
      "properties": {
        "timestamp": {
          "type": "date"
        },
		"level": {
          "type": "keyword"
        },
        "mdc.audit_id": {
          "type": "text",
          "index": false
        },
        "mdc.status": {
          "type": "keyword"
        },
        "mdc.instance_id": {
          "type": "text",
          "index": false
        },
        "mdc.url": {
          "type": "text"
        },
        "mdc.executionTime": {
          "type": "integer"
        },
		"thread": {
          "type": "keyword",
          "index": false
        },
		"logger": {
          "type": "text",
          "index": false
        },
		"message": {
          "type": "keyword"
        },
		"exception": {
          "type": "text"
        }
      }
    }
  },
  "priority": 51,
  "version": 1,
  "_meta": {
    "description": "my custom"
  }
}

```

Logstash Config

```
input {
  redis {
    host => "172.17.0.2"
    key => "ecom"
    data_type => "list"
  }
}

output {
  elasticsearch {
    hosts => ["eslocalhost:9200"]
    index => "index3-*"
    # manage_template => true
    template_name => "index3_template"
    ilm_enabled => true

  }
}
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [October 12, 2020, 5:20pm UTC](https://discuss.elastic.co/t/logstash-output-custom-template/251664/4 "2020-10-12T17:20:02Z")

</div>

> [@](#):
>
> "Invalid index name [index3-\*], must not contain the following characters [, ", \*, \, \<, |, ,, \>, /, ?]"

You cannot call an index "index3-\*". You can use that in the index\_patterns field of the template, but you cannot use it in the index option of an elasticsearch output. What do you want the index to be called?

---

<div class="post-metadata">

**Author:** ![venku](https://avatars.discourse-cdn.com/v4/letter/v/a698b9/32.png) [@venku](https://discuss.elastic.co/u/venku)\
**Post date:** [October 12, 2020, 5:24pm UTC](https://discuss.elastic.co/t/logstash-output-custom-template/251664/5 "2020-10-12T17:24:30Z")

</div>

I want my index be called as "ecom-app-logs". And will use ecom\* to filter all indices in Kibana is the plan.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [October 12, 2020, 5:35pm UTC](https://discuss.elastic.co/t/logstash-output-custom-template/251664/6 "2020-10-12T17:35:07Z")

</div>

OK, so in logstash use

```
output {
    elasticsearch {
        hosts => ["eslocalhost:9200"]
        index => "ecom-app-logs"

```

and in your template use

```
"index_patterns": ["ecom-*"],
```

---

<div class="post-metadata">

**Author:** ![venku](https://avatars.discourse-cdn.com/v4/letter/v/a698b9/32.png) [@venku](https://discuss.elastic.co/u/venku)\
**Post date:** [October 12, 2020, 5:40pm UTC](https://discuss.elastic.co/t/logstash-output-custom-template/251664/7 "2020-10-12T17:40:47Z")

</div>

Thank you. I think it is making sense to me now.  
Quick question before I make the change, do I have to change anything for`template_name`. What is this used for please?

Thanks!

---

<div class="post-metadata">

**Author:** ![venku](https://avatars.discourse-cdn.com/v4/letter/v/a698b9/32.png) [@venku](https://discuss.elastic.co/u/venku)\
**Post date:** [October 12, 2020, 5:59pm UTC](https://discuss.elastic.co/t/logstash-output-custom-template/251664/8 "2020-10-12T17:59:46Z")

</div>

I made the change and tried to create an index in Kibana, I am not sure if the template is picked up. Because, I marked the `level` as a `keyword` but I see it as `string` in Kibana

```
"level": {
          "type": "keyword"
        },

```

 ![kibana_level](https://us1.discourse-cdn.com/elastic/original/3X/1/9/19aa4ef0c039d9c3b9b487533beb7e66ddd5c98c.png)

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [October 12, 2020, 6:48pm UTC](https://discuss.elastic.co/t/logstash-output-custom-template/251664/9 "2020-10-12T18:48:27Z")

</div>

> [@venku](#):
>
> do I have to change anything for `template_name` . What is this used for please?

templates are stored in elasticsearch. They need a name so that you can refer to it if you want to overwrite it. You do not need to change the template name but some people prefer to have a template name that is similar to the index names that it applies to.

If you want to know what mapping (not template) was actually applied to an index then use the [mapping API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-mapping.html).

Prior to logstash 5.0 there was a string type in elasticsearch. In 5.0 strings were [replaced](https://www.elastic.co/blog/strings-are-dead-long-live-strings) with the two types text and keyword. However, I do not think Kibana was update to call them that.

I believe that your template was applied, because Kibana shows the field as aggregatable, which would not be true if it were text.

---

<div class="post-metadata">

**Author:** ![venku](https://avatars.discourse-cdn.com/v4/letter/v/a698b9/32.png) [@venku](https://discuss.elastic.co/u/venku)\
**Post date:** [October 12, 2020, 7:20pm UTC](https://discuss.elastic.co/t/logstash-output-custom-template/251664/10 "2020-10-12T19:20:10Z")

</div>

@Badger, You are correct. Looks like the template is applied. Thank you again for all the help and insights.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 9, 2020, 7:20pm UTC](https://discuss.elastic.co/t/logstash-output-custom-template/251664/11 "2020-11-09T19:20:16Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.