How to apply index templates by using config/templates dir

Elastic Stack Elasticsearch
1

Hi community!

I want to apply index templates in config/templates dir in es docker container.
I tried it in following way , but it did not works well.
Please tell me how to do this.

  • Mount local templates dir to es conifg path dir
version: "3"
services:
  elasticsearch:
    build: ./elasticsearch
    container_name: elasticsearch
    environment:
      discovery.type: single-node
      TZ:
    volumes:
      - ./elasticsearch/config/templates:/usr/share/elasticsearch/config/templates
    ports:
      - 9200:9200
      - 9300:9300

  logstash:
    build: ./logstash
    container_name: logstash
    environment:
      TZ:
    volumes:
      - ./logstash/pipeline:/usr/share/logstash/pipeline
      - ./logstash/driver:/usr/share/logstash/driver
      - ${HOME}:/home
    depends_on:
      - elasticsearch

  kibana:
    build: ./kibana
    container_name: kibana
    environment:
      ELASTICSEARCH_URL: http://elasticsearch:9200
      TZ:
    ports:
      - 5601:5601
    depends_on:
      - elasticsearch
  • In es docker container, templates dir exists
[root@7f27d68d75ea elasticsearch]# ls -l config/templates/
total 4
-rw-r--r-- 1 root root 208 May 18 07:52 zsh_history.json
  • But index template dose not exists.
$ curl -X GET 'http://localhost:9200/_template/zsh_history/?pretty'
{ }
2

which version are you using? that feature was removed in elasticsearch 2.0.

https://www.elastic.co/guide/en/elasticsearch/reference/2.0/breaking_20_index_api_changes.html#_file_based_index_templates

3

Thank you for replying.

which version are you using? that feature was removed in elasticsearch 2.0.

I'm using 7.0.1(docker.elastic.co/elasticsearch/elasticsearch:7.0.1).
So, I got it that file based index template is no longer available as you mentioned.

I would like to ask another question.
What is a best practice to add index templates in executing docker-compose up?

4

Hi,

Not sure I can say if it's the best practice or not, this really depends on your use case and your environment. Who control the template, do you want to change it without restarting your logstash container, etc.

Based on your original post I would say it could be a good idea to configure logstash to load your custom index template into Elasticsearch. The logstash output plugin for elasticsearch has many settings intended to control and load index template into elasticsearch. Look for all the template related settings on this page:
https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-template

Beats also have those feature and they can be configured to load either default or custom index templates into elasticsearch. So a container running something like filebeat could be used, with the proper configuration it could be tasked with loading your index template into elasticsearch. It could even execute only once, load the template then die once it's finished via the "setup" command that Beats have.
For Beats you can't point them at a template file, you need to give the beat a path to a fields.yml file which represent the index template if you will, the Beat will generate an index template from the fields.yml file and load it into elasticsearch.

Of course you can also use simple curl command to load index template json file into elasticsearch. So it's possible to build a docker container whose sole purpose is to load index templates into elasticsearch once it is started and healthy. But if you do that the "orchestration" of it is left for you to figure out and organize. You can obvisouly fail to load an index template into elasticsearch if you do it while elasticsearch is not yet available for example... So you would need to account for that if you use a sidecar container tasked with loading index templates.

Many ways...
I think the best practice you're looking for because you want the thing to work after a docker-compose up is to consider that Elastic has made the Beats and Logstash capable of handling and loading index template into Elasticsearch when they are about to start sending events into ES. Like if the clients were in charge of loading the template they need for the events they will send.

That works for many but it can have some drawbacks depending on your goal and setup that would lead you to use other more custom methods or less automatic methods.

1 Like
5

Hi, martinr_ubi

Thank you for your sincere response.
The logstash output plugin for elasticsearch seems what I have wanted.
I will try it!

6

I tried logstash output plugin with folloing code.

output {
    if "zsh-history" in [tags] {
        elasticsearch {
            hosts => ["http://elasticsearch:9200"]
            index => "zsh-history"
            template => "/usr/share/logstash/templates/zsh-history.json"
            template_name => "zsh-history"
        }
    }
}

This makes logstash create a template in sending data to elasticsearch.
This behavior is what I have wanted.
Thank you very much!

1 Like
7

Thanks for reporting back with your experience!

BTW make sure to remember about this setting and the fact that it defaults to false.
https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-template_overwrite
Which is important if you ever change the template because the default setting makes Logstash load the template only if it doesn't exists in ES. If you change it even a restart of Logstash will not make it overwrite the old version that is already in ES.

The Beats work the same way, they don't overwrite by default.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.