/_template OR /_index_template?

Elastic Stack Logstash
,
1

Hi,

I am using ES 7.11.1 and Logstash 7.11.1 in Docker container.
here is my logstash.conf:

    input {
  file {
    codec => "json"
    path => "/usr/share/logstash/config/sample.log"
  }
}

filter {
}

output {
  elasticsearch {
    hosts => ["http://10.6.226.80:9200"]
    index => "bjb-test-%{+YYYY.MM.dd}"
    manage_template => true
    template => "/usr/share/logstash/config/logstash-test-template.json"
    template_name => "bjb-test"
  }
}

here is the logstash-test-template.json:

    {
  "template": {
    "settings": {
      "number_of_shards": "1",
      "number_of_replicas": "0",
      "refresh_interval": "10s",
      "codec": "best_compression",
      "lifecycle": {
        "name": "bjb-test"
      },
      "mapping": {
        "total_fields": {
          "limit": "50"
        }
      }
    },
    "mappings": {
      "dynamic": true,
      "numeric_detection": false,
      "date_detection": true,
      "dynamic_date_formats": [
        "strict_date_optional_time",
        "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
      ],
      "_source": {
        "enabled": true,
        "includes": [],
        "excludes": []
      },
      "_routing": {
        "required": false
      },
      "dynamic_templates": []
    }
  },
  "index_patterns": [
    "bjb-test-*"
  ],
  "data_stream": {}
}

when I run docker-compose up -d logstash-test, i received the following error:

    [2021-03-08T20:45:59,983][ERROR][logstash.outputs.elasticsearch][main] Failed to install template. {:message=>"Got response code '400' contacting Elasticsearch at URL 'http://10.6.226.80:9200/_template/bjb-test'", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.5.1-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.5.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:332:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.5.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:319:in `block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.5.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:414:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.5.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:318:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.5.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:326:in `block in Pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.5.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:352:in `template_put'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.5.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:86:in `template_install'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.5.1-java/lib/logstash/outputs/elasticsearch/template_manager.rb:28:in `install'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.5.1-java/lib/logstash/outputs/elasticsearch/template_manager.rb:16:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.5.1-java/lib/logstash/outputs/elasticsearch/common.rb:205:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.5.1-java/lib/logstash/outputs/elasticsearch/common.rb:49:in `block in setup_after_successful_connection'"]}

From the error message above, the URL is /_template/

While if I try to create index template using Kibana 7.11.1 Create Template wizard, the URL is PUT _index_template/bjb-test , and successful.

Why Logstash is calling /_template/... instead of _index_template?
Is this the main source of error?

Please enlighten me.
Thank you.

2

The code is using the legacy template feature rather than composable templates. It decides which to use based on the elasticsearch version.

3

can you do it without this one?

because if you put the template with
"index_patterns": ["bjp-test-*"] then when you write to that index it will automatically uses that template.

4

Hi @Badger ,

Thanks for your response.
Ok, so we should be using the new index template instead of the legacy one (since it is deprecated).

I still want to define it via logstash-test-template.json file which is referred by the output section in my logstash-test.conf, instead of defining it manually via Kibana.

Any reference on how to achieve that? and would be great if there is a valid template.json I can refer to.

Thank you.

5

Hi @elasticforme ,

I do want to define the index-template in a file (instead of defining it manually in Kibana.
That way, I can keep all the configuration in a git repo.
I used ES / Logstash 7.8 with the mentioned index-template, and it was running.

But when I try to setup the same config in ES / Logstash 7.11.1, it failed.
I wonder if there is any reference I can look into.

Thank you,

Satria

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.