Multiple templates in single file in logstash's elasticsearch output plugin

Elastic Stack Elasticsearch
1

Hi guys,

I'm trying to install multiple templates (=2) in single index, based on name pattern. The point is to minimize number of fileds in indexes. I want to achieve to have default and proxy fields in logstash-proxy indexes, and default and mail fields in logstash-mail-* indexes.

This is my setup:

/etc/logstash/template.json:
{
"template": "logstash-",
"settings": {
"number_of_shards": 3,
"number_of_replicas": 1,
"index.refresh_interval" : "5s",
"index.mapping.total_fields.limit": 1000
},
"mappings": {
"default":{
"properties": {
"message": { "type": "text"},
"received_at": { "type": "date" },
"collector_ip": { "type": "ip" },
"logstash": { "type": "keyword" },
"program": { "type": "keyword" },
"host": { "type": "ip" },
"logsource": { "type": "ip" },
}
},
"logstash-proxy-":{
"properties": {
"url": { "type": "keyword" }
# more fields located only in logstash-proxy-*
}
},
"logstash-mail-":{
"properties": {
"sender-address": { "type": "keyword" }
# more fields located only in logstash-mail-
}
}
}
}

Logstash output config:
output {
if [program] == "squid" {
elasticsearch {
hosts => "127.0.0.1"
template => ["/etc/logstash/template.json"]
manage_template => true
template_overwrite => true
index => "logstash-proxy-%{+YYYY.MM.dd}"
}
} else if [program] == "postfix" {
elasticsearch {
hosts => "127.0.0.1"
template => ["/etc/logstash/template.json"]
manage_template => true
template_overwrite => true
index => "logstash-mail-%{+YYYY.MM.dd}"
}
}
}

The problem is that multiple mappings in single template are not allowed (Logstash's error: Got response code '400', blabla, block in install_template_after_successful_connection). What's the correct way to do this?

Thank you for help.

2

The correct way is to have two separate files, Elasticsearch won't accept what you have there so it'll never work.

3

Ok, thanx for reply. I did that using ansible templates. For anyone looking for same solution, here you go:

template-basic.j2
{
"template": "logstash-{{ item.index }}-*",
"settings": {
"number_of_shards": 3,
"number_of_replicas": 1,
},
"mappings": {
"doc":{
"properties": {
"message": { "type": "text"},
and other basic mappings
"host": { "type": "keyword"},
{# here goes fields for specific templates #}
{{ item.file | indent(32,true) }}
}
}
}
}

Specific template, for example proxy.txt
"url": { "type": "keyword" },
"domain": { "type": "keyword" },
"http_protocol": { "type": "keyword" },
"http_method": { "type": "keyword" },
"http_status_code": { "type": "integer" },
"squid_result_code": { "type": "keyword" },
"squid_hierarchy_code": { "type": "keyword" },
"duration": { "type": "integer" },
"bytes": { "type": "integer" },
"server": { "type": "ip" }

And now ansible playbook:

  • name: elastics
    hosts: all
    vars:
    logstash_templates:
    - file: "{{ lookup('file', 'proxy.txt' ) }}"
    index: proxy
    - file: "{{ lookup('file', 'asa.txt' ) }}"
    index: asa
    etc...
    tasks:
    • name: Logstash - template for field mapping
      template:
      src: template-basic.j2
      dest: /etc/logstash/templates/{{ item.index }}.json
      mode: 0644
      with_items:
      • "{{ logstash_templates }}"
        notify: Restart Logstash
        tags: logstash

In this scenario I have separate templates for various indexes, which I can now use like this in logstash:
output {
if [program] == "ASA" {
elasticsearch {
hosts => "127.0.0.1"
template => ["/etc/logstash/templates/asa.json"]
template_name => "logstash-asa"
manage_template => true
template_overwrite => true
index => "logstash-asa-%{+YYYY.MM.dd}"
}
if [program] == "squid" {
elasticsearch {
hosts => "127.0.0.1"
template => ["/etc/logstash/templates/proxy.json"]
template_name => "logstash-proxy"
manage_template => true
template_overwrite => true
index => "logstash-proxy-%{+YYYY.MM.dd}"
}
}
}

It's worth mention that default template name in logstash output is logstash, so I needed to change it using template_name, otherwise you'll end up with one template - the last one in your output.

Better solution is welcome :slight_smile:

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.