Multiple templates in single file in logstash's elasticsearch output plugin

[siginigin (sigi)]

Hi guys,

I'm trying to install multiple templates (=2) in single index, based on name pattern. The point is to minimize number of fileds in indexes. I want to achieve to have default and proxy fields in logstash-proxy indexes, and default and mail fields in logstash-mail-* indexes.

This is my setup:

/etc/logstash/template.json:
{
"template": "logstash-",
"settings": {
"number_of_shards": 3,
"number_of_replicas": 1,
"index.refresh_interval" : "5s",
"index.mapping.total_fields.limit": 1000
},
"mappings": {
"default":{
"properties": {
"message": { "type": "text"},
"received_at": { "type": "date" },
"collector_ip": { "type": "ip" },
"logstash": { "type": "keyword" },
"program": { "type": "keyword" },
"host": { "type": "ip" },
"logsource": { "type": "ip" },
}
},
"logstash-proxy-":{
"properties": {
"url": { "type": "keyword" }
# more fields located only in logstash-proxy-*
}
},
"logstash-mail-":{
"properties": {
"sender-address": { "type": "keyword" }
# more fields located only in logstash-mail-
}
}
}
}

Logstash output config:
output {
if [program] == "squid" {
elasticsearch {
hosts => "127.0.0.1"
template => ["/etc/logstash/template.json"]
manage_template => true
template_overwrite => true
index => "logstash-proxy-%{+YYYY.MM.dd}"
}
} else if [program] == "postfix" {
elasticsearch {
hosts => "127.0.0.1"
template => ["/etc/logstash/template.json"]
manage_template => true
template_overwrite => true
index => "logstash-mail-%{+YYYY.MM.dd}"
}
}
}

The problem is that multiple mappings in single template are not allowed (Logstash's error: Got response code '400', blabla, block in install_template_after_successful_connection). What's the correct way to do this?

Thank you for help.

[warkolm (Mark Walkom)]

The correct way is to have two separate files, Elasticsearch won't accept what you have there so it'll never work.

[siginigin (sigi)]

Ok, thanx for reply. I did that using ansible templates. For anyone looking for same solution, here you go:

template-basic.j2
{
"template": "logstash-{{ item.index }}-*",
"settings": {
"number_of_shards": 3,
"number_of_replicas": 1,
},
"mappings": {
"doc":{
"properties": {
"message": { "type": "text"},
and other basic mappings
"host": { "type": "keyword"},
{# here goes fields for specific templates #}
{{ item.file | indent(32,true) }}
}
}
}
}

Specific template, for example proxy.txt
"url": { "type": "keyword" },
"domain": { "type": "keyword" },
"http_protocol": { "type": "keyword" },
"http_method": { "type": "keyword" },
"http_status_code": { "type": "integer" },
"squid_result_code": { "type": "keyword" },
"squid_hierarchy_code": { "type": "keyword" },
"duration": { "type": "integer" },
"bytes": { "type": "integer" },
"server": { "type": "ip" }

And now ansible playbook:

• name: elastics
  hosts: all
  vars:
  logstash_templates:
  - file: "{{ lookup('file', 'proxy.txt' ) }}"
  index: proxy
  - file: "{{ lookup('file', 'asa.txt' ) }}"
  index: asa
  etc...
  tasks:
  • name: Logstash - template for field mapping
    template:
    src: template-basic.j2
    dest: /etc/logstash/templates/{{ item.index }}.json
    mode: 0644
    with_items:
    • "{{ logstash_templates }}"
      notify: Restart Logstash
      tags: logstash

In this scenario I have separate templates for various indexes, which I can now use like this in logstash:
output {
if [program] == "ASA" {
elasticsearch {
hosts => "127.0.0.1"
template => ["/etc/logstash/templates/asa.json"]
template_name => "logstash-asa"
manage_template => true
template_overwrite => true
index => "logstash-asa-%{+YYYY.MM.dd}"
}
if [program] == "squid" {
elasticsearch {
hosts => "127.0.0.1"
template => ["/etc/logstash/templates/proxy.json"]
template_name => "logstash-proxy"
manage_template => true
template_overwrite => true
index => "logstash-proxy-%{+YYYY.MM.dd}"
}
}
}

It's worth mention that default template name in logstash output is logstash, so I needed to change it using template_name, otherwise you'll end up with one template - the last one in your output.

Better solution is welcome