# How to save multiple logs to separate ES index's

**URL:** https://discuss.elastic.co/t/how-to-save-multiple-logs-to-separate-es-indexs/227685
**Category:** Elasticsearch
**Created:** [April 12, 2020, 3:49pm UTC](https://discuss.elastic.co/t/how-to-save-multiple-logs-to-separate-es-indexs/227685 "2020-04-12T15:49:07Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![VSP](https://avatars.discourse-cdn.com/v4/letter/v/e495f1/32.png) [@VSP](https://discuss.elastic.co/u/VSP)
#### Post date: [April 12, 2020, 3:49pm UTC](https://discuss.elastic.co/t/how-to-save-multiple-logs-to-separate-es-indexs/227685/1 "2020-04-12T15:49:07Z")

</div>

Hello,

I am trying to send various types of logs through Filebeat -\> Logstash -\> Elastich Search -\> Kibana  
I used Fields with a variable log\_type and assigned different value to the variable basing on the type of the log and sending it to Logstash. In the output section of the logstash which is sending the data to ES, i am unable to use those fields set in Filebeat to create a index with the name.

Can someone show a sample of how this can be done please.

Code:

Filebeat.yml:

#=========================== Filebeat inputs =============================

filebeat.inputs:

- type: log

# fields\_under\_root: true

- type: log  
paths:
  - /var/www/sites/api/log/debug-\*.log  
fields:  
log\_type: debug\_log

Logstash:

input{  
beats{  
port =\> "5044"  
}  
}

#filter{  
grok {  
match =\> ["message", "%{TIMESTAMP\_ISO8601:timestamp}"]  
}  
date {  
match =\> ["timestamp", "ISO8601"]  
}  
}

output{  
elasticsearch {  
hosts =\> ["xxxx"]  
index =\> "%{[@metadata][fields]}-%{[@metadata][log\_type]}" -\> HOW TO REFER THE FIELDS FROM FILEBEAT TO CREATE SEPARATE INDEX?  
}  
}

---

<div class="post-metadata">

### Author: ![rugenl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rugenl/32/12887_2.png) [@rugenl](https://discuss.elastic.co/u/rugenl)
#### Post date: [April 12, 2020, 4:41pm UTC](https://discuss.elastic.co/t/how-to-save-multiple-logs-to-separate-es-indexs/227685/2 "2020-04-12T16:41:01Z")

</div>

Here's one of our sanatized logstash output sections:

```
output {
  if ([fields][app_id] == "acdt") {
    elasticsearch {
      hosts => [{{ ES_http }}]
      cacert => "/etc/logstash/certs/https_interm.cer"
      user => "{{ elastic.user }}"
      password => "{{elastic.pass }}"
      sniffing => false
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[fields][app_id]}-%{[fields][campus]}"
    }
  }
  else if "use_ingest" in [tags] and [fileset][module] {
    elasticsearch {
      hosts => [{{ ES_http }}]
      cacert => "/etc/logstash/certs/https_interm.cer"
      user => "{{ elastic.user }}"
      password => "{{elastic.pass }}"
      sniffing => false
      manage_template => false
      pipeline => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[fileset][module]}-%{[fileset][name]}-pipeline"
      ilm_enabled => true
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[fields][app_id]}-%{[fields][campus]}"
    }
  }
  else if "use_ingest" in [tags] and [agent][module] {
    elasticsearch {
      hosts => [{{ ES_http }}]
      cacert => "/etc/logstash/certs/https_interm.cer"
      user => "{{ elastic.user }}"
      password => "{{elastic.pass }}"
      sniffing => false
      manage_template => false
      pipeline => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[agent][module]}-%{[fileset][name]}-pipeline"
      ilm_enabled => true
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[fields][app_id]}-%{[fields][campus]}"
    }
  }
  else {
    elasticsearch {
      hosts => [{{ ES_http }}]
      cacert => "/etc/logstash/certs/https_interm.cer"
      user => "{{ elastic.user }}"
      password => "{{elastic.pass }}"
      sniffing => false
      manage_template => false
      ilm_enabled => true
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[fields][app_id]}-%{[fields][campus]}"
    }
  }
}

```

All but the first of these are writing to an ILM alis, but you have to define the ILM parts including creating the initial empty index manually first.

This is an Ansible template so consider {{ vars }} "sanitized". The logic for [agent][module] and [fileste][module] is to accommodate breaking changes (Thanks Elastic).

---

<div class="post-metadata">

### Author: ![VSP](https://avatars.discourse-cdn.com/v4/letter/v/e495f1/32.png) [@VSP](https://discuss.elastic.co/u/VSP)
#### Post date: [April 17, 2020, 4:30pm UTC](https://discuss.elastic.co/t/how-to-save-multiple-logs-to-separate-es-indexs/227685/3 "2020-04-17T16:30:47Z")

</div>

Thanks Rugen!

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 15, 2020, 4:31pm UTC](https://discuss.elastic.co/t/how-to-save-multiple-logs-to-separate-es-indexs/227685/4 "2020-05-15T16:31:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.