# Configuration indices in Filebeat.yaml

**URL:** <https://discuss.elastic.co/t/configuration-indices-in-filebeat-yaml/105636>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [October 28, 2017, 8:33am UTC](https://discuss.elastic.co/t/configuration-indices-in-filebeat-yaml/105636 "2017-10-28T08:33:50Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![YaroslavN](https://avatars.discourse-cdn.com/v4/letter/y/43a26b/32.png) [@YaroslavN](https://discuss.elastic.co/u/YaroslavN)\
**Post date:** [October 28, 2017, 8:33am UTC](https://discuss.elastic.co/t/configuration-indices-in-filebeat-yaml/105636/1 "2017-10-28T08:33:51Z")

</div>

Hi,

I try to setup different index names depends on types specified in different file prospectors. For example:

```auto
- input_type: log

  paths:
    - /path_to_log/application1.log
  fields:
    type: "application1"
  fields_under_root: true
  multiline.pattern: ^[0-9]{4}-[0-9]{2}-[0-9]{2}
  multiline.negate: true
  multiline.match: after  

- input_type: log

  paths:
    - /path_to_log/application2.log
  fields:
    type: "application2"
  fields_under_root: true
  multiline.pattern: ^[0-9]{4}-[0-9]{2}-[0-9]{2}
  multiline.negate: true
  multiline.match: after  

```

Than, I specified output in such way:

```auto
output.elasticsearch:
  hosts: ["https:// ****.us-east-1.aws.found.io:9243"]
  username: ***
  password:***
  index: "service-logs-%{+yyyy.MM.dd}"
  indices:
    - index: "error-%{[@metadata][type]}-%{+yyyy.MM.dd}"
      when.contains:
        fields.type: "error"
    - index: "application1-%{+yyyy.MM.dd}"
      when.equals:
        fields.type: "application1"
    - index: "application2-%{+yyyy.MM.dd}"
      when.equals:
        fields.type: "application2"

```

In Kibana, I could found only service-logs-\* indices, but not application1, application2 or error.

As an example, I use this link: [https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#\_index](https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#_index).

Thanks.

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [October 30, 2017, 12:31pm UTC](https://discuss.elastic.co/t/configuration-indices-in-filebeat-yaml/105636/2 "2017-10-30T12:31:19Z")

</div>

Please format logs and configuration files using the `</>`-Button.

I'm not sure if using `type` and `fields_under_root` can be recommended, as the `document_type` setting also overwrites the `type` field. That is, given your configuration I'm not sure about the value of `type`. Running filebeat with debug logging enabled (e.g. `-d 'publish'`) will print the events before publishing. This might help inspecting the actual events contents.

You are using the setting `fields_under_root: true`. This setting merges configured fields right into the event generated, without creating the namespace `fields`. Yet you reference to the `fields` namespace in your output config. Either remove `fields_under_root: true` from you config or try changing your conditionals from `when.<cond>.fields.type` to `when.<cond>.type: ...` .

I have no idea where/when/why you set the type `"error"` (incomplete config?). If `index` is empty/fails cause the field referenced is missing, the next configuration will be run. If all failed, the `index` setting will be used.

I have no idea why you use `[@metdata][type]` in one of your indices. With filebeat 5.6 this field is hardcoded to `"docs"` + was only shipped to logstash in the past. The field is not known to the elasticsearch output.

Given you have some application logs and some generic logs you can also configure filebeat like this:

```auto
filebeat.prospectors:
- input_type: log
  fields.application: application1
  paths:
    - /path/to/application1.log
  multiline: ...
- input_type: log
  fields.application: application2
  paths:
    - /path/to/application2
  ...
- input_type: log
  paths:
    - /error/logs

...

output.elasticsearch:
  index: "service-logs:%{+yyyy.MM.dd}"
  indices:
    - index: "%{[fields.application]}-%{+yyyy.MM.dd}"
    - index: ...

```

if you can always construct the index name from fields but you want a fallback, you can also make use of 'defaults', like:

```auto
output.elasticsearch:
  index: "%{[fields.application]:service-logs}-%{+yyyy.MM.dd}"

```

---

<div class="post-metadata">

**Author:** ![YaroslavN](https://avatars.discourse-cdn.com/v4/letter/y/43a26b/32.png) [@YaroslavN](https://discuss.elastic.co/u/YaroslavN)\
**Post date:** [October 31, 2017, 8:56am UTC](https://discuss.elastic.co/t/configuration-indices-in-filebeat-yaml/105636/3 "2017-10-31T08:56:58Z")

</div>

Hi, thank you for feedback.

```
- input_type: log
paths:
    - ...\application.log
  fields:
    type: operation-logloader
  fields_under_root: true
  multiline.pattern: ^[0-9]{4}-[0-9]{2}-[0-9]{2}
  multiline.negate: true
  multiline.match: after  

- input_type: log
  paths:
     - ...\error.log
  fields:
     type: operation-logloader-error
  fields_under_root: true
  multiline.pattern: ^[0-9]{4}-[0-9]{2}-[0-9]{2}
  multiline.negate: true
  multiline.match: after

output.elasticsearch:
  ...
  index: "%{[type]:service-logs}-%{+yyyy.MM.dd}"

```

This fix works for me and I could see different indices with different types.

It is possible to change behavior to extend a list of available fields in Kibana with such log configuration:

```
- input_type: log

  paths:
    - ...\application.log
  fields:
    service_name: operation-logloader
  multiline.pattern: ^[0-9]{4}-[0-9]{2}-[0-9]{2}
  multiline.negate: true
  multiline.match: after  

  
- input_type: log

  paths:
    - ...\error.log
  fields:
    service_name: operation-logloader-error
  multiline.pattern: ^[0-9]{4}-[0-9]{2}-[0-9]{2}
  multiline.negate: true
  multiline.match: after

output.elasticsearch:
  index: "%{[fields.service_name]:service-logs}-%{+yyyy.MM.dd}"

```

As result, I expect something like this:

![image](https://us1.discourse-cdn.com/elastic/original/3X/8/a/8a68bf7870e9f0f4fc58611e38f82762e4f1a640.png)

Thank you.

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [November 1, 2017, 12:07pm UTC](https://discuss.elastic.co/t/configuration-indices-in-filebeat-yaml/105636/4 "2017-11-01T12:07:06Z")

</div>

The field should become available in kibana once available. You might have to update the index pattern (check for index patterns in management UI), as an index pattern is kind of a mapping to a set of indices matching.

Depending on `fields_under_root` setting your custom fields should be available via `service_name` or `fields.service_name`.

---

<div class="post-metadata">

**Author:** ![YaroslavN](https://avatars.discourse-cdn.com/v4/letter/y/43a26b/32.png) [@YaroslavN](https://discuss.elastic.co/u/YaroslavN)\
**Post date:** [November 2, 2017, 7:30am UTC](https://discuss.elastic.co/t/configuration-indices-in-filebeat-yaml/105636/5 "2017-11-02T07:30:33Z")

</div>

That works, thank you,

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 30, 2017, 7:30am UTC](https://discuss.elastic.co/t/configuration-indices-in-filebeat-yaml/105636/6 "2017-11-30T07:30:47Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.