Configuration indices in Filebeat.yaml

[YaroslavN (Yaroslav)]

Hi,

I try to setup different index names depends on types specified in different file prospectors. For example:

- input_type: log

  paths:
    - /path_to_log/application1.log
  fields:
    type: "application1"
  fields_under_root: true
  multiline.pattern: ^[0-9]{4}-[0-9]{2}-[0-9]{2}
  multiline.negate: true
  multiline.match: after  

- input_type: log

  paths:
    - /path_to_log/application2.log
  fields:
    type: "application2"
  fields_under_root: true
  multiline.pattern: ^[0-9]{4}-[0-9]{2}-[0-9]{2}
  multiline.negate: true
  multiline.match: after  

Than, I specified output in such way:

output.elasticsearch:
  hosts: ["https://****.us-east-1.aws.found.io:9243"]
  username: ***
  password:***
  index: "service-logs-%{+yyyy.MM.dd}"
  indices:
    - index: "error-%{[@metadata][type]}-%{+yyyy.MM.dd}"
      when.contains:
        fields.type: "error"
    - index: "application1-%{+yyyy.MM.dd}"
      when.equals:
        fields.type: "application1"
    - index: "application2-%{+yyyy.MM.dd}"
      when.equals:
        fields.type: "application2"

In Kibana, I could found only service-logs-* indices, but not application1, application2 or error.

As an example, I use this link: https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#_index.

Thanks.

[steffens (Steffen Siering)]

Please format logs and configuration files using the </>-Button.

I'm not sure if using type and fields_under_root can be recommended, as the document_type setting also overwrites the type field. That is, given your configuration I'm not sure about the value of type. Running filebeat with debug logging enabled (e.g. -d 'publish') will print the events before publishing. This might help inspecting the actual events contents.

You are using the setting fields_under_root: true. This setting merges configured fields right into the event generated, without creating the namespace fields. Yet you reference to the fields namespace in your output config. Either remove fields_under_root: true from you config or try changing your conditionals from when.<cond>.fields.type to when.<cond>.type: ... .

I have no idea where/when/why you set the type "error" (incomplete config?). If index is empty/fails cause the field referenced is missing, the next configuration will be run. If all failed, the index setting will be used.

I have no idea why you use [@metdata][type] in one of your indices. With filebeat 5.6 this field is hardcoded to "docs" + was only shipped to logstash in the past. The field is not known to the elasticsearch output.

Given you have some application logs and some generic logs you can also configure filebeat like this:

filebeat.prospectors:
- input_type: log
  fields.application: application1
  paths:
    - /path/to/application1.log
  multiline: ...
- input_type: log
  fields.application: application2
  paths:
    - /path/to/application2
  ...
- input_type: log
  paths:
    - /error/logs

...

output.elasticsearch:
  index: "service-logs:%{+yyyy.MM.dd}"
  indices:
    - index: "%{[fields.application]}-%{+yyyy.MM.dd}"
    - index: ...

if you can always construct the index name from fields but you want a fallback, you can also make use of 'defaults', like:

output.elasticsearch:
  index: "%{[fields.application]:service-logs}-%{+yyyy.MM.dd}"

[YaroslavN (Yaroslav)]

Hi, thank you for feedback.

- input_type: log
paths:
    - ...\application.log
  fields:
    type: operation-logloader
  fields_under_root: true
  multiline.pattern: ^[0-9]{4}-[0-9]{2}-[0-9]{2}
  multiline.negate: true
  multiline.match: after  

- input_type: log
  paths:
     - ...\error.log
  fields:
     type: operation-logloader-error
  fields_under_root: true
  multiline.pattern: ^[0-9]{4}-[0-9]{2}-[0-9]{2}
  multiline.negate: true
  multiline.match: after

output.elasticsearch:
  ...
  index: "%{[type]:service-logs}-%{+yyyy.MM.dd}"

This fix works for me and I could see different indices with different types.

It is possible to change behavior to extend a list of available fields in Kibana with such log configuration:

- input_type: log

  paths:
    - ...\application.log
  fields:
    service_name: operation-logloader
  multiline.pattern: ^[0-9]{4}-[0-9]{2}-[0-9]{2}
  multiline.negate: true
  multiline.match: after  

  
- input_type: log

  paths:
    - ...\error.log
  fields:
    service_name: operation-logloader-error
  multiline.pattern: ^[0-9]{4}-[0-9]{2}-[0-9]{2}
  multiline.negate: true
  multiline.match: after

output.elasticsearch:
  index: "%{[fields.service_name]:service-logs}-%{+yyyy.MM.dd}"

As result, I expect something like this:

Thank you.

[steffens (Steffen Siering)]

The field should become available in kibana once available. You might have to update the index pattern (check for index patterns in management UI), as an index pattern is kind of a mapping to a set of indices matching.

Depending on fields_under_root setting your custom fields should be available via service_name or fields.service_name.

[YaroslavN (Yaroslav)]

That works, thank you,