Override index name while using the inbuilt filebeat modules

Elastic Stack Beats
,
1

We are in process of standardising index names to ensure permissions are given correctly
(eg my_os_windows_yyyy-mm-dd, my_os_linux_yyyy-mm-dd, my_network_cisco_yyyy-mm-dd etc..) rather than the default "filebeat*" format. This way we can control which index and data and roles/permissions/index patterns etc.

I understand how to put the output settings if I'm developing my on logstash.conf I know how to ensure it goes into correct index.

But if I'm using the inbuilt modules (eg system or audit modules) within filebeat, how would I configure the outputs? The only area I could find is to put "output.elasticsearch" in the main filebeat.yml, but that is not good as you have single index name there? So is there any way, we can put this setting in the module.d itself or an outputs.d folder?

I'm planning to send data from
client => filebeats => logstash => elasticsearch

So how can I make sure it goes into my own index rather than the hardcoded filebeat* nidex?

2

Hi!

If you are planning to send data to Elasticsearch directly you can set the index dynamically by using a format string to access any event. For instance:

output.elasticsearch:
  hosts: ["http://localhost:9200"]
  index: "%{[fields.log_type]}-%{[agent.version]}-%{+yyyy.MM.dd}"

See https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#index-option-es for inforamtion.

Also you can configure Logstash accordingly: https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html

C.

3

thanks for that.

But I thought %{[fields.log_type]}% parameter is just the log severity? I was looking for something at a type-of-data level, like "audit" or "security" "operating" system etc which I can configure based on the data or user custom.

Also where do you update this (as per doc this in the filebeat.yml)? But filebeat.yml is common to entire dataset of the filebeat I thought
I was looking to configure them at the module level, so the data output is configured within the module (and not at the entire filebeat installation level)

4

You can add custom fields in the events on module's level:

- module: system
  period: 10s
  metricsets:
    - cpu
    - load
    - memory
    - network
    - process
    - process_summary
    - socket_summary
  fields:
    index_type: security

Then you can use index_type in place of log_type. See more at https://www.elastic.co/guide/en/beats/filebeat/current/configuration-general-options.html#libbeat-configuration-fields

1 Like
6

@ChrsMark
perfect. That's exactly I was looking for. Thanks mate

7

Also found another method using service.type
To add into filebeat.yml, to customise per index/module

output.elasticsearch:
  hosts: ["192.168.2.26:9200"]
  indices:
    - index: "my_%{[service.type]}-%{[agent.version]}-%{+yyyy.MM.dd}"

need to ensure service.type is filled up correctly

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.