Filebeat Apache Module Change Index Name

[Ryne_Keel]

Been having trouble getting the index to be created as a different index name than filebeat.

#============================= Filebeat modules ===============================

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml

  reload.enabled: true
  reload.period: 10s

#==================== Elasticsearch template setting ==========================


setup.template.name: "apache"
setup.template.name: "apache-*"
setup.template.fields: "fields.yml"
setup.template.overwrite: false
setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 1


#==================== General ==========================

name: agamemnon
#tags: ["apache", "internal"]

#============================== Dashboards =====================================

setup.dashboards:
  index: "apache-*"

#============================== Kibana =====================================

setup.kibana:
  host: "kibana:5601"

#================================ Outputs =====================================
output.elasticsearch:
  hosts: ["elastic1:9200", "elastic2:9200"]
  index: "apache-%{[agent.version]}-%{+yyyy.MM.dd}"

#----------------------------- Logstash output --------------------------------
#output.logstash:
#  hosts: ["logstash2:5044"]

#================================ Processors =====================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

#==================== Logging ==========================

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644

With the config above, I am able to delete my apache dashboard and visualizations, then have it reimport the dashboards, and the dashboard will look for apache-* index pattern as expected.

However, I cannot get the index to write as apache-, and it always just uses the filebeat index. I would like to change this as across all of our hosts I will be sending certain things such as apache, nginx, and mysql for starters with filebeat modules. I was able to get the index name to change using logstash, but logstash strips the fields that filebeat sends which are needed to operate the default dashboard, and are also more robust than the logstash combinedaccess log grok filter, so was hoping to for ease send directly from FB to ES.

I assume it hurts performance to have multiple modules / event patterns sent to ES inside the same index?

Any suggestions?

Filebeat7, ES7, LS7, Kibana7

[jsoriano (Jaime Soriano)]

Hi @Ryne_Keel,

If you want to use different indexes for different modules you can use the indices output setting. It allows you to configure different indices depending on some condition, for example you could do something like this in your case to have a different index for apache module:

output.elasticsearch:
  hosts: ["elastic1:9200", "elastic2:9200"]
  indices:
    - index: "filebeat-%{[agent.version]}-apache-%{+yyyy.MM.dd}"
      when.equals:
        event.module: "apache"
    - default: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"

Notice that here I try to keep the default prefixes, this way the defailt patterns for the index mappings (filebeat-%{[agent.version]}-*) and the dashboard (filebeat-*) should work and you wouldn't need to modify these setup settings.

[Ryne_Keel]

This works although I can't get the:

    - default: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"

working - filebeat won't even start when I have that for some reason, but it works well if I just take that out. Thanks for the advice too on putting apache in the middle of the index name so that I don't have to mess with the default dashboard

[jsoriano (Jaime Soriano)]

Oh sorry, my bad, the default option is for a default mapping, not for a default index.

[Ryne_Keel]

No worries, I am just glad this is working now Thanks again

[subhas (Subhas Takhellambam)]

Hi @jsoriano ,

Can we use two different indexes within the same apache module? If so how do you configure that? Thanks.