Issue with sending apache logs to elasticsearch with different indices

Elastic Stack Beats
1

I am new to ELK and I want to use filebeat to fetch and transfer apache access and error logs to elasticsearch index directly. However, I need to send the logs to different indices (rather than the default filebeat* index), and I also need to enable ILM for both indices. How can I achieve this?

Here is what I have done so far:

  • I have enabled the apache module in filebeat and configured the apache.yml file in modules.d directory. Here is a sample of my configuration:
- module: apache
  # Access logs
  access:
    enabled: true
    var.paths: ["/home/mypath/LOGS/ACCESS/**"]

  # Error logs
  error:
    enabled: true
    var.paths: ["/home/mypath/LOGS/ERROR/**"]
  • I have not enabled any direct filebeat.inputs in the filebeat.yml file, only via modules. Here is a sample of my configuration:
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 60s

setup.template.settings:
  index.number_of_shards: 1

name: "elk-filebeat"
tags: ["elk-filebeat", "web-tier"]

# Elasticsearch output configuration
output.elasticsearch:
  hosts: ["https://<ip>:9200"]
  protocol: "https"
  username: "elastic"
  password: "#######"
  ssl.certificate_authorities: "/cert/path/elasticsearch-ca.pem"

	# indices settings
  indices:
    - index: "apache-access-%{[agent.version]}-%{+yyyy.MM.dd}"
      when.equals:
        event.module: "apache"
        event.dataset: "apache.access"

    - index: "apache-error-%{[agent.version]}-%{+yyyy.MM.dd}"
      when.equals:
        event.module: "apache"
        event.dataset: "apache.error"

# Index lifecycle management(Need correction How to manage for 2 indices???)
setup.ilm:
  enabled: true
  policy_name: "apache"
  overwrite: true
  rollover_alias: "apache-%{[agent.version]}"
  pattern: "{now/d}-0000001"

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

logging.level: error
logging.selectors: ["*"]
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat.log
  keepfiles: 7
  permissions: 0644

I have searched online and found that to send logs to different indices, I need to set setup.ilm.enabled: false (because ilm is enabled by default). But this would disable ILM for all indices, which is not what I want.

Is there a way to send apache access logs to apache-access* index and apache error logs to apache-error* index, and also enable ILM for both indices?

References

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.