Create independent indices

Elastic Stack Elasticsearch
1

Hello,
I am setting up a lab for log collection (Apache, Sophos, ...)
I want to create different indices for each device (Apache, Sophos, Windows, Linux, ...)

Is it possible to create independent indices?
How can this be done?
Could you provide an example link?

I can only get Filebeat to work if the index is the default one.

Thank you.

2

Hi,

Here's a basic example of how you can set this up:

filebeat.inputs:
- type: log
  paths:
    - /path/to/apache/logs/*.log
  fields:
    log_type: apache

- type: log
  paths:
    - /path/to/sophos/logs/*.log
  fields:
    log_type: sophos

output.elasticsearch:
  hosts: ["localhost:9200"]
  index: "%{[fields.log_type]}-%{+yyyy.MM.dd}"

In this example, Filebeat will read logs from the specified paths and assign a log_type field to each log line based on the source (Apache or Sophos in this case). The index option in the Elasticsearch output is then used to dynamically set the index name based on the log_type field and the current date.

Regards

3

-Hello, I have implemented the example you indicated, but it shows the following message. Exiting: setup.template.name and setup.template.pattern have to be set if index name is modified This message appears every time I modify the default index.

I have deleted the indexes, templates, pipelines, and reloaded them with: filebeat setup -e

I have modified the index as you indicated:

yamlCopy code

output.elasticsearch:
  hosts: ["localhost:9200"]
  index: "%{[fields.log_type]}-%{+yyyy.MM.dd}"

And it shows the following error:

Exiting: setup.template.name and setup.template.pattern have to be set if index name is modified

4

Hello, it has finally worked.
The configuration is the same as your example, but it wasn't working for me.
And I don't know the reasons.

Thanks

This the last configuration

filebeat:
  config:
    modules:
      path: /etc/filebeat/modules.d/*.yml
      reload:
        enabled: false
  inputs:
  - enabled: enable
    fields:
      log_type: apache-access
    id: apache-acces-id
    paths:
    - /root/access.log
    type: filestream
  - enabled: disabled
    fields:
      log_type: apache-error
    id: apache-error-id
    paths:
    - /root/error.log
    type: filestream
output:
  elasticsearch:
    hosts:
    - localhost:9200
    index: '%{[fields.log_type]}-%{+yyyy.MM.dd}'
path:
  config: /etc/filebeat
  data: /var/lib/filebeat
  home: /usr/share/filebeat
  logs: /var/log/filebeat
processors:
- add_host_metadata:
    when:
      not:
        contains:
          tags: forwarded
- add_cloud_metadata: null
- add_docker_metadata: null
- add_kubernetes_metadata: null
setup:
  ilm:
    enabled: false
  kibana:
    host: localhost:5601
  template:
    enabled: false
1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.