Create custom index in Filebeat

Elastic Stack Beats
,
1

Sure, here's the edited message:

Hi everyone,

I'm encountering an issue creating custom indexes in Elasticsearch using Filebeat for log flow (Filebeat ==> Elasticsearch).

I have two custom log files: /var/log/nginx/access.log and /var/log/nginx/error.log. In Filebeat, I aim to pass these log files to Elasticsearch and create custom indexes.

Below is my configuration file. Could you please suggest any necessary changes?

filebeat.inputs:
- type: logs
  enabled: true
  paths:
    - /var/log/nginx/access.log
    - /var/log/nginx/error.log
  fields_under_root: true
  fields:
    data_type: "nginx"

setup.template.enabled: true
setup.template.name: "index-%{[beat.version]}"
setup.template.pattern: "index-%{[beat.version]}-*"
setup.template.fields: "fields.yml"
setup.template.overwrite: false
setup.ilm.enabled: false

output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  username: "elastic"
  password: "elasticsearch123456"
  indices:
    - index: "index-%{[beat.version]}-%{[fields.type]:other}-%{+yyyy.MM.dd}"

Despite this configuration, I'm only getting a single index named filebeat-7.12.1-2024.05.03.

Any insights on what adjustments are needed would be greatly appreciated. Thank you!

2

After hours of searching finally claude helping

filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/nginx/access.log
    index: "nginx-access-%{+yyyy.MM.dd}"

  - type: log
    enabled: true
    paths:
      - /var/log/nginx/error.log
    index: "nginx-error-%{+yyyy.MM.dd}"

setup.template.enabled: false
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.fields: "fields.yml"
setup.template.overwrite: false

setup.ilm.enabled: false

output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  username: "elastic"
  password: "elasticsearch123456"

ilm.enabled: false

this how we can create seperate index for seperate log files