Setup filebeat to send different logs to different indexes (to elasticsearch)

Elastic Stack Elasticsearch
1

Hello, I setup a filebeat 8.8.2 on redhat host and configured my filebeat.yml like this, Im sending all my log data to ES directly,

filebeat.inputs:
- type: filestream
  id: my_id
  enabled: true
  paths:
    - /home/custom/logs/*

filebeat.modules:
- module: system
  enabled: True
  syslog.enabled: true
  auth.enabled: true

setup.template:
  enabled: true
  name: "my-custom-filebeat"
  pattern: "my-custom-filebeat-%{+yyyy.MM.dd}"
  fields: "/etc/filebeat/filebeat_fields.yml"
  overwrite: false
  settings:
    index.number_of_shards: 2
    index.number_of_replicas: 1
    setup.template.enabled: true
    index.codec: best_compression

setup.kibana:
  host: "https://elk:5601"
  api_key: "xxxx"
  ssl:
    certificate_authorities: "/etc/pki/elastic/ca.crt"
    verification_mode: "certificate"

output.elasticsearch:
  hosts: ["https://elk:9200"]
  protocol: "https"
  api_key: "xxxxxx"
  ssl:
    certificate_authorities: ["/etc/pki/elastic/ca.crt"]
    verification_mode: "certificate"
    

  indices:
    - index: "syslog-%{+yyyy.MM.dd}"
      when.equals:
        event.module: "system"
  index: "my-custom-filebeat-%{+yyyy.MM.dd}"

basically I am sending custom logs to my-custom-filebeat-YYYY-mm-dd index

and want to send anything from System module to syslog-YYYY-mm-dd index

I am seeing the filestream data for custom logs coming into my custom index (the filebeat is creating the index template for it), but I dont see anything coming in for Syslog index, it doesnt exist.

  1. should I create syslog index template manually?
  2. is there a way to add multiple "setup.template" parameters to automatically setup these indexes from filebeat?

How do others configure these settings?

Thanks.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.