Filebeat configuration to create different elasticsearch index

Elastic Stack Beats
1

Hi,

We are trying to configure Filebeat to send logs to Elasticsearch. We want to create different index than the default one. Here is the configuration

- input_type: log
  paths:
     - /var/log/application1/*.err
     - /var/log/application1/*.log
  tags: "kapacitor-logs"
  document_type: log
  tail_files: true
  
  output.elasticsearch:
    hosts: ["localhost:9200"]
    protocol: "https"
    output.elasticsearch.index: "application1-%{+yyyy.MM.dd}"
    setup.template.name: "application1"
    setup.template.pattern: "application1-*"
    setup.dashboards.index: "application1-*"

We found logs in filebeat-6.2.4-2018.04.18 always. How can we send logs to application1-6.2.4-2018.04.18 index of Elasticsearch.

Thanks
Ferdous Shibly

2

I have the same issue as well, i tried to send the logs using filebeats to logstash and have a grok filter to create the index but not successful.

I read that we can do it by document_type but seems thats is deprecated.

filebeat.prospectors:

  • type: log
    paths:
    - /var/log/messages
    fields:
    document_type: vlm

  • type: log
    enabled: true
    paths:

    • /var/log/ocphc*.log
      fields:
      document_type: ocphc
3

Hi,

could you please try with below format.

output.elasticsearch:
hosts: ["http://localhost:9200"]
index: "application1-%{+yyyy.MM.dd}"

Thanks,
Harsh

1 Like
4

As @harshbajaj16 mentioned it looks like the config option is wrong + the indentation. Here are the docs: https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html

2 Likes
5

@rufin and @harshbajaj16 I have updated my configuration accordingly but when I restart filebeat, it is giving me the following error message.

2018-04-19T10:22:37.587-0400	ERROR	instance/beat.go:667	Exiting: setup.template.name and setup.template.pattern have to be set if index name is modified.
Exiting: setup.template.name and setup.template.pattern have to be set if index name is modified.

Here is my updated configuration

output.elasticsearch:
  hosts: ["localhost:9200"]
  protocol: "https"
  index: "application1-%{+yyyy.MM.dd}"
6

Hi,

when you set index name Beats force you to also set template and pattern. However, If you don't want to set this just put below line in your config file.

setup.template.enabled: false

Thanks,
Harsh

2 Likes
7

setup.template.enabled: false worked, thanks

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.