Filebeat V7.4.1 output.elasticsearch not writing to defined index "logstash"

Elastic Stack Beats
1

Hello

The goal is to write the filebeat-Logs via the output.elasticsearch into the logstash index.
We have done the following entry in the filebeat.yml

setup.template:
name: "logstash"
pattern: "logstash-*"

output.elasticsearch:
hosts: [":9200" ]
pipeline: "filebeat-7.4.0-%{[fields.pipelinect]}"
index: "logstash-%{[agent.version]}-%{+yyyy.MM.dd}"

After restarting the filebeat service we receive the logs not in the logstash-index. We still receive them in the filebeat-7.4.0-Index.

What do we wrong in the configuration.
Thank for tipps
Ritzo

2

Hi could you share your config files

3

Hi BM

Thanks for your response. I copied the filebeat.yml into this post

filebeat.inputs:

  • type: log
    enabled: true
    paths:

    • /var/log/apache2/*access.log
      tags:
    • "apache.access.logs"
      fields:
      logct: "apache.access.logs"
      envct: "Prod"
      appct: "webapache"
      pipelinect: "apache-access-default"

  • type: log
    enabled: true
    paths:

    • /var/log/apache2/*error.log
      tags:
    • "apache.error.logs"
      fields:
      logct: "apache.error.logs"
      envct: "Produktion"
      appct: "webapache"
      pipelinect: "apache-error-pipeline"

filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false

setup.template.settings:
index.number_of_shards: 1

setup.template:
name: "logstash"
pattern: "logstash-*"

output.elasticsearch:
hosts: ["es.weblocal:9200" ]
pipeline: "filebeat-7.4.1-%{[fields.pipelinect]}"
index: "logstash-%{[agent.version]}-%{+yyyy.MM.dd}"

processors:

  • add_host_metadata: ~

logging.selectors: ["*"]
logging.level: info

logging.to_files: true
logging.files:
path: /var/log/filebeat74
name: filebeat_debug
keepfiles: 2
permissions: 0644

xpack.monitoring.enabled: false

We also enabled the following modules
filebeat modules enable apache

and set up the pipelines for them:

filebeat setup --pipelines # result: Loaded Ingest pipelines
filebeat setup --pipelines --modules system,apache # result: Loaded Ingest pipelines
filebeat modules list # Enabled: apache

check config

filebeat test config # config OK
filebeat test output # parse url OK / connection OK,..... / talk to server OK / version 7.4.1

If you need further infos. Let me know
Ritzo

4

In your configuration you are not enabling ILM, but it is activated by default if your Elasticsearch cluster supports it.

Index Lifecycle Management by default creates its own index name.

Try to disable it first: setup.ilm.enabled: false
https://www.elastic.co/guide/en/beats/filebeat/current/ilm.html

To confirm that it is what's happening, please check your logs for messages from index-management

5

Cool. It works with this option as excpeted

setup.ilm.enabled: false

Thanks a lot

6

Happy to help :smiley:

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.