Disabling of Index management not possible

[sopraf]

Hey,
I'm currently working on setting up a filebeat system to send logs of my application server to logstash. I'm using multiple inputs to monitor my application logs as well as a nginx proxy. When I start the service or run filebeat setup -e -d '*' filebeat won't start due to no given elasticsearch output. The complete log is here:

Log

2024-03-19T16:45:11.641+0100    INFO    instance/beat.go:686    Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat] Hostfs Path: [/]
2024-03-19T16:45:11.641+0100    DEBUG   [beat]  instance/beat.go:744    Beat metadata path: /var/lib/filebeat/meta.json
2024-03-19T16:45:11.641+0100    INFO    instance/beat.go:694    Beat ID: 3c9d0829-894c-4598-bbfc-d55bed25d27e
2024-03-19T16:45:11.641+0100    INFO    [beat]  instance/beat.go:1040   Beat info       {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "[...]"}}}
2024-03-19T16:45:11.641+0100    INFO    [beat]  instance/beat.go:1049   Build info      {"system_info": {"build": {"commit": "d420ccdaf201e32a524632b5da729522e50257ae", "libbeat": "7.16.3", "time": "2022-01-07T00:36:57.000Z", "version": "7.16.3"}}}
2024-03-19T16:45:11.642+0100    INFO    [beat]  instance/beat.go:1052   Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.17.5"}}}
2024-03-19T16:45:11.642+0100    INFO    [beat]  instance/beat.go:1056   Host info       {"system_info": {"host": [...]}}}
2024-03-19T16:45:11.643+0100    INFO    [beat]  instance/beat.go:1085   Process info    {"system_info": {"process": [...]}}
2024-03-19T16:45:11.643+0100    INFO    instance/beat.go:328    Setup Beat: filebeat; Version: 7.16.3
2024-03-19T16:45:11.643+0100    DEBUG   [beat]  instance/beat.go:354    Initializing output plugins
2024-03-19T16:45:11.643+0100    DEBUG   [publisher]     pipeline/consumer.go:148        start pipeline event consumer
2024-03-19T16:45:11.644+0100    INFO    [publisher]     pipeline/module.go:113  Beat name: app
2024-03-19T16:45:11.644+0100    WARN    beater/filebeat.go:202  Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.
2024-03-19T16:45:11.644+0100    ERROR   instance/beat.go:1015   Exiting: Index management requested but the Elasticsearch output is not configured/enabled
Exiting: Index management requested but the Elasticsearch output is not configured/enabled

I understand the log but I don't see a problem with my configuration here:

filebeat.yml

#=========================== Filebeat inputs via conf.d=============================
filebeat.config.inputs:
  enabled: true
  path: "/etc/filebeat/conf.d/*.yml"

#----------------------------- Logstash logging --------------------------------
logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644

#----------------------------- Logstash output --------------------------------
output.logstash:
  hosts: ["my_logstash_server.com:5044"]

My further inputs are placed in separate .yml files in the conf.d directory.

As I have no index management configured in Kibana or Elasticsearch, I don't understand why filebeat won't start.

Maybe you have an idea.

Thanks in advance

[leandrojmp (Leandro Pereira)]

@sopraf

path: "/etc/filebeat/conf.d/*.yml"

Can you share the inputs you have here?

Are you using any built-in module from filebeat? Like the nginx module or things like that?

[stephenb (Stephen Brown)]

Hi @sopraf Welcome to the community

setup can only be run when the filebeat output is pointing out to elasticsearch.

setup load assets into Kibana and Elasticsearch it does not ship telemetry

Filebeat needs to point to elasticsearch to load the assets.

Then if you want to switch to just filebeat running through logstash. Change the output in the filebeat.yml and start

Also if you intend to use logstash to passthrough filebeat telemetry is suggest looking at this.

Use ingest pipelines for parsing | Logstash Reference [7.16] | Elastic

But this line

index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"

Should be

index => "%{[@metadata][beat]}-%{[@metadata][version]}

[sopraf]

Thank you, once again I've noticed that it's better to read what the command you're running actually does. Instead of debugging my system with setup I should've used filebeat run -e -d "*".

I'll check ingest pipelines out, to debug my system.

Thanks again!