Filebeat setup without Kibana?

[davidwhthomas (David Thomas)]

Working on a setup where log data is stored in elasticsearch using filebeat.

However, we don't need Kibana (lovely although it is) for this application. The data is queried separately via an API.

I read: Configure Kibana dashboard loading | Filebeat Reference [7.12] | Elastic

I also read How to run filebeat without Kibana? - #7 by Wahyu however that doesn't appear to fix the issue.

If I specify, in filebeat.yml

setup.dashboards.enabled: false

Then filebeat setup will still try to contact Kibana and error out.

|2021-05-03T09:48:43.520Z|INFO|kibana/client.go:119|Kibana url: http://localhost:5601|
|---|---|---|---|
|2021-05-03T09:48:43.521Z|ERROR|instance/beat.go:971|Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to http://localhost:5601/api/status fails: fail to execute the HTTP GET request: Get http://localhost:5601/api/status: dial tcp 127.0.0.1:5601: connect: connection refused. Response: .|

I also tried replacing the kibana directory with an empty volume mount, but it has the same error.

volumes:
  - './filebeat/filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro'
  - './filebeat/kibana:/usr/share/filebeat/kibana'

Is there a way to disable Kibana loading in filebeat.yml ?
I still need to configure additional ILM options such as:

setup.ilm:
  overwrite: true
  enabled: auto
  rollover_alias: "metrics"
  pattern: "{now/M{yyyy.MM}}"

Maybe there's another way to apply ILM config, without running setup?

[davidwhthomas (David Thomas)]

To answer my own question, it appears it's not necessary to run

filebeat setup

With the ILM config in filebeat.yml it is applied automatically

Seen in the following log:

2021-05-03T11:14:47.155Z  INFO  [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.12.1
2021-05-03T11:14:47.189Z  INFO  [index-management]  idxmgmt/std.go:261  Auto ILM enable success.
2021-05-03T11:14:47.318Z  INFO  [index-management]  idxmgmt/std.go:274  ILM policy successfully loaded.
2021-05-03T11:14:47.318Z  INFO  [index-management]  idxmgmt/std.go:407  Set setup.template.name to '{metrics {now/M{yyyy.MM}}}' as ILM is enabled.
2021-05-03T11:14:47.318Z  INFO  [index-management]  idxmgmt/std.go:412  Set setup.template.pattern to 'metrics-*' as ILM is enabled.
2021-05-03T11:14:47.318Z  INFO  [index-management]  idxmgmt/std.go:446  Set settings.index.lifecycle.rollover_alias in template to {metrics {now/M{yyyy.MM}}} as ILM is enabled.
2021-05-03T11:14:47.319Z  INFO  [index-management]  idxmgmt/std.go:450  Set settings.index.lifecycle.name in template to {filebeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2021-05-03T11:14:48.505Z  INFO  template/load.go:117  Try loading template metrics to Elasticsearch
2021-05-03T11:14:48.871Z  INFO  template/load.go:109  template with name 'metrics' loaded.
2021-05-03T11:14:48.871Z  INFO  [index-management]  idxmgmt/std.go:298  Loaded index template.
2021-05-03T11:14:49.697Z  INFO  [index-management]  idxmgmt/std.go:309  Write alias successfully generated.
2021-05-03T11:14:49.700Z  INFO  [publisher_pipeline_output] pipeline/output.go:151  Connection to backoff(elasticsearch(http://elasticsearch:9200)) established

[Marius_Iversen]

@davidwhthomas A few different ways, just to add for others that might visit this later.

You should comment out the "setup.kibana" part of the filebeat.yml, since if you do not need to setup the dashboards then that part might as well be removed.

For the second part, it is possible to choose what part you want to setup, as referenced on the flag section here: Filebeat command reference | Filebeat Reference [7.12] | Elastic

So you could do filebeat setup --pipelines for example. You do want to apply at least the ILM (if needed) and the ingest pipelines if you are using modules.