Working on a setup where log data is stored in elasticsearch using filebeat.
However, we don't need Kibana (lovely although it is) for this application. The data is queried separately via an API.
I read: Configure Kibana dashboard loading | Filebeat Reference [7.12] | Elastic
I also read How to run filebeat without Kibana? - #7 by Wahyu however that doesn't appear to fix the issue.
If I specify, in filebeat.yml
setup.dashboards.enabled: false
Then filebeat setup will still try to contact Kibana and error out.
|2021-05-03T09:48:43.520Z|INFO|kibana/client.go:119|Kibana url: http://localhost:5601|
|---|---|---|---|
|2021-05-03T09:48:43.521Z|ERROR|instance/beat.go:971|Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to http://localhost:5601/api/status fails: fail to execute the HTTP GET request: Get http://localhost:5601/api/status: dial tcp 127.0.0.1:5601: connect: connection refused. Response: .|
I also tried replacing the kibana directory with an empty volume mount, but it has the same error.
volumes:
- './filebeat/filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro'
- './filebeat/kibana:/usr/share/filebeat/kibana'
Is there a way to disable Kibana loading in filebeat.yml ?
I still need to configure additional ILM options such as:
setup.ilm:
overwrite: true
enabled: auto
rollover_alias: "metrics"
pattern: "{now/M{yyyy.MM}}"
Maybe there's another way to apply ILM config, without running setup?
To answer my own question, it appears it's not necessary to run
filebeat setup
With the ILM config in filebeat.yml it is applied automatically
Seen in the following log:
2021-05-03T11:14:47.155Z INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.12.1
2021-05-03T11:14:47.189Z INFO [index-management] idxmgmt/std.go:261 Auto ILM enable success.
2021-05-03T11:14:47.318Z INFO [index-management] idxmgmt/std.go:274 ILM policy successfully loaded.
2021-05-03T11:14:47.318Z INFO [index-management] idxmgmt/std.go:407 Set setup.template.name to '{metrics {now/M{yyyy.MM}}}' as ILM is enabled.
2021-05-03T11:14:47.318Z INFO [index-management] idxmgmt/std.go:412 Set setup.template.pattern to 'metrics-*' as ILM is enabled.
2021-05-03T11:14:47.318Z INFO [index-management] idxmgmt/std.go:446 Set settings.index.lifecycle.rollover_alias in template to {metrics {now/M{yyyy.MM}}} as ILM is enabled.
2021-05-03T11:14:47.319Z INFO [index-management] idxmgmt/std.go:450 Set settings.index.lifecycle.name in template to {filebeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2021-05-03T11:14:48.505Z INFO template/load.go:117 Try loading template metrics to Elasticsearch
2021-05-03T11:14:48.871Z INFO template/load.go:109 template with name 'metrics' loaded.
2021-05-03T11:14:48.871Z INFO [index-management] idxmgmt/std.go:298 Loaded index template.
2021-05-03T11:14:49.697Z INFO [index-management] idxmgmt/std.go:309 Write alias successfully generated.
2021-05-03T11:14:49.700Z INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(elasticsearch(http://elasticsearch:9200)) established
@davidwhthomas A few different ways, just to add for others that might visit this later.
You should comment out the "setup.kibana" part of the filebeat.yml, since if you do not need to setup the dashboards then that part might as well be removed.
For the second part, it is possible to choose what part you want to setup, as referenced on the flag section here: Filebeat command reference | Filebeat Reference [7.12] | Elastic
So you could do filebeat setup --pipelines for example. You do want to apply at least the ILM (if needed) and the ingest pipelines if you are using modules.