Setting the number of shards and replicas in /etc/filebeat/filebeat.yml

Elastic Stack Beats
1

According to filebeat documentation (Configure Elasticsearch index template loading | Filebeat Reference [8.1] | Elastic), it should be possible to set the number of shards and replicas for Filebeat indexes with the following settings:

setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 0

I have tried these settings together with

setup.template.overwrite: true

which should set the number of shards to 1 and the number of replicas to 0.

However, when running filebeat-7.16.3 as follows

filebeat -e setup --pipelines --index-management --modules microsoft

in order to set up pipelines and index management in Elasticsearch, the above settings have no effect whatsoever (even after removing the filebeat index templates, pipelines and ilm policy by hand).

Is it a know bug of filebeat?

--
risto

2

What about setting the replica number in template instead?

PUT /<Index / template>/_settings
{
  "index" : {
    "number_of_replicas" : 0
  }
}
3

I know that it can be done, but I am rather interested why setting the number of shards and replicas in the filebeat configuration file does not work, although the documentation advertises it.
risto

4

For that answer you need to share more of the filebeat.yml ...

5

Here is the relevant section from the configuration file:

setup.ilm.overwrite: true

setup.template.overwrite: true

setup.template.settings:
  index.number_of_shards: 2
  index.number_of_replicas: 0

The other parts in the configuration file set up access to Elasticsearch and it works without issues -- after deleting the template, it appears again when 'filebeat setup' gets executed. However, what is not working as expected are the directives for configuring the number of shards and replicas (they will both remain set to 1 which is the default).

6

After running the setup command ?

sudo filebeat setup -e --strict.perms=false  \
  -E output.logstash.enabled=false \
  -E output.elasticsearch.hosts=['https://0.0.0.0:9200'] \
  -E output.elasticsearch.username=elastic \
  -E output.elasticsearch.password=changeme \
  -E setup.kibana.host=https://0.0.0.0:5601
7

I think I have found the reason for this issue. I needed to change the number of replicas from 1 to 0, and in addition to my setup.template.settings entry, the filebeat.yml file contained another entry that looked like this:

setup.template.settings:
  index.number_of_shards: 1

As you can see, It specifies just the number of shards without the number of replicas. Initially, I didn't notice the second instance and even though the number of replicas was not provided there, it appears that the mere appearance of the second setup.template.settings: directive will override the previous number_of_replicas setting with the default (that is, 1). After getting rid of the duplicate block my original command line worked as expected.

risto

8

Usually that is why its better to share your full configuration for eagle eyes to spot these thing.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.