When do you run the filebeat setup command?

Elastic Stack Beats
1

This is a general question that is probably more obvious to others than to me, sorry in advance.

Question: When do you run the filebeat setup command under the below scenario's? I know based on this great answer (thanks again @Marius_Iversen), and from the docs that it should be run once before starting up filebeat for the first time or after upgrading. But what about when:

  1. If I have 5 different application servers pushing logs to a shared ES/Kibana instance - should it be run on each server when installing filebeat or only the first server? What if 1 of these servers has different log modules activated (4 are apache and 1 has nginx)?
  2. if I update the filebeat.yml on a server or enable a new log module, should I re-run the startup or just restart filebeat?
2

I think the best way to answer the question is to explain what setup actually does:
When you run filebeat setup it will:

  1. If its the first time you run setup ever (against Kibana and Elasticsearch that is), it will create the filebeat-* index pattern, alias, index mapping and setup ILM.
  2. It will look at what modules you have enabled, and then install the relevant Ingest pipelines.
  3. It will look at what modules you have enabled, and then install the relevant dashboards.

As you can see, the setup command doesn't actually do anything on the local machine, it simply prepares your Elasticsearch (and kibana if any dashboards are included) to ingest and store the data.

Based on that, you simply want to run setup whenever you want to change something on ES/Kibana, for example if you upgrade, you enable a new module, or perform any changes on ingest pipelines yourself.

You could even run filebeat setup on a machine that isn't actually going to run filebeat in the end, but it could simply prepare your cluster with all the setup it needs, before deploying filebeat through something like automation.

So to answer your examples, you would only need to rerun setup from someplace if you update your list of modules, but not if you simply update filebeat.yml.

It is also good to mention that it often (though not always), will check if certain things exist the first time it starts up, but this is not always accurate, so running setup is usually the best way to go.

If you want to only install certain aspects of a module, for example if you are not interested in a dashboard, or your filebeat can only reach Elasticsearch and not Kibana, you can add certain options to only install what you want, and all of the commands are documented here:

1 Like
3

Recent versions of filebeat can run setup automatically, when it's needed, but it's not a trivial adventure getting it working properly. I wouldn't allow it on a "client" system because the account used needs permission to load the templates and bootstrap the ILM index but I use it where I also own the filebeat install.

2 Likes
4

That is also a great addition, running the setup procedure from somewhere else is common as your deployed filebeat usually runs with a limited user (that is only allowed to write and not read/delete data, or create anything else like dashboards or modify ILM).

5

This is all awesome additional information, thank you both for it!

So in theory I could run the setup command as often as I like. In my case it doesn't seem necessary as it's not preparing the host server but instead preparing ES/Kibana which is already up and configured. But it sounds like running it wont cause any adverse side-effects?

For the idea of running the setup centrally (which sounds awesome), does this mean you'd install filebeat on the central machine, run setup and potentially remove filebeat? And if this central machine didn't have any of the specified log locations (e.g. /var/log/nginx/*.log) it wouldn't matter as again, it's actually not doing anything for the local machine but preparing ES/Kibana based on the configurations?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.