Filebeat, ILM and multiple indices

Elastic Stack Beats
1

Hi all, im having a bit of a hard time understanding the best config for our setup. We are running filebeat to ship several totally different types of logs to elastic that need their own template and index. But at the same time, id like to manage those using ILM. I dont think filebeat really supports that out of the box, is that correct?

So, am I right in understanding that if i want the above, i need to manage ILM policy and templates manually, and then configure each input to ship to the write alias for the ilm policy linked to the index pattern?

1 Like
2

Hi @corbosman, just a quick clarification: will each type of log be going into a separate index (each with it's own template and ILM policy)? Or are all types of logs going into a single (but custom) index?

I think it's the former, in which case you're right — a single Filebeat instance cannot manage multiple indices.

However, if I'm mistaken and it's the latter, you might be able to achieve the desired customization from within Filebeat itself, using the setup.ilm.* and setup.template.* configuration settings.

3

Hi, @shaunak , it's indeed the former. Separate index for each, with their own template and ILM. It's not a big deal, we just manage the index and ILM manually. What kinda messed with my brain at first is that we need to tell filebeat to send to the alias. Took a while for me to get that. But it makes sense.

4

Hi Corbosman,

I am actually trying to do the same: storing multiple index with specifi ilm and template.
Can you tell me want you mean by tell filebeat to send to the alias ?

Thanks,

David

5

If you use ilm to manage your indices, it manages an alias that always points to the currently active index. I didn't quite understand that bit at first. So lets say you have ILM roll over indices like foo-2020-03-01, foo-2020-03-02 etc, then you might have an alias 'foo' that always points to the most recent index. See https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html, the bit about aliases.

6

Thanks @corbosman,

It took me a bit of time too to understand how elastic works with the ilm and aliases. What I don't understand is how to do that part Separate index for each, with their own template and ILM.
As @shaunak wrote a single Filebeat instance cannot manage multiple indices. So if you can only configure filebeat to ship data to a single index name (or write alias in that case) let say foo-* , how you manage to have in the end multiple index : foo-* , bar-* , toto-* , ... with their own template and ilm ?

7

We use the inputs.d folder, and each entry there looks like this:

- type: log
  enabled: true
  paths:
    - /var/log/apache2/*access*
  exclude_files: ['\.gz$']
  ignore_older: "720h"
  pipeline: "apache-access"
  index: "web-%{[agent.version]}"

As you can see, you can tell filebeat which index, and if needed, which ingest pipeline to use.

The ILM and template config you have to manage manually. I used the dev console for that for a bit. But what I ended up doing is a bit of a hack. I created a folder on my mac with multiple filebeat configs, one for each template/ilm setup we needed, and just made a shell script to run filebeat to setup the template and everything.

exec /usr/local/Cellar/filebeat-full/7.6.0/libexec/bin/filebeat --path.config /usr/local/etc/filebeat/apache --path.data /usr/local/var/lib/filebeat --path.home /usr/local/Cellar/filebeat-full/7.6.0/libexec --path.logs /usr/local/var/log/filebeat setup --index-management
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.