Handling multiples modules output to multiples indexes, good practice?

[TanguyB]

Hello,

In my current deployment, I've many filebeats shipping logs from many sources ( system / audit / mysql modules / docker processor ...).

I'd like filebeat to send them in different indexes to ES instead of everything mixed under filebeat-*..

I saw this post which contain a big part of the solution
Output to multiple indexes straight from filebeat.yml? :

filebeat.prospectors:
- ...
  fields:
    type: "system"
- ...
  fields:
    type: "audit"


setup.template.name: "index-%{[beat.version]}"
setup.template.pattern: "index-%{[beat.version]}-*"


output.elasticsearch:
  ...
  index: "index-%{[beat.version]}-%{[fields.type]:other}-%{+yyyy.MM.dd}"

As the setup.template.pattern match them all I guess a templates will be created covering all of them ?

However, how can I add custom filed like show above :

  fields:
    type: "system"

to a module ? (as I use module instead of processor) I couldn't find the answer on the web.

Also is the the good way to do this ? I feel like letting everything under a filebeat-%{+yyyy.MM.dd} is a bad idea (I end up with 1300+ fields on the index), but I didn't find a "best practices" section about filebeat --> multiples indexes too..

Thanks !

[elastock (Guillaume Dufrenne)]

Hello Tanguy .
Im not sure to completely understand your problem
There are differents ways to do what you want .

If you want to have different index names and , it's simple

output.elasticsearch:
  ...
  index: "%{[fields.type]}-putwhatyouwanthere"

Also is the the good way to do this ? I feel like letting everything under a filebeat-%{+yyyy.MM.dd} is a bad idea (I end up with 1300+ fields on the index), but I didn't find a "best practices" section about filebeat --> multiples indexes too..

Letting everything under filebeat-%{+yyyy.MM.dd} is clearly a bad idea if you have thousands of different fields
But if you do filebeat-%{[fields.type]}
you will have many different filebeat index as you have differents fields.type, and you will avoid the heavy monolithic trash index

However, how can I add custom filed like show above :

```
  fields:
    type: "system"
```

to a module ? (as I use module instead of processor) I couldn't find the answer on the web.

I don't know , you should test you can try to add
- module: thenameofthemodule
and then

- ...
  fields:
    type: "audit

And lets see if it works .

[TanguyB]

Hey

Thanks for the help,

It seems that I can't add a field to a module like that, but instead I can try to filter between modules and processors like that :

  indices:
    - index: "filebeat-%{[agent.version]}-system-%{+yyyy.MM.dd}"
      when.equals:
        event.module: "system"
    - index: "filebeat-%{[agent.version]}-audit-%{+yyyy.MM.dd}"
      when.equals:
        event.module: "audit"
    - index: "filebeat-%{[agent.version]}-docker-%{+yyyy.MM.dd}"

As the doc say "Filebeat uses the first matching rule in the array." the log having no module (comming from docker processor should go in the last index)

Or maybe I could use a more simpler :

  index: "index-%{[agent.version]}-%{[fileset.module]:other}-%{+yyyy.MM.dd}"

And every non-module should go into "other" (which limit the sorting if someday I had many processor and less modules)

That's the only way I found to mix modules and processors and have multiple output

I'll try that and update this thread !

[TanguyB]

Update !

It's working with :

  index: "filebeat-%{[agent.version]}-%{[event.module]:other}-%{+yyyy.MM.dd}"

However I had to disable ilm (enabled by default) else it erase this setting and ILM don't work with dynamic index name from FB ..........