# Handling multiples modules output to multiples indexes, good practice? **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [June 4, 2019, 3:37am UTC](https://discuss.elastic.co/t/handling-multiples-modules-output-to-multiples-indexes-good-practice/184085 "2019-06-04T03:37:38Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![TanguyB](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tanguyb/32/47408_2.png) [@TanguyB](https://discuss.elastic.co/u/TanguyB)\ **Post date:** [June 4, 2019, 3:37am UTC](https://discuss.elastic.co/t/handling-multiples-modules-output-to-multiples-indexes-good-practice/184085/1 "2019-06-04T03:37:38Z")
Hello, In my current deployment, I've many filebeats shipping logs from many sources ( system / audit / mysql modules / docker processor ...). I'd like filebeat to send them in different indexes to ES instead of everything mixed under filebeat-\*.. I saw this post which contain a big part of the solution 😀 [Output to multiple indexes straight from filebeat.yml?](https://discuss.elastic.co/t/output-to-multiple-indexes-straight-from-filebeat-yml/130165/2) : ```auto filebeat.prospectors: - ... fields: type: "system" - ... fields: type: "audit" setup.template.name: "index-%{[beat.version]}" setup.template.pattern: "index-%{[beat.version]}-*" output.elasticsearch: ... index: "index-%{[beat.version]}-%{[fields.type]:other}-%{+yyyy.MM.dd}" ``` As the setup.template.pattern match them all I guess a templates will be created covering all of them ? However, how can I add custom filed like show above : ```auto fields: type: "system" ``` to a module ? (as I use module instead of processor) I couldn't find the answer on the web. Also is the the good way to do this ? I feel like letting everything under a filebeat-%{+yyyy.MM.dd} is a bad idea (I end up with 1300+ fields on the index), but I didn't find a "best practices" section about filebeat --\> multiples indexes too.. Thanks 👍 ! ---
**Author:** ![elastock](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/elastock/32/35672_2.png) [@elastock](https://discuss.elastic.co/u/elastock)\ **Post date:** [June 4, 2019, 8:18am UTC](https://discuss.elastic.co/t/handling-multiples-modules-output-to-multiples-indexes-good-practice/184085/2 "2019-06-04T08:18:26Z")
Hello Tanguy . Im not sure to completely understand your problem There are differents ways to do what you want . If you want to have different index names and , it's simple > ``` > output.elasticsearch: > ... > index: "%{[fields.type]}-putwhatyouwanthere" > > ``` > Also is the the good way to do this ? I feel like letting everything under a filebeat-%{+yyyy.MM.dd} is a bad idea (I end up with 1300+ fields on the index), but I didn't find a "best practices" section about filebeat --\> multiples indexes too.. Letting everything under filebeat-%{+yyyy.MM.dd} is clearly a bad idea if you have thousands of different fields But if you do `filebeat-%{[fields.type]}` you will have many different filebeat index as you have differents fields.type, and you will avoid the heavy monolithic trash index ```` However, how can I add custom filed like show above : ``` fields: type: "system" ``` to a module ? (as I use module instead of processor) I couldn't find the answer on the web. ```` I don't know , you should test you can try to add `- module: thenameofthemodule` and then > ``` > - ... > fields: > type: "audit > > ``` And lets see if it works . ---
**Author:** ![TanguyB](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tanguyb/32/47408_2.png) [@TanguyB](https://discuss.elastic.co/u/TanguyB)\ **Post date:** [June 6, 2019, 3:25am UTC](https://discuss.elastic.co/t/handling-multiples-modules-output-to-multiples-indexes-good-practice/184085/3 "2019-06-06T03:25:37Z")
Hey Thanks for the help, It seems that I can't add a field to a module like that, but instead I can try to filter between modules and processors like that : ```auto indices: - index: "filebeat-%{[agent.version]}-system-%{+yyyy.MM.dd}" when.equals: event.module: "system" - index: "filebeat-%{[agent.version]}-audit-%{+yyyy.MM.dd}" when.equals: event.module: "audit" - index: "filebeat-%{[agent.version]}-docker-%{+yyyy.MM.dd}" ``` As the doc say "Filebeat uses the first matching rule in the array." the log having no module (comming from docker processor should go in the last index) Or maybe I could use a more simpler : ```auto index: "index-%{[agent.version]}-%{[fileset.module]:other}-%{+yyyy.MM.dd}" ``` And every non-module should go into "other" (which limit the sorting if someday I had many processor and less modules) That's the only way I found to mix modules and processors and have multiple output 🧐 I'll try that and update this thread ! ---
**Author:** ![TanguyB](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tanguyb/32/47408_2.png) [@TanguyB](https://discuss.elastic.co/u/TanguyB)\ **Post date:** [June 27, 2019, 7:34am UTC](https://discuss.elastic.co/t/handling-multiples-modules-output-to-multiples-indexes-good-practice/184085/4 "2019-06-27T07:34:28Z")
Update ! It's working with : ``` index: "filebeat-%{[agent.version]}-%{[event.module]:other}-%{+yyyy.MM.dd}" ``` ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/9/d94322f769818373072b802cf924ab409406e326.png) However I had to disable ilm (enabled by default) else it erase this setting and ILM don't work with dynamic index name from FB .......... ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 25, 2019, 7:34am UTC](https://discuss.elastic.co/t/handling-multiples-modules-output-to-multiples-indexes-good-practice/184085/5 "2019-07-25T07:34:37Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.