# How to optimize the multiple configuration files to improve the performance for too many filters **URL:** \ **Category:** Logstash\ **Created:** [June 20, 2018, 3:06pm UTC](https://discuss.elastic.co/t/how-to-optimize-the-multiple-configuration-files-to-improve-the-performance-for-too-many-filters/136723 "2018-06-20T15:06:58Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![hyzhang6639](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hyzhang6639/32/32515_2.png) [@hyzhang6639](https://discuss.elastic.co/u/hyzhang6639)\ **Post date:** [June 20, 2018, 3:06pm UTC](https://discuss.elastic.co/t/how-to-optimize-the-multiple-configuration-files-to-improve-the-performance-for-too-many-filters/136723/1 "2018-06-20T15:06:58Z")
We have several configuration files include about 50 filters. It take about 9 mins to filter 10,000,000 events. After reducing to 30 filters, it take around 6 mins to complete to filter 10,000,000 events. In the future, we need to increase more filter and concerned the performance if too many filters are used. We thought to develop the new filter plugin to improve the performance. It appears to be time-consuming to develop a new filter plugin. We're wondering if some other solutions can improve the performance if many filters(100+) are used. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 20, 2018, 3:21pm UTC](https://discuss.elastic.co/t/how-to-optimize-the-multiple-configuration-files-to-improve-the-performance-for-too-many-filters/136723/2 "2018-06-20T15:21:20Z")
Are you able to share your config? ---
**Author:** ![hyzhang6639](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hyzhang6639/32/32515_2.png) [@hyzhang6639](https://discuss.elastic.co/u/hyzhang6639)\ **Post date:** [June 20, 2018, 3:47pm UTC](https://discuss.elastic.co/t/how-to-optimize-the-multiple-configuration-files-to-improve-the-performance-for-too-many-filters/136723/3 "2018-06-20T15:47:45Z")
Thanks for your updates. The below is the example for the config. The configuration file includes some "if...else if" statement. if [Type1] == "..." { if [message] { csv { columns =\> ["..."] autogenerate\_column\_names =\> false } ``` if [Type2] == "..." { csv { columns => [ ... ]} else if [Type2]=="..."{ csv { columns => [ ... ]} .... ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 20, 2018, 3:50pm UTC](https://discuss.elastic.co/t/how-to-optimize-the-multiple-configuration-files-to-improve-the-performance-for-too-many-filters/136723/4 "2018-06-20T15:50:08Z")
Is it all a lot of conditional around different csv formats? Any other type of processing that could be slow? What inputs do you have? How many columns do the csv files typically have? ---
**Author:** ![hyzhang6639](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hyzhang6639/32/32515_2.png) [@hyzhang6639](https://discuss.elastic.co/u/hyzhang6639)\ **Post date:** [June 20, 2018, 3:56pm UTC](https://discuss.elastic.co/t/how-to-optimize-the-multiple-configuration-files-to-improve-the-performance-for-too-many-filters/136723/5 "2018-06-20T15:56:03Z")
Yes. The conditional is all around csv formats. The filter are mostly used csv and the configuration options are mostly use "columns" and "convert" for csv filter. The event source is the log file for the OS and some applications. ---
**Author:** ![hyzhang6639](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hyzhang6639/32/32515_2.png) [@hyzhang6639](https://discuss.elastic.co/u/hyzhang6639)\ **Post date:** [June 20, 2018, 11:09pm UTC](https://discuss.elastic.co/t/how-to-optimize-the-multiple-configuration-files-to-improve-the-performance-for-too-many-filters/136723/6 "2018-06-20T23:09:39Z")
In addition typically there are around 50+ columns in the csv event source. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 21, 2018, 5:40am UTC](https://discuss.elastic.co/t/how-to-optimize-the-multiple-configuration-files-to-improve-the-performance-for-too-many-filters/136723/7 "2018-06-21T05:40:24Z")
You may want to look into using the[dissect filter](https://www.elastic.co/guide/en/logstash/6.3/plugins-filters-dissect.html) instead of the csv filter as it [can be considerably faster](https://www.elastic.co/blog/logstash-dude-wheres-my-chainsaw-i-need-to-dissect-my-logs). ---
**Author:** ![hyzhang6639](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hyzhang6639/32/32515_2.png) [@hyzhang6639](https://discuss.elastic.co/u/hyzhang6639)\ **Post date:** [June 22, 2018, 1:37am UTC](https://discuss.elastic.co/t/how-to-optimize-the-multiple-configuration-files-to-improve-the-performance-for-too-many-filters/136723/8 "2018-06-22T01:37:30Z")
Hi Christian Thanks for your updates. I will test if filter dissect can improve the performance in our scenario. In addition I also found another symptom. If the lowercase option is commented in "mutate" filter as the following, the transfer rate will be increased from 24KiB/s to 30KiB/s. Is it working as designed? Whether some solution can replace "lowercase" in "mutate" filter to improve the performance? #lowercase =\> ["host", "Column1"] ---
**Author:** ![hyzhang6639](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hyzhang6639/32/32515_2.png) [@hyzhang6639](https://discuss.elastic.co/u/hyzhang6639)\ **Post date:** [June 22, 2018, 3:38am UTC](https://discuss.elastic.co/t/how-to-optimize-the-multiple-configuration-files-to-improve-the-performance-for-too-many-filters/136723/9 "2018-06-22T03:38:14Z")
In addition I tried the similar scenario in another box, there is no obvious difference between enabling lowercase and disabling lowercase option as the following. I am wondering which parameter or setting can trigger the difference for the option lowercase? 9.54MiB 0:05:44 [28.4kiB/s] (Enable lowercase) 9.54MiB 0:05:35 [29.1kiB/s] (Disable lowercase) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 20, 2018, 3:38am UTC](https://discuss.elastic.co/t/how-to-optimize-the-multiple-configuration-files-to-improve-the-performance-for-too-many-filters/136723/10 "2018-07-20T03:38:15Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.