# Best practice to handle logstash configuratioon **URL:** \ **Category:** Logstash\ **Created:** [June 14, 2018, 6:14pm UTC](https://discuss.elastic.co/t/best-practice-to-handle-logstash-configuratioon/135977 "2018-06-14T18:14:55Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![tag\_v](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tag_v/32/47466_2.png) [@tag\_v](https://discuss.elastic.co/u/tag_v)\ **Post date:** [June 14, 2018, 6:14pm UTC](https://discuss.elastic.co/t/best-practice-to-handle-logstash-configuratioon/135977/1 "2018-06-14T18:14:55Z")
Hello All, My logstash configuration have almost 1500+ lines of code where i am taking input from beats and applying filters for extracting fields of various log types and output to kafka. Is there any way to handle huge configuration files or any best practices for logstash parsers? Thanks in advance. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 14, 2018, 6:58pm UTC](https://discuss.elastic.co/t/best-practice-to-handle-logstash-configuratioon/135977/2 "2018-06-14T18:58:48Z")
I'd split that file into multiple files, but apart from that it's hard to give advice without seeing what it looks like. I'd also strongly suggest using my tool [Logstash Filter Verifier](https://github.com/magnusbaeck/logstash-filter-verifier) to test your filter configurations. ---
**Author:** ![tag\_v](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tag_v/32/47466_2.png) [@tag\_v](https://discuss.elastic.co/u/tag_v)\ **Post date:** [June 15, 2018, 1:49am UTC](https://discuss.elastic.co/t/best-practice-to-handle-logstash-configuratioon/135977/3 "2018-06-15T01:49:58Z")
> [@magnusbaeck](#): > > I'd split that file into multiple files, Does this means splitting 1 large conf to many confs and run multiple logstash instances? > [@magnusbaeck](#): > > [Logstash Filter Verifier](https://github.com/magnusbaeck/logstash-filter-verifier) This is awsome i will try this tool. Thanks in advance. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 15, 2018, 5:59am UTC](https://discuss.elastic.co/t/best-practice-to-handle-logstash-configuratioon/135977/4 "2018-06-15T05:59:53Z")
> Does this means splitting 1 large conf to many confs and run multiple logstash instances? That's one possibility, but I was thinking of splitting into multiple files and running a single instance. ---
**Author:** ![tag\_v](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tag_v/32/47466_2.png) [@tag\_v](https://discuss.elastic.co/u/tag_v)\ **Post date:** [June 15, 2018, 6:53am UTC](https://discuss.elastic.co/t/best-practice-to-handle-logstash-configuratioon/135977/5 "2018-06-15T06:53:08Z")
> [@magnusbaeck](#): > > but apart from that it's hard to give advice without seeing what it looks like. Heres the filter part of my conf. Please suggest on this. Thanks in advance. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 18, 2018, 6:13am UTC](https://discuss.elastic.co/t/best-practice-to-handle-logstash-configuratioon/135977/6 "2018-06-18T06:13:31Z")
I obviously haven't studied all 2000 lines in any detail, but a few things stand out: - You're overusing DATA and GREEDYDATA in your grok expression. That's very inefficient and can lead to incorrect matches. - Having conditionals that inspect the exact source filename (`if [source] == "/archives/log/10.90.250.151.log"` etc) is probably not a great idea. ---
**Author:** ![tag\_v](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tag_v/32/47466_2.png) [@tag\_v](https://discuss.elastic.co/u/tag_v)\ **Post date:** [June 19, 2018, 7:21am UTC](https://discuss.elastic.co/t/best-practice-to-handle-logstash-configuratioon/135977/7 "2018-06-19T07:21:24Z")
can u please suggest any best practices for maintaining logstash parser confs where we have custom log patterns (other than usual standard log patterns provided by vendors). ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 19, 2018, 8:36am UTC](https://discuss.elastic.co/t/best-practice-to-handle-logstash-configuratioon/135977/8 "2018-06-19T08:36:30Z")
I don't think I have any particular suggestions. In what way is your current situation unmanageable? ---
**Author:** ![tag\_v](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tag_v/32/47466_2.png) [@tag\_v](https://discuss.elastic.co/u/tag_v)\ **Post date:** [June 19, 2018, 8:48am UTC](https://discuss.elastic.co/t/best-practice-to-handle-logstash-configuratioon/135977/9 "2018-06-19T08:48:49Z")
with number of conf lines increased, we were facing latency for logs getting parsed. How to efficiently handle in this scenario? By writing multiple confs will improve performance? If so how to do this? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 19, 2018, 9:51am UTC](https://discuss.elastic.co/t/best-practice-to-handle-logstash-configuratioon/135977/10 "2018-06-19T09:51:17Z")
Have you looked into using the Logstash monitoring API to see which filters are adding the most time to the processing? But as I said, reduce the amount of DATA and GREEDYDATA patterns. Don't spend time on other optimizations until your grok expressions have been cleaned up. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 17, 2018, 9:51am UTC](https://discuss.elastic.co/t/best-practice-to-handle-logstash-configuratioon/135977/11 "2018-07-17T09:51:22Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.