# Issue with Multiple Logstash Conf files **URL:** \ **Category:** Logstash\ **Created:** [May 31, 2017, 2:23pm UTC](https://discuss.elastic.co/t/issue-with-multiple-logstash-conf-files/87762 "2017-05-31T14:23:51Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![Sameer\_Panicker](https://avatars.discourse-cdn.com/v4/letter/s/d9b06d/32.png) [@Sameer\_Panicker](https://discuss.elastic.co/u/Sameer_Panicker)\ **Post date:** [May 31, 2017, 2:23pm UTC](https://discuss.elastic.co/t/issue-with-multiple-logstash-conf-files/87762/1 "2017-05-31T14:23:51Z")
I am facing a problem when multiple logstash conf are used within a folder and trying to connect the same using filebeat. I had posted this issue under filebeat section and now I am asked to post it under logstash as well. Can anyone please let me know how to resolve this issue using multiple conf files ... > [@Issue with Filebeat and Multiple Logstash Conf files](https://discuss.elastic.co/t/issue-with-filebeat-and-multiple-logstash-conf-files/87301): > > I have 2 Logstash.conf files under Configurations folder. Logstash is configured to run using logstash -f \Configurations command. 2 conf files are nothing but IIS.conf and Services.conf Everything worked smoothly using logstash to ES. Now, I have FB to send data from multiple paths to logstash. No change to logstash command line. So, when trigger my services...i can see logs getting captured for IIS only and not for my services. Here is my FB configuration - - input\_type: log paths: … ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 1, 2017, 5:25am UTC](https://discuss.elastic.co/t/issue-with-multiple-logstash-conf-files/87762/2 "2017-06-01T05:25:35Z")
You can't have more than one Beats input plugin listening on the same port in more than one configuration file. Logstash concatenates the contents of all configuration files and keeping them separate is meant as an organizational aid for you. Logstash doesn't care about how you divide the input, output, and filter blocks into different files. ---
**Author:** ![Sameer\_Panicker](https://avatars.discourse-cdn.com/v4/letter/s/d9b06d/32.png) [@Sameer\_Panicker](https://discuss.elastic.co/u/Sameer_Panicker)\ **Post date:** [June 1, 2017, 5:54pm UTC](https://discuss.elastic.co/t/issue-with-multiple-logstash-conf-files/87762/3 "2017-06-01T17:54:58Z")
That means if I have multiple beats port configured under multiple logstash conf files, things should work ? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 1, 2017, 8:03pm UTC](https://discuss.elastic.co/t/issue-with-multiple-logstash-conf-files/87762/4 "2017-06-01T20:03:36Z")
Yes, but why would you want to have multiple beats listeners? ---
**Author:** ![Sameer\_Panicker](https://avatars.discourse-cdn.com/v4/letter/s/d9b06d/32.png) [@Sameer\_Panicker](https://discuss.elastic.co/u/Sameer_Panicker)\ **Post date:** [June 1, 2017, 9:32pm UTC](https://discuss.elastic.co/t/issue-with-multiple-logstash-conf-files/87762/5 "2017-06-01T21:32:47Z")
I would like to have multiple conf files for multiple file path. Because I have multiple patterns and easy to maintain a file with few lines. Easy to understand and edit later. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 2, 2017, 5:11am UTC](https://discuss.elastic.co/t/issue-with-multiple-logstash-conf-files/87762/6 "2017-06-02T05:11:31Z")
Sure, but you can still have multiple configuration files for your filters even if you have a single beats input (that you can put in a file of its own if you like). ---
**Author:** ![Sameer\_Panicker](https://avatars.discourse-cdn.com/v4/letter/s/d9b06d/32.png) [@Sameer\_Panicker](https://discuss.elastic.co/u/Sameer_Panicker)\ **Post date:** [June 2, 2017, 2:27pm UTC](https://discuss.elastic.co/t/issue-with-multiple-logstash-conf-files/87762/7 "2017-06-02T14:27:34Z")
But in that case only my alphabetically ordered 1st conf is capturing logs.The other ones are ignored or it is not capturing anything. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 3, 2017, 3:04pm UTC](https://discuss.elastic.co/t/issue-with-multiple-logstash-conf-files/87762/8 "2017-06-03T15:04:08Z")
So have you removed one of the beats inputs then? What do your configuration files contain now? ---
**Author:** ![Sameer\_Panicker](https://avatars.discourse-cdn.com/v4/letter/s/d9b06d/32.png) [@Sameer\_Panicker](https://discuss.elastic.co/u/Sameer_Panicker)\ **Post date:** [June 3, 2017, 3:41pm UTC](https://discuss.elastic.co/t/issue-with-multiple-logstash-conf-files/87762/9 "2017-06-03T15:41:20Z")
You mean a single prospector in filebeat ? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 4, 2017, 1:54pm UTC](https://discuss.elastic.co/t/issue-with-multiple-logstash-conf-files/87762/10 "2017-06-04T13:54:48Z")
No, I mean that you should have a single beats input plugin on the Logstash side. The number of prospectors doesn't matter. ---
**Author:** ![Sameer\_Panicker](https://avatars.discourse-cdn.com/v4/letter/s/d9b06d/32.png) [@Sameer\_Panicker](https://discuss.elastic.co/u/Sameer_Panicker)\ **Post date:** [June 5, 2017, 8:56pm UTC](https://discuss.elastic.co/t/issue-with-multiple-logstash-conf-files/87762/11 "2017-06-05T20:56:28Z")
I have only 1 beat port configure. Here is my FB conf files. - - input\_type: log paths: - D:\ServiceLogs\Zephyr\*\* - D:\Zephyr\inetpub\LogFiles\*\* tags: ["Zephyr"] ignore\_older: 1h - input\_type: log paths: - D:\ServiceLogs\Nightingale\*\* - D:\Nightingale\inetpub\LogFiles\*\* tags: ["IVR"] ignore\_older: 1h - input\_type: log paths: - D:\FCServices\inetpub\LogFiles\*\* tags: ["FCServices"] ignore\_older: 1h - input\_type: log paths: - D:\BreezeServices\inetpub\LogFiles\*\* tags: ["BreezeServices"] ignore\_older: 1h Logstash 1 conf - input { beats { port =\> 5044 } } filter { kv { value\_split =\> ":" remove\_char\_key =\> "[]" remove\_char\_value =\> "[]" include\_keys =\> ["method", "reasonPhrase", "requestUri", "content", "Payload", "id", "ClientID"] recursive =\> "true" } ``` if "Zephyr" in [tags] { mutate { replace => { "type" => "Zephyr" } } } if "IVR" in [tags] { mutate { replace => { "type" => "IVR" } } } mutate { add_field => { "LogType" => "Services" } remove_field => ["tags", "offset", "input_type", "beat"] } ``` } output { if [type] == "Zephyr" { elasticsearch { index =\> "zephyr-%{+YYYY.MM.dd}" hosts =\> ["server:9200"] } } ``` if [type] == "IVR" { elasticsearch { index => "ivr-%{+YYYY.MM.dd}" hosts => ["server:9200"] } } #stdout #{ # codec => rubydebug #} ``` } Logstash 2 conf - input { beats { port =\> 5044 } } filter { #Ignore log comments if [message] =~ "^#" { drop {} } ``` grok { match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{IPORHOST:site} %{WORD:HttpVerb} %{URIPATH:RequestUri} %{NOTSPACE:querystring} %{NUMBER:Port} %{NOTSPACE:username} %{IPORHOST:clienthost} %{NOTSPACE:useragent} %{NOTSPACE:username} %{NUMBER:ResponseCode} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:time_taken}" } } #Set the Event Timestamp from the log date { match => ["log_timestamp", "YYYY-MM-dd HH:mm:ss"] timezone => "Etc/UTC" } #Ignore all values for which GROK pattern is not set at this moment. if "_grokparsefailure" in [tags] { drop {} } if "Zephyr" in [tags] { mutate { replace => { "type" => "Zephyr" } } } if "IVR" in [tags] { mutate { replace => { "type" => "IVR" } } } if "FCServices" in [tags] { mutate { replace => { "type" => "FCServices" } } } if "BreezeServices" in [tags] { mutate { replace => { "type" => "BreezeServices" } } } mutate { add_field => { "LogType" => "IIS" } remove_field => ["@version", "log_timestamp", "site", "querystring", "username", "clienthost", "useragent", "subresponse", "scstatus", "tags", "offset", "input_type", "beat"] } ``` } output { if [type] == "Zephyr" { elasticsearch { index =\> "zephyr-%{+YYYY.MM.dd}" hosts =\> ["server:9200"] } } ``` if [type] == "IVR" { elasticsearch { index => "ivr-%{+YYYY.MM.dd}" hosts => ["server:9200"] } } if [type] == "FCServices" { elasticsearch { index => "fcservices-%{+YYYY.MM.dd}" hosts => ["server:9200"] } } if [type] == "BreezeServices" { elasticsearch { index => "breezeservices-%{+YYYY.MM.dd}" hosts => ["server:9200"] } } #stdout #{ # codec => rubydebug #} ``` } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 7, 2017, 5:57am UTC](https://discuss.elastic.co/t/issue-with-multiple-logstash-conf-files/87762/12 "2017-06-07T05:57:02Z")
> I have only 1 beat port configure. No, you have one in each configuration file. You can only have one **in total** (listening on the same port). ---
**Author:** ![Sameer\_Panicker](https://avatars.discourse-cdn.com/v4/letter/s/d9b06d/32.png) [@Sameer\_Panicker](https://discuss.elastic.co/u/Sameer_Panicker)\ **Post date:** [June 7, 2017, 2:49pm UTC](https://discuss.elastic.co/t/issue-with-multiple-logstash-conf-files/87762/13 "2017-06-07T14:49:47Z")
So in my case what would be the ideal solution that you can suggest. I wanted to keep these conf sepearte for future maintenance and easy understanding. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 7, 2017, 6:16pm UTC](https://discuss.elastic.co/t/issue-with-multiple-logstash-conf-files/87762/14 "2017-06-07T18:16:29Z")
You can organize your configuration in any way you want as long as you make sure you don't have more than one beats input listening on the same port. You could e.g. have one configuration file containing all inputs, multiple files containing filters for various purposes, and one file with output plugins. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 6:16pm UTC](https://discuss.elastic.co/t/issue-with-multiple-logstash-conf-files/87762/15 "2017-07-05T18:16:32Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.