# Logstash index template **URL:** \ **Category:** Logstash\ **Created:** [November 30, 2017, 10:34am UTC](https://discuss.elastic.co/t/logstash-index-template/109732 "2017-11-30T10:34:11Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![pavuk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pavuk/32/28914_2.png) [@pavuk](https://discuss.elastic.co/u/pavuk)\ **Post date:** [November 30, 2017, 10:34am UTC](https://discuss.elastic.co/t/logstash-index-template/109732/1 "2017-11-30T10:34:11Z")
Sorry for stupid question, it's my first Logstash experience, but I would like to ask about following issue. I have two LS configs: ``` # cat /etc/logstash/conf.d/nginx.conf input { file { type => "nginx" start_position => "beginning" path => ["/var/log/nginx/*.log"] exclude => ["*.gz"] } } filter { grok { # type => "nginx-access" match => [ "message", "%{IPORHOST:http_host} %{IPORHOST:client_ip} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:http_verb} %{NOTSPACE:http_request}(?: HTTP/%{NUMBER:http_version})?|%{DATA:raw_http_request})\" %{NUMBER:http_status_code} (?:%{NUMBER:bytes_read}|-) %{QS:referrer} %{QS:agent} %{NUMBER:time_duration:float} %{NUMBER:time_backend_response:float}", "message", "%{IPORHOST:http_host} %{IPORHOST:client_ip} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:http_verb} %{NOTSPACE:http_request}(?: HTTP/%{NUMBER:http_version})?|%{DATA:raw_http_request})\" %{NUMBER:http_status_code} (?:%{NUMBER:bytes_read}|-) %{QS:referrer} %{QS:agent} %{NUMBER:time_duration:float}" ] } # mutate { # convert => ["response", "integer"] # convert => ["bytes", "integer"] # convert => ["responsetime", "float"] # } geoip { source => "client_ip" target => "geoip" add_tag => ["nginx-geoip"] } date { match => ["timestamp" , "dd/MMM/YYYY:HH:mm:ss Z"] remove_field => ["timestamp"] } useragent { source => "agent" } } output { elasticsearch { hosts => ["http://talkkib.plesk.com:9200/"] index => "logstash-logs_%{+YYYY.MM.dd}" document_type => "nginx_logs" } # stdout { codec => rubydebug } } ``` And: ``` # cat /etc/logstash/conf.d/mysqlBA.conf input { jdbc { jdbc_driver_library => "/usr/share/java/mysql-connector-java.jar" jdbc_driver_class => "com.mysql.jdbc.Driver" jdbc_connection_string => "jdbc:mysql://localhost:3306/xf" jdbc_user => "root" jdbc_password => "xxxxx" schedule => "0 * * * *" statement => "select xf_post.username, count(*) as posts, count(xf_thread.best_answer_id) as best_answers, sum(xf_post.likes) as likes from xf_post left join xf_thread on (xf_post.post_id = xf_thread.best_answer_id and xf_thread.best_answer_id > 0) where xf_post.post_date between UNIX_TIMESTAMP('2017-11-01 00:00:00') and UNIX_TIMESTAMP('2017-12-01 00:00:00') group by xf_post.username order by posts desc limit 25" } } output { elasticsearch { hosts => ["http://talkkib.plesk.com:9200/"] index => "forum_stat_%{+YYYY.MM.dd}" document_type => "forum_stat" } # stdout { codec => rubydebug } } ``` All works good. But the problem in that in Kibana I see the same index structure for forum\_stat\_%{+YYYY.MM.dd} like forum\_stat\_%{+YYYY.MM.dd} with additional fields 'posts', 'likes', 'best\_answers', 'username'. I suspect that size of this index forum\_stat\_%{+YYYY.MM.dd} grows so fast dut to data from nginx logs. Could you please clarify - why? I need forum\_stat\_%{+YYYY.MM.dd} index with very simple structure and small size. Please help me with this. Additionally it would be very good if you correct my configs with any useful customizations from your great experience! Thanks! ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 30, 2017, 11:01am UTC](https://discuss.elastic.co/t/logstash-index-template/109732/2 "2017-11-30T11:01:17Z")
When you have multiple configuration files in a single pipeline (the only option prior to Logstash 6), all events from all inputs will be passed to all filters and all outputs unless you use conditionals to apply filters and outputs conditionally. Your current configuration is equivalent to this: ```nohighlight cat /etc/logstash/conf.d/mysqlBA.conf /etc/logstash/conf.d/nginx.conf > all-of-it.conf rm /etc/logstash/conf.d/mysqlBA.conf /etc/logstash/conf.d/nginx.conf ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 30, 2017, 11:01am UTC](https://discuss.elastic.co/t/logstash-index-template/109732/3 "2017-11-30T11:01:35Z")
Logstash concatenates all config files in that directory, which means all event will go to all outputs unless you use conditionals to separate the flows. There should be numerous examples in this forum. ---
**Author:** ![pavuk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pavuk/32/28914_2.png) [@pavuk](https://discuss.elastic.co/u/pavuk)\ **Post date:** [November 30, 2017, 12:10pm UTC](https://discuss.elastic.co/t/logstash-index-template/109732/4 "2017-11-30T12:10:28Z")
Ok, I see, Thanks. I use Logstash 6. Will try to find the mentioned examples how to use conditionals to separate the flows. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 30, 2017, 12:27pm UTC](https://discuss.elastic.co/t/logstash-index-template/109732/5 "2017-11-30T12:27:58Z")
If you use Logstash 6.0 you can specify separate pipelines in the pipelines.yml file and avoid conditionals. ---
**Author:** ![pavuk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pavuk/32/28914_2.png) [@pavuk](https://discuss.elastic.co/u/pavuk)\ **Post date:** [December 1, 2017, 5:59am UTC](https://discuss.elastic.co/t/logstash-index-template/109732/6 "2017-12-01T05:59:26Z")
Unfortunately still doesn't work. I have created file ``` # cat /etc/logstash/pipeline.yml - pipeline.id: nginx path.config: "/etc/logstash/conf.d/nginx.conf" #queue.type: persisted - pipeline.id: forum_stat path.config: "/etc/logstash/conf.d/mysqlBA.conf" #pipeline.workers: 3 ``` Restarted logstash service and got error like `Ignoring the 'pipelines.yml' file because modules or command line options are specified` According to [https://github.com/elastic/logstash/issues/8420#issuecomment-333956090](https://github.com/elastic/logstash/issues/8420#issuecomment-333956090) I commented line path.config in logstash.yml, restarted service and see endless messages like ``` [2017-12-01T05:57:14,204][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"} [2017-12-01T05:57:14,206][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"} [2017-12-01T05:57:14,359][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.0.0-java/modules/arcsight/configuration"} [2017-12-01T05:57:24,493][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"} [2017-12-01T05:57:24,495][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"} [2017-12-01T05:57:24,643][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.0.0-java/modules/arcsight/configuration"} [2017-12-01T05:57:34,847][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"} [2017-12-01T05:57:34,849][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"} [2017-12-01T05:57:34,983][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.0.0-java/modules/arcsight/configuration"} [2017-12-01T05:57:45,214][INFO][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"} ``` Logstash doesn't work. What's wrong with it? How can I run two separate pipelines? Please guide me. Thanks. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 29, 2017, 5:59am UTC](https://discuss.elastic.co/t/logstash-index-template/109732/7 "2017-12-29T05:59:55Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.