Logstash index template

Elastic Stack Logstash
1

Sorry for stupid question, it's my first Logstash experience, but I would like to ask about following issue.

I have two LS configs:

# cat /etc/logstash/conf.d/nginx.conf 
input {
  file {
    type => "nginx"
    start_position => "beginning"
    path => [ "/var/log/nginx/*.log" ]
    exclude => [ "*.gz" ]
  }
}
filter {

grok {
#    type => "nginx-access"
    match => [
      "message", "%{IPORHOST:http_host} %{IPORHOST:client_ip} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:http_verb} %{NOTSPACE:http_request}(?: HTTP/%{NUMBER:http_version})?|%{DATA:raw_http_request})\" %{NUMBER:http_status_code} (?:%{NUMBER:bytes_read}|-) %{QS:referrer} %{QS:agent} %{NUMBER:time_duration:float} %{NUMBER:time_backend_response:float}",
      "message", "%{IPORHOST:http_host} %{IPORHOST:client_ip} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:http_verb} %{NOTSPACE:http_request}(?: HTTP/%{NUMBER:http_version})?|%{DATA:raw_http_request})\" %{NUMBER:http_status_code} (?:%{NUMBER:bytes_read}|-) %{QS:referrer} %{QS:agent} %{NUMBER:time_duration:float}"
    ]
  }

# mutate {
#   convert => ["response", "integer"]
#   convert => ["bytes", "integer"]
#   convert => ["responsetime", "float"]
# }
 geoip {
   source => "client_ip"
   target => "geoip"
   add_tag => [ "nginx-geoip" ]
 }
 date {
   match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
   remove_field => [ "timestamp" ]
 }
 useragent {
   source => "agent"
 }
}
output {
 elasticsearch {
   hosts => ["http://talkkib.plesk.com:9200/"]
   index => "logstash-logs_%{+YYYY.MM.dd}"
   document_type => "nginx_logs"
 }
# stdout { codec => rubydebug }
}

And:

# cat /etc/logstash/conf.d/mysqlBA.conf 
input {
  jdbc {
    jdbc_driver_library => "/usr/share/java/mysql-connector-java.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://localhost:3306/xf"
    jdbc_user => "root"
    jdbc_password => "xxxxx"
    schedule => "0 * * * *"
    statement => "select xf_post.username, count(*) as posts, count(xf_thread.best_answer_id) as best_answers, sum(xf_post.likes) as likes from xf_post left join xf_thread on (xf_post.post_id = xf_thread.best_answer_id and xf_thread.best_answer_id > 0) where xf_post.post_date between UNIX_TIMESTAMP('2017-11-01 00:00:00') and UNIX_TIMESTAMP('2017-12-01 00:00:00') group by xf_post.username order by posts desc limit 25"  
}
}
output {
 elasticsearch {
   hosts => ["http://talkkib.plesk.com:9200/"]
   index => "forum_stat_%{+YYYY.MM.dd}"
   document_type => "forum_stat"
 }
# stdout { codec => rubydebug }
}

All works good. But the problem in that in Kibana I see the same index structure for forum_stat_%{+YYYY.MM.dd} like forum_stat_%{+YYYY.MM.dd} with additional fields 'posts', 'likes', 'best_answers', 'username'. I suspect that size of this index forum_stat_%{+YYYY.MM.dd} grows so fast dut to data from nginx logs. Could you please clarify - why? I need forum_stat_%{+YYYY.MM.dd} index with very simple structure and small size.
Please help me with this. Additionally it would be very good if you correct my configs with any useful customizations from your great experience!

Thanks!

2

When you have multiple configuration files in a single pipeline (the only option prior to Logstash 6), all events from all inputs will be passed to all filters and all outputs unless you use conditionals to apply filters and outputs conditionally. Your current configuration is equivalent to this:

cat /etc/logstash/conf.d/mysqlBA.conf /etc/logstash/conf.d/nginx.conf > all-of-it.conf
rm /etc/logstash/conf.d/mysqlBA.conf /etc/logstash/conf.d/nginx.conf
1 Like
3

Logstash concatenates all config files in that directory, which means all event will go to all outputs unless you use conditionals to separate the flows. There should be numerous examples in this forum.

4

Ok, I see, Thanks. I use Logstash 6.
Will try to find the mentioned examples how to use conditionals to separate the flows.

5

If you use Logstash 6.0 you can specify separate pipelines in the pipelines.yml file and avoid conditionals.

1 Like
6

Unfortunately still doesn't work. I have created file

# cat /etc/logstash/pipeline.yml 
- pipeline.id: nginx
  path.config: "/etc/logstash/conf.d/nginx.conf"
  #queue.type: persisted
- pipeline.id: forum_stat
  path.config: "/etc/logstash/conf.d/mysqlBA.conf"
  #pipeline.workers: 3

Restarted logstash service and got error like

Ignoring the 'pipelines.yml' file because modules or command line options are specified

According to https://github.com/elastic/logstash/issues/8420#issuecomment-333956090 I commented line path.config in logstash.yml, restarted service and see endless messages like

[2017-12-01T05:57:14,204][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-12-01T05:57:14,206][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-12-01T05:57:14,359][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.0.0-java/modules/arcsight/configuration"}
[2017-12-01T05:57:24,493][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-12-01T05:57:24,495][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-12-01T05:57:24,643][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.0.0-java/modules/arcsight/configuration"}
[2017-12-01T05:57:34,847][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-12-01T05:57:34,849][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-12-01T05:57:34,983][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.0.0-java/modules/arcsight/configuration"}
[2017-12-01T05:57:45,214][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}

Logstash doesn't work.
What's wrong with it? How can I run two separate pipelines?
Please guide me.
Thanks.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.