I had a working 5.6.3 environment which I upgraded to 6.0.0-rc1. metricbeat requests the Apache server-status page and filebeat is capturing both the apache logs and some GC logs. After a trip through Kafka, the data hits logstash which uses this output
output { if [type] == "metricsets" { elasticsearch { hosts => "localhost" index => "metricbeat-%{+YYYY.MM.dd}" } } else if [filter] == "j9gclog" { elasticsearch { hosts => "localhost" index => "logs.j9gc-%{+YYYY.MM}" } } else if [filter] == "G1gclog" { elasticsearch { hosts => "localhost" index => "logs.g1gc-%{+YYYY.MM}" } } else { elasticsearch { hosts => "localhost" index => "logstash-%{+YYYY.MM.dd}" } } }
The problem I had was that zero events got written to the logstash-* indexes. filebeat through kafka was working, because the g1gc-* and j9gc-* indexes were being written. trace logging showed the event reaching the output of logstash
[2017-10-16T11:00:51,438][DEBUG][logstash.pipeline ] output received {"event"=>{"request"=>"/server-status", "referer"=>"-", "sbytes"=>440, "useragent"=>"Go-http-client/1.1", "source"=>"/var/log/httpd/access_log.1508112000", "clienthost"=>"127.0.0.1", "sysdate"=>"[16/Oct/2017:10:44:03 -0400]", "timetaken"=>194, "beat"=>{"name"=>"elided", "hostname"=>"elided", "version"=>"6.0.0-rc1"}, "host"=>"127.0.0.1", "@version"=>"1", "win32status"=>"0", "cookie"=>"-", "method"=>"GET", "offset"=>2132610, "cbytes"=>0, "querystring"=>"?auto=", "responsestatus"=>"200", "prospector"=>{"type"=>"log"}, "filter"=>"apacheAccess", "site"=>"127.0.0.1", "@timestamp"=>2017-10-16T14:44:03.139Z, "port"=>"80", "subresponse"=>"0", "httpversion"=>"HTTP/1.1", "username"=>"-"}}
The problem turned out to be this (old, no longer required) template that I had been using to have a couple of fields treated as ip addresses. Once I deleted this template and deleted logstash-2017-10-16 the events started to arrive in the newly recreated index.
{ "template" : "logstash-*", "order" : 10, "version" : 5, "mappings" : { "apacheaccess": { "properties": { "site": { "type": "ip"}, "clienthost": { "type": "ip"} } } } }
However, it took me over a day to find it, because nothing logs an error anywhere