Logstash not sending events to es with this template


#1

I had a working 5.6.3 environment which I upgraded to 6.0.0-rc1. metricbeat requests the Apache server-status page and filebeat is capturing both the apache logs and some GC logs. After a trip through Kafka, the data hits logstash which uses this output

output {
        if [type] == "metricsets" {
                elasticsearch {
                        hosts => "localhost"
                        index => "metricbeat-%{+YYYY.MM.dd}"
                }
        } else if [filter] == "j9gclog" {
                elasticsearch {
                        hosts => "localhost"
                        index => "logs.j9gc-%{+YYYY.MM}"
                }
        } else if [filter] == "G1gclog" {
                elasticsearch {
                        hosts => "localhost"
                        index => "logs.g1gc-%{+YYYY.MM}"
                }
        } else {
                elasticsearch {
                        hosts => "localhost"
                        index => "logstash-%{+YYYY.MM.dd}"
                }
        }
}

The problem I had was that zero events got written to the logstash-* indexes. filebeat through kafka was working, because the g1gc-* and j9gc-* indexes were being written. trace logging showed the event reaching the output of logstash

[2017-10-16T11:00:51,438][DEBUG][logstash.pipeline ] output received {"event"=>{"request"=>"/server-status", "referer"=>"-", "sbytes"=>440, "useragent"=>"Go-http-client/1.1", "source"=>"/var/log/httpd/access_log.1508112000", "clienthost"=>"127.0.0.1", "sysdate"=>"[16/Oct/2017:10:44:03 -0400]", "timetaken"=>194, "beat"=>{"name"=>"elided", "hostname"=>"elided", "version"=>"6.0.0-rc1"}, "host"=>"127.0.0.1", "@version"=>"1", "win32status"=>"0", "cookie"=>"-", "method"=>"GET", "offset"=>2132610, "cbytes"=>0, "querystring"=>"?auto=", "responsestatus"=>"200", "prospector"=>{"type"=>"log"}, "filter"=>"apacheAccess", "site"=>"127.0.0.1", "@timestamp"=>2017-10-16T14:44:03.139Z, "port"=>"80", "subresponse"=>"0", "httpversion"=>"HTTP/1.1", "username"=>"-"}}

The problem turned out to be this (old, no longer required) template that I had been using to have a couple of fields treated as ip addresses. Once I deleted this template and deleted logstash-2017-10-16 the events started to arrive in the newly recreated index.

{
    "template" : "logstash-*",
    "order" : 10,
    "version" : 5,
    "mappings" : {
        "apacheaccess": { 
            "properties": {
                "site": { "type": "ip"},
                "clienthost": { "type": "ip"}
            }
        }
    }
}

However, it took me over a day to find it, because nothing logs an error anywhere :frowning:


(system) #2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.