Possible Bug? Filter Not Applying Before Output into Elasticsearch

So I'm getting input from Redis, filtering and transforming it, and then outputting it into Elasticsearch. However, I have run into a problem where the filter is not being applied for output to Elasticsearch. I tested it to see if the filter is being applied, and it is for output directly to stdout. I am a little befuddled. Can anyone help? Does this seem like a possible bug?

This is ElasticSearch output, filter is not being applied

"_index": "bro-2016.07.19", "_type": "ssl", "_id": "AVYDnigW4FpzjhIkBVAo", "_score": null, "_source": { "message": "1468939511.398594\tCQ8Xtm4cpBrjPieTLh\t130.85.86.87\t55751\t199.16.156.73\t443\tTLSv12\tTLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256\tsecp256r1\tanalytics.twitter.com\tF\t-\th2\tT\tF0VRoggjZTPAmyQOg,FkfpY93yjN8dTfBmV2\t(empty)\tCN=*.twitter.com,OU=Twitter Security,O=Twitter\\\\, Inc.,L=San Francisco,ST=CA,C=US\tCN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US\t-\t-\tok", "@version": "1", "@timestamp": "2016-07-19T14:45:12.853Z", "path": "/local/bro/logs/current/ssl.log", "host": "redacted", "type": "ssl" },

This is Stdout, filter is being applied

"@timestamp" => "2016-07-19T14:45:11.398Z", "path" => "/local/bro/logs/current/ssl.log", "host" => "redacted", "type" => "ssl", "ts" => "1468939511.398594", "uid" => "CQ8Xtm4cpBrjPieTLh", "orig_IP" => "", "orig_port" => "55751", "resp_IP" => "", "resp_port" => "443", "version" => "TLSv12", "cipher" => "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "curve" => "secp256r1", "server_name" => "analytics.twitter.com", "resumed" => "F", "last_alert" => "-", "next_protocol" => "h2", "establishedcert_chain_fuids" => "T", "client_cert_chain_fuids" => "F0VRoggjZTPAmyQOg,FkfpY93yjN8dTfBmV2", "subject" => "(empty)", "issuer" => "CN=*.twitter.com,OU=Twitter Security,O=Twitter\\\\, Inc.,L=San Francisco,ST=CA,C=US", "client_subject" => "CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US", "client_issuer" => "-", "validation_status" => "-",

Here is the relevant part of the configuration, the same exact output is being sent to stdout and ElasticSearch.



        elasticsearch {

        hosts => [""]
        index => "bro-%{+YYYY.MM.dd}"


That's not really relevant as we can't see the filters you are using! :frowning:

I didn't think the filter is pertinent because it is clearly being applied correctly for one output and not the other. I can't post the full config, it's too long.

I have posted the full configuration here along with additional information https://github.com/elastic/logstash/issues/5670.