So I'm getting input from Redis, filtering and transforming it, and then outputting it into Elasticsearch. However, I have run into a problem where the filter is not being applied for output to Elasticsearch. I tested it to see if the filter is being applied, and it is for output directly to stdout. I am a little befuddled. Can anyone help? Does this seem like a possible bug?
This is ElasticSearch output, filter is not being applied
"_index": "bro-2016.07.19", "_type": "ssl", "_id": "AVYDnigW4FpzjhIkBVAo", "_score": null, "_source": { "message": "1468939511.398594\tCQ8Xtm4cpBrjPieTLh\t130.85.86.87\t55751\t199.16.156.73\t443\tTLSv12\tTLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256\tsecp256r1\tanalytics.twitter.com\tF\t-\th2\tT\tF0VRoggjZTPAmyQOg,FkfpY93yjN8dTfBmV2\t(empty)\tCN=*.twitter.com,OU=Twitter Security,O=Twitter\\\\, Inc.,L=San Francisco,ST=CA,C=US\tCN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US\t-\t-\tok", "@version": "1", "@timestamp": "2016-07-19T14:45:12.853Z", "path": "/local/bro/logs/current/ssl.log", "host": "redacted", "type": "ssl" },
This is Stdout, filter is being applied
"@timestamp" => "2016-07-19T14:45:11.398Z", "path" => "/local/bro/logs/current/ssl.log", "host" => "redacted", "type" => "ssl", "ts" => "1468939511.398594", "uid" => "CQ8Xtm4cpBrjPieTLh", "orig_IP" => "130.85.86.87", "orig_port" => "55751", "resp_IP" => "199.16.156.73", "resp_port" => "443", "version" => "TLSv12", "cipher" => "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "curve" => "secp256r1", "server_name" => "analytics.twitter.com", "resumed" => "F", "last_alert" => "-", "next_protocol" => "h2", "establishedcert_chain_fuids" => "T", "client_cert_chain_fuids" => "F0VRoggjZTPAmyQOg,FkfpY93yjN8dTfBmV2", "subject" => "(empty)", "issuer" => "CN=*.twitter.com,OU=Twitter Security,O=Twitter\\\\, Inc.,L=San Francisco,ST=CA,C=US", "client_subject" => "CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US", "client_issuer" => "-", "validation_status" => "-",
Here is the relevant part of the configuration, the same exact output is being sent to stdout and ElasticSearch.
output{
stdout{codec=>rubydebug}
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "bro-%{+YYYY.MM.dd}"
}