So the json inputs are not always consistent. For example, one field is usually true/false, but the source occasionally tosses in a "not_supported" just to be fun. So after some digging, I thought that I could just do:
prune {
blacklist_values => [
"expires", "not_supported"
]
}
but no. The help page says that "blacklist_values" accepts a hash, and even helpfully links to the help pages for hashes, as well as shows an example... except that the example hash looks like:
match => {
"field1" => "value1"
"field2" => "value2"
...
}
and the example prune looks like:
prune {
blacklist_values => [ "uripath", "/index.php",
"method", "(HEAD|OPTIONS)",
"status", "^[^2]" ]
}
In experimenting, it seems like the values in "blacklist_values" need to be in pairs (eg: logstash gives an error if the only thing in "blacklist_values" is "not_supported") , but no matter how I use this, I dont get the desired results.
The json that might come in might be something like:
{"item": "first thing", "is_working": true, "is_configured": true}
{"item": "second thing", "is_working": "not_supported", "is_configured": true}
And I want logstash to pump the data to Elasticsearch:
{"item": "first thing", "is_working": true, "is_configured": true}
{"item": "second thing", "is_configured": true}
It would be lovely if someone could point out what I am missing.