Prune not parsing date component of custom elasticsearch index

[iiz]

I have json logs being shipped to logstash by filebeat on multiple server instances. I would like to whitelist a specific set of fields to include. The whitelist feature works quite well, except for the fact that it doesn't parse my custom index name for my elastic search output. When the whitelist config is added to prune the index name used is "custom_index-" rather than "custom_index-2019-08-28".

It seemed pretty logical to include interpolate => true in my config since the prune plugin docs describes not parsing fields prior to doing checks for whitelists/blacklists. However, when I added that configuration option logstash began raising an exception.

org.logstash.FieldReference$IllegalSyntaxException: Invalid FieldReference: `[^`

Here is the full console output when the exception occurs: https://pastebin.com/hx01S1kS

Here is my configuration that fails, for testing I reduced the whitelist to a single item to make config changes easier:

input {
        beats {
                port => 5044
                ssl => true
                ssl_certificate_authorities => ["/etc/ssl/certs/ca_cert.pem"]
                ssl_certificate => "/etc/logstash/ssl/certs/logstash.pem"
                ssl_key => "/etc/logstash/ssl/keys/logstash-pkcs8.key"
                ssl_verify_mode => "force_peer"
        }
}

filter {
        prune {
                interpolate => true
                whitelist_names => ["message"]
        }
}

output {
        elasticsearch {
                hosts => ["http://localhost:9200"]
                index => "custom_index-%{+YYYY.MM.dd}"
                user => "username"
                password => "password"
        }
}

Logstash Version: 7.3.1
Filebeat Version: 7.3.1
Libbeat Version: 7.3.1

Thank you for your help!

[Badger]

I would say that is a bug, at the very least it should be documented. You can work around it by adding

 blacklist_names => []

The problem is that the default function of prune is to remove fields that look they were sprintf failures. The default value for blacklist_names is "%{[^}]+}" and when the filter tries to sprintf that it all goes pear shaped.

[iiz]

Thanks Badger, that solve the issue of logstash producing a fatal exception. I'll submit a bug report for this.

Unfortunately the custom index is still not be parsed with the inclusion of the interpolate => true settings. I figured that was somewhat of a long shot.

I have tested a configuration that excluded the whitelist directive in the prune filter. That resulted in logs being placed into the appropriate index. However, as soon as I add a whitelist directive back logstash will not parse %{+YYYY.MM.dd} to an appropriate date.