# Prune not parsing date component of custom elasticsearch index

**URL:** <https://discuss.elastic.co/t/prune-not-parsing-date-component-of-custom-elasticsearch-index/197129>\
**Category:** Logstash\
**Created:** [August 28, 2019, 1:51pm UTC](https://discuss.elastic.co/t/prune-not-parsing-date-component-of-custom-elasticsearch-index/197129 "2019-08-28T13:51:42Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![iiz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iiz/32/53176_2.png) [@iiz](https://discuss.elastic.co/u/iiz)\
**Post date:** [August 28, 2019, 1:51pm UTC](https://discuss.elastic.co/t/prune-not-parsing-date-component-of-custom-elasticsearch-index/197129/1 "2019-08-28T13:51:42Z")

</div>

I have json logs being shipped to logstash by filebeat on multiple server instances. I would like to whitelist a specific set of fields to include. The whitelist feature works quite well, except for the fact that it doesn't parse my custom index name for my elastic search output. When the whitelist config is added to prune the index name used is "custom\_index-" rather than "custom\_index-2019-08-28".

It seemed pretty logical to include `interpolate => true` in my config since the prune plugin docs describes not parsing fields prior to doing checks for whitelists/blacklists. However, when I added that configuration option logstash began raising an exception.

`org.logstash.FieldReference$IllegalSyntaxException: Invalid FieldReference: `[^``

Here is the full console output when the exception occurs: [https://pastebin.com/hx01S1kS](https://pastebin.com/hx01S1kS)

Here is my configuration that fails, for testing I reduced the whitelist to a single item to make config changes easier:

```
input {
        beats {
                port => 5044
                ssl => true
                ssl_certificate_authorities => ["/etc/ssl/certs/ca_cert.pem"]
                ssl_certificate => "/etc/logstash/ssl/certs/logstash.pem"
                ssl_key => "/etc/logstash/ssl/keys/logstash-pkcs8.key"
                ssl_verify_mode => "force_peer"
        }
}

filter {
        prune {
                interpolate => true
                whitelist_names => ["message"]
        }
}

output {
        elasticsearch {
                hosts => ["http://localhost:9200"]
                index => "custom_index-%{+YYYY.MM.dd}"
                user => "username"
                password => "password"
        }
}

```

Logstash Version: 7.3.1  
Filebeat Version: 7.3.1  
Libbeat Version: 7.3.1

Thank you for your help!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 28, 2019, 2:14pm UTC](https://discuss.elastic.co/t/prune-not-parsing-date-component-of-custom-elasticsearch-index/197129/2 "2019-08-28T14:14:09Z")

</div>

I would say that is a bug, at the very least it should be documented. You can work around it by adding

```
 blacklist_names => []

```

The problem is that the default function of prune is to remove fields that look they were sprintf failures. The default value for blacklist\_names is "%{[^}]+}" and when the filter [tries](https://github.com/logstash-plugins/logstash-filter-prune/blob/fd4b14398a31c6bf6e1cdbfa52024c837dd2ceeb/lib/logstash/filters/prune.rb#L126) to sprintf that it all goes pear shaped.

---

<div class="post-metadata">

**Author:** ![iiz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iiz/32/53176_2.png) [@iiz](https://discuss.elastic.co/u/iiz)\
**Post date:** [August 28, 2019, 2:26pm UTC](https://discuss.elastic.co/t/prune-not-parsing-date-component-of-custom-elasticsearch-index/197129/3 "2019-08-28T14:26:03Z")

</div>

Thanks Badger, that solve the issue of logstash producing a fatal exception. I'll submit a bug report for this.

Unfortunately the custom index is still not be parsed with the inclusion of the `interpolate => true` settings. I figured that was somewhat of a long shot.

I have tested a configuration that excluded the whitelist directive in the prune filter. That resulted in logs being placed into the appropriate index. However, as soon as I add a whitelist directive back logstash will not parse `%{+YYYY.MM.dd}` to an appropriate date.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 25, 2019, 2:26pm UTC](https://discuss.elastic.co/t/prune-not-parsing-date-component-of-custom-elasticsearch-index/197129/4 "2019-09-25T14:26:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.