Elasticseach input plugin - docinfo field is not respected?

[wmathews]

I'm trying to do some work with the elastic input plugin, and part of it is that I want to access the index for use in the eventual output. In the documentation there is the docinfo field, but (unless I don't understand how this is supposed to work) I can't get it to actually populate the [@metadata] fields.

input {
  elasticsearch {
   docinfo => true
   hosts => "http://localhost:9200"
   query => removed
   add_field => {
      'logstashCreatedIndex' => "%{['@metadata']['_index']}"
    }
  }
}

To my understanding of this plugin based on the documentation this should allow me to have the index saved in that field but what is happening is that the metadata index field isn't being recognized as a field. I've tried with and without the internal quotes and I've tried moving the 'add field' to one of the filters later in the document. I've tried specifically asking for the index field and the metadata location, as well as accessing the field with the event api in the ruby filter and accessing it directly in my output filter.

Is this something simple I just don't understand about the plugin or about the docinfo field?

[magnusbaeck (Magnus Bäck)]

This is a bug in the plugin:

Issue: 'add_field' and 'tags' can't reference metadata fields

If you add the field in a subsequent mutate filter it works:

input {
  elasticsearch {
    hosts => "hallonet:9200"
    index => "logstash*"
    docinfo => true
  }
}
filter {
  mutate {
    add_field => {
      'logstashCreatedIndex' => "%{[@metadata][_index]}"
    }
  }
}

[wmathews]

That worked, thanks.

Is there some property of the mutate filter that allows it access to the metadata fields that other plugins don't have? For instance, if I try to put [@metadata][_index] as is in a csv output plugin it doesn't work, but the example in the documentation uses the elasticsearch output to access the field so I'm not sure where it can and can't be seen and it seems inconsistent.

Thanks for your help.