# Extract elasticsearch "index" field from event field
**URL:** https://discuss.elastic.co/t/extract-elasticsearch-index-field-from-event-field/64503
**Category:** Beats
**Tags:** filebeat
**Created:** [November 1, 2016, 12:10am UTC](https://discuss.elastic.co/t/extract-elasticsearch-index-field-from-event-field/64503 "2016-11-01T00:10:27Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@surajs](https://discuss.elastic.co/u/surajs)
#### Post date: [November 1, 2016, 12:10am UTC](https://discuss.elastic.co/t/extract-elasticsearch-index-field-from-event-field/64503/1 "2016-11-01T00:10:27Z")
In filebeat for Kafka output, we are able to dynamically select the topic-name using data from the an event field using something like `%{[type]}`.
In a similar fashion, is it possible to dynamically select the index name for an elasticsearch output using data from the event field ?
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [November 1, 2016, 9:30am UTC](https://discuss.elastic.co/t/extract-elasticsearch-index-field-from-event-field/64503/2 "2016-11-01T09:30:51Z")
It should work in 5.0 with format strings, but I never tested it TBH. Let me know if it works as expected.
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [November 1, 2016, 1:24pm UTC](https://discuss.elastic.co/t/extract-elasticsearch-index-field-from-event-field/64503/3 "2016-11-01T13:24:49Z")
See [index](https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#_index) and [indices](https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#_indices) settings.
---
### Author:  [@surajs](https://discuss.elastic.co/u/surajs)
#### Post date: [November 2, 2016, 10:06pm UTC](https://discuss.elastic.co/t/extract-elasticsearch-index-field-from-event-field/64503/4 "2016-11-02T22:06:07Z")
Thanks @steffens , @ruflin!
Another quick question around the same lines.
Would it make sense for filebeat to expose the filename that its reading from in order to determine the index name( or kafka topic for that matter).
For example,
if its currently tailing from a file `"/var/log/docker/api.log"` then,
include a field in the event`{"event_source" : "api" }` or even `{"event_source" : "api.log" }`along with other metadata?
I know that filebeat already exposes an absolute path in the `source` field, but that is not enough for determining which index / kafka topic to write to if you're doing this at scale. Also, adding a field in the event is going to require change in the way the events are logged which makes the transition to filebeat much more difficult.
What we need is something very similar to : [Using filename from filebeat in index pattern](https://discuss.elastic.co/t/using-filename-from-filebeat-in-index-pattern/61602)
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [November 3, 2016, 9:35am UTC](https://discuss.elastic.co/t/extract-elasticsearch-index-field-from-event-field/64503/5 "2016-11-03T09:35:09Z")
It would be totally possible but for advance processing / routing I recommend to use Logstash in the middle.
The reason we expose the full path and no the filename because the file name is not necessarly unique.
---
### Author:  [@surajs](https://discuss.elastic.co/u/surajs)
#### Post date: [November 3, 2016, 5:48pm UTC](https://discuss.elastic.co/t/extract-elasticsearch-index-field-from-event-field/64503/6 "2016-11-03T17:48:41Z")
I totally see the worth in exposing the full path.
As more and more large scale organizations start to consider beats as their option for log-tailing, as seen in few other questions on Stackoverflow as well as the Elastic forum, this feature is going to be something that could be really helpful to add.
Additional overhead of maintaining logstash for doing simple extractions/inductions based on either fields in events or path is something that I feel will hamper the adoption of filebeat (or even worse, could potentially lead to adopters maintaining their own versions of filebeat) when used at scale.
Thoughts?
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [November 3, 2016, 10:06pm UTC](https://discuss.elastic.co/t/extract-elasticsearch-index-field-from-event-field/64503/7 "2016-11-03T22:06:49Z")
One workaround would be to make use of prospector fields.
e.g. adds create a prospector per file type and set document\_type accordingly or use ```
```auto
filebeat.prospectors.X.fields:
source_type: "api"
```
then you can use `%{[fields.source_type]}`.
Using `indices` or `topics` for kafka one can use conditionals todo some more processing.
But your request makes me think about introducing some kind of template-processors/functions as supported by more common templating systems. This could look somewhat like `%{[source:basename]}` or `%{[source]|basename}`. The former only on fields extracted from events, the second potentially on other value sources. I kind of like the pipe-symbol here. Imagine `%{[source]|basename|trimRight('.log')}`. Well, just some idea so far. Will have to think more about this.
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [November 7, 2016, 4:36pm UTC](https://discuss.elastic.co/t/extract-elasticsearch-index-field-from-event-field/64503/8 "2016-11-07T16:36:11Z")
@surajs Are you just referring the the feature of the file name or more general processing?
---
### Author:  [@surajs](https://discuss.elastic.co/u/surajs)
#### Post date: [November 8, 2016, 1:34am UTC](https://discuss.elastic.co/t/extract-elasticsearch-index-field-from-event-field/64503/9 "2016-11-08T01:34:11Z")
@ruflin,
I was originally thinking to extract index/topic name using the basename as an original request.
However, what @steffens mentioned, i can totally see the value of providing a scripting interface to existing metadata.
I'd be happy to contribute on that feature should you feel that we need to add that to filebeat or need to discuss potential use-cases that this request may suffice.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 22, 2016, 12:10am UTC](https://discuss.elastic.co/t/extract-elasticsearch-index-field-from-event-field/64503/10 "2016-11-22T00:10:53Z")
This topic was automatically closed after 21 days. New replies are no longer allowed.