# Defining a custom ES index based on an existing field

**URL:** <https://discuss.elastic.co/t/defining-a-custom-es-index-based-on-an-existing-field/84244>\
**Category:** Logstash\
**Created:** [May 2, 2017, 10:54am UTC](https://discuss.elastic.co/t/defining-a-custom-es-index-based-on-an-existing-field/84244 "2017-05-02T10:54:06Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![ticktockhouse](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ticktockhouse/32/16801_2.png) [@ticktockhouse](https://discuss.elastic.co/u/ticktockhouse)\
**Post date:** [May 2, 2017, 10:54am UTC](https://discuss.elastic.co/t/defining-a-custom-es-index-based-on-an-existing-field/84244/1 "2017-05-02T10:54:06Z")

</div>

I have a logstash pipeline designed to break container logs coming from coreos into separate indexes, but it's not quite working:

My pipeline looks like this:

```
input {
  beats {
    port => 5044
  }
}

filter {
  grok {
    match => { "source" => "%{GREEDYDATA}/%{CONTAINERAPP:prefix}*" }
    patterns_dir => ["/patterns"]
  }
......
  translate {
    add_tag => ["monitoring"]
    field => "prefix"
    regex => true
    exact => true
    dictionary_path => "/dict/monitoring.yml"
    destination => "container"
  }
.......
[Several more blocks like this adding tags to different groups of "prefix" fields]
.......
}
output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    manage_template => false
    index => "logtest-%{[@metadata][container]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
    }
}

```

I'm trying to use the syntax suggested [here](https://discuss.elastic.co/t/using-tag-found-in-array-as-index-name/46057) to get logstash to write the events to an index based on the name of the container. However, what I'm ending up with are indexes with the literal string, rather than the contents of the field "container", which is what I'm after.

```
$ curl -v elasticsearch:9200/_cat/indices?v |grep logtest
.....
green open logtest-%{[@metadata][container]}-2017.05.02 unVVBghgRra974trYtDPJA 5 1 9168 0 18.9mb 9.5mb

```

This doesn't happen with certain other fields, for instance, up until now I have had the "type" field in the pipeline config:

```
.......
    index => "logtest-%{[@metadata][type]}-%{+YYYY.MM.dd}"
.......

```

and have ended up with indices like:

```
.......
green open logtest-log-2017.04.06 EzNg2CMKSx-mMMpbDVF-9g 5 1 116131362 0 202.1gb 101gb
.......

```

Thanks in advance.

---

<div class="post-metadata">

**Author:** ![Nico-DF](https://avatars.discourse-cdn.com/v4/letter/n/ed8c4c/32.png) [@Nico-DF](https://discuss.elastic.co/u/Nico-DF)\
**Post date:** [May 2, 2017, 10:57am UTC](https://discuss.elastic.co/t/defining-a-custom-es-index-based-on-an-existing-field/84244/2 "2017-05-02T10:57:40Z")

</div>

I'm not sure about this but @metadata isn't for fields like \_type, etc.?  
Did you try simply %{container}?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [May 2, 2017, 10:59am UTC](https://discuss.elastic.co/t/defining-a-custom-es-index-based-on-an-existing-field/84244/3 "2017-05-02T10:59:58Z")

</div>

You are storing the data in the field `container`, not `@metadata.container`. Change the index name specification to`"logtest-%{container}-%{+YYYY.MM.dd}"`. Having said that, be careful with this approach as you could easily end up with a lot of very small indices, which is very, very inefficient.

---

<div class="post-metadata">

**Author:** ![ticktockhouse](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ticktockhouse/32/16801_2.png) [@ticktockhouse](https://discuss.elastic.co/u/ticktockhouse)\
**Post date:** [May 2, 2017, 11:08am UTC](https://discuss.elastic.co/t/defining-a-custom-es-index-based-on-an-existing-field/84244/4 "2017-05-02T11:08:53Z")

</div>

Thanks for the very quick replies!

Both were along the right lines. It's now working.. 😃👍

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 30, 2017, 11:08am UTC](https://discuss.elastic.co/t/defining-a-custom-es-index-based-on-an-existing-field/84244/5 "2017-05-30T11:08:53Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.