# Output configuration question

**URL:** <https://discuss.elastic.co/t/output-configuration-question/121186>\
**Category:** Logstash\
**Created:** [February 23, 2018, 9:04am UTC](https://discuss.elastic.co/t/output-configuration-question/121186 "2018-02-23T09:04:30Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![elkstarter](https://avatars.discourse-cdn.com/v4/letter/e/22d042/32.png) [@elkstarter](https://discuss.elastic.co/u/elkstarter)\
**Post date:** [February 23, 2018, 9:04am UTC](https://discuss.elastic.co/t/output-configuration-question/121186/1 "2018-02-23T09:04:30Z")

</div>

Hi everyone,

I am curious about the code described [here](https://www.elastic.co/guide/en/beats/filebeat/5.6/logstash-output.html#CO3-1) :

```auto
output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" 
    document_type => "%{[@metadata][type]}" 
  }
}
```

I've run this config for a year now and this is creating a lot of indexes (one or more per day).  
Is this supposed to be a code example to illustrate the possibilities offered in output, or the best way to organize the data ?  
Should I have simply created a single index like this ?

```auto
index => "%{[@metadata][beat]}"
```

Thanks for your help!

---

<div class="post-metadata">

**Author:** ![rcowart](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rcowart/32/88091_2.png) [@rcowart](https://discuss.elastic.co/u/rcowart)\
**Post date:** [February 23, 2018, 9:15am UTC](https://discuss.elastic.co/t/output-configuration-question/121186/2 "2018-02-23T09:15:32Z")

</div>

How large are your daily indicies? This will be the determining factor. I would not remove the date element completely as eventually the indicies will become too large, and deleting data to make space will be a more complex process. However, you may be able to move from daily to monthly indicies if your current daily indicies are really small (e.g. \< 1GB).

---

<div class="post-metadata">

**Author:** ![elkstarter](https://avatars.discourse-cdn.com/v4/letter/e/22d042/32.png) [@elkstarter](https://discuss.elastic.co/u/elkstarter)\
**Post date:** [February 23, 2018, 9:33am UTC](https://discuss.elastic.co/t/output-configuration-question/121186/3 "2018-02-23T09:33:02Z")

</div>

> [@rcowart](#):
>
> How large are your daily indicies?

This is a good question, I think they may be \<100mb / day.  
Is there quick way to get this info (like average size) ?

---

<div class="post-metadata">

**Author:** ![rcowart](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rcowart/32/88091_2.png) [@rcowart](https://discuss.elastic.co/u/rcowart)\
**Post date:** [February 23, 2018, 9:37am UTC](https://discuss.elastic.co/t/output-configuration-question/121186/4 "2018-02-23T09:37:51Z")

</div>

You can access data about the indicies by querying the REST API. In Kibana go to Dev Tools -\> Console and execute the query:

```auto
GET _cat/indices?v

```

---

<div class="post-metadata">

**Author:** ![elkstarter](https://avatars.discourse-cdn.com/v4/letter/e/22d042/32.png) [@elkstarter](https://discuss.elastic.co/u/elkstarter)\
**Post date:** [February 23, 2018, 10:11am UTC](https://discuss.elastic.co/t/output-configuration-question/121186/5 "2018-02-23T10:11:32Z")

</div>

It seems I was wrong about the size. My indices are between 200mb and 500mb / day

EDIT: I did a quick average on a year on my two outputs.  
The first one is approx sending 20mb/day, but the second one is around 350mb/day.

I think i'll follow your input and change how the first is configured to something like

```auto
"%{[@metadata][beat]}-%{+YYYY.MM}"
```

---

<div class="post-metadata">

**Author:** ![rcowart](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rcowart/32/88091_2.png) [@rcowart](https://discuss.elastic.co/u/rcowart)\
**Post date:** [February 23, 2018, 10:41am UTC](https://discuss.elastic.co/t/output-configuration-question/121186/6 "2018-02-23T10:41:16Z")

</div>

That is pretty small. I if changed to monthly indices, by changing this...

```auto
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"

```

to this...

```auto
index => "%{[@metadata][beat]}-%{+YYYY.MM}"

```

You will reduce the number of indices significantly. The only downside is that if you want to drop old data by simply deleting old indices, you would have to drop a full month at a time, but this is likely not an issue in your case.

---

<div class="post-metadata">

**Author:** ![elkstarter](https://avatars.discourse-cdn.com/v4/letter/e/22d042/32.png) [@elkstarter](https://discuss.elastic.co/u/elkstarter)\
**Post date:** [February 23, 2018, 11:08am UTC](https://discuss.elastic.co/t/output-configuration-question/121186/7 "2018-02-23T11:08:50Z")

</div>

Thanks for your help, i'll think about it.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 23, 2018, 11:09am UTC](https://discuss.elastic.co/t/output-configuration-question/121186/8 "2018-03-23T11:09:10Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.