# How should I index the 100 billion log

**URL:** <https://discuss.elastic.co/t/how-should-i-index-the-100-billion-log/129751>\
**Category:** Logstash\
**Created:** [April 26, 2018, 11:17pm UTC](https://discuss.elastic.co/t/how-should-i-index-the-100-billion-log/129751 "2018-04-26T23:17:26Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![woon\_minika](https://avatars.discourse-cdn.com/v4/letter/w/f475e1/32.png) [@woon\_minika](https://discuss.elastic.co/u/woon_minika)\
**Post date:** [April 26, 2018, 11:17pm UTC](https://discuss.elastic.co/t/how-should-i-index-the-100-billion-log/129751/1 "2018-04-26T23:17:26Z")

</div>

Hi,

I have multi services which generating at least 2 million of log message per day, I’m using logstash to form a doc and I create new index for every new day log, but I’m feeling it should have better way to handle it? Should I put all the log into one index regardless how many day? And make it 30 shards

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [April 27, 2018, 2:45am UTC](https://discuss.elastic.co/t/how-should-i-index-the-100-billion-log/129751/2 "2018-04-27T02:45:26Z")

</div>

> [@woon\_minika](#):
>
> Should I put all the log into one index regardless how many day?

No, use time based indices.

---

<div class="post-metadata">

**Author:** ![woon\_minika](https://avatars.discourse-cdn.com/v4/letter/w/f475e1/32.png) [@woon\_minika](https://discuss.elastic.co/u/woon_minika)\
**Post date:** [April 27, 2018, 3:40am UTC](https://discuss.elastic.co/t/how-should-i-index-the-100-billion-log/129751/3 "2018-04-27T03:40:42Z")

</div>

Hi Markolm,

If I just want to have one month logs rotation, Should my logstash ouput like this :  
elasticsearch {  
hosts =\> "127.0.0.1:9200"  
manage\_template =\> false  
index =\> "appslog-%{+dd}"  
}

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [April 27, 2018, 3:48am UTC](https://discuss.elastic.co/t/how-should-i-index-the-100-billion-log/129751/4 "2018-04-27T03:48:20Z")

</div>

Almost, you just need to adjust the time parameter, see [https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-index](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-index)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 25, 2018, 3:48am UTC](https://discuss.elastic.co/t/how-should-i-index-the-100-billion-log/129751/5 "2018-05-25T03:48:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.