# Trying to understand "indexes" and fields to store logs

**URL:** <https://discuss.elastic.co/t/trying-to-understand-indexes-and-fields-to-store-logs/182370>\
**Category:** Elasticsearch\
**Created:** [May 23, 2019, 7:50am UTC](https://discuss.elastic.co/t/trying-to-understand-indexes-and-fields-to-store-logs/182370 "2019-05-23T07:50:12Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Marteyboy](https://avatars.discourse-cdn.com/v4/letter/m/3e96dc/32.png) [@Marteyboy](https://discuss.elastic.co/u/Marteyboy)\
**Post date:** [May 23, 2019, 7:50am UTC](https://discuss.elastic.co/t/trying-to-understand-indexes-and-fields-to-store-logs/182370/1 "2019-05-23T07:50:12Z")

</div>

Hi,

I'm new to Elasticsearch and I'm trying to figure out the best practices on storing logs.

What I've understood it's good to have one index for all logs and increment the index name by day, weeks, months, whatever suits (logs-2019.05.21). But now what I'm confused about, is when there are many different log sources, there are many different fields.

For example, lets say these are the logs and fields:  
cloudfront-access-log: status, ip **, referrer, time\_taken, location**  
apigateway-access-log: status, ip **, user, stage, exection\_time**

In this case, both have couple similar fields, but also completely different fields. Now let's say there are many more logs with many more fields that one has and the others don't. Should I be concerned about it at all? What is the best way to think about this?

I'm trying to keep in mind best practices and make sure there are no performance bottleneck from this step when scaling up.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [May 23, 2019, 10:19pm UTC](https://discuss.elastic.co/t/trying-to-understand-indexes-and-fields-to-store-logs/182370/2 "2019-05-23T22:19:05Z")

</div>

Welcome 😃

> [@Marteyboy](#):
>
> What I've understood it's good to have one index for all logs

Depends, if they are the same sort of log, then yep. If they are totally different formats then best to separate em.

> [@Marteyboy](#):
>
> nd increment the index name by day, weeks, months, whatever suits (logs-2019.05.21)

> [@Marteyboy](#):
>
> and increment the index name by day, weeks, months, whatever suits (logs-2019.05.21)

Yep! Or we'd probably recommend using [index lifecycle management](https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html) these days, it does the same thing but is much easier to manage and has heaps of other benefits.

> [@Marteyboy](#):
>
> Now let's say there are many more logs with many more fields that one has and the others don't. Should I be concerned about it at all? What is the best way to think about this?

Yep. That goes back to my first comment. Group similar things and avoid having too much disparity in the one index.

---

<div class="post-metadata">

**Author:** ![Marteyboy](https://avatars.discourse-cdn.com/v4/letter/m/3e96dc/32.png) [@Marteyboy](https://discuss.elastic.co/u/Marteyboy)\
**Post date:** [May 24, 2019, 10:13am UTC](https://discuss.elastic.co/t/trying-to-understand-indexes-and-fields-to-store-logs/182370/3 "2019-05-24T10:13:13Z")

</div>

Thank you! That clears it up for me.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 21, 2019, 10:13am UTC](https://discuss.elastic.co/t/trying-to-understand-indexes-and-fields-to-store-logs/182370/4 "2019-06-21T10:13:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.