# One index or seperate indices for logfiles

**URL:** <https://discuss.elastic.co/t/one-index-or-seperate-indices-for-logfiles/196656>\
**Category:** Elasticsearch\
**Created:** [August 25, 2019, 10:10am UTC](https://discuss.elastic.co/t/one-index-or-seperate-indices-for-logfiles/196656 "2019-08-25T10:10:46Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![lcer00](https://avatars.discourse-cdn.com/v4/letter/l/d2c977/32.png) [@lcer00](https://discuss.elastic.co/u/lcer00)\
**Post date:** [August 25, 2019, 10:10am UTC](https://discuss.elastic.co/t/one-index-or-seperate-indices-for-logfiles/196656/1 "2019-08-25T10:10:46Z")

</div>

Hallo,

I'm implementing elasticsearch for storing log-data from different sources. Its log-data from windows dhcp an nps/radius servers but also syslog data from different sources. I have a logstash instance that filters the data and generates a lot of different fields for each source.

Now I have to decide to put the data into one common index or into different indices. At the moment I have the following indices (using ilm):  
filebeat-dhcp-{now/d}-00001  
filebeat-nps-{now/d}-00001  
filebeat-...….  
logstash-%{[vendor]}-00001

It would be less work for me to use one common index or only one filebeat and one logstash  
index. But this "common" index will contain a large number of fields. Is this a problem for elasticsearch performance? Also I had problems with the 1024-field-limit when upgrading from 6 to 7 so - less fields would be preferable?

Thanks

lcer

---

<div class="post-metadata">

**Author:** ![Emanuil](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/emanuil/32/36783_2.png) [@Emanuil](https://discuss.elastic.co/u/Emanuil)\
**Post date:** [August 25, 2019, 11:56pm UTC](https://discuss.elastic.co/t/one-index-or-seperate-indices-for-logfiles/196656/2 "2019-08-25T23:56:58Z")

</div>

Welcome to the forums!

Using ILM and thinking about your index management are definitely very much on the right track and what we'd recommend doing in such a situation.

> [@lcer00](#):
>
> But this "common" index will contain a large number of fields. Is this a problem for elasticsearch performance?

How many fields are you expecting the common index to have, roughly? Generally I wouldn't have said more fields is such a big problem. I'd have expected 10-20 new fields per log type (dhcp, nps, etc.), with a number of common fields like timestamp. But hitting the 1024 limit is on a different level entirely.

---

<div class="post-metadata">

**Author:** ![lcer00](https://avatars.discourse-cdn.com/v4/letter/l/d2c977/32.png) [@lcer00](https://discuss.elastic.co/u/lcer00)\
**Post date:** [August 26, 2019, 6:30am UTC](https://discuss.elastic.co/t/one-index-or-seperate-indices-for-logfiles/196656/3 "2019-08-26T06:30:49Z")

</div>

Hallo,

> I'd have expected 10-20 new fields per log type (dhcp, nps, etc.), with a number of common fields like timestamp. But hitting the 1024 limit is on a different level entirely.

Yes, it will be about 6-10 types with 10-20 fields. The 1024-problem occured, because I used the preloaded filebeat template - which got larger when I bulk-changed fieldnames.

So - for my setup I should:

- use ILM
- use my own index template (not one that comes with the installation)

With this I can use one common index for all my logging-stuff. Right?

Regards

lcer

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 23, 2019, 6:30am UTC](https://discuss.elastic.co/t/one-index-or-seperate-indices-for-logfiles/196656/4 "2019-09-23T06:30:50Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.