# New index file for every log file getting transferred to ElasticSearch

**URL:** <https://discuss.elastic.co/t/new-index-file-for-every-log-file-getting-transferred-to-elasticsearch/164422>\
**Category:** Elasticsearch\
**Created:** [January 16, 2019, 9:08am UTC](https://discuss.elastic.co/t/new-index-file-for-every-log-file-getting-transferred-to-elasticsearch/164422 "2019-01-16T09:08:19Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![hemenkotecha](https://avatars.discourse-cdn.com/v4/letter/h/34f0e0/32.png) [@hemenkotecha](https://discuss.elastic.co/u/hemenkotecha)\
**Post date:** [January 16, 2019, 9:08am UTC](https://discuss.elastic.co/t/new-index-file-for-every-log-file-getting-transferred-to-elasticsearch/164422/1 "2019-01-16T09:08:19Z")

</div>

We use elasticsearch for the EMC Networker Log analysis.. So we transfer Networker server daemon logs to ES.. These are log files are generally huge in size... So for the better analysis perspective, we need new indices for every log that we transfer to Elastic-search.  
How to achieve this ?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [January 16, 2019, 9:12am UTC](https://discuss.elastic.co/t/new-index-file-for-every-log-file-getting-transferred-to-elasticsearch/164422/2 "2019-01-16T09:12:10Z")

</div>

How large are the files? How many are generated per day?

Please be aware that [having lots of small indices and shards is inefficient and can cause performance problems](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster).

---

<div class="post-metadata">

**Author:** ![hemenkotecha](https://avatars.discourse-cdn.com/v4/letter/h/34f0e0/32.png) [@hemenkotecha](https://discuss.elastic.co/u/hemenkotecha)\
**Post date:** [January 16, 2019, 9:24am UTC](https://discuss.elastic.co/t/new-index-file-for-every-log-file-getting-transferred-to-elasticsearch/164422/3 "2019-01-16T09:24:44Z")

</div>

File size couldvary from 500MB to 40+GB ... mostly 4 to 5 logs everyday.

I am fully aware bout the possible performance issues.  
As of now our focus is to analyze micro details of each log files and single index set for multiple log files would create confusion hence we wanted solution that way.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [January 16, 2019, 9:30am UTC](https://discuss.elastic.co/t/new-index-file-for-every-log-file-getting-transferred-to-elasticsearch/164422/4 "2019-01-16T09:30:59Z")

</div>

Why not just extract the file name into a separate field and let the user filter on this?

What does your ingest architecture look like?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 13, 2019, 9:38am UTC](https://discuss.elastic.co/t/new-index-file-for-every-log-file-getting-transferred-to-elasticsearch/164422/5 "2019-02-13T09:38:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.