# Every log line sent to ES create new index

**URL:** <https://discuss.elastic.co/t/every-log-line-sent-to-es-create-new-index/318340>\
**Category:** Logs\
**Created:** [November 7, 2022, 2:33pm UTC](https://discuss.elastic.co/t/every-log-line-sent-to-es-create-new-index/318340 "2022-11-07T14:33:59Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![florind](https://avatars.discourse-cdn.com/v4/letter/f/5f8ce5/32.png) [@florind](https://discuss.elastic.co/u/florind)\
**Post date:** [November 7, 2022, 2:33pm UTC](https://discuss.elastic.co/t/every-log-line-sent-to-es-create-new-index/318340/1 "2022-11-07T14:33:59Z")

</div>

Hello,  
I'm using fluentd to send pod logs to ES 8.5

My problem is that each log sent is creating a new index, so i end up having a very large number of indices.

What I'd like to have is one index per day, and keep them for only 7 days.

this is the config i have for the index name:  
index\_name odc-dev-new

And what i see in kibana are a lot of indexes created like this:

odc-dev-new-1970.02.10  
odc-dev-new-1970.02.23  
odc-dev-new-1970.03.12

Another problem is that if i setup in Discover an view for odc-dev-new\* i don't see any info there.

I'm not sure what I'm missing in my configuration.

Could you please help with a suggestion?

thank you,  
Florin

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [November 7, 2022, 2:41pm UTC](https://discuss.elastic.co/t/every-log-line-sent-to-es-create-new-index/318340/2 "2022-11-07T14:41:44Z")

</div>

If you are sending data using fluentd, this issue is related to fluentd.

Something in fluentd config is making it write to indices with the date in the name, you need to check with the fluentd community.

From what you shared, if you have indices with date of `1970-*`, you probably have some error parsing the data in epoch.

> [@florind](#):
>
> Another problem is that if i setup in Discover an view for odc-dev-new\* i don't see any info there.

You have indices with `1970-*` in the nane, try to change the time range in discover to show logs from this date.

---

<div class="post-metadata">

**Author:** ![florind](https://avatars.discourse-cdn.com/v4/letter/f/5f8ce5/32.png) [@florind](https://discuss.elastic.co/u/florind)\
**Post date:** [November 7, 2022, 2:53pm UTC](https://discuss.elastic.co/t/every-log-line-sent-to-es-create-new-index/318340/3 "2022-11-07T14:53:04Z")

</div>

hello,  
thank you for the very prompt reply.  
this is my config to send the logs to ES:  
  
@type tail  
#@log\_level debug  
path /var/log/containers/_.log  
exclude\_path /var/log/containers/fluent.log  
pos\_file /tmp/es-containers.log.pos  
pos\_file\_compaction\_interval 72h  
tag kubernetes._  
read\_from\_head false  
format json  
refresh\_interval 3  
time\_format %Y-%m-%dT%H:%M:%S.%sZ

```
<filter kubernetes.**>
  @type stdout
  output_type json
  key_name log
  time_key time
</filter>

```

thank you,  
Florin

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [November 7, 2022, 2:59pm UTC](https://discuss.elastic.co/t/every-log-line-sent-to-es-create-new-index/318340/4 "2022-11-07T14:59:58Z")

</div>

I do not use fluentd and cannot help further.

As I said, your issue is related on how fluentd is sending data to Elasticsearch, you need to check this issue with the Fluentd community.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 5, 2022, 3:00pm UTC](https://discuss.elastic.co/t/every-log-line-sent-to-es-create-new-index/318340/5 "2022-12-05T15:00:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.