Every log line sent to ES create new index

Elastic Observability Logs
1

Hello,
I'm using fluentd to send pod logs to ES 8.5

My problem is that each log sent is creating a new index, so i end up having a very large number of indices.

What I'd like to have is one index per day, and keep them for only 7 days.

this is the config i have for the index name:
index_name odc-dev-new

And what i see in kibana are a lot of indexes created like this:

odc-dev-new-1970.02.10
odc-dev-new-1970.02.23
odc-dev-new-1970.03.12

Another problem is that if i setup in Discover an view for odc-dev-new* i don't see any info there.

I'm not sure what I'm missing in my configuration.

Could you please help with a suggestion?

thank you,
Florin

2

If you are sending data using fluentd, this issue is related to fluentd.

Something in fluentd config is making it write to indices with the date in the name, you need to check with the fluentd community.

From what you shared, if you have indices with date of 1970-*, you probably have some error parsing the data in epoch.

You have indices with 1970-* in the nane, try to change the time range in discover to show logs from this date.

1 Like
3

hello,
thank you for the very prompt reply.
this is my config to send the logs to ES:

@type tail
#@log_level debug
path /var/log/containers/.log
exclude_path /var/log/containers/fluent.log
pos_file /tmp/es-containers.log.pos
pos_file_compaction_interval 72h
tag kubernetes.
read_from_head false
format json
refresh_interval 3
time_format %Y-%m-%dT%H:%M:%S.%sZ

<filter kubernetes.**>
  @type stdout
  output_type json
  key_name log
  time_key time
</filter>

thank you,
Florin

4

I do not use fluentd and cannot help further.

As I said, your issue is related on how fluentd is sending data to Elasticsearch, you need to check this issue with the Fluentd community.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.