Which index is receiving data and where is that configured

Elastic Stack Kibana
1

I am a newbie to ELK, and this is very probably a n00b question.
I have set up ELK, and those are currently the indices that I have on my host:

curl 'http://localhost:9200/_cat/indices?v'
health status index                            uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   filebeat-2020.03.10              9vYa2M9QR1-sBMtDnIaekw   1   1       1666            0    839.7kb        839.7kb
yellow open   logstash-2020.03.03-000001       035LbhSGRv-_yAW9UNV4Xw   1   1       2842            0      1.5mb          1.5mb
yellow open   test                             79GZ8OKpQzOWrEanGL1Z4w   1   1          0            0       283b           283b
green  open   .kibana_task_manager_1           ca5hQQYIT62JKee4euSSHQ   1   0          2            2     20.8kb         20.8kb
green  open   .apm-agent-configuration         MAAyG6-zSWWl61C_0ne43w   1   0          0            0       283b           283b
green  open   ilm-history-1-000001             JcOJ8kX5TIWcPd1kSRCy3w   1   0         27            0     34.3kb         34.3kb
yellow open   filebeat-7.6.1-2020.03.10-000001 pvS64-dvSmekPbJNV8xZXQ   1   1     173561            0       45mb           45mb
green  open   .kibana_1                        oQN_ZPreTaifBdHEbq6paQ   1   0         25            8    227.7kb        227.7kb
yellow open   filebeat-2020.03.09              8hQ2yayeQuqanX8lRuVrKw   1   1      12363            0      7.1mb          7.1mb

I am using filebeat as you see, that sends data to logstash.
I do not understand where those indexes are configured. For instance, it seems that the latest index is filebeat-7.6.1-2020.03.10-000001 and that the other ones are old (temporary tests). But my logstash configuration file has this config (below) and you can see it does not include the filebeat version: my index pattern should rather be filebeat-2020.03.13.

elasticsearch {
            hosts => ["localhost:9200"]
	    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
        }
  • Can I safely delete the other filebeat indexes? (Update: I know how to delete indexes)
  • How can I know which index is currently receiving data? With the previous command, I just see that filebeat-7.6.1-2020.03.10-000001 has the most data. Would it be possible to see how long ago it received data?
  • How can I view the current content of a given index using CLI? Is there a URL to grab for instance the last few records it got?
  • And of course, why isn't the name of my index filebeat-2020... given my /etc/logstash/conf.d/logstash.conf?

Thanks for your help!
PS. I am using version 7.6.1 on Linux.

2

I can give you some pointers there, but this is really not a Kibana question as Kibana is not involved anywhere there. The Logstash subforum or Beats one seem like better choices.
For the second and third question you can solve those easily by using Kibana.
Just create an index pattern in Kibana that matches and index and then in Discover you can see all documents in that index.

1 Like
3

okay thanks, I'll post to logstash. Thanks!

4

Look at the ILM section here, the default for 7.x is to use ILM unless you turn it off. That's why you are getting the ILM style indices.

1 Like
5

Oh nice! Thanks, indeed, it was there by default... so in that case, the index name in my logstash configuration is purely ignored ?

elasticsearch {
            hosts => ["localhost:9200"]
	    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
        }
6

Yes, from the doc

The index setting is ignored when index lifecycle management is enabled

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.