ES and Kibana 5.2.2 how to use timelion to view data size ingestion per day

Elastic Stack Kibana
1

Hi all, Is there a way for me to visualize in Timelion the actual data size of the total indexes being ingested per day?

.es(index=app_log*) shows total count. In the legend I see 'q:* > count'

Can someone please provide a sample query? I see references to store.size and pri.store.size but how do I use that in the query field in Timelion.

Help would be greatly appreciated. Is there a tutorial of some kind for the various types of queries I can run?

Version (yes its old): 5.2.2.

Thanks

2

Hi, I don't think it is possible to get the disk size of indices ingested per day, as that's not a piece of information stored anywhere. As you mentioned, it is possible to track the count of documents rising over time.

If you really need this information, one way I could think of doing this is by creating a script that checks _cat/indices that runs at midnight everyday and creates a new document in a index that could be called indices_stats. Of course, to get the increase in size of the indices per day, you will need to subtract the previous day's size. Then you can use indices_stats to visualize the size of your indices over time.

3

Hi Jen,

hm...ok thanks for this info. Appreciate the quick response.

Thanks!

4

Hi, I have been looking into this further, and found that ES does in fact track index sizes.

GET /_cat/indices?v&h=i,tm&s=tm:desc

is there a way to graph this? so far I have been unsuccessful. I cant figure out where to even add metrics under visualization or timelion.

5

Hi, sorry - that is what I was getting at. That API endpoint will retrieve the current size on disk, but it is not tracked through time. The history is not persisted anywhere. Visualizations and Timelion works from data stored in your ES indices only.

That's why my suggestion was to create a custom script that indexes data from that endpoint periodically, so that you are able to track the size through an ES index.

6

Hi Jen... I see got it, thanks for clarifying!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.