How to build indexes for Object Storage Statistics collection

Hi Friends,

Newbie to ELK, so patience with me.

I have a grid product (Object storage) which is a cluster that is compound of nodes.
It is based on nodes (server with disks) grouped into Zones, and the data is stored using object policies with replicas.

The cluster has a HTTP API to where I can get its info & stats using HTTP calls & get json replies, which are easy to push into elasticsearch. The problem is that the product doesn't have historical logging, and I want to use elastic-search to store (mainly statistics historically) that data over time & be able to present it using Kibana.

The bricks that I can get via the HTTP calls are:
0 - Cluster info
1 - Node info
2 - Zones
3 - Policies info
4 - Statistics summary cluster (sum of all nodes)
5 - Statistics detailed cluster (sum of all nodes)
6 - Statistics detailed per node

Most will remain static, (will check updates daily) and what is the most dynamic and I plan to collect every 5min is #6.

my plan is to wget/curl the per node statistics every 5min (returns json output, see below) and push it into elasticsearch.

Given the above, how would you recommend I build the indexes, so that I can easily use Kibana to build a nice dashboard for historical table/graphic view ?


wget -q --user=XXX --password=XXX http://10.12.11.51:8088/mgmt/statistics_detail?address=10.12.11.51 -O - | /usr/bin/python -mjson.tool > statistics_detail_single

{
"avgDelLatency": [
0,
"ms"
],
"avgGetLatency": [
0,
"ms"
],
"avgPutLatency": [
0,
"ms"
],
"fileDeletesPerSec": [
0,
"FDPS"
],
"fileReadsPerSec": [
0,
"FRPS"
],
"fileWritesPerSec": [
0,
"FWPS"
],
"getThroughput": [
0.0,
"MB/s"
],
"putThroughput": [
0.0,
"MB/s"
],
"totalDelFailureCount": [
0,
"requests"
],
"totalDelSuccessCount": [
0,
"requests"
],
"totalGetFailureCount": [
0,
"requests"
],
"totalGetSuccessCount": [
0,
"requests"
],
"totalPutFailureCount": [
0,
"requests"
],
"totalPutSuccessCount": [
66761,
"requests"
]
}

I'd just create daily/weekly indices and put it in as time based data.

Hi Mark,

What do you mean by indices ? Indexs ?
Create a Index for a day/week, and than each time-stamp will be an entry ?
But Say I have 4 nodes, and for each node I need to collect, so I create 4 x Day indexes ?

I thought to make an index for each node, and than each entry will have the statistics, and a parameter in the statistic json would be the date/time, no ?

Do you have any sample configuration implementation which I can review & get an idea ?

Thanks !
Ran

Indices = more than one index.

Yes.

Depends, is the datas structure the same for all 4? If so then just one index is fine.