To compute ingestion rate to elasticsearch cluster

[Varun_S]

Hi Team,

We are using fluentd to send data to Elasticsearch and fluentd is transmitting data to Elasticsearch using the HTTP layer. We want to compute the ingestion rate to the Elasticsearch cluster, but there are no metrics available to get the ingestion rate to an Elasticsearch cluster. Kindly please help here to compute the ingestion rate to the Elasticsearch cluster.

[warkolm (Mark Walkom)]

Check out Elasticsearch Monitoring Metrics | Kibana Guide [7.16] | Elastic and Monitor a cluster | Elasticsearch Guide [7.16] | Elastic

[Varun_S]

Hi @warkolm ,

Thanks for your reply, In the link that is shared there are no metrics related to ingestion rate. Can you please help to get the ingestion rate metrics.

Regards,
Varun S

[FALEN (Ali Can Saykal)]

Do you want to see Event Per Second EPS?

Stack Monitoring>Logstash>Pipelines>

This is a part of stack monitoring feature that you need to enable first

Monitor a cluster | Elasticsearch Guide [7.16] | Elastic

[Varun_S]

Hi @FALEN,

Thanks for your reply, We want to see at what rate data is getting ingested into Elasticsearch and the Elasticsearch used here is not enterprise edition.

Regards,
Varun S

[FALEN (Ali Can Saykal)]

It seems you can do this in basic license too, have you tried?

--EDIT--

this might block you, im not sure, didnt tested it with basic license : "centralized pipeline management"

You can also try visualizations to create an EPS dashboard

1. Go to Panel Options > Index pattern and choose the right index pattern
2. Go back to Data and configure Count, then Math, then type params.count / params._interval * 1000
3. You'll start seeing average values per second
4. Switch to Metric to reduce this to a single number
5. Important : Go back to Panel Options > Data timerange mode > Entire time range

source: Viewing EPS graph in Kibana - #9 by Falikou1

[philippkahr]

Hi

if you use Stack monitoring whether legacy or using metricbeat you will see this data in the free and open stack as well. Just click on Stack Monitoring in Kibana and it will walk you through the process of setting things up.