Elasticsearch Latency Metrics

[animageofmine (Ankit Parikh)]

We are trying to set up monitoring for Elasticsearch using statsd (not using Kibana). We also enabled slow logs:

curl  -XPUT 'localhost:9200/_template/template_1?pretty' -d'
    {
      "template": "*",
      "settings": {
          "index.indexing.slowlog.level" : "info",
          "index.indexing.slowlog.source" : "1000",
          "index.indexing.slowlog.threshold.index.debug" : "0ms",
          "index.indexing.slowlog.threshold.index.info" : "0ms",
          "index.indexing.slowlog.threshold.index.trace" : "0ms",
          "index.indexing.slowlog.threshold.index.warn" : "0ms",
          "index.search.slowlog.threshold.fetch.debug" : "0ms",
          "index.search.slowlog.threshold.fetch.info" : "0ms",
          "index.search.slowlog.threshold.fetch.trace" : "0ms",
          "index.search.slowlog.threshold.fetch.warn" : "0ms",
          "index.search.slowlog.threshold.query.debug" : "0ms",
          "index.search.slowlog.threshold.query.info" : "0ms",
          "index.search.slowlog.threshold.query.trace" : "0ms",
          "index.search.slowlog.threshold.query.warn" : "0ms"
      } 
}

Following are some of the metrics I see logged by statsd and slow logs

Statsd:

indexing_index_time_in_millis value
search_query_time_in_millis value

slow logs:

Search Log:

[2016-12-13T00:11:28,171][WARN ][index.search.slowlog.fetch] [elasticsearch-data1-host] [testfieldset][0] took[364.2micros], took_millis[0], types[], stats[], search_type[QUERY_THEN_FETCH], total_shards[5], source[{
  "ext" : { }
}],

Index Log:

[2016-12-12T22:41:19,850][WARN ][index.indexing.slowlog.index] [elasticsearch-data1-host] [2906a5de043548ca9ffe93b0/IFHyGrjoQzq9XW0O1T9ajQ] took[780.3micros], took_millis[0], type[records], id[AVj1Mohnpy7Iwavve0Dw], routing[]

Questions:

1. I was wondering if these metrics from slow log and statsd are same? i.e. is index.indexing.slowlog.index => took_millis same as indexing_index_time_in_millis and is index.search.slowlog.fetch => took_millis same as search_query_time_in_millis?
2. Assuming answer to 1 is yes, do we have to enable slow logs for elasticsearch to log indexing_index_time_in_millis & search_query_time_in_millis that are later collected by statsd?

Please let me know. Thank you.