Questions on system.diskio latency metrics

Elastic Stack Beats
1

Hello, I am trying to get a better understanding of the disk io metrics below.
Included are the definitions for each per the Elastic documentation online.

system.diskio.read.time - total number of milliseconds spent by all reads
system.diskio.write.time - total number of milliseconds spent by all writes
system.diskio.iostat.await - average time spent for requests issued to the device to be served
system.diskio.iostat.service_time - average service time (in milliseconds) for I/O requests that were issued to the device

Q1: Is system.diskio.io.time equal to the average of system.diskio.read.time and system.diskio.write.time for a given time interval on a given device?

Q2: How does system.diskio.io.time relate to system.diskio.iostat.await?

Q3: Which metric is best to use to represent overall disk latencies - both reads and writes?

Thank you.

2

  1. According to the kernel docs, time is "number of milliseconds spent doing I/Os". Based on my understanding, these fields measure I/O time differently, so read.time + write.time != io.time. There's a good summary here..

  2. time is time spent in I/O operations, await is time spent waiting for I/O requests to be serviced. In theory, these two metrics should have no overlap.

  3. Personally, I would use diskio.read and diskio.write as they would provide a fairly realistic measure individual IO times, but it depends on what your use case is.