Measuring response time from ES cluster

Hi all,

I tried googling, but didn't see a satisfactory answer.

We're trying to measure the execution time of our ES queries against an ES
cluster. Measuring on the cluster isn't a good option in our case. So we'd
like to do it through our java TransportClient. I don't however know how to
wrap the Runnable or Callable that gets sent to the thread pool with code
that does any stat collection. There are two ways in which this could be
done:

  1. Modify the process by which the runnable is constructed.
  2. Modify the thread pool object or the thread factory associated with the
    thread pool.

Is there any way to do either of these? Going through the documentation and
code didn't really get me anywhere. I didn't see any way to use guice
injection to get me what I wanted. Can anyone offer any suggestions on how
we might collect time statistics?

Sunil

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi Sunil,

I'm not sure I understand what exactly you are trying to time... ES has a
stats API, but presumably you know this and it doesn't meet the needs for
some reason? Can you explain a bit what you are trying to get to, at what
level, and why?

Thanks,
Otis

ELASTICSEARCH Performance Monitoring - Sematext Monitoring | Infrastructure Monitoring Service

On Tuesday, April 16, 2013 8:50:34 PM UTC-4, Sunil Mishra wrote:

Hi all,

I tried googling, but didn't see a satisfactory answer.

We're trying to measure the execution time of our ES queries against an ES
cluster. Measuring on the cluster isn't a good option in our case. So we'd
like to do it through our java TransportClient. I don't however know how to
wrap the Runnable or Callable that gets sent to the thread pool with code
that does any stat collection. There are two ways in which this could be
done:

  1. Modify the process by which the runnable is constructed.
  2. Modify the thread pool object or the thread factory associated with the
    thread pool.

Is there any way to do either of these? Going through the documentation
and code didn't really get me anywhere. I didn't see any way to use guice
injection to get me what I wanted. Can anyone offer any suggestions on how
we might collect time statistics?

Sunil

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I add to Otis's answer that for each SearchResponse, you have access to 'took' field which gives you the total duration before sending back the response.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 17 avr. 2013 à 05:54, Otis Gospodnetic otis.gospodnetic@gmail.com a écrit :

Hi Sunil,

I'm not sure I understand what exactly you are trying to time... ES has a stats API, but presumably you know this and it doesn't meet the needs for some reason? Can you explain a bit what you are trying to get to, at what level, and why?

Thanks,
Otis

ELASTICSEARCH Performance Monitoring - Sematext Monitoring | Infrastructure Monitoring Service

On Tuesday, April 16, 2013 8:50:34 PM UTC-4, Sunil Mishra wrote:

Hi all,

I tried googling, but didn't see a satisfactory answer.

We're trying to measure the execution time of our ES queries against an ES cluster. Measuring on the cluster isn't a good option in our case. So we'd like to do it through our java TransportClient. I don't however know how to wrap the Runnable or Callable that gets sent to the thread pool with code that does any stat collection. There are two ways in which this could be done:

  1. Modify the process by which the runnable is constructed.
  2. Modify the thread pool object or the thread factory associated with the thread pool.

Is there any way to do either of these? Going through the documentation and code didn't really get me anywhere. I didn't see any way to use guice injection to get me what I wanted. Can anyone offer any suggestions on how we might collect time statistics?

Sunil

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi Otis, David, thanks for the pointers! We would be able to use the time
taken in the result (I had forgotten about that) as an approximation, but
ideally we'd be able to get the time taken statistics from the java client
directly. Which would include network time. We're setting up a monitoring
system for a server we're developing, and the time taken for Elasticsearch
calls is one of the stats we want to track, independent from the stats for
the ES cluster.

I didn't know about the stats API. I googled and found stats api to get
information about nodes, but not client calls. Could you give me a pointer?

Thanks in advance!

Sunil

On Tuesday, April 16, 2013 5:50:34 PM UTC-7, Sunil Mishra wrote:

Hi all,

I tried googling, but didn't see a satisfactory answer.

We're trying to measure the execution time of our ES queries against an ES
cluster. Measuring on the cluster isn't a good option in our case. So we'd
like to do it through our java TransportClient. I don't however know how to
wrap the Runnable or Callable that gets sent to the thread pool with code
that does any stat collection. There are two ways in which this could be
done:

  1. Modify the process by which the runnable is constructed.
  2. Modify the thread pool object or the thread factory associated with the
    thread pool.

Is there any way to do either of these? Going through the documentation
and code didn't really get me anywhere. I didn't see any way to use guice
injection to get me what I wanted. Can anyone offer any suggestions on how
we might collect time statistics?

Sunil

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi,

Couldn't you always use Coda Hale's metrics library (assuming your client
app is written in Java) and wrap your calls to ES with a timer?

Otis

ELASTICSEARCH Performance Monitoring - Sematext Monitoring | Infrastructure Monitoring Service

On Wednesday, April 17, 2013 9:45:48 AM UTC-4, Sunil Mishra wrote:

Hi Otis, David, thanks for the pointers! We would be able to use the time
taken in the result (I had forgotten about that) as an approximation, but
ideally we'd be able to get the time taken statistics from the java client
directly. Which would include network time. We're setting up a monitoring
system for a server we're developing, and the time taken for Elasticsearch
calls is one of the stats we want to track, independent from the stats for
the ES cluster.

I didn't know about the stats API. I googled and found stats api to get
information about nodes, but not client calls. Could you give me a pointer?

Thanks in advance!

Sunil

On Tuesday, April 16, 2013 5:50:34 PM UTC-7, Sunil Mishra wrote:

Hi all,

I tried googling, but didn't see a satisfactory answer.

We're trying to measure the execution time of our ES queries against an
ES cluster. Measuring on the cluster isn't a good option in our case. So
we'd like to do it through our java TransportClient. I don't however know
how to wrap the Runnable or Callable that gets sent to the thread pool with
code that does any stat collection. There are two ways in which this could
be done:

  1. Modify the process by which the runnable is constructed.
  2. Modify the thread pool object or the thread factory associated with
    the thread pool.

Is there any way to do either of these? Going through the documentation
and code didn't really get me anywhere. I didn't see any way to use guice
injection to get me what I wanted. Can anyone offer any suggestions on how
we might collect time statistics?

Sunil

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

The TransportClient returns an ActionFuture. Timing the execution of the
Future itself isn't particularly meaningful. We should really be timing the
underlying function that's executed asynchronously. I haven't been able to
find a way to influence either how that function is constructed, or modify
the threadpool that executes it. I took a look at Coda Hale's library, and
while it gives lots of mechanisms for collecting statistics, I didn't see
how it might be able to hook into an arbitrary method's execution in the
JVM. Even if we could do that, I wouldn't know which method to hook into.

If it makes a difference, our server's being implemented in scala.

Thanks,

Sunil

On Wednesday, April 17, 2013 5:35:27 PM UTC-7, Otis Gospodnetic wrote:

Hi,

Couldn't you always use Coda Hale's metrics library (assuming your client
app is written in Java) and wrap your calls to ES with a timer?

Otis

ELASTICSEARCH Performance Monitoring - Sematext Monitoring | Infrastructure Monitoring Service

On Wednesday, April 17, 2013 9:45:48 AM UTC-4, Sunil Mishra wrote:

Hi Otis, David, thanks for the pointers! We would be able to use the time
taken in the result (I had forgotten about that) as an approximation, but
ideally we'd be able to get the time taken statistics from the java client
directly. Which would include network time. We're setting up a monitoring
system for a server we're developing, and the time taken for Elasticsearch
calls is one of the stats we want to track, independent from the stats for
the ES cluster.

I didn't know about the stats API. I googled and found stats api to get
information about nodes, but not client calls. Could you give me a pointer?

Thanks in advance!

Sunil

On Tuesday, April 16, 2013 5:50:34 PM UTC-7, Sunil Mishra wrote:

Hi all,

I tried googling, but didn't see a satisfactory answer.

We're trying to measure the execution time of our ES queries against an
ES cluster. Measuring on the cluster isn't a good option in our case. So
we'd like to do it through our java TransportClient. I don't however know
how to wrap the Runnable or Callable that gets sent to the thread pool with
code that does any stat collection. There are two ways in which this could
be done:

  1. Modify the process by which the runnable is constructed.
  2. Modify the thread pool object or the thread factory associated with
    the thread pool.

Is there any way to do either of these? Going through the documentation
and code didn't really get me anywhere. I didn't see any way to use guice
injection to get me what I wanted. Can anyone offer any suggestions on how
we might collect time statistics?

Sunil

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Stopwatch watch = new Stopwatch();
watch.start();
//build your request
*.execute().actionGet();
watch.stop();
log("Response took: {} ms", watch.elapsed(TimeUnit.MILLISECONDS));

On Thursday, April 18, 2013 5:49:40 AM UTC+4, Sunil Mishra wrote:

The TransportClient returns an ActionFuture. Timing the execution of the
Future itself isn't particularly meaningful. We should really be timing the
underlying function that's executed asynchronously. I haven't been able to
find a way to influence either how that function is constructed, or modify
the threadpool that executes it. I took a look at Coda Hale's library, and
while it gives lots of mechanisms for collecting statistics, I didn't see
how it might be able to hook into an arbitrary method's execution in the
JVM. Even if we could do that, I wouldn't know which method to hook into.

If it makes a difference, our server's being implemented in scala.

Thanks,

Sunil

On Wednesday, April 17, 2013 5:35:27 PM UTC-7, Otis Gospodnetic wrote:

Hi,

Couldn't you always use Coda Hale's metrics library (assuming your client
app is written in Java) and wrap your calls to ES with a timer?

Otis

ELASTICSEARCH Performance Monitoring - Sematext Monitoring | Infrastructure Monitoring Service

On Wednesday, April 17, 2013 9:45:48 AM UTC-4, Sunil Mishra wrote:

Hi Otis, David, thanks for the pointers! We would be able to use the
time taken in the result (I had forgotten about that) as an approximation,
but ideally we'd be able to get the time taken statistics from the java
client directly. Which would include network time. We're setting up a
monitoring system for a server we're developing, and the time taken for
Elasticsearch calls is one of the stats we want to track, independent from
the stats for the ES cluster.

I didn't know about the stats API. I googled and found stats api to get
information about nodes, but not client calls. Could you give me a pointer?

Thanks in advance!

Sunil

On Tuesday, April 16, 2013 5:50:34 PM UTC-7, Sunil Mishra wrote:

Hi all,

I tried googling, but didn't see a satisfactory answer.

We're trying to measure the execution time of our ES queries against an
ES cluster. Measuring on the cluster isn't a good option in our case. So
we'd like to do it through our java TransportClient. I don't however know
how to wrap the Runnable or Callable that gets sent to the thread pool with
code that does any stat collection. There are two ways in which this could
be done:

  1. Modify the process by which the runnable is constructed.
  2. Modify the thread pool object or the thread factory associated with
    the thread pool.

Is there any way to do either of these? Going through the documentation
and code didn't really get me anywhere. I didn't see any way to use guice
injection to get me what I wanted. Can anyone offer any suggestions on how
we might collect time statistics?

Sunil

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.