# Logging queries and response times **URL:** \ **Category:** Elasticsearch\ **Created:** [July 5, 2011, 2:51pm UTC](https://discuss.elastic.co/t/logging-queries-and-response-times/4780 "2011-07-05T14:51:16Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Alexander\_Reelsen](https://avatars.discourse-cdn.com/v4/letter/a/85e7bf/32.png) [@Alexander\_Reelsen](https://discuss.elastic.co/u/Alexander_Reelsen)\ **Post date:** [July 5, 2011, 2:51pm UTC](https://discuss.elastic.co/t/logging-queries-and-response-times/4780/1 "2011-07-05T14:51:16Z")
Hi there, by judging my questions from the last few days, you might have noticed I am evaluating elasticsearch for further usage ๐Ÿ™‚ Our current custom implementation has the nice feature of logging incoming queries as well as query times in order to find bottlenecks (or monitoring rising response times with a growing index via munin). Can I change the logging configuration somehow in this direction? Or is there a different way of monitoring something like that? Thanks a lot for all the answers I got so far to my questions, they were pretty helpful! --Alexander ---
**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\ **Post date:** [July 5, 2011, 4:37pm UTC](https://discuss.elastic.co/t/logging-queries-and-response-times/4780/2 "2011-07-05T16:37:32Z")
I'm using the java API where I can use: searchResponse.getTookInMillis() If you are not using a layer between ES and your UI I would suggest that you should do it ๐Ÿ™‚ not only for security reasons ... Regards, Peter. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [July 6, 2011, 1:13am UTC](https://discuss.elastic.co/t/logging-queries-and-response-times/4780/3 "2011-07-06T01:13:04Z")
Yea, you need to do it yourself currently. I do want to build more stats into search and query times, a bit challenging when it comes to distributed systems ๐Ÿ™‚ On Tuesday, July 5, 2011 at 7:37 PM, Karussell wrote: > I'm using the java API where I can use: > searchResponse.getTookInMillis() > > If you are not using a layer between ES and your UI I would suggest > that you should do it ๐Ÿ™‚ > > not only for security reasons ... > > Regards, > Peter. ---
**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\ **Post date:** [July 6, 2011, 5:57pm UTC](https://discuss.elastic.co/t/logging-queries-and-response-times/4780/4 "2011-07-06T17:57:39Z")
Not sure if this is really necessary in Elastics' core, because it is very simple to feed a stat-index with your own data and build a simple UI via JS or something around it. I would prefer to see the index feed as plugin and the UI for that in ES-HEAD ๐Ÿ˜‰ But also if we would have that, requirements of 'stats' are often very different besides the top terms and query time ... On 6 Jul., 03:13, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > Yea, you need to do it yourself currently. I do want to build more stats into search and query times, a bit challenging when it comes to distributed systems ๐Ÿ™‚ > > On Tuesday, July 5, 2011 at 7:37 PM, Karussell wrote: > > > I'm using the java API where I can use: > > searchResponse.getTookInMillis() > > > If you are not using a layer between ES and your UI I would suggest > > that you should do it ๐Ÿ™‚ > > > not only for security reasons ... > > > Regards, > > Peter. ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [July 6, 2011, 7:26pm UTC](https://discuss.elastic.co/t/logging-queries-and-response-times/4780/5 "2011-07-06T19:26:36Z")
Hi, this is interesting topic. Actually, I want to store similar info/stats directly into separate ES index and use it for further analysis (one use case in the future can be query suggestions once they are implemented based on new Lucene 3.3 API). Now the question is: what information do you think people usually want to track? The obvious candidates would be: - raw user query text - may be the QueryDSL that is relevant to the user query (assuming that the DSL query is not directly built by the user but it is created behind the ES proxy). So the point here is to track what kind of query is created from given user input. - ES response (which includes time, number of hits and error as well as details for errors, number of involved shards... etc) may be not very obvious but very useful too (although this has to be provided by external system): - user id - something like "query session" id - which search result hit did use click - did user click some suggestions instead of search result hit? (for example spellchecker suggestion) BTW I think Otis G. had an interesting talk about this on #bbuzz this year. Regards, Lukas On Wed, Jul 6, 2011 at 7:57 PM, Karussell [tableyourtime@googlemail.com](mailto:tableyourtime@googlemail.com)wrote: > Not sure if this is really necessary in Elastics' core, because it is > very simple to feed a stat-index with your own data and build a simple > UI via JS or something around it. I would prefer to see the index feed > as plugin and the UI for that in ES-HEAD ๐Ÿ˜‰ > > But also if we would have that, requirements of 'stats' are often very > different besides the top terms and query time ... > > On 6 Jul., 03:13, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > Yea, you need to do it yourself currently. I do want to build more stats > > into search and query times, a bit challenging when it comes to distributed > > systems ๐Ÿ™‚ > > > > On Tuesday, July 5, 2011 at 7:37 PM, Karussell wrote: > > > > > I'm using the java API where I can use: > > > searchResponse.getTookInMillis() > > > > > If you are not using a layer between ES and your UI I would suggest > > > that you should do it ๐Ÿ™‚ > > > > > not only for security reasons ... > > > > > Regards, > > > Peter. ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [July 6, 2011, 9:22pm UTC](https://discuss.elastic.co/t/logging-queries-and-response-times/4780/6 "2011-07-06T21:22:55Z")
Yeah, I went to Otis G' talk in Berlin recently (Search Analytics). The slides and video are available online. You may also be interested in: - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) As well as: - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) -- the original poster, Alexander, may be more after something like this. ## Otis Sematext is hiring search people! [Jobs](http://sematext.com/about/jobs.html) On Jul 6, 3:26 pm, Lukรกลก Vlฤek [lukas.vl...@gmail.com](mailto:lukas.vl...@gmail.com) wrote: > Hi, > > this is interesting topic. Actually, I want to store similar info/stats > directly into separate ES index and use it for further analysis (one use > case in the future can be query suggestions once they are implemented based > on new Lucene 3.3 API). > > Now the question is: what information do you think people usually want to > track? > The obvious candidates would be: > > - raw user query text > - may be the QueryDSL that is relevant to the user query (assuming that the > DSL query is not directly built by the user but it is created behind the ES > proxy). So the point here is to track what kind of query is created from > given user input. > - ES response (which includes time, number of hits and error as well as > details for errors, number of involved shards... etc) > > may be not very obvious but very useful too (although this has to be > provided by external system): > > - user id > - something like "query session" id > - which search result hit did use click > - did user click some suggestions instead of search result hit? (for example > spellchecker suggestion) > > BTW I think Otis G. had an interesting talk about this on #bbuzz this year. > > Regards, > Lukas > > On Wed, Jul 6, 2011 at 7:57 PM, Karussell [tableyourt...@googlemail.com](mailto:tableyourt...@googlemail.com)wrote: > > > Not sure if this is really necessary in Elastics' core, because it is > > very simple to feed a stat-index with your own data and build a simple > > UI via JS or something around it. I would prefer to see the index feed > > as plugin and the UI for that in ES-HEAD ๐Ÿ˜‰ > > > But also if we would have that, requirements of 'stats' are often very > > different besides the top terms and query time ... > > > On 6 Jul., 03:13, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > Yea, you need to do it yourself currently. I do want to build more stats > > > into search and query times, a bit challenging when it comes to distributed > > > systems ๐Ÿ™‚ > > > > On Tuesday, July 5, 2011 at 7:37 PM, Karussell wrote: > > > > > > > I'm using the java API where I can use: > > > > searchResponse.getTookInMillis() > > > > > If you are not using a layer between ES and your UI I would suggest > > > > that you should do it ๐Ÿ™‚ > > > > > not only for security reasons ... > > > > > Regards, > > > > Peter. ---
**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\ **Post date:** [July 16, 2011, 10:32pm UTC](https://discuss.elastic.co/t/logging-queries-and-response-times/4780/7 "2011-07-16T22:32:59Z")
Hi Lukรกลก, do you think about to embed this into your bigdesk app? this would be cool ๐Ÿ™‚ All the suggested properties would be nice. The problem for the not so obvious candidates: how would you get that info into elasticsearch? Would you somehow append that to the original ES query and enable ES to persist the query (would be cool :))? Or would you simply feed to a special formatted index after the original query (a lot easier) and make this viewable it via big desk? Regards, Peter. -- [http://jetsli.de](http://jetsli.de) News 4 Geeks On 6 Jul., 21:26, Lukรกลก Vlฤek [lukas.vl...@gmail.com](mailto:lukas.vl...@gmail.com) wrote: > Hi, > > this is interesting topic. Actually, I want to store similar info/stats > directly into separate ES index and use it for further analysis (one use > case in the future can be query suggestions once they are implemented based > on new Lucene 3.3 API). > > Now the question is: what information do you think people usually want to > track? > The obvious candidates would be: > > - raw user query text > - may be the QueryDSL that is relevant to the user query (assuming that the > DSL query is not directly built by the user but it is created behind the ES > proxy). So the point here is to track what kind of query is created from > given user input. > - ES response (which includes time, number of hits and error as well as > details for errors, number of involved shards... etc) > > may be not very obvious but very useful too (although this has to be > provided by external system): > > - user id > - something like "query session" id > - which search result hit did use click > - did user click some suggestions instead of search result hit? (for example > spellchecker suggestion) > > BTW I think Otis G. had an interesting talk about this on #bbuzz this year. > > Regards, > Lukas > > On Wed, Jul 6, 2011 at 7:57 PM, Karussell [tableyourt...@googlemail.com](mailto:tableyourt...@googlemail.com)wrote: > > > Not sure if this is really necessary in Elastics' core, because it is > > very simple to feed a stat-index with your own data and build a simple > > UI via JS or something around it. I would prefer to see the index feed > > as plugin and the UI for that in ES-HEAD ๐Ÿ˜‰ > > > But also if we would have that, requirements of 'stats' are often very > > different besides the top terms and query time ... > > > On 6 Jul., 03:13, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > Yea, you need to do it yourself currently. I do want to build more stats > > > into search and query times, a bit challenging when it comes to distributed > > > systems ๐Ÿ™‚ > > > > On Tuesday, July 5, 2011 at 7:37 PM, Karussell wrote: > > > > > > > I'm using the java API where I can use: > > > > searchResponse.getTookInMillis() > > > > > If you are not using a layer between ES and your UI I would suggest > > > > that you should do it ๐Ÿ™‚ > > > > > not only for security reasons ... > > > > > Regards, > > > > Peter. ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [July 17, 2011, 8:05pm UTC](https://discuss.elastic.co/t/logging-queries-and-response-times/4780/8 "2011-07-17T20:05:32Z")
Hi, I do not plan to include this into BigDesk for now unless ES has a direct support for such statistics (may be Shay plans something?). Regards, Lukas On Sun, Jul 17, 2011 at 12:32 AM, Karussell [tableyourtime@googlemail.com](mailto:tableyourtime@googlemail.com)wrote: > Hi Lukรกลก, > > do you think about to embed this into your bigdesk app? this would be > cool ๐Ÿ™‚ > > All the suggested properties would be nice. The problem for the not so > obvious candidates: how would you get that info into elasticsearch? > Would you somehow append that to the original ES query and enable ES > to persist the query (would be cool :))? > > Or would you simply feed to a special formatted index after the > original query (a lot easier) and make this viewable it via big desk? > > Regards, > Peter. > > -- > > [http://jetsli.de](http://jetsli.de) News 4 Geeks > > On 6 Jul., 21:26, Lukรกลก Vlฤek [lukas.vl...@gmail.com](mailto:lukas.vl...@gmail.com) wrote: > > > Hi, > > > > this is interesting topic. Actually, I want to store similar info/stats > > directly into separate ES index and use it for further analysis (one use > > case in the future can be query suggestions once they are implemented > > based > > on new Lucene 3.3 API). > > > > Now the question is: what information do you think people usually want to > > track? > > The obvious candidates would be: > > > > - raw user query text > > - may be the QueryDSL that is relevant to the user query (assuming that > > the > > DSL query is not directly built by the user but it is created behind the > > ES > > proxy). So the point here is to track what kind of query is created from > > given user input. > > - ES response (which includes time, number of hits and error as well as > > details for errors, number of involved shards... etc) > > > > may be not very obvious but very useful too (although this has to be > > provided by external system): > > > > - user id > > - something like "query session" id > > - which search result hit did use click > > - did user click some suggestions instead of search result hit? (for > > example > > spellchecker suggestion) > > > > BTW I think Otis G. had an interesting talk about this on #bbuzz this > > year. > > > > Regards, > > Lukas > > > > On Wed, Jul 6, 2011 at 7:57 PM, Karussell \<[tableyourt...@googlemail.com](mailto:tableyourt...@googlemail.com) > > wrote: > > > > > Not sure if this is really necessary in Elastics' core, because it is > > > very simple to feed a stat-index with your own data and build a simple > > > UI via JS or something around it. I would prefer to see the index feed > > > as plugin and the UI for that in ES-HEAD ๐Ÿ˜‰ > > > > > But also if we would have that, requirements of 'stats' are often very > > > different besides the top terms and query time ... > > > > > On 6 Jul., 03:13, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > > > Yea, you need to do it yourself currently. I do want to build more > > > > stats > > > > into search and query times, a bit challenging when it comes to > > > > distributed > > > > systems ๐Ÿ™‚ > > > > > > On Tuesday, July 5, 2011 at 7:37 PM, Karussell wrote: > > > > > > > > > I'm using the java API where I can use: > > > > > searchResponse.getTookInMillis() > > > > > > > If you are not using a layer between ES and your UI I would suggest > > > > > that you should do it ๐Ÿ™‚ > > > > > > > not only for security reasons ... > > > > > > > Regards, > > > > > Peter. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:00am UTC](https://discuss.elastic.co/t/logging-queries-and-response-times/4780/9 "2017-07-06T04:00:21Z")