# What are they searching for and what are they getting?
**URL:** https://discuss.elastic.co/t/what-are-they-searching-for-and-what-are-they-getting/3210
**Category:** Elasticsearch
**Created:** [August 12, 2010, 7:03am UTC](https://discuss.elastic.co/t/what-are-they-searching-for-and-what-are-they-getting/3210 "2010-08-12T07:03:16Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)
#### Post date: [August 12, 2010, 7:03am UTC](https://discuss.elastic.co/t/what-are-they-searching-for-and-what-are-they-getting/3210/1 "2010-08-12T07:03:16Z")
Hi,
Does ES allow me to collect data about the question in ${Subject}?
I would like to know what queries the users are issuing and what kind of
response they get. It seems that there is no simple way how to get this data
by setting log levels (even if I set rootLogger to TRACE level then no
relevant data is recorded for searches and responses). May be the better
option would be to have a specific plugin that could record this type of
data (configurabe output to log file or pluggable custom storage
implementation?) and it would be great if it would be able to return some
data via REST (eg. statistic and few recent queries and responses and their
processing times and the like...). What do you think? Would it be possible
to implement plugin like that?
Regards,
Lukas
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [August 12, 2010, 8:35am UTC](https://discuss.elastic.co/t/what-are-they-searching-for-and-what-are-they-getting/3210/2 "2010-08-12T08:35:25Z")
No, there is no stats gathering on that level. The plan is to have stats
gathering on the index level, but more general ones, like indexing TPS and
so on.
Regarding the question at hand, it would be very cool if it was provided.
The problems is that its quite complex to do from within elasticsearch. For
example, not all queries are "user" driven (i.e. did not result from a
direct user search request). Or, many times, the queries gets augmented. If
I had to implement something like this in a search centric application, I
would do that in a layer on top of elasticsearch that has a bit more
knowledge of where, why, and what the user did.
Of course, this information can be stored within elasticsearch so you can
search on it 😉
-shay.banon
On Thu, Aug 12, 2010 at 10:03 AM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com) wrote:
> Hi,
>
> Does ES allow me to collect data about the question in ${Subject}?
>
> I would like to know what queries the users are issuing and what kind of
> response they get. It seems that there is no simple way how to get this data
> by setting log levels (even if I set rootLogger to TRACE level then no
> relevant data is recorded for searches and responses). May be the better
> option would be to have a specific plugin that could record this type of
> data (configurabe output to log file or pluggable custom storage
> implementation?) and it would be great if it would be able to return some
> data via REST (eg. statistic and few recent queries and responses and their
> processing times and the like...). What do you think? Would it be possible
> to implement plugin like that?
>
> Regards,
> Lukas
---
### Author:  [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)
#### Post date: [August 12, 2010, 8:44am UTC](https://discuss.elastic.co/t/what-are-they-searching-for-and-what-are-they-getting/3210/3 "2010-08-12T08:44:21Z")
For now I would be happy to get all user queries issued via REST interface
(although even at this level some queries may not be user driven). I just
want to know what users are looking for and all I give them is REST API. Is
there any easy way how to get queries and responses data in this case? For
now I can live with the fact that I will just put more logging message into
some class, is there any central class in this case which could be used for
this purpose?
Regards,
Lukas
On Thu, Aug 12, 2010 at 10:35 AM, Shay Banon
[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:
> No, there is no stats gathering on that level. The plan is to have stats
> gathering on the index level, but more general ones, like indexing TPS and
> so on.
>
> Regarding the question at hand, it would be very cool if it was provided.
> The problems is that its quite complex to do from within elasticsearch. For
> example, not all queries are "user" driven (i.e. did not result from a
> direct user search request). Or, many times, the queries gets augmented. If
> I had to implement something like this in a search centric application, I
> would do that in a layer on top of elasticsearch that has a bit more
> knowledge of where, why, and what the user did.
>
> Of course, this information can be stored within elasticsearch so you can
> search on it 😉
>
> -shay.banon
>
> On Thu, Aug 12, 2010 at 10:03 AM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
>
> > Hi,
> >
> > Does ES allow me to collect data about the question in ${Subject}?
> >
> > I would like to know what queries the users are issuing and what kind of
> > response they get. It seems that there is no simple way how to get this data
> > by setting log levels (even if I set rootLogger to TRACE level then no
> > relevant data is recorded for searches and responses). May be the better
> > option would be to have a specific plugin that could record this type of
> > data (configurabe output to log file or pluggable custom storage
> > implementation?) and it would be great if it would be able to return some
> > data via REST (eg. statistic and few recent queries and responses and their
> > processing times and the like...). What do you think? Would it be possible
> > to implement plugin like that?
> >
> > Regards,
> > Lukas
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [August 12, 2010, 9:03am UTC](https://discuss.elastic.co/t/what-are-they-searching-for-and-what-are-they-getting/3210/4 "2010-08-12T09:03:00Z")
There isn't one, and storing all the user queries means you need to build a
storage for that since it can be quite big. Thats something that you can do
on the client level by indexing it into elasticsearch (with the extra
knowledge you have about the query itself).
I can add logging for that, but its not always textual (it is with json, but
not with xson), and I work really hard at trying to reduce the number of fs
operations on critical path requests, and that logging would add to it 😉
-shay.banon
On Thu, Aug 12, 2010 at 11:44 AM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com) wrote:
> For now I would be happy to get all user queries issued via REST interface
> (although even at this level some queries may not be user driven). I just
> want to know what users are looking for and all I give them is REST API. Is
> there any easy way how to get queries and responses data in this case? For
> now I can live with the fact that I will just put more logging message into
> some class, is there any central class in this case which could be used for
> this purpose?
>
> Regards,
> Lukas
>
> On Thu, Aug 12, 2010 at 10:35 AM, Shay Banon \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)
>
> > wrote:
>
> > No, there is no stats gathering on that level. The plan is to have stats
> > gathering on the index level, but more general ones, like indexing TPS and
> > so on.
> >
> > Regarding the question at hand, it would be very cool if it was provided.
> > The problems is that its quite complex to do from within elasticsearch. For
> > example, not all queries are "user" driven (i.e. did not result from a
> > direct user search request). Or, many times, the queries gets augmented. If
> > I had to implement something like this in a search centric application, I
> > would do that in a layer on top of elasticsearch that has a bit more
> > knowledge of where, why, and what the user did.
> >
> > Of course, this information can be stored within elasticsearch so you can
> > search on it 😉
> >
> > -shay.banon
> >
> > On Thu, Aug 12, 2010 at 10:03 AM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com)wrote:
> >
> > > Hi,
> > >
> > > Does ES allow me to collect data about the question in ${Subject}?
> > >
> > > I would like to know what queries the users are issuing and what kind of
> > > response they get. It seems that there is no simple way how to get this data
> > > by setting log levels (even if I set rootLogger to TRACE level then no
> > > relevant data is recorded for searches and responses). May be the better
> > > option would be to have a specific plugin that could record this type of
> > > data (configurabe output to log file or pluggable custom storage
> > > implementation?) and it would be great if it would be able to return some
> > > data via REST (eg. statistic and few recent queries and responses and their
> > > processing times and the like...). What do you think? Would it be possible
> > > to implement plugin like that?
> > >
> > > Regards,
> > > Lukas
---
### Author:  [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)
#### Post date: [August 12, 2010, 9:14am UTC](https://discuss.elastic.co/t/what-are-they-searching-for-and-what-are-they-getting/3210/5 "2010-08-12T09:14:34Z")
On Thu, 2010-08-12 at 12:03 +0300, Shay Banon wrote:
> There isn't one, and storing all the user queries means you need to
> build a storage for that since it can be quite big. Thats something
> that you can do on the client level by indexing it into elasticsearch
> (with the extra knowledge you have about the query itself).
I agree with you - the most meaningful query logging can happen in the
client. In ES, the same query could be expressed in a number of
different ways, and the end result probably wouldn't look terribly
meaningful to the programmer anyway.
Much better to just log queries to a text file or DB in a format that
suits your application, and analyse them later.
clint
---
### Author:  [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)
#### Post date: [August 12, 2010, 9:27am UTC](https://discuss.elastic.co/t/what-are-they-searching-for-and-what-are-they-getting/3210/6 "2010-08-12T09:27:01Z")
Guys, I fully agree with you. Implementing this properly is not trivial but
all I need now is just simple workaround or hack if you like (nothing of a
production quality).
Anyway, I will probably gum
up org.elasticsearch.rest.action.search.RestSearchAction a little bit. This
could give me what I want and I should be able to configure logging to
output it into rolling files, that is all I need for now.
On Thu, Aug 12, 2010 at 11:14 AM, Clinton Gormley
[clinton@iannounce.co.uk](mailto:clinton@iannounce.co.uk)wrote:
> On Thu, 2010-08-12 at 12:03 +0300, Shay Banon wrote:
>
> > There isn't one, and storing all the user queries means you need to
> > build a storage for that since it can be quite big. Thats something
> > that you can do on the client level by indexing it into elasticsearch
> > (with the extra knowledge you have about the query itself).
>
> I agree with you - the most meaningful query logging can happen in the
> client. In ES, the same query could be expressed in a number of
> different ways, and the end result probably wouldn't look terribly
> meaningful to the programmer anyway.
>
> Much better to just log queries to a text file or DB in a format that
> suits your application, and analyse them later.
>
> clint
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 4:20am UTC](https://discuss.elastic.co/t/what-are-they-searching-for-and-what-are-they-getting/3210/7 "2017-07-06T04:20:54Z")