Logging queries and response times


(Alexander Reelsen) #1

Hi there,

by judging my questions from the last few days, you might have noticed
I am evaluating elasticsearch for further usage :slight_smile:

Our current custom implementation has the nice feature of logging
incoming queries as well as query times in order to find bottlenecks
(or monitoring rising response times with a growing index via munin).
Can I change the logging configuration somehow in this direction? Or
is there a different way of monitoring something like that?

Thanks a lot for all the answers I got so far to my questions, they
were pretty helpful!

--Alexander


(Karussell) #2

I'm using the java API where I can use:
searchResponse.getTookInMillis()

If you are not using a layer between ES and your UI I would suggest
that you should do it :slight_smile:

not only for security reasons ...

Regards,
Peter.


(Shay Banon) #3

Yea, you need to do it yourself currently. I do want to build more stats into search and query times, a bit challenging when it comes to distributed systems :slight_smile:

On Tuesday, July 5, 2011 at 7:37 PM, Karussell wrote:

I'm using the java API where I can use:
searchResponse.getTookInMillis()

If you are not using a layer between ES and your UI I would suggest
that you should do it :slight_smile:

not only for security reasons ...

Regards,
Peter.


(Karussell) #4

Not sure if this is really necessary in Elastics' core, because it is
very simple to feed a stat-index with your own data and build a simple
UI via JS or something around it. I would prefer to see the index feed
as plugin and the UI for that in ES-HEAD :wink:

But also if we would have that, requirements of 'stats' are often very
different besides the top terms and query time ...

On 6 Jul., 03:13, Shay Banon shay.ba...@elasticsearch.com wrote:

Yea, you need to do it yourself currently. I do want to build more stats into search and query times, a bit challenging when it comes to distributed systems :slight_smile:

On Tuesday, July 5, 2011 at 7:37 PM, Karussell wrote:

I'm using the java API where I can use:
searchResponse.getTookInMillis()

If you are not using a layer between ES and your UI I would suggest
that you should do it :slight_smile:

not only for security reasons ...

Regards,
Peter.


(Lukáš Vlček) #5

Hi,

this is interesting topic. Actually, I want to store similar info/stats
directly into separate ES index and use it for further analysis (one use
case in the future can be query suggestions once they are implemented based
on new Lucene 3.3 API).

Now the question is: what information do you think people usually want to
track?
The obvious candidates would be:

  • raw user query text
  • may be the QueryDSL that is relevant to the user query (assuming that the
    DSL query is not directly built by the user but it is created behind the ES
    proxy). So the point here is to track what kind of query is created from
    given user input.
  • ES response (which includes time, number of hits and error as well as
    details for errors, number of involved shards... etc)

may be not very obvious but very useful too (although this has to be
provided by external system):

  • user id
  • something like "query session" id
  • which search result hit did use click
  • did user click some suggestions instead of search result hit? (for example
    spellchecker suggestion)

BTW I think Otis G. had an interesting talk about this on #bbuzz this year.

Regards,
Lukas

On Wed, Jul 6, 2011 at 7:57 PM, Karussell tableyourtime@googlemail.comwrote:

Not sure if this is really necessary in Elastics' core, because it is
very simple to feed a stat-index with your own data and build a simple
UI via JS or something around it. I would prefer to see the index feed
as plugin and the UI for that in ES-HEAD :wink:

But also if we would have that, requirements of 'stats' are often very
different besides the top terms and query time ...

On 6 Jul., 03:13, Shay Banon shay.ba...@elasticsearch.com wrote:

Yea, you need to do it yourself currently. I do want to build more stats
into search and query times, a bit challenging when it comes to distributed
systems :slight_smile:

On Tuesday, July 5, 2011 at 7:37 PM, Karussell wrote:

I'm using the java API where I can use:
searchResponse.getTookInMillis()

If you are not using a layer between ES and your UI I would suggest
that you should do it :slight_smile:

not only for security reasons ...

Regards,
Peter.


(Otis Gospodnetić) #6

Yeah, I went to Otis G' talk in Berlin recently (Search Analytics).
The slides and video are available online.
You may also be interested in:

As well as:

Otis

Sematext is hiring search people! http://sematext.com/about/jobs.html

On Jul 6, 3:26 pm, Lukáš Vlček lukas.vl...@gmail.com wrote:

Hi,

this is interesting topic. Actually, I want to store similar info/stats
directly into separate ES index and use it for further analysis (one use
case in the future can be query suggestions once they are implemented based
on new Lucene 3.3 API).

Now the question is: what information do you think people usually want to
track?
The obvious candidates would be:

  • raw user query text
  • may be the QueryDSL that is relevant to the user query (assuming that the
    DSL query is not directly built by the user but it is created behind the ES
    proxy). So the point here is to track what kind of query is created from
    given user input.
  • ES response (which includes time, number of hits and error as well as
    details for errors, number of involved shards... etc)

may be not very obvious but very useful too (although this has to be
provided by external system):

  • user id
  • something like "query session" id
  • which search result hit did use click
  • did user click some suggestions instead of search result hit? (for example
    spellchecker suggestion)

BTW I think Otis G. had an interesting talk about this on #bbuzz this year.

Regards,
Lukas

On Wed, Jul 6, 2011 at 7:57 PM, Karussell tableyourt...@googlemail.comwrote:

Not sure if this is really necessary in Elastics' core, because it is
very simple to feed a stat-index with your own data and build a simple
UI via JS or something around it. I would prefer to see the index feed
as plugin and the UI for that in ES-HEAD :wink:

But also if we would have that, requirements of 'stats' are often very
different besides the top terms and query time ...

On 6 Jul., 03:13, Shay Banon shay.ba...@elasticsearch.com wrote:

Yea, you need to do it yourself currently. I do want to build more stats
into search and query times, a bit challenging when it comes to distributed
systems :slight_smile:

On Tuesday, July 5, 2011 at 7:37 PM, Karussell wrote:

I'm using the java API where I can use:
searchResponse.getTookInMillis()

If you are not using a layer between ES and your UI I would suggest
that you should do it :slight_smile:

not only for security reasons ...

Regards,
Peter.


(Karussell) #7

Hi Lukáš,

do you think about to embed this into your bigdesk app? this would be
cool :slight_smile:

All the suggested properties would be nice. The problem for the not so
obvious candidates: how would you get that info into elasticsearch?
Would you somehow append that to the original ES query and enable ES
to persist the query (would be cool :))?

Or would you simply feed to a special formatted index after the
original query (a lot easier) and make this viewable it via big desk?

Regards,
Peter.

--

http://jetsli.de News 4 Geeks

On 6 Jul., 21:26, Lukáš Vlček lukas.vl...@gmail.com wrote:

Hi,

this is interesting topic. Actually, I want to store similar info/stats
directly into separate ES index and use it for further analysis (one use
case in the future can be query suggestions once they are implemented based
on new Lucene 3.3 API).

Now the question is: what information do you think people usually want to
track?
The obvious candidates would be:

  • raw user query text
  • may be the QueryDSL that is relevant to the user query (assuming that the
    DSL query is not directly built by the user but it is created behind the ES
    proxy). So the point here is to track what kind of query is created from
    given user input.
  • ES response (which includes time, number of hits and error as well as
    details for errors, number of involved shards... etc)

may be not very obvious but very useful too (although this has to be
provided by external system):

  • user id
  • something like "query session" id
  • which search result hit did use click
  • did user click some suggestions instead of search result hit? (for example
    spellchecker suggestion)

BTW I think Otis G. had an interesting talk about this on #bbuzz this year.

Regards,
Lukas

On Wed, Jul 6, 2011 at 7:57 PM, Karussell tableyourt...@googlemail.comwrote:

Not sure if this is really necessary in Elastics' core, because it is
very simple to feed a stat-index with your own data and build a simple
UI via JS or something around it. I would prefer to see the index feed
as plugin and the UI for that in ES-HEAD :wink:

But also if we would have that, requirements of 'stats' are often very
different besides the top terms and query time ...

On 6 Jul., 03:13, Shay Banon shay.ba...@elasticsearch.com wrote:

Yea, you need to do it yourself currently. I do want to build more stats
into search and query times, a bit challenging when it comes to distributed
systems :slight_smile:

On Tuesday, July 5, 2011 at 7:37 PM, Karussell wrote:

I'm using the java API where I can use:
searchResponse.getTookInMillis()

If you are not using a layer between ES and your UI I would suggest
that you should do it :slight_smile:

not only for security reasons ...

Regards,
Peter.


(Lukáš Vlček) #8

Hi,

I do not plan to include this into BigDesk for now unless ES has a direct
support for such statistics (may be Shay plans something?).

Regards,
Lukas

On Sun, Jul 17, 2011 at 12:32 AM, Karussell tableyourtime@googlemail.comwrote:

Hi Lukáš,

do you think about to embed this into your bigdesk app? this would be
cool :slight_smile:

All the suggested properties would be nice. The problem for the not so
obvious candidates: how would you get that info into elasticsearch?
Would you somehow append that to the original ES query and enable ES
to persist the query (would be cool :))?

Or would you simply feed to a special formatted index after the
original query (a lot easier) and make this viewable it via big desk?

Regards,
Peter.

--

http://jetsli.de News 4 Geeks

On 6 Jul., 21:26, Lukáš Vlček lukas.vl...@gmail.com wrote:

Hi,

this is interesting topic. Actually, I want to store similar info/stats
directly into separate ES index and use it for further analysis (one use
case in the future can be query suggestions once they are implemented
based
on new Lucene 3.3 API).

Now the question is: what information do you think people usually want to
track?
The obvious candidates would be:

  • raw user query text
  • may be the QueryDSL that is relevant to the user query (assuming that
    the
    DSL query is not directly built by the user but it is created behind the
    ES
    proxy). So the point here is to track what kind of query is created from
    given user input.
  • ES response (which includes time, number of hits and error as well as
    details for errors, number of involved shards... etc)

may be not very obvious but very useful too (although this has to be
provided by external system):

  • user id
  • something like "query session" id
  • which search result hit did use click
  • did user click some suggestions instead of search result hit? (for
    example
    spellchecker suggestion)

BTW I think Otis G. had an interesting talk about this on #bbuzz this
year.

Regards,
Lukas

On Wed, Jul 6, 2011 at 7:57 PM, Karussell <tableyourt...@googlemail.com
wrote:

Not sure if this is really necessary in Elastics' core, because it is
very simple to feed a stat-index with your own data and build a simple
UI via JS or something around it. I would prefer to see the index feed
as plugin and the UI for that in ES-HEAD :wink:

But also if we would have that, requirements of 'stats' are often very
different besides the top terms and query time ...

On 6 Jul., 03:13, Shay Banon shay.ba...@elasticsearch.com wrote:

Yea, you need to do it yourself currently. I do want to build more
stats

into search and query times, a bit challenging when it comes to
distributed

systems :slight_smile:

On Tuesday, July 5, 2011 at 7:37 PM, Karussell wrote:

I'm using the java API where I can use:
searchResponse.getTookInMillis()

If you are not using a layer between ES and your UI I would suggest
that you should do it :slight_smile:

not only for security reasons ...

Regards,
Peter.


(system) #9