by judging my questions from the last few days, you might have noticed
I am evaluating elasticsearch for further usage
Our current custom implementation has the nice feature of logging
incoming queries as well as query times in order to find bottlenecks
(or monitoring rising response times with a growing index via munin).
Can I change the logging configuration somehow in this direction? Or
is there a different way of monitoring something like that?
Thanks a lot for all the answers I got so far to my questions, they
were pretty helpful!
Yea, you need to do it yourself currently. I do want to build more stats into search and query times, a bit challenging when it comes to distributed systems
On Tuesday, July 5, 2011 at 7:37 PM, Karussell wrote:
I'm using the java API where I can use:
searchResponse.getTookInMillis()
If you are not using a layer between ES and your UI I would suggest
that you should do it
Not sure if this is really necessary in Elastics' core, because it is
very simple to feed a stat-index with your own data and build a simple
UI via JS or something around it. I would prefer to see the index feed
as plugin and the UI for that in ES-HEAD
But also if we would have that, requirements of 'stats' are often very
different besides the top terms and query time ...
Yea, you need to do it yourself currently. I do want to build more stats into search and query times, a bit challenging when it comes to distributed systems
On Tuesday, July 5, 2011 at 7:37 PM, Karussell wrote:
I'm using the java API where I can use:
searchResponse.getTookInMillis()
If you are not using a layer between ES and your UI I would suggest
that you should do it
this is interesting topic. Actually, I want to store similar info/stats
directly into separate ES index and use it for further analysis (one use
case in the future can be query suggestions once they are implemented based
on new Lucene 3.3 API).
Now the question is: what information do you think people usually want to
track?
The obvious candidates would be:
raw user query text
may be the QueryDSL that is relevant to the user query (assuming that the
DSL query is not directly built by the user but it is created behind the ES
proxy). So the point here is to track what kind of query is created from
given user input.
ES response (which includes time, number of hits and error as well as
details for errors, number of involved shards... etc)
may be not very obvious but very useful too (although this has to be
provided by external system):
user id
something like "query session" id
which search result hit did use click
did user click some suggestions instead of search result hit? (for example
spellchecker suggestion)
BTW I think Otis G. had an interesting talk about this on #bbuzz this year.
Not sure if this is really necessary in Elastics' core, because it is
very simple to feed a stat-index with your own data and build a simple
UI via JS or something around it. I would prefer to see the index feed
as plugin and the UI for that in ES-HEAD
But also if we would have that, requirements of 'stats' are often very
different besides the top terms and query time ...
Yea, you need to do it yourself currently. I do want to build more stats
into search and query times, a bit challenging when it comes to distributed
systems
On Tuesday, July 5, 2011 at 7:37 PM, Karussell wrote:
I'm using the java API where I can use:
searchResponse.getTookInMillis()
If you are not using a layer between ES and your UI I would suggest
that you should do it
this is interesting topic. Actually, I want to store similar info/stats
directly into separate ES index and use it for further analysis (one use
case in the future can be query suggestions once they are implemented based
on new Lucene 3.3 API).
Now the question is: what information do you think people usually want to
track?
The obvious candidates would be:
raw user query text
may be the QueryDSL that is relevant to the user query (assuming that the
DSL query is not directly built by the user but it is created behind the ES
proxy). So the point here is to track what kind of query is created from
given user input.
ES response (which includes time, number of hits and error as well as
details for errors, number of involved shards... etc)
may be not very obvious but very useful too (although this has to be
provided by external system):
user id
something like "query session" id
which search result hit did use click
did user click some suggestions instead of search result hit? (for example
spellchecker suggestion)
BTW I think Otis G. had an interesting talk about this on #bbuzz this year.
Not sure if this is really necessary in Elastics' core, because it is
very simple to feed a stat-index with your own data and build a simple
UI via JS or something around it. I would prefer to see the index feed
as plugin and the UI for that in ES-HEAD
But also if we would have that, requirements of 'stats' are often very
different besides the top terms and query time ...
Yea, you need to do it yourself currently. I do want to build more stats
into search and query times, a bit challenging when it comes to distributed
systems
On Tuesday, July 5, 2011 at 7:37 PM, Karussell wrote:
I'm using the java API where I can use:
searchResponse.getTookInMillis()
If you are not using a layer between ES and your UI I would suggest
that you should do it
do you think about to embed this into your bigdesk app? this would be
cool
All the suggested properties would be nice. The problem for the not so
obvious candidates: how would you get that info into elasticsearch?
Would you somehow append that to the original ES query and enable ES
to persist the query (would be cool :))?
Or would you simply feed to a special formatted index after the
original query (a lot easier) and make this viewable it via big desk?
this is interesting topic. Actually, I want to store similar info/stats
directly into separate ES index and use it for further analysis (one use
case in the future can be query suggestions once they are implemented based
on new Lucene 3.3 API).
Now the question is: what information do you think people usually want to
track?
The obvious candidates would be:
raw user query text
may be the QueryDSL that is relevant to the user query (assuming that the
DSL query is not directly built by the user but it is created behind the ES
proxy). So the point here is to track what kind of query is created from
given user input.
ES response (which includes time, number of hits and error as well as
details for errors, number of involved shards... etc)
may be not very obvious but very useful too (although this has to be
provided by external system):
user id
something like "query session" id
which search result hit did use click
did user click some suggestions instead of search result hit? (for example
spellchecker suggestion)
BTW I think Otis G. had an interesting talk about this on #bbuzz this year.
Not sure if this is really necessary in Elastics' core, because it is
very simple to feed a stat-index with your own data and build a simple
UI via JS or something around it. I would prefer to see the index feed
as plugin and the UI for that in ES-HEAD
But also if we would have that, requirements of 'stats' are often very
different besides the top terms and query time ...
Yea, you need to do it yourself currently. I do want to build more stats
into search and query times, a bit challenging when it comes to distributed
systems
On Tuesday, July 5, 2011 at 7:37 PM, Karussell wrote:
I'm using the java API where I can use:
searchResponse.getTookInMillis()
If you are not using a layer between ES and your UI I would suggest
that you should do it
do you think about to embed this into your bigdesk app? this would be
cool
All the suggested properties would be nice. The problem for the not so
obvious candidates: how would you get that info into elasticsearch?
Would you somehow append that to the original ES query and enable ES
to persist the query (would be cool :))?
Or would you simply feed to a special formatted index after the
original query (a lot easier) and make this viewable it via big desk?
this is interesting topic. Actually, I want to store similar info/stats
directly into separate ES index and use it for further analysis (one use
case in the future can be query suggestions once they are implemented
based
on new Lucene 3.3 API).
Now the question is: what information do you think people usually want to
track?
The obvious candidates would be:
raw user query text
may be the QueryDSL that is relevant to the user query (assuming that
the
DSL query is not directly built by the user but it is created behind the
ES
proxy). So the point here is to track what kind of query is created from
given user input.
ES response (which includes time, number of hits and error as well as
details for errors, number of involved shards... etc)
may be not very obvious but very useful too (although this has to be
provided by external system):
user id
something like "query session" id
which search result hit did use click
did user click some suggestions instead of search result hit? (for
example
spellchecker suggestion)
BTW I think Otis G. had an interesting talk about this on #bbuzz this
year.
Not sure if this is really necessary in Elastics' core, because it is
very simple to feed a stat-index with your own data and build a simple
UI via JS or something around it. I would prefer to see the index feed
as plugin and the UI for that in ES-HEAD
But also if we would have that, requirements of 'stats' are often very
different besides the top terms and query time ...
Yea, you need to do it yourself currently. I do want to build more
stats
into search and query times, a bit challenging when it comes to
distributed
systems
On Tuesday, July 5, 2011 at 7:37 PM, Karussell wrote:
I'm using the java API where I can use:
searchResponse.getTookInMillis()
If you are not using a layer between ES and your UI I would suggest
that you should do it
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
