Stats about response times

Florian_Munz · December 9, 2011, 2:47pm

Hello,

is there any way to get the response time for a query in the logfile
or an aggregate for a specific time frame? I'd like to graph the
response times we're seeing from elasticsearch.

Cheers,
Florian

Karussell1 · December 9, 2011, 7:44pm

You could enable slow query into log feature:

github.com/elastic/elasticsearch

Index shard search slow log (query and fetch)

opened 12:10PM - 14 Nov 11 UTC

closed 12:11PM - 14 Nov 11 UTC

kimchy

>feature v0.19.0.RC1 v0.18.3

Shard level slow search log allows to log slow search (query and fetch execution…s) into a dedicated log file. Thresholds can be set for both the query phase of the execution, and fetch phase, here is a sample: ``` #index.search.slowlog.threshold.query.warn: 10s #index.search.slowlog.threshold.query.info: 5s #index.search.slowlog.threshold.query.debug: 2s #index.search.slowlog.threshold.query.trace: 500ms #index.search.slowlog.threshold.fetch.warn: 1s #index.search.slowlog.threshold.fetch.info: 800ms #index.search.slowlog.threshold.fetch.debug: 500ms #index.search.slowlog.threshold.fetch.trace: 200ms ``` By default, none are enabled (set to `-1`). Levels (`warn`, `info`, `debug`, `trace`) allow to control under which logging level the log will be logged. Not all are required to be configured (for example, only `warn` threshold can be set). The benefit of several levels is the ability to quickly "grep" for specific thresholds breached. The logging is done on the shard level scope, meaning the execution of a search request within a specific shard. It does not encompass the whole search request, which can be broadcast to several shards in order to execute. Some of the benefits of shard level logging is the association of the actual execution on the specific machine, compared with request level. All settings are index level settings (and each index can have different values for it), and can be changed in runtime using the index update settings API. The logging file is configured by default using the following configuration (found in `logging.yml`): ``` index_search_slow_log_file: type: dailyRollingFile file: ${path.logs}/${cluster.name}_index_search_slowlog.log datePattern: "'.'yyyy-MM-dd" layout: type: pattern conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n" ```

But you can also index this into elasticsearch itself (or use
logstash), then you could search, facet them etc ...

Peter.

On 9 Dez., 15:47, Florian Munz m...@podio.com wrote:

Hello,

is there any way to get the response time for a query in the logfile
or an aggregate for a specific time frame? I'd like to graph the
response times we're seeing from elasticsearch.

Cheers,
Florian

dungtc · December 9, 2011, 7:50pm

Hello
I have use Elastic search (machine 1) to index 50000 documents in BigCouch (cluster version of CouchDB 1.1.0) on the machine 2. At this moment, cluster contain only one node.
I use couchdb river to do it in the same way with couchdb, because the bigcouch API is the same as couchdb API . All is successful after typing curl. An index has been created and I can search on them. But When I restart Elastic Search, it cann't be retrieve changes from BigCouch any more (chang stream donot work). Here is the error message:
[2011-12-09 16:09:08,643][WARN ][river.couchdb ] [NFL Superpro] [couchdb][couchdb_indexation_test_river_1] failed to read from _changes, throttling....
java.net.SocketException: Unexpected end of file from server
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:770)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:767)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1162)
at org.elasticsearch.river.couchdb.CouchdbRiver$Slurper.run(CouchdbRiver.java:429)
at java.lang.Thread.run(Thread.java:636)

In the machine 2 of BigCouch, I see:
[error] [<0.1515.0>] [04d73d10] Uncaught error in HTTP request: {exit,normal}
[info] [<0.1515.0>] [04d73d10] Stacktrace: [{mochiweb_request,send,2},
{chttpd,send_chunk,2},
{chttpd_db,changes_callback,2},
{fabric_view_changes,
keep_sending_changes,6},
{fabric_view_changes,go,5},
{chttpd,handle_request,1},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]
Please explain it to me and how to solve it? For Couchdb (not BigCouch), all works well but this problem happen with BigCouch and Elastic Search 0.18.4
Thanks a lot
TCD

dadoonet · December 9, 2011, 8:10pm

It's a big couch problem. Not an ES problem.
It could be relative to the way Bigcouch modify sequence numbers in Strings.

Let me check that.

David
@dadoonet

Le 9 déc. 2011 à 20:50, Chi Dung Tran dungtctin4@yahoo.com a écrit :

Hello
I have use Elastic search (machine 1) to index 50000 documents in BigCouch (cluster version of CouchDB 1.1.0) on the machine 2. At this moment, cluster contain only one node.
I use couchdb river to do it in the same way with couchdb, because the bigcouch API is the same as couchdb API . All is successful after typing curl. An index has been created and I can search on them. But When I restart Elastic Search, it cann't be retrieve changes from BigCouch any more (chang stream donot work). Here is the error message:
[2011-12-09 16:09:08,643][WARN ][river.couchdb ] [NFL Superpro] [couchdb][couchdb_indexation_test_river_1] failed to read from _changes, throttling....
java.net.SocketException: Unexpected end of file from server
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:770)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:767)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1162)
at org.elasticsearch.river.couchdb.CouchdbRiver$Slurper.run(CouchdbRiver.java:429)
at java.lang.Thread.run(Thread.java:636)

In the machine 2 of BigCouch, I see:
[error] [<0.1515.0>] [04d73d10] Uncaught error in HTTP request: {exit,normal}
[info] [<0.1515.0>] [04d73d10] Stacktrace: [{mochiweb_request,send,2},
{chttpd,send_chunk,2},
{chttpd_db,changes_callback,2},
{fabric_view_changes,
keep_sending_changes,6},
{fabric_view_changes,go,5},
{chttpd,handle_request,1},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]
Please explain it to me and how to solve it? For Couchdb (not BigCouch), all works well but this problem happen with BigCouch and Elastic Search 0.18.4
Thanks a lot
TCD

dadoonet · December 9, 2011, 8:38pm

It has been fixed 3 weeks ago in 0.18 and now in the external plugin
(GitHub - elastic/elasticsearch-river-couchdb: CouchDB River Plugin for elasticsearch (STOPPED)).

See also here : BigCouch returns JSON array for sequence · Issue #1478 · elastic/elasticsearch · GitHub

Which version of ES are you using and which version of couchDb plugin are you
using ?

David.

Le 9 décembre 2011 à 21:10, David Pilato david@pilato.fr a écrit :

It's a big couch problem. Not an ES problem.
It could be relative to the way Bigcouch modify sequence numbers in Strings.

Let me check that.

David
@dadoonet

Le 9 déc. 2011 à 20:50, Chi Dung Tran dungtctin4@yahoo.com a écrit :

Hello
I have use Elastic search (machine 1) to index 50000 documents in BigCouch
(cluster version of CouchDB 1.1.0) on the machine 2. At this moment, cluster
contain only one node.
I use couchdb river to do it in the same way with couchdb, because the
bigcouch API is the same as couchdb API . All is successful after typing
curl. An index has been created and I can search on them. But When I restart
Elastic Search, it cann't be retrieve changes from BigCouch any more (chang
stream donot work). Here is the error message:
[2011-12-09 16:09:08,643][WARN ][river.couchdb ] [NFL Superpro]
[couchdb][couchdb_indexation_test_river_1] failed to read from _changes,
throttling....
java.net.SocketException: Unexpected end of file from server
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:770)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:767)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1162)
at
org.elasticsearch.river.couchdb.CouchdbRiver$Slurper.run(CouchdbRiver.java:429)
at java.lang.Thread.run(Thread.java:636)

In the machine 2 of BigCouch, I see:
[error] [<0.1515.0>] [04d73d10] Uncaught error in HTTP request:
{exit,normal}
[info] [<0.1515.0>] [04d73d10] Stacktrace: [{mochiweb_request,send,2},
{chttpd,send_chunk,2},
{chttpd_db,changes_callback,2},
{fabric_view_changes,
keep_sending_changes,6},
{fabric_view_changes,go,5},
{chttpd,handle_request,1},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]
Please explain it to me and how to solve it? For Couchdb (not BigCouch), all
works well but this problem happen with BigCouch and Elastic Search 0.18.4
Thanks a lot
TCD
--
David Pilato
http://dev.david.pilato.fr/
Twitter : @dadoonet

kimchy · December 9, 2011, 10:16pm

Also, check node stats and index stats APIs, which include search/index/get
stats.

On Fri, Dec 9, 2011 at 9:44 PM, Karussell tableyourtime@googlemail.comwrote:

You could enable slow query into log feature:

Index shard search slow log (query and fetch) · Issue #1462 · elastic/elasticsearch · GitHub

But you can also index this into elasticsearch itself (or use
logstash), then you could search, facet them etc ...

Peter.

On 9 Dez., 15:47, Florian Munz m...@podio.com wrote:

Hello,

is there any way to get the response time for a query in the logfile
or an aggregate for a specific time frame? I'd like to graph the
response times we're seeing from elasticsearch.

Cheers,
Florian

dungtc · December 9, 2011, 11:05pm

I'm using Elastic Search 0.18.4 and 0.17.6
This problem happen with both. BigCOuch is 0.4.a (cluster of CouchDB 1.1.0)
I'll try new river
Thanks

Ngu ngoc lon nhat cua doi nguoi la yeu duong - dieu ran thu 15 cua duc Phat (tai sao khong)

TINH YEU KHONG TU NHIEN SINH RA, KHONG TU NHIEN MAT DI, NO CHI CHUYEN TU NGUOI NAY SANG NGUOI KHAC

     NGAY MAI EM DI

BIEN NAM CHAN EM GIAT VE

De : "david@pilato.fr" david@pilato.fr
À : elasticsearch@googlegroups.com
Envoyé le : Vendredi 9 Décembre 2011 21h38
Objet : Re: Problem with river BigCouch

It has been fixed 3 weeks ago in 0.18 and now in the external plugin ( https://github.com/elasticsearch/elasticsearch-river-couchdb ).

See also here : https://github.com/elasticsearch/elasticsearch/issues/1478

Which version of ES are you using and which version of couchDb plugin are you using ?

David.

Le 9 décembre 2011 à 21:10, David Pilato david@pilato.fr a écrit : > It's a big couch problem. Not an ES problem. > It could be relative to the way Bigcouch modify sequence numbers in Strings. > > Let me check that. > > David > @dadoonet > > > Le 9 déc. 2011 à 20:50, Chi Dung Tran dungtctin4@yahoo.com a écrit : > > > Hello > > I have use Elastic search (machine 1) to index 50000 documents in BigCouch (cluster version of CouchDB 1.1.0) on the machine 2. At this moment, cluster contain only one node. > > I use couchdb river to do it in the same way with couchdb, because the bigcouch API is the same as couchdb API . All is successful after typing curl. An index has been created and I can search on them. But When I restart Elastic Search, it cann't be retrieve changes from BigCouch any more (chang stream donot work). Here is the error message: > > [2011-12-09 16:09:08,643][WARN ][river.couchdb ] [NFL Superpro]
[couchdb][couchdb_indexation_test_river_1] failed to read from _changes, throttling.... > > java.net.SocketException: Unexpected end of file from server > > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:770) > > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:767) > > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > > at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1162) > > at org.elasticsearch.river.couchdb.CouchdbRiver$Slurper.run(CouchdbRiver.java:429) > > at java.lang.Thread.run(Thread.java:636) > > > > In the machine 2 of BigCouch, I see: > > [error] [<0.1515.0>] [04d73d10] Uncaught error in HTTP request: {exit,normal} > > [info] [<0.1515.0>] [04d73d10] Stacktrace: [{mochiweb_request,send,2}, > >
{chttpd,send_chunk,2}, > > {chttpd_db,changes_callback,2}, > > {fabric_view_changes, > > keep_sending_changes,6}, > > {fabric_view_changes,go,5}, > > {chttpd,handle_request,1}, > > {mochiweb_http,headers,5}, > > {proc_lib,init_p_do_apply,3}] > > Please explain it to me and how to solve it? For Couchdb (not BigCouch), all works well but this problem happen with BigCouch and Elastic Search 0.18.4 > > Thanks a lot > > TCD

--
David Pilato
http://dev.david.pilato.fr/
Twitter : @dadoonet

dungtc · December 9, 2011, 11:25pm

I'm using ES 0.18.4 with plugin couchdb river 0.18.4, not 1.1.0 on your link

But I think my problem is different, but I'll try to use ES 0.18.4 with this new plugin couchdb 1.1.0
Thanks

Ngu ngoc lon nhat cua doi nguoi la yeu duong - dieu ran thu 15 cua duc Phat (tai sao khong)

TINH YEU KHONG TU NHIEN SINH RA, KHONG TU NHIEN MAT DI, NO CHI CHUYEN TU NGUOI NAY SANG NGUOI KHAC

     NGAY MAI EM DI

BIEN NAM CHAN EM GIAT VE

De : "david@pilato.fr" david@pilato.fr
À : elasticsearch@googlegroups.com
Envoyé le : Vendredi 9 Décembre 2011 21h38
Objet : Re: Problem with river BigCouch

It has been fixed 3 weeks ago in 0.18 and now in the external plugin ( https://github.com/elasticsearch/elasticsearch-river-couchdb ).

See also here : https://github.com/elasticsearch/elasticsearch/issues/1478

Which version of ES are you using and which version of couchDb plugin are you using ?

David.

Le 9 décembre 2011 à 21:10, David Pilato david@pilato.fr a écrit : > It's a big couch problem. Not an ES problem. > It could be relative to the way Bigcouch modify sequence numbers in Strings. > > Let me check that. > > David > @dadoonet > > > Le 9 déc. 2011 à 20:50, Chi Dung Tran dungtctin4@yahoo.com a écrit : > > > Hello > > I have use Elastic search (machine 1) to index 50000 documents in BigCouch (cluster version of CouchDB 1.1.0) on the machine 2. At this moment, cluster contain only one node. > > I use couchdb river to do it in the same way with couchdb, because the bigcouch API is the same as couchdb API . All is successful after typing curl. An index has been created and I can search on them. But When I restart Elastic Search, it cann't be retrieve changes from BigCouch any more (chang stream donot work). Here is the error message: > > [2011-12-09 16:09:08,643][WARN ][river.couchdb ] [NFL Superpro]
[couchdb][couchdb_indexation_test_river_1] failed to read from _changes, throttling.... > > java.net.SocketException: Unexpected end of file from server > > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:770) > > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:767) > > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) > > at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1162) > > at org.elasticsearch.river.couchdb.CouchdbRiver$Slurper.run(CouchdbRiver.java:429) > > at java.lang.Thread.run(Thread.java:636) > > > > In the machine 2 of BigCouch, I see: > > [error] [<0.1515.0>] [04d73d10] Uncaught error in HTTP request: {exit,normal} > > [info] [<0.1515.0>] [04d73d10] Stacktrace: [{mochiweb_request,send,2}, > >
{chttpd,send_chunk,2}, > > {chttpd_db,changes_callback,2}, > > {fabric_view_changes, > > keep_sending_changes,6}, > > {fabric_view_changes,go,5}, > > {chttpd,handle_request,1}, > > {mochiweb_http,headers,5}, > > {proc_lib,init_p_do_apply,3}] > > Please explain it to me and how to solve it? For Couchdb (not BigCouch), all works well but this problem happen with BigCouch and Elastic Search 0.18.4 > > Thanks a lot > > TCD

--
David Pilato
http://dev.david.pilato.fr/
Twitter : @dadoonet

Topic		Replies	Views
Why the slow query doesn't log in slowlog file Elasticsearch	0	83	May 10, 2024
Slow log threshold question Elasticsearch	4	346	May 23, 2022
Capturing all the queries Elasticsearch	7	2222	July 5, 2017
Slow log took time questions Elasticsearch rally	4	540	July 31, 2023
Elasticsearch Slowlog methodology clarification Elasticsearch	1	257	November 29, 2020

Stats about response times

Related topics