Slow Log Duplicates

Elastic Stack Elasticsearch
1

Hi,

I've set up elasticsearch and it works as expected.
However, I need to log all the queries that get fetched. I set up the
configuration in the elasticsearch.yml as follows :

index.search.slowlog.level: TRACE
index.search.slowlog.threshold.query.trace: 0

and this in the logging.yml

additivity:
index.search.slowlog: true

The logging works, however, I get duplicate entries for each search query.
I'm not using any client library, I've written a small client myself to
check out elasticsearch.
Here is a sample from the slow log :

TRACE - [Zynger] [dI][1] took[63.3ms], took_millis[63],
search_type[QUERY_THEN_FETCH], total_shards[5],
source[{"from":0,"size":"25","explain":0,"sort":["_score"],"min_score":0.1,"fields":["Gr","Ic","Vert","Title"],"query":{"bool":{"should":[{"query_string":{"auto_generate_phrase_queries":0,"enable_position_increments":0,"fields":["Title.partial^2","DB","Typ","Lang","Are","keys"],"defaultOperator":"AND","query":"lamp"}}]}}}],
extra_source[],
TRACE - [Zynger] [dI][3] took[67.2ms], took_millis[67],
search_type[QUERY_THEN_FETCH], total_shards[5],
source[{"from":0,"size":"25","explain":0,"sort":["_score"],"min_score":0.1,"fields":["Gr","Ic","Vert","Title"],"query":{"bool":{"should":[{"query_string":{"auto_generate_phrase_queries":0,"enable_position_increments":0,"fields":["Title.partial^2","DB","Typ","Lang","Are","keys"],"defaultOperator":"AND","query":"lamp"}}]}}}],
extra_source[],
TRACE - [Zynger] [dI][2] took[3.8ms], took_millis[3],
search_type[QUERY_THEN_FETCH], total_shards[5],
source[{"from":0,"size":"25","explain":0,"sort":["_score"],"min_score":0.1,"fields":["Gr","Ic","Vert","Title"],"query":{"bool":{"should":[{"query_string":{"auto_generate_phrase_queries":0,"enable_position_increments":0,"fields":["Title.partial^2","DB","Typ","Lang","Are","keys"],"defaultOperator":"AND","query":"lamp"}}]}}}],
extra_source[],

Am I missing something?
Thanks

--

2

You are seeing the "fetch" stage from each shard, note it says:

TRACE - [Zynger] [dI][1]
TRACE - [Zynger] [dI][3]
TRACE - [Zynger] [dI][2]

On Fri, Nov 23, 2012 at 11:51 PM, Aaditi Sharma
aaditi.p.sharma@gmail.comwrote:

Hi,

I've set up elasticsearch and it works as expected.
However, I need to log all the queries that get fetched. I set up the
configuration in the elasticsearch.yml as follows :

index.search.slowlog.level: TRACE
index.search.slowlog.threshold.query.trace: 0

and this in the logging.yml

additivity:
index.search.slowlog: true

The logging works, however, I get duplicate entries for each search query.
I'm not using any client library, I've written a small client myself to
check out elasticsearch.
Here is a sample from the slow log :

TRACE - [Zynger] [dI][1] took[63.3ms], took_millis[63],
search_type[QUERY_THEN_FETCH], total_shards[5],
source[{"from":0,"size":"25","explain":0,"sort":["_score"],"min_score":0.1,"fields":["Gr","Ic","Vert","Title"],"query":{"bool":{"should":[{"query_string":{"auto_generate_phrase_queries":0,"enable_position_increments":0,"fields":["Title.partial^2","DB","Typ","Lang","Are","keys"],"defaultOperator":"AND","query":"lamp"}}]}}}],
extra_source,
TRACE - [Zynger] [dI][3] took[67.2ms], took_millis[67],
search_type[QUERY_THEN_FETCH], total_shards[5],
source[{"from":0,"size":"25","explain":0,"sort":["_score"],"min_score":0.1,"fields":["Gr","Ic","Vert","Title"],"query":{"bool":{"should":[{"query_string":{"auto_generate_phrase_queries":0,"enable_position_increments":0,"fields":["Title.partial^2","DB","Typ","Lang","Are","keys"],"defaultOperator":"AND","query":"lamp"}}]}}}],
extra_source,
TRACE - [Zynger] [dI][2] took[3.8ms], took_millis[3],
search_type[QUERY_THEN_FETCH], total_shards[5],
source[{"from":0,"size":"25","explain":0,"sort":["_score"],"min_score":0.1,"fields":["Gr","Ic","Vert","Title"],"query":{"bool":{"should":[{"query_string":{"auto_generate_phrase_queries":0,"enable_position_increments":0,"fields":["Title.partial^2","DB","Typ","Lang","Are","keys"],"defaultOperator":"AND","query":"lamp"}}]}}}],
extra_source,

Am I missing something?
Thanks

--

--

3

Hi Clinton,

I'm not familiar with logging, could you guide me in what do I need to
change to log entries after the fetch stage?
Thanks.

On Sat, Nov 24, 2012 at 5:30 AM, Clinton Gormley clint@traveljury.comwrote:

You are seeing the "fetch" stage from each shard, note it says:

TRACE - [Zynger] [dI][1]
TRACE - [Zynger] [dI][3]
TRACE - [Zynger] [dI][2]

On Fri, Nov 23, 2012 at 11:51 PM, Aaditi Sharma <aaditi.p.sharma@gmail.com

wrote:

Hi,

I've set up elasticsearch and it works as expected.
However, I need to log all the queries that get fetched. I set up the
configuration in the elasticsearch.yml as follows :

index.search.slowlog.level: TRACE
index.search.slowlog.threshold.query.trace: 0

and this in the logging.yml

additivity:
index.search.slowlog: true

The logging works, however, I get duplicate entries for each search query.
I'm not using any client library, I've written a small client myself to
check out elasticsearch.
Here is a sample from the slow log :

TRACE - [Zynger] [dI][1] took[63.3ms], took_millis[63],
search_type[QUERY_THEN_FETCH], total_shards[5],
source[{"from":0,"size":"25","explain":0,"sort":["_score"],"min_score":0.1,"fields":["Gr","Ic","Vert","Title"],"query":{"bool":{"should":[{"query_string":{"auto_generate_phrase_queries":0,"enable_position_increments":0,"fields":["Title.partial^2","DB","Typ","Lang","Are","keys"],"defaultOperator":"AND","query":"lamp"}}]}}}],
extra_source,
TRACE - [Zynger] [dI][3] took[67.2ms], took_millis[67],
search_type[QUERY_THEN_FETCH], total_shards[5],
source[{"from":0,"size":"25","explain":0,"sort":["_score"],"min_score":0.1,"fields":["Gr","Ic","Vert","Title"],"query":{"bool":{"should":[{"query_string":{"auto_generate_phrase_queries":0,"enable_position_increments":0,"fields":["Title.partial^2","DB","Typ","Lang","Are","keys"],"defaultOperator":"AND","query":"lamp"}}]}}}],
extra_source,
TRACE - [Zynger] [dI][2] took[3.8ms], took_millis[3],
search_type[QUERY_THEN_FETCH], total_shards[5],
source[{"from":0,"size":"25","explain":0,"sort":["_score"],"min_score":0.1,"fields":["Gr","Ic","Vert","Title"],"query":{"bool":{"should":[{"query_string":{"auto_generate_phrase_queries":0,"enable_position_increments":0,"fields":["Title.partial^2","DB","Typ","Lang","Are","keys"],"defaultOperator":"AND","query":"lamp"}}]}}}],
extra_source,

Am I missing something?
Thanks

--

--

--

4

Each shard is performing the search. So all calls are logged.
You can not change that.

You can disable TRACE logs for all classes, for a single class or for a method but AFAIK you can not achieve what you want to do. You have to filter that on your side.

I'm wondering: what is your use case?
Perhaps you can handle it from your client application and trace the ES call when you send the search request?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 24 nov. 2012 à 01:44, Aaditi Sharma aaditi.p.sharma@gmail.com a écrit :

Hi Clinton,

I'm not familiar with logging, could you guide me in what do I need to change to log entries after the fetch stage?
Thanks.

On Sat, Nov 24, 2012 at 5:30 AM, Clinton Gormley clint@traveljury.com wrote:

You are seeing the "fetch" stage from each shard, note it says:

TRACE - [Zynger] [dI][1]
TRACE - [Zynger] [dI][3]
TRACE - [Zynger] [dI][2]

On Fri, Nov 23, 2012 at 11:51 PM, Aaditi Sharma aaditi.p.sharma@gmail.com wrote:

Hi,

I've set up elasticsearch and it works as expected.
However, I need to log all the queries that get fetched. I set up the configuration in the elasticsearch.yml as follows :

index.search.slowlog.level: TRACE
index.search.slowlog.threshold.query.trace: 0

and this in the logging.yml

additivity:
index.search.slowlog: true

The logging works, however, I get duplicate entries for each search query.
I'm not using any client library, I've written a small client myself to check out elasticsearch.
Here is a sample from the slow log :

TRACE - [Zynger] [dI][1] took[63.3ms], took_millis[63], search_type[QUERY_THEN_FETCH], total_shards[5], source[{"from":0,"size":"25","explain":0,"sort":["_score"],"min_score":0.1,"fields":["Gr","Ic","Vert","Title"],"query":{"bool":{"should":[{"query_string":{"auto_generate_phrase_queries":0,"enable_position_increments":0,"fields":["Title.partial^2","DB","Typ","Lang","Are","keys"],"defaultOperator":"AND","query":"lamp"}}]}}}], extra_source,
TRACE - [Zynger] [dI][3] took[67.2ms], took_millis[67], search_type[QUERY_THEN_FETCH], total_shards[5], source[{"from":0,"size":"25","explain":0,"sort":["_score"],"min_score":0.1,"fields":["Gr","Ic","Vert","Title"],"query":{"bool":{"should":[{"query_string":{"auto_generate_phrase_queries":0,"enable_position_increments":0,"fields":["Title.partial^2","DB","Typ","Lang","Are","keys"],"defaultOperator":"AND","query":"lamp"}}]}}}], extra_source,
TRACE - [Zynger] [dI][2] took[3.8ms], took_millis[3], search_type[QUERY_THEN_FETCH], total_shards[5], source[{"from":0,"size":"25","explain":0,"sort":["_score"],"min_score":0.1,"fields":["Gr","Ic","Vert","Title"],"query":{"bool":{"should":[{"query_string":{"auto_generate_phrase_queries":0,"enable_position_increments":0,"fields":["Title.partial^2","DB","Typ","Lang","Are","keys"],"defaultOperator":"AND","query":"lamp"}}]}}}], extra_source,

Am I missing something?
Thanks

--

--

--

--