Elasticsearch logformat:

Elastic Stack Elasticsearch
1

HI All,

We are using elasticsearch 1.3.7,
below the sample slow log:

[2016-07-25 23:16:50,654][INFO ][index.search.slowlog.query] [NODE_NAME] [INDEX_NAME][2] took[5.2s], took_millis[5200], types[244], stats[], search_type[QUERY_AND_FETCH], total_shards[1], source[{QUERY}], extra_source[],

Here let me know what is that [2] stands for, is it shard id?
[NODE_NAME] [INDEX_NAME][2] took[5.2s],

If it is shard id mean, we are storing data in shard number 1. why it shows shard number 2 ?

2

Hi @Selvam_ayyanar,

this is the shard id if I am not mistaken.

Can you please run GET /YOUR_INDEX_NAME/_settings and post the result here?

Daniel

3

Hi @danielmitterdorfer

Thanks for the response, Here the details you asked

{
"INDEX_NAME" : {
"settings" : {
"index" : {
"number_of_replicas" : "2",
filters.. analyzers.. tokernizers..
{
...
...
...
}
"translog" : {
"disable_flush" : "false"
},
"number_of_shards" : "5",
"refresh_interval" : "2s",
"version" : {
"created" : "900299"
}
}
}
}
}

4

Hi @Selvam_ayyanar,

isn't the output then entirely reasonable given that you have 5 shards?

Daniel

5

HI @danielmitterdorfer,

Yes, we can't share the full o/p of _settings . We have 5 shards,
My question is our data is stored in shard number 1, but in slow logs it shows shard number 2.

6

Hi @Selvam_ayyanar,

Elasticsearch cannot know upfront in which shard data can be found. That's why a search hits all shards and this is completely normal and expected behavior. Makes sense?

Daniel

7

Yes that's right,

Here we are using routing, for example

curl -XPOST 'http://localhost:9200/store/order?routing=1' -d '
{
"productName":"sample",
"customerID":"user123"
}

if we use routing=1 data stored in shard number 2,
if we use routing=5 data stored in shard number 0

8

Did you imagine that the routing value you provide is literally the shard number?

What would you expect to happen if you used "routing=9"? Just fail?

Providing a routing assures the given document will be in the same shard as any document which was indexed with the same routing.

Why did it even bother to search shard number 2? Obviously the document is stored in shard number 1!

You haven't shown your search query, but you should inspect it and consider: what information in your search would allow a system to uniquely associate it with exactly and exclusively the document that ended up (by normal or manual routing) on shard 1. How would it "know", with no uncertainty, on which shard the matching document resided? And if that information was available before dispatching the search to the shards, why not just return it? What are the shards for?

Ok, run that command, indexing that document. Now run

curl -XPOST 'http://localhost:9200/store/order?routing=2' -d '
{   
    "productName":"sample",
    "customerID":"user123"
}

Do that over and over for routing values from 3 through 999.

Now how should your search behave?

A search has to query one copy of every shard in the index. It's not a key/value store.

Sometimes, there are no results from a shard for a given search. And sometimes even a search that comes up empty takes time, like when someone performs a regex query with a leading wildcard. Just because your search forces a full scan of the index doesn't mean it's going to find something.

9

Hi @DiscussBuster

Missed to update here,
we got routing is just a key to push and get the data from es :slight_smile: