>30s to create an index

Elastic Stack Elasticsearch
1

Hi guys,

On a 4 nodes cluster a new index creation takes more than 30 seconds.
Is it something usual ?

$ time curl -XPUT http://localhost:9023/test/user/kimchy -d '{
"name" : "Shay Banon"
}'
{"ok":true,"_index":"test","_type":"user","_id":"kimchy","_version":1}
real 0m32.836s
user 0m0.002s
sys 0m0.005s

I enabled the cluster.service logs to see what happens. I put the logs on a
gist https://gist.github.com/Filirom1/4754342

Cheers
Romain

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Filirom1 wrote:

On a 4 nodes cluster a new index creation takes more than 30 seconds.
Is it something usual ?

$ time curl -XPUT http://localhost:9023/test/user/kimchy -d '{
"name" : "Shay Banon"
}'
{"ok":true,"_index":"test","_type":"user","_id":"kimchy","_version":1}
real 0m32.836s
user 0m0.002s
sys 0m0.005s

I enabled the cluster.service logs to see what happens. I put the logs on a
gist ElasticSearch log with cluster.service in DEBUG · GitHub

It looks like you've created, or are creating, a bunch of unique
indices and types as you index. Can we see the indexing code?
Perhaps you're doing something inefficiently we can help you think
through.

-Drew

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

When I did the curl command to create a new index, the cluster was doing
nothing else.
The indexing code is the curl command.

This 4 nodes cluster is not used in production. It just for test purpose,
so I can try what you want on it.

2013/2/11 Drew Raines aaraines@gmail.com

Filirom1 wrote:

On a 4 nodes cluster a new index creation takes more than 30 seconds.
Is it something usual ?

$ time curl -XPUT http://localhost:9023/test/user/kimchy -d '{
"name" : "Shay Banon"
}'
{"ok":true,"_index":"test","_type":"user","_id":"kimchy","_version":1}
real 0m32.836s
user 0m0.002s
sys 0m0.005s

I enabled the cluster.service logs to see what happens. I put the logs
on a
gist ElasticSearch log with cluster.service in DEBUG · GitHub

It looks like you've created, or are creating, a bunch of unique
indices and types as you index. Can we see the indexing code?
Perhaps you're doing something inefficiently we can help you think
through.

-Drew

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Are there any other indices in the cluster? Types?

It looks like you have a lot of mappings that are continuously getting
refreshed.

On Monday, February 11, 2013 8:46:03 AM UTC-5, Filirom1 wrote:

Hi guys,

On a 4 nodes cluster a new index creation takes more than 30 seconds.
Is it something usual ?

$ time curl -XPUT http://localhost:9023/test/user/kimchy -d '{
"name" : "Shay Banon"
}'
{"ok":true,"_index":"test","_type":"user","_id":"kimchy","_version":1}
real 0m32.836s
user 0m0.002s
sys 0m0.005s

I enabled the cluster.service logs to see what happens. I put the logs on
a gist ElasticSearch log with cluster.service in DEBUG · GitHub

Cheers
Romain

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

There are 51 indices in the cluster, and for each indices there is 6 shards
and 1 replica.

Here is the number of types per index :
[ 6,
9,
6,
1,
15,
2,
3,
31,
1,
3,
4,
2,
0,
3,
5,
5,
86,
4,
1,
50,
1,
0,
1,
3,
21,
3,
8,
51,
6,
3,
2,
27,
2,
318,
3,
711,
2,
4,
21,
11,
3,
2,
2,
4,
5,
3,
1,
3,
8,
9,
1 ]

The download the _mapping, it weights 5.2Mo

2013/2/11 egaumer egaumer@gmail.com

Are there any other indices in the cluster? Types?

It looks like you have a lot of mappings that are continuously getting
refreshed.

On Monday, February 11, 2013 8:46:03 AM UTC-5, Filirom1 wrote:

Hi guys,

On a 4 nodes cluster a new index creation takes more than 30 seconds.
Is it something usual ?

$ time curl -XPUT http://localhost:9023/test/**user/kimchyhttp://localhost:9023/test/user/kimchy-d '{
"name" : "Shay Banon"
}'
{"ok":true,"index":"test","type":"user","id":"kimchy","
version":1}
real 0m32.836s
user 0m0.002s
sys 0m0.005s

I enabled the cluster.service logs to see what happens. I put the logs on
a gist https://gist.github.com/**Filirom1/4754342https://gist.github.com/Filirom1/4754342

Cheers
Romain

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

As said, you created many indices, each with many types, with many
shards, and even more dynamic fields that are refreshing dynamically.

Remember: each new field mapping is a cluster upgrade. Such an operation
is costly as it propagates throughout the whole cluster.

30 seconds are very fast according to the some thousands of cluster
state field mapping upgrade operations you have provoked.

The index creation time scales over the number of nodes, indices, types,
shards, and fields. Use less and you will observe faster creation times.

Best regards,

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

7

Thanks a lot for the explanation !

2013/2/12 Jörg Prante joergprante@gmail.com

As said, you created many indices, each with many types, with many shards,
and even more dynamic fields that are refreshing dynamically.

Remember: each new field mapping is a cluster upgrade. Such an operation
is costly as it propagates throughout the whole cluster.

30 seconds are very fast according to the some thousands of cluster state
field mapping upgrade operations you have provoked.

The index creation time scales over the number of nodes, indices, types,
shards, and fields. Use less and you will observe faster creation times.

Best regards,

Jörg

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@**googlegroups.comelasticsearch%2Bunsubscribe@googlegroups.com
.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

8

I would still love to see the cluster state if you provide, would like to simulate it on our end and see if we can improve things. Can you gist the result of /_cluster/state on the cluster with all the indices?

On Feb 12, 2013, at 2:02 PM, Romain filirom1@gmail.com wrote:

Thanks a lot for the explanation !

2013/2/12 Jörg Prante joergprante@gmail.com
As said, you created many indices, each with many types, with many shards, and even more dynamic fields that are refreshing dynamically.

Remember: each new field mapping is a cluster upgrade. Such an operation is costly as it propagates throughout the whole cluster.

30 seconds are very fast according to the some thousands of cluster state field mapping upgrade operations you have provoked.

The index creation time scales over the number of nodes, indices, types, shards, and fields. Use less and you will observe faster creation times.

Best regards,

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

9

I would prefer not to publish the mapping on internet. I sent it to you by
email.

Thank you
Romain

2013/2/12 kimchy@gmail.com

I would still love to see the cluster state if you provide, would like to
simulate it on our end and see if we can improve things. Can you gist the
result of /_cluster/state on the cluster with all the indices?

On Feb 12, 2013, at 2:02 PM, Romain filirom1@gmail.com wrote:

Thanks a lot for the explanation !

2013/2/12 Jörg Prante joergprante@gmail.com

As said, you created many indices, each with many types, with many
shards, and even more dynamic fields that are refreshing dynamically.

Remember: each new field mapping is a cluster upgrade. Such an operation
is costly as it propagates throughout the whole cluster.

30 seconds are very fast according to the some thousands of cluster state
field mapping upgrade operations you have provoked.

The index creation time scales over the number of nodes, indices, types,
shards, and fields. Use less and you will observe faster creation times.

Best regards,

Jörg

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@**googlegroups.comelasticsearch%2Bunsubscribe@googlegroups.com
.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.