ES seems to be stuck in resyncing mappings

lukeforehand_2 · August 18, 2011, 2:33pm

I am revisiting this archived thread:
http://groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/fe64dd72d233b1f7/30d8df648ae32beb?lnk=gst&q=ES+seems+to+be+stuck+in+resyncing+mappings#30d8df648ae32beb

I have gisted my log here:

gist.github.com

https://gist.github.com/lukeforehand/1152872

gistfile1.txt

[2011-08-17 16:59:51,293][INFO ][cluster.metadata         ] [Agony] [post_1970_01] update_mapping [POST] (dynamic)
[2011-08-17 16:59:51,295][INFO ][cluster.metadata         ] [Agony] [post_2009_36] update_mapping [POST] (dynamic)
[2011-08-17 16:59:51,298][INFO ][cluster.metadata         ] [Agony] [post_2009_31] update_mapping [POST] (dynamic)
[2011-08-17 16:59:51,301][INFO ][cluster.metadata         ] [Agony] [post_2011_31] update_mapping [POST] (dynamic)
[2011-08-17 16:59:53,644][INFO ][cluster.metadata         ] [Agony] [post_2011_21] creating index, cause [auto(bulk api)], shards [4]/[1], mappings [default, POST]
[2011-08-17 16:59:53,699][INFO ][cluster.metadata         ] [Agony] [post_2011_12] creating index, cause [auto(bulk api)], shards [4]/[1], mappings [default, POST]
[2011-08-17 16:59:53,756][INFO ][cluster.metadata         ] [Agony] [post_2009_28] creating index, cause [auto(bulk api)], shards [4]/[1], mappings [default, POST]
[2011-08-17 16:59:53,812][INFO ][cluster.metadata         ] [Agony] [post_2011_10] creating index, cause [auto(bulk api)], shards [4]/[1], mappings [default, POST]
[2011-08-17 16:59:53,870][INFO ][cluster.metadata         ] [Agony] [post_2011_20] creating index, cause [auto(bulk api)], shards [4]/[1], mappings [default, POST]
[2011-08-17 16:59:53,917][INFO ][cluster.metadata         ] [Agony] [post_2011_08] creating index, cause [auto(bulk api)], shards [4]/[1], mappings [default, POST]

This file has been truncated. show original

We are also experiencing this problem with version 0.17.6

Does anyone have any insight as to how this can happen?

-Luke

kimchy · August 18, 2011, 4:24pm

Is it something you can easily recreate? Can you post your mappings
definition for "default"?

On Thu, Aug 18, 2011 at 5:33 PM, lukeforehand lukeforehand@gmail.comwrote:

I am revisiting this archived thread:

http://groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/fe64dd72d233b1f7/30d8df648ae32beb?lnk=gst&q=ES+seems+to+be+stuck+in+resyncing+mappings#30d8df648ae32beb

I have gisted my log here:
gist:1152872 · GitHub

We are also experiencing this problem with version 0.17.6

Does anyone have any insight as to how this can happen?

-Luke

lukeforehand_2 · August 18, 2011, 4:43pm

It is easy to re-create, we don't run into this problem when indexing
to a single index. This seems to happen if several indices are being
created in a burst.

Here is the default mapping def:

gist.github.com

https://gist.github.com/lukeforehand/1154470

gistfile1.txt

cat /etc/elasticsearch/mappings/_default/default.json

{
    "default" : {
        "dynamic_templates" : [
            {
                "template_1" : {
                    "match" : "*",
                    "match_mapping_type" : "string",
                    "mapping" : {

This file has been truncated. show original

-Luke

On Aug 18, 11:24 am, Shay Banon kim...@gmail.com wrote:

Is it something you can easily recreate? Can you post your mappings
definition for "default"?

On Thu, Aug 18, 2011 at 5:33 PM, lukeforehand lukeforeh...@gmail.comwrote:

I am revisiting this archived thread:

http://groups.google.com/a/elasticsearch.com/group/users/browse_threa...

I have gisted my log here:
gist:1152872 · GitHub

We are also experiencing this problem with version 0.17.6

Does anyone have any insight as to how this can happen?

-Luke

kimchy · August 18, 2011, 4:56pm

Can you post a sample curl indexing that recreates it? I assume that just
starting two nodes, with the below configuration and indexing a doc causes
that?

On Thu, Aug 18, 2011 at 7:43 PM, lukeforehand lukeforehand@gmail.comwrote:

It is easy to re-create, we don't run into this problem when indexing
to a single index. This seems to happen if several indices are being
created in a burst.

Here is the default mapping def:

gist:1154470 · GitHub

-Luke

On Aug 18, 11:24 am, Shay Banon kim...@gmail.com wrote:

Is it something you can easily recreate? Can you post your mappings
definition for "default"?

On Thu, Aug 18, 2011 at 5:33 PM, lukeforehand <lukeforeh...@gmail.com
wrote:

I am revisiting this archived thread:

http://groups.google.com/a/elasticsearch.com/group/users/browse_threa.
..

I have gisted my log here:
gist:1152872 · GitHub

We are also experiencing this problem with version 0.17.6

Does anyone have any insight as to how this can happen?

-Luke

kimchy · August 18, 2011, 6:15pm

I did not manage to recreate it, but I think I found where it might happen.
Pushed a fix: dynamic templates might cause elasticsearch to keep resync mappings · Issue #1257 · elastic/elasticsearch · GitHub(its
on 0.17 branch as well). Can you check it if helps?

On Thu, Aug 18, 2011 at 7:56 PM, Shay Banon kimchy@gmail.com wrote:

Can you post a sample curl indexing that recreates it? I assume that just
starting two nodes, with the below configuration and indexing a doc causes
that?

On Thu, Aug 18, 2011 at 7:43 PM, lukeforehand lukeforehand@gmail.comwrote:

It is easy to re-create, we don't run into this problem when indexing
to a single index. This seems to happen if several indices are being
created in a burst.

Here is the default mapping def:

gist:1154470 · GitHub

-Luke

On Aug 18, 11:24 am, Shay Banon kim...@gmail.com wrote:

Is it something you can easily recreate? Can you post your mappings
definition for "default"?

On Thu, Aug 18, 2011 at 5:33 PM, lukeforehand <lukeforeh...@gmail.com
wrote:

I am revisiting this archived thread:

http://groups.google.com/a/elasticsearch.com/group/users/browse_threa.
..

I have gisted my log here:
gist:1152872 · GitHub

We are also experiencing this problem with version 0.17.6

Does anyone have any insight as to how this can happen?

-Luke

lukeforehand_2 · August 19, 2011, 2:02pm

Thanks Shay, we have patched and will test today.

On Aug 18, 1:15 pm, Shay Banon kim...@gmail.com wrote:

I did not manage to recreate it, but I think I found where it might happen.
Pushed a fix:dynamic templates might cause elasticsearch to keep resync mappings · Issue #1257 · elastic/elasticsearch · GitHub(its
on 0.17 branch as well). Can you check it if helps?

lukeforehand_2 · August 22, 2011, 6:48pm

We have been testing with the patch for this and cannot reproduce the
problem. More testing to follow, will keep you posted. Thanks,

-Luke

On Aug 19, 9:02 am, lukeforehand lukeforeh...@gmail.com wrote:

Thanks Shay, we have patched and will test today.

On Aug 18, 1:15 pm, Shay Banon kim...@gmail.com wrote:

I did not manage to recreate it, but I think I found where it might happen.
Pushed a fix:dynamic templates might cause elasticsearch to keep resync mappings · Issue #1257 · elastic/elasticsearch · GitHub(its
on 0.17 branch as well). Can you check it if helps?

kimchy · August 22, 2011, 6:51pm

Great, thanks for the update!

On Mon, Aug 22, 2011 at 9:48 PM, lukeforehand lukeforehand@gmail.comwrote:

We have been testing with the patch for this and cannot reproduce the
problem. More testing to follow, will keep you posted. Thanks,

-Luke

On Aug 19, 9:02 am, lukeforehand lukeforeh...@gmail.com wrote:

Thanks Shay, we have patched and will test today.

On Aug 18, 1:15 pm, Shay Banon kim...@gmail.com wrote:

I did not manage to recreate it, but I think I found where it might
happen.
Pushed a fix:
dynamic templates might cause elasticsearch to keep resync mappings · Issue #1257 · elastic/elasticsearch · GitHub(its
on 0.17 branch as well). Can you check it if helps?