Index deleted when putMapping is used twice


(Alexandre) #1

Hi !

I am developping a small library that integrates Morphia (MongoDB) and
ElasticSearch using the ES java API.
During application startup I use Annotated Entities to generate the ES
index Mapping.

Everything works pretty fine and I am able to create the index, then
analyzers and the type mappings properly . Indexing works great too.

However I have noted that when restarting the application and going
through the index mapping generation process again (using putMapping
API) the content of the index gets deleted.

I do perform a check to see if the index already exists to avoid
recreating it, however, in my Library the mappings are automatically
uploaded each time the app starts which maybe is the problem ...

Is it a normal behavior of ES? Is the content wiped away when type
mappings are redefined, even if they are exactly the same as the
prevoius ones?
Is there a way to get existing mappings and compare them to the ones
I'm about to upload to avoid this problem?

Hope I'm clear enough :slight_smile:

Thanks for your help

Cheers !

Alexandre


(Karussell) #2

the putMapping api should merge the mapping of the fields:
http://www.elasticsearch.org/guide/reference/api/admin-indices-put-mapping.html

but content should stay or are you doing a delete mapping reqest?
http://www.elasticsearch.org/guide/reference/api/admin-indices-delete-mapping.html

Can you track down the issues to some curl or java statements?

and yes, you can also get the current mapping
http://www.elasticsearch.org/guide/reference/api/admin-indices-get-mapping.html

I would recommend to reindex the data if the mapping changed otherwise
you possibly get wrong mapping merges and incorrect indexed fields...

Regards,
Peter.

On 21 Okt., 09:23, Alexandre azli...@gmail.com wrote:

Hi !

I am developping a small library that integrates Morphia (MongoDB) and
ElasticSearch using the ES java API.
During application startup I use Annotated Entities to generate the ES
index Mapping.

Everything works pretty fine and I am able to create the index, then
analyzers and the type mappings properly . Indexing works great too.

However I have noted that when restarting the application and going
through the index mapping generation process again (using putMapping
API) the content of the index gets deleted.

I do perform a check to see if the index already exists to avoid
recreating it, however, in my Library the mappings are automatically
uploaded each time the app starts which maybe is the problem ...

Is it a normal behavior of ES? Is the content wiped away when type
mappings are redefined, even if they are exactly the same as the
prevoius ones?
Is there a way to get existing mappings and compare them to the ones
I'm about to upload to avoid this problem?

Hope I'm clear enough :slight_smile:

Thanks for your help

Cheers !

Alexandre


(Karussell) #3

I would recommend to reindex the data

only if you changed or removed a field ... for new fields updating
without reindexing should be fine


(Alexandre) #4

Can you track down the issues to some curl or java statements?

Sure !

Create index is called at framework startup :

then each entity is analyzed and a mapping is generated using an
XContentBuilder. Here is an example of a generated mapping :

and here are the instructions used to put mapping in es :

Finally insertion of random data .

Works great on first launch

BUT, if I relaunch it without indexing random stuff ... my previous
indexed content has dissapeard from ES ...

Thanks again for you help


(Shay Banon) #5

Thanks for the gists, one way to simulate it faster is if you have a simpel
"main" program that uses the APIs to shows it. Can you do it?

On Fri, Oct 21, 2011 at 2:36 PM, Alexandre azlist1@gmail.com wrote:

Can you track down the issues to some curl or java statements?

Sure !

Create index is called at framework startup :

https://gist.github.com/1303673

then each entity is analyzed and a mapping is generated using an
XContentBuilder. Here is an example of a generated mapping :

https://gist.github.com/1303707

and here are the instructions used to put mapping in es :

https://gist.github.com/1303712

Finally insertion of random data .

https://gist.github.com/1303716

Works great on first launch

BUT, if I relaunch it without indexing random stuff ... my previous
indexed content has dissapeard from ES ...

Thanks again for you help


(Alexandre) #6

Hey !

Thanks for following up !

Here's a full test that you can run.

You will jsut need a small lorem ipsum lib to generate the random
content and its available here :

http://loremipsum.sourceforge.net/

As you will see content is wiped out of the index ... -_-

Let me know what you think.

Regards,

Alexandre


(Shay Banon) #7

Hey,

I took your code and ran it (on 0.17.9, and master), and it works for me.
Started both a 1 node cluster, and a 2 nodes cluster, and getting the
correct doc count on both cases. Any more data on your usage?

Btw, few notes on elasticsearch usage. You don't have to close an index to
put the mappings. Also, you can provide the mappings on index creation time
(so you remove the need to have the put mapping call). Last, if you want to
make things searchable "right away", you can simply call the refresh API.

On Sat, Oct 22, 2011 at 12:27 AM, Alexandre azlist1@gmail.com wrote:

Hey !

Thanks for following up !

Here's a full test that you can run.

https://gist.github.com/1305141

You will jsut need a small lorem ipsum lib to generate the random
content and its available here :

http://loremipsum.sourceforge.net/

As you will see content is wiped out of the index ... -_-

Let me know what you think.

Regards,

Alexandre


(Alexandre) #8

Hi,

I just upgraded to 0.17.9 and re-ran the test I gave you... and the
problem was still here for me ...

Then I also modified my program a bit to take some of your comments
into account such as using refresh instead of sleep and NOT taking the
index down before putMapping.

And swoosh!! problem seems to have disappeared ...

I don't know what the source of the problem was but maybe there is a
hidden bug to investigate ...

Anyway thank you very much for your help.
I'll probably be bugging you again soon ...

Have a nice day !


(Shay Banon) #9

Heya, it still should not happen. Can you give more details on your
deployment? How big is the cluster (how many nodes), configuration of it.

On Sat, Oct 22, 2011 at 2:13 PM, Alexandre azlist1@gmail.com wrote:

Hi,

I just upgraded to 0.17.9 and re-ran the test I gave you... and the
problem was still here for me ...

Then I also modified my program a bit to take some of your comments
into account such as using refresh instead of sleep and NOT taking the
index down before putMapping.

And swoosh!! problem seems to have disappeared ...

I don't know what the source of the problem was but maybe there is a
hidden bug to investigate ...

Anyway thank you very much for your help.
I'll probably be bugging you again soon ...

Have a nice day !


(Alexandre) #10

This is a dev / test server so I only have 1 node + 1 shard + 0
replicas

  • 1 client connecting to it on the same machine.

(Alexandre) #11

Configuration is the default config I think ... I didnt change
anything as far as I can remember.


(Shay Banon) #12

tried to recreate it again with a single node, 1 shard 0 replica index, and
still seems to work.. . Maybe you can check what config you used?

On Sat, Oct 22, 2011 at 11:42 PM, Alexandre azlist1@gmail.com wrote:

Configuration is the default config I think ... I didnt change
anything as far as I can remember.


(Alexandre) #13

Ok so first of all here are all the configuration infos I can find in
my GUI :

Next I noticed something strange in my console that occurs when
triggering the createIndex(true) method of the test I gave you.
I don0t if its related to the same problem ...

In the logs I get the following :

Exception caught on netty layer [[id: 0x552297d7, /192.168.0.52:55124
=> /192.168.0.52:9301]]

here is the full log of what happened just when creating the index.


(Alexandre) #14

Plus I am now sure that the index wiping problem ONLY occurs when the
index is taken down before putMapping calls in the test ...

If I comment out the close + open instructions the problem does not
occur ...

//
getClient().admin().indices().prepareClose(globalIndexname).execute().actionGet();

//
getClient().admin().indices().prepareOpen(globalIndexname).execute().actionGet();


(Alexandre) #15

Hey !

Just to let you know :

I just tested this again today on a different machine (32bit vista)
with a fresh install of ES 0.17.9 with no configuration changes from
what was downloaded and the weird index wiping still occurred.

When the index is taken down before putMapping (even though its not
necessary), the first putMapping call works fine and documents are
indexed but the seccond putMapping call (with same mapping) wipes
the index.

When the index is NOT taken down both putMapping call work fine and
the documents are still there after both calls.

Hope this helps you.

Cheers !

Alex


(system) #16