Help with MergeMappingException please


(Graham Jones) #1

I have an ES index with 370 million records in it.

I added 5 new records with fields what were all going to be objects.
It then turned out that these new objects might have values in their keys
that are unacceptable to ES and our other databases.
I therefore changed the data that is going into these fields to just be a
JSON blob.

But now my mapping is all wrong and I can't see to find any way to delete
the fields or change the type of them - all for 5 records!

If I try to update the mapping with this:

curl -XPUT 'http://hs3:9200/index/user/_mapping' -d '

{

"user": {

"properties": {

  "something_cloud": {

    "type": "string"

  }

}

}

}'

I get this:

{

"error": "MergeMappingException[Merge failed with failures {[Can't merge
a non object mapping [something_cloud] with an object mapping
[something_cloud]]}]",

"status": 400

}

It might be possible to reload all 370 million records but I would have to
do this to another index and then rename it to the index people are
expecting to use. I can't see a way to rename an index either.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/424e50d7-d2ac-4733-955c-b3eb13fbdfa6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(David Pilato) #2

A common good practice is using aliases.
With aliases you can easily create a new index do whatever you want and then switch your alias from old index to new one.

You can not "mutate" an object to a String.

Can't you modify the source document?
How looks like a "good" document vs a "bad" one?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 19 mars 2014 à 08:47:09, Graham Jones (netsinque@gmail.com) a écrit:

I have an ES index with 370 million records in it.

I added 5 new records with fields what were all going to be objects.
It then turned out that these new objects might have values in their keys that are unacceptable to ES and our other databases.
I therefore changed the data that is going into these fields to just be a JSON blob.

But now my mapping is all wrong and I can't see to find any way to delete the fields or change the type of them - all for 5 records!

If I try to update the mapping with this:
curl -XPUT 'http://hs3:9200/index/user/_mapping' -d '
{
"user": {
"properties": {
"something_cloud": {
"type": "string"
}
}
}
}'

I get this:
{
"error": "MergeMappingException[Merge failed with failures {[Can't merge a non object mapping [something_cloud] with an object mapping [something_cloud]]}]",
"status": 400
}

It might be possible to reload all 370 million records but I would have to do this to another index and then rename it to the index people are expecting to use. I can't see a way to rename an index either.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/424e50d7-d2ac-4733-955c-b3eb13fbdfa6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.53295185.2a487cb0.97ca%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.


(Graham Jones) #3

I have deleted all 5 offending records but I still can't change the
mapping.

Do you suspect that there might be some other record that I missed holding
at the object type?

A "good" record is simply a json blob of the object instead of the object.

On Wednesday, March 19, 2014 7:12:53 PM UTC+11, David Pilato wrote:

A common good practice is using aliases.
With aliases you can easily create a new index do whatever you want and
then switch your alias from old index to new one.

You can not "mutate" an object to a String.

Can't you modify the source document?
How looks like a "good" document vs a "bad" one?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr

Le 19 mars 2014 à 08:47:09, Graham Jones (nets...@gmail.com <javascript:>)
a écrit:

I have an ES index with 370 million records in it.

I added 5 new records with fields what were all going to be objects.
It then turned out that these new objects might have values in their keys
that are unacceptable to ES and our other databases.
I therefore changed the data that is going into these fields to just be a
JSON blob.

But now my mapping is all wrong and I can't see to find any way to delete
the fields or change the type of them - all for 5 records!

If I try to update the mapping with this:

curl -XPUT 'http://hs3:9200/index/user/_mapping' -d '

{

"user": {

"properties": {

  "something_cloud": {

    "type": "string"

  }

}

}

}'

I get this:

{

"error": "MergeMappingException[Merge failed with failures {[Can't merge
a non object mapping [something_cloud] with an object mapping
[something_cloud]]}]",

"status": 400

}

It might be possible to reload all 370 million records but I would have to
do this to another index and then rename it to the index people are
expecting to use. I can't see a way to rename an index either.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/424e50d7-d2ac-4733-955c-b3eb13fbdfa6%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/424e50d7-d2ac-4733-955c-b3eb13fbdfa6%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d88b2c58-61d9-4fd8-b0f0-490bf3349079%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(David Pilato) #4

I'm afraid I don't understand.
Could you illustrate with an example how a "good" record looks like?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 19 mars 2014 à 09:23:41, Graham Jones (netsinque@gmail.com) a écrit:

I have deleted all 5 offending records but I still can't change the mapping.

Do you suspect that there might be some other record that I missed holding at the object type?

A "good" record is simply a json blob of the object instead of the object.

On Wednesday, March 19, 2014 7:12:53 PM UTC+11, David Pilato wrote:
A common good practice is using aliases.
With aliases you can easily create a new index do whatever you want and then switch your alias from old index to new one.

You can not "mutate" an object to a String.

Can't you modify the source document?
How looks like a "good" document vs a "bad" one?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 19 mars 2014 à 08:47:09, Graham Jones (nets...@gmail.com) a écrit:

I have an ES index with 370 million records in it.

I added 5 new records with fields what were all going to be objects.
It then turned out that these new objects might have values in their keys that are unacceptable to ES and our other databases.
I therefore changed the data that is going into these fields to just be a JSON blob.

But now my mapping is all wrong and I can't see to find any way to delete the fields or change the type of them - all for 5 records!

If I try to update the mapping with this:
curl -XPUT 'http://hs3:9200/index/user/_mapping' -d '
{
"user": {
"properties": {
"something_cloud": {
"type": "string"
}
}
}
}'

I get this:
{
"error": "MergeMappingException[Merge failed with failures {[Can't merge a non object mapping [something_cloud] with an object mapping [something_cloud]]}]",
"status": 400
}

It might be possible to reload all 370 million records but I would have to do this to another index and then rename it to the index people are expecting to use. I can't see a way to rename an index either.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/424e50d7-d2ac-4733-955c-b3eb13fbdfa6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d88b2c58-61d9-4fd8-b0f0-490bf3349079%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.532954ae.57e4ccaf.97ca%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.


(Graham Jones) #5

A good record looks like a string.

Here is an example but it really could be anything:

record = {'test': 1}

record

{'test': 1} # Bad - this is an actual object

json.dumps(record)

'{"test": 1}' # Good - this is the object json serialized as a string

So for the other question?
Do you suspect that there is some record still left with an object in that
field? Is that why I can't change the mapping?
In other words, if there are no records with those fields, should I be able
to change the mapping however I wish?

I assume there is no way to remove the field completely?
On Wednesday, March 19, 2014 7:26:22 PM UTC+11, David Pilato wrote:

I'm afraid I don't understand.
Could you illustrate with an example how a "good" record looks like?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr

Le 19 mars 2014 à 09:23:41, Graham Jones (nets...@gmail.com <javascript:>)
a écrit:

I have deleted all 5 offending records but I still can't change the
mapping.

Do you suspect that there might be some other record that I missed holding
at the object type?

A "good" record is simply a json blob of the object instead of the object.

On Wednesday, March 19, 2014 7:12:53 PM UTC+11, David Pilato wrote:

A common good practice is using aliases.
With aliases you can easily create a new index do whatever you want and
then switch your alias from old index to new one.

You can not "mutate" an object to a String.

Can't you modify the source document?
How looks like a "good" document vs a "bad" one?

 -- 

David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr

Le 19 mars 2014 à 08:47:09, Graham Jones (nets...@gmail.com) a écrit:

I have an ES index with 370 million records in it.

I added 5 new records with fields what were all going to be objects.
It then turned out that these new objects might have values in their keys
that are unacceptable to ES and our other databases.
I therefore changed the data that is going into these fields to just be a
JSON blob.

But now my mapping is all wrong and I can't see to find any way to delete
the fields or change the type of them - all for 5 records!

If I try to update the mapping with this:

curl -XPUT 'http://hs3:9200/index/user/_mapping' -d '

{

"user": {

"properties": {

  "something_cloud": {

    "type": "string"

  }

}

}

}'

I get this:

{

"error": "MergeMappingException[Merge failed with failures {[Can't
merge a non object mapping [something_cloud] with an object mapping
[something_cloud]]}]",

"status": 400

}

It might be possible to reload all 370 million records but I would have
to do this to another index and then rename it to the index people are
expecting to use. I can't see a way to rename an index either.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/424e50d7-d2ac-4733-955c-b3eb13fbdfa6%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/424e50d7-d2ac-4733-955c-b3eb13fbdfa6%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d88b2c58-61d9-4fd8-b0f0-490bf3349079%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/d88b2c58-61d9-4fd8-b0f0-490bf3349079%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2e3cc2b0-25ff-400d-8cbc-4337ed951d48%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(David Pilato) #6

Definitely you can't change the mapping afterwards.
You can merge mapping (adding new fields) but you can not change an existing field.

Removing some documents won't help here as the mapping will still the same.
You can not remove a field from a mapping AFAIK.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 19 mars 2014 à 09:38:34, Graham Jones (netsinque@gmail.com) a écrit:

A good record looks like a string.

Here is an example but it really could be anything:

record = {'test': 1}
record
{'test': 1} # Bad - this is an actual object

json.dumps(record)
'{"test": 1}' # Good - this is the object json serialized as a string

So for the other question?
Do you suspect that there is some record still left with an object in that field? Is that why I can't change the mapping?
In other words, if there are no records with those fields, should I be able to change the mapping however I wish?

I assume there is no way to remove the field completely?
On Wednesday, March 19, 2014 7:26:22 PM UTC+11, David Pilato wrote:
I'm afraid I don't understand.
Could you illustrate with an example how a "good" record looks like?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 19 mars 2014 à 09:23:41, Graham Jones (nets...@gmail.com) a écrit:

I have deleted all 5 offending records but I still can't change the mapping.

Do you suspect that there might be some other record that I missed holding at the object type?

A "good" record is simply a json blob of the object instead of the object.

On Wednesday, March 19, 2014 7:12:53 PM UTC+11, David Pilato wrote:
A common good practice is using aliases.
With aliases you can easily create a new index do whatever you want and then switch your alias from old index to new one.

You can not "mutate" an object to a String.

Can't you modify the source document?
How looks like a "good" document vs a "bad" one?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 19 mars 2014 à 08:47:09, Graham Jones (nets...@gmail.com) a écrit:

I have an ES index with 370 million records in it.

I added 5 new records with fields what were all going to be objects.
It then turned out that these new objects might have values in their keys that are unacceptable to ES and our other databases.
I therefore changed the data that is going into these fields to just be a JSON blob.

But now my mapping is all wrong and I can't see to find any way to delete the fields or change the type of them - all for 5 records!

If I try to update the mapping with this:
curl -XPUT 'http://hs3:9200/index/user/_mapping' -d '
{
"user": {
"properties": {
"something_cloud": {
"type": "string"
}
}
}
}'

I get this:
{
"error": "MergeMappingException[Merge failed with failures {[Can't merge a non object mapping [something_cloud] with an object mapping [something_cloud]]}]",
"status": 400
}

It might be possible to reload all 370 million records but I would have to do this to another index and then rename it to the index people are expecting to use. I can't see a way to rename an index either.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/424e50d7-d2ac-4733-955c-b3eb13fbdfa6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d88b2c58-61d9-4fd8-b0f0-490bf3349079%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2e3cc2b0-25ff-400d-8cbc-4337ed951d48%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.53295ba6.684a481a.97ca%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.


(system) #7