Force all the JSON values as String type - Error - Merging dynamic updates triggered a conflict

Elastic Stack Elasticsearch
#1

Hello,

We are trying to store a lot of JSON but ElasticSearch throws the following exception:

Merging dynamic updates triggered a conflict: mapper [our_type] of different type, current_type [string], merged_type [long].

Could we force ElasticSearch to store with the same type String for all the values from the JSON?

Many thanks.

Regards,
Paco.

Drawbacks of mapping all types as string?
#2

You can use dynamic templates: https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-templates.html

I think that something like the following should work:

{
  "mappings": {
    "_default_": {
      "dynamic_templates": [
        {
          "rule1": {
            "mapping": {
              "type": "string"
            },
            "match": "*",
            "match_mapping_type": "*"
          }
        }
      ]
    }
  }
}
```
#3

Many thanks, I will try it :wink:

Could it affect to the performance?

Thanks again :slightly_smiling:

Regards,
Paco.

#4

Could it affect to the performance?

Nothing you should worried about :wink:

#5

Many thanks!

#6

Hello again,

After creating a Java method

	    CreateIndexRequestBuilder createIndexRequestBuilder = client.admin().indices().prepareCreate(INDEX_NAME);
		// CREATE SETTINGS
		String settings_json = new String(Files.readAllBytes((Paths.get("C:\\mapping.json")))) ;
		createIndexRequestBuilder.setSettings(settings_json);
		// CREATE MAPPING
		String mapping_json = new String(Files.readAllBytes((Paths.get("C:\\mapping.json")))) ;
		createIndexRequestBuilder.addMapping(INDEX_NAME, mapping_json);
		CreateIndexResponse indexResponse = createIndexRequestBuilder.execute().actionGet();

I got the following exception:

Caused by: MapperParsingException[Root mapping definition has unsupported parameters:  [mappings : {_default_={dynamic_templates=[{rule1={mapping={type=string}, match_mapping_type=*, match=*}}]}}]]
	at org.elasticsearch.index.mapper.DocumentMapperParser.checkNoRemainingFields(DocumentMapperParser.java:192)
	at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:180)
	at org.elasticsearch.index.mapper.DocumentMapperParser.parseCompressed(DocumentMapperParser.java:121)
	at org.elasticsearch.index.mapper.MapperService.parse(MapperService.java:391)
	at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:265)
	at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$2.execute(MetaDataCreateIndexService.java:378)
	... 6 more

I am using Elastic Search 2.1.1.

Many thanks again.

Regards,
Paco.

#7

Very very strange...because of after using the "curl" command and the following script I could do it (but it was impossible using the Java API):

curl -XPOST "http://localhost:9200/my_index" -d "{\"mappings\": {\"_default_\": {\"dynamic_templates\": [{\"rule1\": {\"mapping\": {\"type\": \"string\"}, \"match\": \"*\", \"match_mapping_type\": \"*\"}}]}}}"

On the other hand, after configuring the index with the previous script, I launched my application but I receive the following exception from Elastic Search:

Exception 'failed to parse [tag_a]' while indexing the following content: {"tag_a":{"tag_b":{"tag_c": ... }}
Exception in thread "main" MapperParsingException[failed to parse [tag_a]]; nested: IllegalArgumentException[unknown property [tag_b]];
	at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:339)
	at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:314)
	at org.elasticsearch.index.mapper.DocumentParser.parseAndMergeUpdate(DocumentParser.java:762)
	at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:357)
	at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:257)
	at org.elasticsearch.index.mapper.DocumentParser.innerParseDocument(DocumentParser.java:127)
	at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:79)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:304)
	at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:551)
	at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:542)
	at org.elasticsearch.action.support.replication.TransportReplicationAction.prepareIndexOperationOnPrimary(TransportReplicationAction.java:1049)
	at org.elasticsearch.action.support.replication.TransportReplicationAction.executeIndexRequestOnPrimary(TransportReplicationAction.java:1060)
	at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:170)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.performOnPrimary(TransportReplicationAction.java:579)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase$1.doRun(TransportReplicationAction.java:452)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: unknown property [tag_b]
	at org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateFieldForString(StringFieldMapper.java:348)
	at org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:294)
	at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:331)
	... 18 more

I can confirm to you it is because after configuring the default mapping, ES thinks everything is a String, and in this case it is a subdocument. Is there another way to specify that all the values different than a subdocument or arrays, i.e. simple types, are Strings, or do I have to specify one by one all the types that I want to define as Strings? This last choice it cannot be an option because I have thousand of types.

Many thanks.

Regards,
Paco.

#8

Is there another way to specify that all the values different than a subdocument or arrays, i.e. simple types, are Strings, or do I have to specify one by one all the types that I want to define as Strings? This last choice it cannot be an option because I have thousand of types.

Only these types can be automatically detected: boolean, date, double, long, object, string, so you'll have to define a rule for boolean, date, double, long and string only.
You can check this page for further documentation:
https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-templates.html#match-mapping-type

#9

Many thanks

I think that it could be the expected configuration:

{
  "mappings": { "my_type": {
      "dynamic_templates": [
        {
          "booleans": {
            "match_mapping_type": "boolean",
            "mapping": {
              "type": "string"
            }
          }
        },
		{
          "dates": {
            "match_mapping_type": "date",
            "mapping": {
              "type": "string"
            }
          }
        },
		{
          "doubles": {
            "match_mapping_type": "double",
            "mapping": {
              "type": "string"
            }
          }
        },
		{
          "longs": {
            "match_mapping_type": "long",
            "mapping": {
              "type": "string"
            }
          }
        }	
      ]
    }
  }
}

But now, I have another problem when the tag could be a simple type or an object, for example:

<a>
  <b>1234</b>
  <b myAttr="attrValue">crash!</b>
</a>
  • If you see, the first "a.b" tag, its simple value is 1234 but the mapper will use the type String.
  • The problem is in the second tag "a.b", because the content is a "object" due to the internal attribute. Then, when I launch the application I see the following exception:

MapperParsingException[Merging dynamic updates triggered a conflict: mapper [a.b] of different type, current_type [string], merged_type [ObjectMapper]]

If I redefine the previous configuration and I add the "object" as "String":

{
 "objects": {
 "match_mapping_type": "object",
 "mapping": {
   "type": "string"
   }
}

I will get the following exception:

MapperParsingException[failed to parse [a]]; nested: IllegalArgumentException[unknown property [b]];

When I use a simple XML like this:

<a><b>hello</b></a>

Any other idea?

Many thanks.

Regards,
Paco.

#10

Could anyone help to me in this topic?

Thanks a lot! :slightly_smiling:

#11

Hi Paco_B!!
Did you solve your problem, I have a similar situation and I can not find the solution. I will appreciate if you share with me your knowledge.

#12

Nobody helped to me :frowning:

I used "Spark" technology to solve the issue.

Good luck!

#13