Problem with auto-mapping in ElasticSearch 5

Elastic Stack Elasticsearch
1

Hello,

We are testing and want to move to ES 5 from ES 1.5.2, the log source PUT into ES looks like this:
{code}
{
"_index": "esb-2016.08.15",
"_type": "ESB",
"_id": "AVaM_VjDQw5s6KmmdycR",
"_score": null,
"_source": {
"eventUUID": "7f95b9a3-439e-4c11-ba10-a5649ec0a83d",
"category": "osgi",
"eventType": "OSGiEvent",
"logMessage": "org.osgi.framework.BundleEvent[source=locator-monitor_6.3.0.SNAPSHOT [310]]",
"logSource": {
"host.name": "xldai-PC",
"process.id": "8244",
"topic": "org/osgi/framework/BundleEvent/STARTED"
},
"logTimestamp": "2016-08-15T06:57:07.330+0000",
"agentId": "agentXilai",
"agentTimestamp": "2016-08-15T06:57:07.330+0000",
"serverTimestamp": "2016-08-15T06:57:08.776+0000",
"audit": false,
"customInfo": {
"bundle": "locator-monitor_6.3.0.SNAPSHOT [310]",
"bundle.id": "310",
"bundle.symbolicName": "locator-monitor",
"timestamp": "1471244227330"
}
}
}
{code}

It worked well when PUT this log into ES 1.5.2, but failed on ES 5 with this exception:
{code}
[2016-08-15 15:50:35,680][DEBUG][rest.suppressed ] path: /talendesb-2016.08.16/ESB/123456, params: {pretty=true
, index=esb-2016.08.15, id=123456, type=ESB}
RemoteTransportException[[node-xld][127.0.0.1:9300][indices:data/write/index[p]]]; nested: IllegalArgumentException[Can'
t merge a non object mapping [customInfo.bundle] with an object mapping [customInfo.bundle]];
Caused by: java.lang.IllegalArgumentException: Can't merge a non object mapping [customInfo.bundle] with an object mappi
ng [customInfo.bundle]
at org.elasticsearch.index.mapper.object.ObjectMapper.merge(ObjectMapper.java:459)
{code}

If rename the "bundle" field into something else e.g. "bundleObj", then it can be PUT into ES 5 successfully.
{code}
"customInfo": {
"bundleObj": "locator-monitor_6.3.0.SNAPSHOT [310]",
"bundle.id": "310",
"bundle.symbolicName": "locator-monitor",
"timestamp": "1471244227330"
}
{code}

Any suggestions for this problem? (rename field name is not good because will incompatible with the existing our product)

Thanks in advance!
Xldai

2

If you want to code format text, use the </> button.

3

You will need first upgrade to 2.4 when out and apply a new setting to support dot in field names.
Then you'll be able to upgrade to 5.0.

Upgrading directly from 1.x to 5.0 won't work if I'm not mistaken.

4

Thanks David!

But seems the field names with dot (e.g. bundle.id) works well, the problem is happen only when the "bundle" and "bundle.id" co-existing in a same json element.

5

What is your mapping?

6

We didn't POST any mapping into ES 5, just using the dynamic mapping. (this worked in ES 1.x)

After post the log json source above, the http://localhost:9200/_mapping?pretty shows:
"mappings" : { }

and got this exception in the ES 5 logs:
[2016-08-17 15:47:35,266][DEBUG][rest.suppressed ] path: /talendesb-2016.08.17/ESB, params: {index=talendesb-20
16.08.17, type=ESB}
RemoteTransportException[[_-Moa3H][127.0.0.1:9300][indices:data/write/index[p]]]; nested: IllegalArgumentException[Can't
merge a non object mapping [customInfo.bundle] with an object mapping [customInfo.bundle]];
Caused by: java.lang.IllegalArgumentException: Can't merge a non object mapping [customInfo.bundle] with an object mappi
ng [customInfo.bundle]
at org.elasticsearch.index.mapper.object.ObjectMapper.merge(ObjectMapper.java:459)
at org.elasticsearch.index.mapper.object.ObjectMapper.merge(ObjectMapper.java:57)
at org.elasticsearch.index.mapper.DocumentParser.createDynamicUpdate(DocumentParser.java:211)
at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:91)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:286)
at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:511)

7

May be open an issue then with a full recreation script?

8

It can be reproduce with cURL.

curl -H "Content-type: application/json" -X PUT -d "{ "eventUUID" : "0e8789c6-e9fb-4924-96eb-f8410c457fb9
", "customInfo" : {"bundle" : "111", "bundle.id" : "222", "bundle.symbolicName": "locator-monitor", "tim
estamp": "1471244227330"} }" http://localhost:9200/my_index2/my_type/3

{"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[_-Moa3H][127.0.0.1:9300][indices:data/write/inde
x[p]]"}],"type":"illegal_argument_exception","reason":"Can't merge a non object mapping [customInfo.bundle] with an obje
ct mapping [customInfo.bundle]"},"status":400}

I just want to be sure here it's an ES 5 mapping issue or invalid data issue before I open an issue ticket.

9

Could you please point me that which setting I should configure to support dot in field names in ElasticSearch 2.4?

10

The issue is that Elasticsearch interprets:

{
  "bundle.id": "222"
}

the same way as

{
  "bundle": {
    "id": "222"
  }
}

So it maps "bundle" as on object. But then "bundle" also needs to be mapped as a string due to

{
  "bundle": "111"
}

which causes mapping conflicts since bundle is also mapped as an object. You would need to organize you documents differently in order not to have such conflicts.

11

Thanks Adrien!

Yes we solved this issue by better organize our json data.

I just asked another question related to ES 2.4.0, could you please also maybe give suggestions? Thanks

Could you please point me that which setting I should configure to support dot in field names in ElasticSearch 2.4?

12

See https://www.elastic.co/guide/en/elasticsearch/reference/2.4/dots-in-names.html#_enabling_support_for_dots_in_field_names

13

This setting works well! Thank you Adrien!