Problem with auto-mapping in ElasticSearch 5


(xldai) #1

Hello,

We are testing and want to move to ES 5 from ES 1.5.2, the log source PUT into ES looks like this:
{code}
{
"_index": "esb-2016.08.15",
"_type": "ESB",
"_id": "AVaM_VjDQw5s6KmmdycR",
"_score": null,
"_source": {
"eventUUID": "7f95b9a3-439e-4c11-ba10-a5649ec0a83d",
"category": "osgi",
"eventType": "OSGiEvent",
"logMessage": "org.osgi.framework.BundleEvent[source=locator-monitor_6.3.0.SNAPSHOT [310]]",
"logSource": {
"host.name": "xldai-PC",
"process.id": "8244",
"topic": "org/osgi/framework/BundleEvent/STARTED"
},
"logTimestamp": "2016-08-15T06:57:07.330+0000",
"agentId": "agentXilai",
"agentTimestamp": "2016-08-15T06:57:07.330+0000",
"serverTimestamp": "2016-08-15T06:57:08.776+0000",
"audit": false,
"customInfo": {
"bundle": "locator-monitor_6.3.0.SNAPSHOT [310]",
"bundle.id": "310",
"bundle.symbolicName": "locator-monitor",
"timestamp": "1471244227330"
}
}
}
{code}

It worked well when PUT this log into ES 1.5.2, but failed on ES 5 with this exception:
{code}
[2016-08-15 15:50:35,680][DEBUG][rest.suppressed ] path: /talendesb-2016.08.16/ESB/123456, params: {pretty=true
, index=esb-2016.08.15, id=123456, type=ESB}
RemoteTransportException[[node-xld][127.0.0.1:9300][indices:data/write/index[p]]]; nested: IllegalArgumentException[Can'
t merge a non object mapping [customInfo.bundle] with an object mapping [customInfo.bundle]];
Caused by: java.lang.IllegalArgumentException: Can't merge a non object mapping [customInfo.bundle] with an object mappi
ng [customInfo.bundle]
at org.elasticsearch.index.mapper.object.ObjectMapper.merge(ObjectMapper.java:459)
{code}

If rename the "bundle" field into something else e.g. "bundleObj", then it can be PUT into ES 5 successfully.
{code}
"customInfo": {
"bundleObj": "locator-monitor_6.3.0.SNAPSHOT [310]",
"bundle.id": "310",
"bundle.symbolicName": "locator-monitor",
"timestamp": "1471244227330"
}
{code}

Any suggestions for this problem? (rename field name is not good because will incompatible with the existing our product)

Thanks in advance!
Xldai


(Mark Walkom) #2

If you want to code format text, use the </> button.


(David Pilato) #3

You will need first upgrade to 2.4 when out and apply a new setting to support dot in field names.
Then you'll be able to upgrade to 5.0.

Upgrading directly from 1.x to 5.0 won't work if I'm not mistaken.


(xldai) #4

Thanks David!

But seems the field names with dot (e.g. bundle.id) works well, the problem is happen only when the "bundle" and "bundle.id" co-existing in a same json element.


(David Pilato) #5

What is your mapping?


(xldai) #6

We didn't POST any mapping into ES 5, just using the dynamic mapping. (this worked in ES 1.x)

After post the log json source above, the http://localhost:9200/_mapping?pretty shows:
"mappings" : { }

and got this exception in the ES 5 logs:
[2016-08-17 15:47:35,266][DEBUG][rest.suppressed ] path: /talendesb-2016.08.17/ESB, params: {index=talendesb-20
16.08.17, type=ESB}
RemoteTransportException[[_-Moa3H][127.0.0.1:9300][indices:data/write/index[p]]]; nested: IllegalArgumentException[Can't
merge a non object mapping [customInfo.bundle] with an object mapping [customInfo.bundle]];
Caused by: java.lang.IllegalArgumentException: Can't merge a non object mapping [customInfo.bundle] with an object mappi
ng [customInfo.bundle]
at org.elasticsearch.index.mapper.object.ObjectMapper.merge(ObjectMapper.java:459)
at org.elasticsearch.index.mapper.object.ObjectMapper.merge(ObjectMapper.java:57)
at org.elasticsearch.index.mapper.DocumentParser.createDynamicUpdate(DocumentParser.java:211)
at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:91)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:286)
at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:511)


(David Pilato) #7

May be open an issue then with a full recreation script?


(xldai) #8

It can be reproduce with cURL.

curl -H "Content-type: application/json" -X PUT -d "{ "eventUUID" : "0e8789c6-e9fb-4924-96eb-f8410c457fb9
", "customInfo" : {"bundle" : "111", "bundle.id" : "222", "bundle.symbolicName": "locator-monitor", "tim
estamp": "1471244227330"} }" http://localhost:9200/my_index2/my_type/3

{"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[_-Moa3H][127.0.0.1:9300][indices:data/write/inde
x[p]]"}],"type":"illegal_argument_exception","reason":"Can't merge a non object mapping [customInfo.bundle] with an obje
ct mapping [customInfo.bundle]"},"status":400}

I just want to be sure here it's an ES 5 mapping issue or invalid data issue before I open an issue ticket.


(xldai) #9

Could you please point me that which setting I should configure to support dot in field names in ElasticSearch 2.4?


(Adrien Grand) #10

The issue is that Elasticsearch interprets:

{
  "bundle.id": "222"
}

the same way as

{
  "bundle": {
    "id": "222"
  }
}

So it maps "bundle" as on object. But then "bundle" also needs to be mapped as a string due to

{
  "bundle": "111"
}

which causes mapping conflicts since bundle is also mapped as an object. You would need to organize you documents differently in order not to have such conflicts.


(xldai) #11

Thanks Adrien!

Yes we solved this issue by better organize our json data.

I just asked another question related to ES 2.4.0, could you please also maybe give suggestions? Thanks

Could you please point me that which setting I should configure to support dot in field names in ElasticSearch 2.4?


(Adrien Grand) #12

See https://www.elastic.co/guide/en/elasticsearch/reference/2.4/dots-in-names.html#_enabling_support_for_dots_in_field_names


(xldai) #13

This setting works well! Thank you Adrien!


(system) #14