ElasticSearch - transport-couchbase


#1

We have run into a scenario where one of the fields in our couchbase documents has a "long" (0) instead of the string ("0"). We are using dynamic mapping.

There are conflicting data types for this field "doc.gameJournalEntries.gameAction.gameType" In most cases, values for this field is a string. However, a few documents now have a long value of 0. This is throwing the dynamic mapper of ElasticSearch into an exception and data stops replicating from Couchbase.

Technically, ES should be able to handle this and I tried adding dynamic templates to handle this, but ES seems to be picky, in that, once dynamic mapping identifies a field data type, it expects all data for that field to be in the same data type. Here is the couchbase_template.json

{
"template" : "*",
"order" : 10,
"mappings" : {
    "couchbaseCheckpoint" : {
        "_source" : {
            "includes" : ["doc.*"]
        },
        "dynamic_templates": [
            {
                    "longs": {
                            "match_mapping_type": "long",
                            "mapping": {
                                    "type": "text"
                            }
                    }
            },
            {
                    "strings": {
                            "match_mapping_type": "string",
                            "mapping": {
                                    "type": "text"
                            }
                    }
            },
            {
                    "longs_as_strings": {
                            "match":   "[doc.id]",
                            "mapping": {
                                    "type": "text"
                            }
                    }
            },
            {
                    "longs_as_floats": {
                            "match":   "[doc.bridgeServerBalanceAmount]",
                            "mapping": {
                                    "type": "float"
                            }
                    }
            },
            {
                "store_no_index": {
                    "match": "*",
                    "mapping": {
                        "store" : "no",
                        "index" : "no",
                        "include_in_all" : false
                    }
                }
            }
        ]
    },
    "doc": {
            "properties": {
                    "id":    { "type": "text"  } ,
                    "doc.id":    { "type": "text"  }
            }
    },
    "_default_" : {
        "_source" : {
            "includes" : ["meta.*","doc.*"]
        },
        "properties" : {
            "meta" : {
                "type" : "object",
                "include_in_all" : false
            }
        }
    }
}

}

And here is the exception:

[2018-01-05T23:34:20,083][ERROR][o.e.t.c.c.CouchbaseCAPITransportImpl] [wKd0fSJ] Error indexing document, will NOT retry. Document id: XiGpGameDetails969a43fe-21f4-42ba-b378-4ea278d69f9a_5 exception: java.lang.IllegalArgumentException: mapper [doc.gameJournalEntries.gameAction.gameType] of different type, current_type [text], merged_type [long]

[2018-01-05T23:34:20,084][ERROR][o.e.t.c.c.CouchbaseCAPITransportImpl] [wKd0fSJ] bulk index requestId=100 Failing due to errors: java.lang.IllegalArgumentException: mapper [doc.gameJournalEntries.gameAction.gameType] of different type, current_type [text], merged_type [long]

[2018-01-05T23:34:20,084][WARN ][o.e.j.s.ServletHandler ] /poc-sandbox%2f62/_bulk_docs
java.lang.RuntimeException: java.lang.IllegalArgumentException: mapper [doc.gameJournalEntries.gameAction.gameType] of different type, current_type [text], merged_type [long]

at org.elasticsearch.transport.couchbase.capi.ElasticSearchCAPIBehavior.bulkDocs(ElasticSearchCAPIBehavior.java:548) ~[elasticsearch-transport-couchbase-3.0.0-cypress-es5.6.4-all.jar:?]
at com.couchbase.capi.servlet.CAPIServlet.handleBulkDocs(CAPIServlet.java:532) ~[couchbase-capi-server-1.6.3.jar:?]
at com.couchbase.capi.servlet.CAPIServlet.service(CAPIServlet.java:89) ~[couchbase-capi-server-1.6.3.jar:?]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845) ~[jetty-servlet-9.3.13.v20161014.jar:9.3.13.v20161014]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:584) [jetty-servlet-9.3.13.v20161014.jar:9.3.13.v20161014]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [jetty-server-9.3.13.v20161014.jar:9.3.13.v20161014]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:513) [jetty-security-9.3.13.v20161014.jar:9.3.13.v20161014]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) [jetty-server-9.3.13.v20161014.jar:9.3.13.v20161014]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) [jetty-server-9.3.13.v20161014.jar:9.3.13.v20161014]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) [jetty-servlet-9.3.13.v20161014.jar:9.3.13.v20161014]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-server-9.3.13.v20161014.jar:9.3.13.v20161014]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) [jetty-server-9.3.13.v20161014.jar:9.3.13.v20161014]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.3.13.v20161014.jar:9.3.13.v20161014]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [jetty-server-9.3.13.v20161014.jar:9.3.13.v20161014]
at org.eclipse.jetty.server.Server.handle(Server.java:523) [jetty-server-9.3.13.v20161014.jar:9.3.13.v20161014]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) [jetty-server-9.3.13.v20161014.jar:9.3.13.v20161014]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [jetty-server-9.3.13.v20161014.jar:9.3.13.v20161014]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) [jetty-io-9.3.13.v20161014.jar:9.3.13.v20161014]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) [jetty-io-9.3.13.v20161014.jar:9.3.13.v20161014]
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [jetty-io-9.3.13.v20161014.jar:9.3.13.v20161014]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [jetty-util-9.3.13.v20161014.jar:9.3.13.v20161014]
at

#2

I also tried adding a dynamic rule:
{ "type_as_strings": { "match": "[doc.gameJournalEntries.gameAction.gameType]", "mapping": { "type": "text" } } },

This did not help


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.