[FATAL] Elasticsearch BulkHandler: Unrecoverable indexing failure [400]

Elastic Stack Elasticsearch
1

Hello,

We are using Elasticesearch 6.5.4 with SugarCRM Professional version 8.0.2 on site version

Global search is working fine within Sugar, but we are seeing this in the elasticesearch.log

It is showing the same index "ea842a77ff2db255971c5b09f02b0028_cases" over and over again.

[root@sugarcrm-01 tmp]# tail -f /var/log/elasticsearch/elasticsearch.log

[2019-01-11T00:00:20,008][DEBUG][o.e.a.b.TransportShardBulkAction] [2e6qEfw] [ea842a77ff2db255971c5b09f02b0028_cases][2] failed to execute bulk item (index) index {[ea842a77ff2db255971c5b09f02b0028_cases][Cases][f190e234-e136-3a37-d495-543d6eb545c0], source[n/a, actual length: [35.3kb], max length: 2kb]}
java.lang.IllegalArgumentException: DocValuesField "description" is too large, must be <= 32766
at org.apache.lucene.index.SortedSetDocValuesWriter.addValue(SortedSetDocValuesWriter.java:82) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13]
at org.apache.lucene.index.DefaultIndexingChain.indexDocValue(DefaultIndexingChain.java:615) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13]
at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:467) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13]
at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:394) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13]
at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:251) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13]
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:494) ~[lucene-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - jimczi - 2018-09-18 13:01:13]
org.elasticsearch.action.bulk.TransportShardBulkAction.executeOnPrimaryWhileHandlingMappingUpdates(TransportShardBulkAction.java:483) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:459) [elasticsearch-6.5.4.jar:6.5.4]
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:79) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1022) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1000) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:102) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:356) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:296) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:963) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:960) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:271) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:238) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2327) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:972) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction.access$500(TransportReplicationAction.java:97) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:317) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:292) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:279) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:251) [x-pack-security-6.5.4.jar:6.5.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:309) [x-pack-security-6.5.4.jar:6.5.4]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:717) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) [elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.4.jar:6.5.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_191]
at java.lang.Thread.run(Unknown Source) [?:1.8.0_191]
[2019-01-11T00:00:28,694][DEBUG][o.e.a.b.TransportShardBulkAction] [2e6qEfw] [ea842a77ff2db255971c5b09f02b0028_cases][4] failed to execute bulk item (index) index {[ea842a77ff2db255971c5b09f02b0028_cases][Cases][af6e7235-7c12-b075-ed1b-543d8ee6cdc1], source[n/a, actual length: [53.5kb], max length: 2kb]}
java.lang.IllegalArgumentException: DocValuesField "description" is too large, must be <= 32766

2

[root@sugarcrm-01 tmp]# java -version
java version "1.8.0_191"
Java(TM) SE Runtime Environment (build 1.8.0_191-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode)

[root@sugarcrm-01 tmp]# ps -ef | grep elasticsearch
elastic+ 11797 1 6 Jan10 ? 01:04:35 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch.K9GK8AO7 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/var/log/elasticsearch/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=rpm -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet
elastic+ 11860 11797 0 Jan10 ? 00:00:00 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller
root 26704 26086 0 15:39 pts/1 00:00:00 grep --color=auto elasticsearch
[root@sugarcrm-01 tmp]#

3
  1. Please format your code for better readability.
  2. The relevant error message is DocValuesField "description" is too large, must be <= 32766. The relevant information for this is in the mapping of the description field — please get that information and add it. Overall the problem is that the value of the description field is longer than expected (32KB). and you'll either have to fix your data, the way you store that information, or your mapping.
4

Thank you xeraa

Sorry, I am new to this and Elasticsearch..

Please can you provide details on how do I update the mapping in Elasticsearh ?

Thanks again.

5

Updating mappings is unfortunately not as simple as in relational databases. But first we need the actual mapping. The command should look something like the following, but you'll have to change hostname and port if they are not the defaults:

curl -XGET http://localhost:9200/ea842a77ff2db255971c5b09f02b0028_cases/_mapping
6

Thank you again

Here is the info. it display

[root@sugarcrm-01 ~]# curl -XGET http://localhost:9200/ea842a77ff2db255971c5b09f02b0028_cases/_mapping
{"ea842a77ff2db255971c5b09f02b0028_cases":{"mappings":{"Cases":{"dynamic":"false","_all":{"enabled":false},"properties":{"Cases__assigned_user_id":{"type":"keyword","index":false,"fields":{"gs_not_analyzed":{"type":"keyword","store":true}}},"Cases__case_number":{"type":"keyword","index":false,"fields":{"gs_integer":{"type":"integer","index":false},"gs_string":{"type":"text","store":true,"analyzer":"gs_analyzer_string"},"gs_string_wildcard":{"type":"text","store":true,"analyzer":"gs_analyzer_string_ngram","search_analyzer":"gs_analyzer_string"}}},"Cases__created_by":{"type":"keyword","index":false,"fields":{"gs_not_analyzed":{"type":"keyword","store":true}}},"Cases__date_entered":{"type":"keyword","index":false,"fields":{"gs_datetime":{"type":"date","format":"YYYY-MM-dd HH:mm:ss"}}},"Cases__date_modified":{"type":"keyword","index":false,"fields":{"gs_datetime":{"type":"date","format":"YYYY-MM-dd HH:mm:ss"}}},"Cases__description":{"type":"keyword","index":false,"fields":{"gs_string":{"type":"text","store":true,"analyzer":"gs_analyzer_string"}}},"Cases__initial_contact_email_c":{"type":"keyword","index":false,"fields":{"gs_string":{"type":"text","store":true,"analyzer":"gs_analyzer_string"},"gs_string_wildcard":{"type":"text","store":true,"analyzer":"gs_analyzer_string_ngram","search_analyzer":"gs_analyzer_string"}}},"Cases__initial_contact_officephone_c":{"type":"keyword","index":false,"fields":{"gs_string":{"type":"text","store":true,"analyzer":"gs_analyzer_string"},"gs_string_wildcard":{"type":"text","store":true,"analyzer":"gs_analyzer_string_ngram","search_analyzer":"gs_analyzer_string"}}},"Cases__modified_user_id":{"type":"keyword","index":false,"fields":{"gs_not_analyzed":{"type":"keyword","store":true}}},"Cases__name":{"type":"keyword","index":false,"fields":{"gs_string":{"type":"text","store":true,"analyzer":"gs_analyzer_string"},"gs_string_wildcard":{"type":"text","store":true,"analyzer":"gs_analyzer_string_ngram","search_analyzer":"gs_analyzer_string"}}},"Cases__resolution":{"type":"keyword","index":false,"fields":{"gs_string":{"type":"text","store":true,"analyzer":"gs_analyzer_string"}}},"Cases__slx_ticket_no_c":{"type":"keyword","index":false,"fields":{"gs_string":{"type":"text","store":true,"analyzer":"gs_analyzer_string"},"gs_string_wildcard":{"type":"text","store":true,"analyzer":"gs_analyzer_string_ngram","search_analyzer":"gs_analyzer_string"}}},"Cases__team_set_id":{"type":"keyword","index":false,"fields":{"set":{"type":"keyword"}}},"Cases__work_log":{"type":"keyword","index":false,"fields":{"gs_string":{"type":"text","store":true,"analyzer":"gs_analyzer_string"}}},"Common__assigned_user_id":{"type":"keyword","index":false,"fields":{"agg":{"type":"keyword"}}},"Common__created_by":{"type":"keyword","index":false,"fields":{"agg":{"type":"keyword"}}},"Common__date_modified":{"type":"keyword","index":false,"fields":{"gs_datetime":{"type":"date","format":"YYYY-MM-dd HH:mm:ss"}}},"Common__erased_fields":{"type":"keyword","index":false,"fields":{"erased_fields":{"type":"keyword"}}},"Common__favorite_link":{"type":"keyword","index":false,"fields":{"agg":{"type":"keyword"}}},"Common__modified_user_id":{"type":"keyword","index":false,"fields":{"agg":{"type":"keyword"}}},"Common__owner_id":{"type":"keyword","index":false,"fields":{"owner":{"type":"keyword"}}},"Common__tags":{"type":"keyword","index":false,"fields":{"tags":{"type":"keyword"}}},"Common__user_favorites":{"type":"keyword","index":false,"fields":{"agg":{"type":"keyword"}}},"assigned_user_id":{"type":"keyword","index":false,"copy_to":["Cases__assigned_user_id","Common__assigned_user_id"]},"case_number":{"type":"keyword","index":false,"copy_to":["Cases__case_number"]},"created_by":{"type":"keyword","index":false,"copy_to":["Cases__created_by","Common__created_by"]},"date_entered":{"type":"keyword","index":false,"copy_to":["Cases__date_entered"]},"date_modified":{"type":"keyword","index":false,"copy_to":["Cases__date_modified","Common__date_modified"]},"description":{"type":"keyword","index":false,"copy_to":["Cases__description"]},"erased_fields":{"type":"keyword","index":false,"copy_to":["Common__erased_fields"]},"favorite_link":{"type":"keyword","index":false,"copy_to":["Common__favorite_link"]},"initial_contact_email_c":{"type":"keyword","index":false,"copy_to":["Cases__initial_contact_email_c"]},"initial_contact_officephone_c":{"type":"keyword","index":false,"copy_to":["Cases__initial_contact_officephone_c"]},"modified_user_id":{"type":"keyword","index":false,"copy_to":["Cases__modified_user_id","Common__modified_user_id"]},"name":{"type":"keyword","index":false,"copy_to":["Cases__name"]},"owner_id":{"type":"keyword","index":false,"copy_to":["Common__owner_id"]},"resolution":{"type":"keyword","index":false,"copy_to":["Cases__resolution"]},"slx_ticket_no_c":{"type":"keyword","index":false,"copy_to":["Cases__slx_ticket_no_c"]},"tags":{"type":"keyword","index":false,"copy_to":["Common__tags"]},"team_set_id":{"type":"keyword","index":false,"copy_to":["Cases__team_set_id"]},"user_favorites":{"type":"keyword","index":false,"copy_to":["Common__user_favorites"]},"work_log":{"type":"keyword","index":false,"copy_to":["Cases__work_log"]}}}}}}

7

Hello xeraa,

Any update please ?

I am not sure how to update the mapping properly with curl -XPUT http://localhost:9200/.......

I also came across this link when researching this issue.

Here is an example of how I use it in php:

'mapping'    => [
    'type'   => 'multi_field',
    'path'   => 'full',
    'fields' => [
        '{name}' => [
            'type'     => 'string',
            'index'    => 'analyzed',
            'analyzer' => 'standard',
        ],
        'raw' => [
            'type'         => 'string',
            'index'        => 'not_analyzed',
            'ignore_above' => 256,
        ],
    ],
],

In your case you probably want to do as John Petrone told you and set "index": "no" but for anyone else finding this question after, like me, searching on that Exception then your options are:

  • set "index": "no"
  • set "index": "analyze"
  • set "index": "not_analyzed" and "ignore_above": 256
8

Yes, those 3 are your options. The main question is why description would be a keyword / not analyzed? That's normally helpful for sorting by that field or running aggregations on it. Whereas analyzed fields are for full-text search. I'd assume that description should be searchable, so you'd want it to be analyzed.

Unfortunately you can't change that mapping after creation. If you're still prototyping, delete the data and change your mapping. If you have production data you'll want to create a new index with the right mapping and _reindex your existing data into the new index.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.