Unable to add "total_fields.limit" in template in ELK 6.3.0

deepsing · August 30, 2018, 5:21am

Hi

Followed this link to update the template Increase total fields limit via creation of index in logstash to add "index.mapping.total_fields.limit": 10000 in template.json

Below is the part of my updated template.json file.
I tried to add "index.mapping.total_fields.limit: 10000" under setting. But still it is not working

{
  "template": "event-%{+YYYY.MM.dd}",
  "settings": {
    "index.refresh_interval" : "5s",
    "index.mapping.total_fields.limit": 10000
  },
  "mappings": {
    "logs": {
      "dynamic_templates": [
        {
          "boolean_value": {
            "match": "*_boolean",
            "mapping": {
              "type": "boolean"
            }
          }
        },

Updated logstash.conf

output {
    elasticsearch { 
	    index => "event-%{+YYYY.MM.dd}" 
	    hosts => ["elasticsearch-machine"]
        template => "C:\apps\logstash-6.3.0\config\templates\template.json"	
        template_name => 'event' # The default is logstash
        template_overwrite => true		
	}

Run GET event-/_settings command from kibana UI, There is no "index.mapping.total_fields.limit": 10000 in the setting of "event-"

Error on Elastic Search cmd console

[2018-08-28T02:44:44,268][DEBUG][o.e.a.a.i.m.p.TransportPutMappingAction] [TIsOAqH] failed to put mappings on indices [[[event-2018.08.28/lYtCv2TZSvikfNhXHklUuQ]]], type [doc]
java.lang.IllegalArgumentException: Limit of total fields [1000] in index [event-2018.08.28] has been exceeded
        at org.elasticsearch.index.mapper.MapperService.checkTotalFieldsLimit(MapperService.java:630) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:463) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:356) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:288) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest(MetaDataMappingService.java:313) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.execute(MetaDataMappingService.java:230) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:630) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:267) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:197) [elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:132) [elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-6.3.0.jar:6.3.0]

        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [elasticsearch-6.3.0.jar:6.3.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
[2018-08-28T02:44:44,417][DEBUG][o.e.a.b.TransportShardBulkAction] [event-2018.08.28][1] failed to execute bulk item (index) BulkShardRequest [[event-2018.08.28][1]] containing [10] requests
java.lang.IllegalArgumentException: Limit of total fields [1000] in index [event-2018.08.28] has been exceeded
        at org.elasticsearch.index.mapper.MapperService.checkTotalFieldsLimit(MapperService.java:630) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:463) ~[elasticsearch-6.3.0.jar:6.3.0]

deepsing · August 30, 2018, 5:22am

More information

Christian_Dahlqvist · August 30, 2018, 5:28am

Try changing this to "template": "event-*",. You can not use the Logstash date pattern construct here.

deepsing · August 30, 2018, 11:20am

Hi @Christian_Dahlqvist

When I changed it in the template.json.
I am getting below error in ElasticSearch console:

[2018-08-31T04:15:27,878][DEBUG][o.e.a.b.TransportShardBulkAction] [event-2018.08.31][0] failed to execute bulk item (index) BulkShardRequest [[event-2018.08.31][0]] containing [125] requests
java.lang.IllegalArgumentException: Rejecting mapping update to [event-2018.08.31] as the final mapping would have more than 1 type: [doc, logs]
        at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:408) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:356) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:288) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest(MetaDataMappingService.java:313) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.execute(MetaDataMappingService.java:230) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:630) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:267) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:197) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:132) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) ~[elasticsearch-6.3.0.jar:6.3.0
]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) ~[elasticsearch-6.3.0.jar:6.3.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
[2018-08-31T04:15:27,941][ERROR][i.n.u.c.D.rejectedExecution] Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated
        at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:821) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:327) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:320) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:746) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:760) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:428) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:113) ~[?:?]
        at io.netty.channel.DefaultChannelPromise.setFailure(DefaultChannelPromise.java:87) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.safeExecute(AbstractChannelHandlerContext.java:1010) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:825) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1036) ~[?:?]
        at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:305) ~[?:?]
        at org.elasticsearch.http.netty4.Netty4HttpChannel.sendResponse(Netty4HttpChannel.java:146) ~[?:?]
        at org.elasticsearch.rest.RestController$ResourceHandlingHttpChannel.sendResponse(RestController.java:496) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:37) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:47) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:85) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:81) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) [elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:379) [elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onFailure(TransportBulkAction.java:374) [elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:91) [elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:897) [elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleException(TransportReplicationAction.java:855) [elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1095) [elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:268) [elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:724) [elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.3.0.jar:6.3.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]

deepsing · August 30, 2018, 11:21am

Also getting error on logstash console:

deepsing · August 31, 2018, 1:11pm

Hi @Christian_Dahlqvist

Can you please help me to solve this issue.

Christian_Dahlqvist · August 31, 2018, 2:51pm

In your template you have specified the type as logs:

"mappings": {
    "logs": {
      "dynamic_templates": [

The Elasticsearch output plugin writes documents with type doc which causes a failure as an index now can have only one type.

Rejecting mapping update to [event-2018.08.31] as the final mapping would have more than 1 type: [doc, logs]

Change from logs to doc in your template and I think it should work for new indices.

deepsing · September 1, 2018, 7:28am

Hi

Changed the templates, but still I am getting issues:

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
[2018-09-02T00:12:13,177][DEBUG][o.e.a.b.TransportShardBulkAction] [event-2018.09.02][0] failed to execute bulk item (index) BulkShardRequest [[event-2018.09.02][0]] containing [125] requ
org.elasticsearch.index.mapper.MapperParsingException: object mapping for [event.action.actionTypes] tried to parse field [null] as object, but found a concrete value
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:357) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:478) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:603) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.mapper.DocumentParser.parseNonDynamicArray(DocumentParser.java:590) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.mapper.DocumentParser.parseArray(DocumentParser.java:538) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:392) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:380) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:478) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:496) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:390) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:380) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:478) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:496) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:390) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:380) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:95) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:69) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:261) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:700) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:677) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:658) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.lambda$executeIndexRequestOnPrimary$2(TransportShardBulkAction.java:553) ~[elasticsearch-6.3.0.jar:6.3.0]

deepsing · September 1, 2018, 7:29am

Error on logstash console:

deepsing · September 5, 2018, 2:32am

Hi Christian

Can you help me on this

Christian_Dahlqvist · September 5, 2018, 5:10am

It looks like you may have a field ‘event.action.actionTypes’ that sometimes contains an object and sometimes the value ‘null’. As each field in an index need to have a single mapping, this is causing problems.

deepsing · September 6, 2018, 11:41am

Hi Chrisitian

I want to make this field 'event.action.actionTypes' optional.
Earlier, this template was running fine, when I used logs instead of doc, and event-YYYY-MM-dd.

Now, why it started to show this issue?

Thanks
Deepak

Christian_Dahlqvist · September 6, 2018, 11:51am

What does the document that fails look like? What does successful documents look like?

deepsing · September 6, 2018, 12:04pm

Didn't get you @Christian_Dahlqvist

Christian_Dahlqvist · September 6, 2018, 12:06pm

In order to identify what has changed it would help to see mappings (before and after) and well as sample documents.

deepsing · September 6, 2018, 12:12pm

Do you want to see the template.json or sample data?

deepsing · September 6, 2018, 12:14pm

Sample Data before changes:

Christian_Dahlqvist · September 6, 2018, 12:56pm

That document does as far as I can see not have a event.action.actionTypes field.

deepsing · September 6, 2018, 1:01pm

Yes, I know

Do you want to see that document, where actionTypes feed is present, but why?

Christian_Dahlqvist · September 6, 2018, 1:03pm

That is what Elasticsearch is complaining about.

Topic		Replies	Views
How to increase number of fields in index template? Logstash	2	957	April 4, 2018
Limit of total fields [1000] in index [p_bs_api_abcd_2020_23] has been exceeded Logstash	9	1047	July 3, 2020
Setting 'index.mapping.total_fields.limit' on Index Template Elasticsearch	4	8932	April 17, 2017
Increase total fields limit via creation of index in logstash Logstash	7	14780	September 27, 2017
Total Fields Limit setting Elasticsearch	6	124456	July 5, 2017

Unable to add "total_fields.limit" in template in ELK 6.3.0

Related topics