Maximum size for index limit size

Hi Elastic search Team

I was facing below issue:
java.lang.IllegalArgumentException: Limit of total fields [1000] in index [event-2018.08.24] has been exceeded at org.elasticsearch.index.mapper.MapperService.checkTotalFieldsLimit(MapperService.java:630)

But I resolved it by increasing index size using below command:

PUT [elasticsearch-hostname]:9200/event-*/_settings
{
index.mapping.total_fields.limit: 5000
}

I want to know that what could be the maximum value of total_fields.limit?
Is it dynamic or static?

It's dynamic up to the 1000 limit. Beyond that it's static based on whatever you set.

I don't know if there's a hard-hard limit in the code, but if you need to increase to a huge number then you may want to review your indexing strategy anyway.

Hi

There is a problem, user has to increase the limit size on every day basis, because below command increase the size of existing event-* setting, not the new event-%{+YYYY.MM.dd}:

PUT event-*/_settings

{
  "index.mapping.total_fields.limit": 10000
}

Could we add below setting in template.json or somewhere else in elastic search conf?
index.mapping.total_fields.limit: 5000
so we don't need to manually update the index limit.

Use a template that contains it for your index pattern and you should be right.

Hi

Followed this link to update the template Increase total fields limit via creation of index in logstash

Below is the part of my updated template.json file.
I tried to add "index.mapping.total_fields.limit: 10000" under setting. But still it is not working

{
  "template": "event-%{+YYYY.MM.dd}",
  "settings": {
    "index.refresh_interval" : "5s",
    "index.mapping.total_fields.limit": 10000
  },
  "mappings": {
    "logs": {
      "dynamic_templates": [
        {
          "boolean_value": {
            "match": "*_boolean",
            "mapping": {
              "type": "boolean"
            }
          }
        },

Updated logstash.conf

output {
    elasticsearch { 
	    index => "event-%{+YYYY.MM.dd}" 
	    hosts => ["elasticsearch-machine"]
        template => "C:\apps\logstash-6.3.0\config\templates\template.json"	
        template_name => 'event' # The default is logstash
        template_overwrite => true		
	}

Run GET event-/_settings command from kibana UI, There is no "index.mapping.total_fields.limit": 10000 in the setting of "event-"

Error on Elastic Search cmd console

[2018-08-28T02:44:44,268][DEBUG][o.e.a.a.i.m.p.TransportPutMappingAction] [TIsOAqH] failed to put mappings on indices [[[event-2018.08.28/lYtCv2TZSvikfNhXHklUuQ]]], type [doc]
java.lang.IllegalArgumentException: Limit of total fields [1000] in index [event-2018.08.28] has been exceeded
        at org.elasticsearch.index.mapper.MapperService.checkTotalFieldsLimit(MapperService.java:630) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:463) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:356) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:288) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest(MetaDataMappingService.java:313) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.execute(MetaDataMappingService.java:230) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:630) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:267) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:197) [elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:132) [elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-6.3.0.jar:6.3.0]

        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [elasticsearch-6.3.0.jar:6.3.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
[2018-08-28T02:44:44,417][DEBUG][o.e.a.b.TransportShardBulkAction] [event-2018.08.28][1] failed to execute bulk item (index) BulkShardRequest [[event-2018.08.28][1]] containing [10] requests
java.lang.IllegalArgumentException: Limit of total fields [1000] in index [event-2018.08.28] has been exceeded
        at org.elasticsearch.index.mapper.MapperService.checkTotalFieldsLimit(MapperService.java:630) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:463) ~[elasticsearch-6.3.0.jar:6.3.0]

Logs on logstash console, I can see "index.mapping.total_fields.limit": 10000 in the logs

C:\apps\logstash-6.3.0\bin>logstash -f ..\config\logstash.conf
Sending Logstash's logs to C:/apps/logstash-6.3.0/logs which is now configured via log4j2.properties
[2018-08-28T02:58:55,313][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-08-28T02:58:56,499][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.3.0"}
[2018-08-28T02:59:09,582][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>6, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-08-28T02:59:11,250][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch-machine:9200/]}}
[2018-08-28T02:59:11,273][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elasticsearch-machine:9200/, :path=>"/"}
[2018-08-28T02:59:11,674][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http:/elasticsearch-machine:9200/"}
[2018-08-28T02:59:11,826][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2018-08-28T02:59:11,840][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2018-08-28T02:59:11,877][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>"C:\\apps\\logstash-6.3.0\\config\\templates\\template.json"}
[2018-08-28T02:59:11,917][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"event-%{+YYYY.MM.dd}", "settings"=>{"index.number_of_shards"=>1, "index
.refresh_interval"=>"5s", "index.mapping.total_fields.limit"=>10000}, "mappings"=>{"logs"=>{"dynamic_templates"=>[{"boolean_value"=>{"match"=>"*_boolean", "mapping"=>{"type"=>"boolean"}}}, {"string_va
lue"=>{"match"=>"*_string", "mapping"=>{"type"=>"text"}}}, {"number_value"=>{"match"=>"*_number", "mapping"=>{"type"=>"text"}}}, {"boolean_strings"=>{"match_mapping_type"=>"boolean", "mapping"=>{"type
"=>"text"}}}, {"date_strings"=>{"match_mapping_type"=>"date", "mapping"=>{"type"=>"text"}}}, {"double_strings"=>{"match_mapping_type"=>"double", "mapping"=>{"type"=>"text"}}}, {"long_strings"=>{"match
_mapping_type"=>"long", "mapping"=>{"type"=>"text"}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "format"=>"yyyy-MM-dd'T'HH:mm:ss.SSSZ"}, "@version"=>{"type"=>"text"}, "event"=>{"type"=>"nested",
 "properties"=>{"resource"=>{"type"=>"nested", "properties"=>{"isSensitive"=>{"type"=>"boolean"}, "uri"=>{"type"=>"text"}, "id"=>{"type"=>"text"}}}, "subject"=>{"type"=>"nested", "properties"=>{"anony
mousId"=>{"type"=>"text"}, "id"=>{"type"=>"text"}, "role"=>{"type"=>"text"}, "sessionId"=>{"type"=>"text"}}}, "action"=>{"type"=>"nested", "properties"=>{"name"=>{"type"=>"text"}, "arguments"=>{"type"
=>"object"}, "outcome"=>{"type"=>"nested", "properties"=>{"result"=>{"type"=>"object"}, "error"=>{"type"=>"text"}}}, "actionTypes"=>{"type"=>"object"}, "time"=>{"type"=>"nested", "properties"=>{"actio
nTime"=>{"type"=>"date", "format"=>"yyyy-MM-dd'T'HH:mm:ss.SSSZ"}}}}}, "environment"=>{"type"=>"nested", "properties"=>{"hostName"=>{"type"=>"text"}, "ip"=>{"type"=>"text"}, "serviceName"=>{"type"=>"te
xt"}, "serviceInstance"=>{"type"=>"text"}}}, "trackingId"=>{"type"=>"text"}, "auditLevel"=>{"type"=>"integer"}}}}}}}}
[2018-08-28T02:59:12,019][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/event
[2018-08-28T02:59:12,228][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//elasticsearch-machine"]}
[2018-08-28T02:59:13,895][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x34390077 run>"}
[2018-08-28T02:59:14,071][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2018-08-28T02:59:14,810][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
{
          "host" => "elasticsearch-machine",
    "@timestamp" => 2018-08-28T10:07:14.632Z,
         "event" => {
        "environment" => {
                   "hostName" => "elasticsearch-machine",
                "serviceName" => "hello-service",
            "serviceInstance" => "hello-service"
        },
            "subject" => {
            "sessionId" => "37e078e5-e112-48f0-b4af-63cce5b11327",
                   "id" => "helloDeepak@example.com"
        },
         "auditLevel" => 1,
           "resource" => {
            "isSensitive" => "false"
        },
             "action" => {
            "arguments" => {
                "lastRenderedTime_string" => "2018-08-28T03:00:43",
                             "preference" => {
                    "enableVisitTypeFilter_boolean" => false,
                                  "timezone_string" => "-0700",
                            "ignoreConsent_boolean" => false
                }
            },
                 "time" => {
                "actionTime" => "2018-08-28T10:07:13.866Z"
            },
                 "name" => "notification.checkForNewNotifications",
              "outcome" => {
                "result" => {
                            "result_boolean" => false,
                    "lastUpdatedDTTM_string" => "2018-08-28T03:07:13"
                }
            }
        },
         "trackingId" => "16dff1c1-91e9-49ca-b2a4-aca2bde04ef8"
    },
          "path" => "C:/Temp/Tomcat/log/hello-service-event.log",
       "message" => "[EVENT] 2018-08-28T03:07:13,877-0700 {\"environment\":{\"hostName\":\"elasticsearch-machine\",\"serviceName\":\"hello-service\",\"serviceInstance\":\"hello-service\"},\"resource\":{\"isSensit
ive\":\"false\"},\"subject\":{\"id\":\"helloDeepak@example.com\",\"sessionId\":\"37e078e5-e112-48f0-b4af-63cce5b11327\"},\"action\":{\"name\":\"notification.checkForNewNotifications\",\"arguments\":{\"l
astRenderedTime_string\":\"2018-08-28T03:00:43\",\"preference\":{\"timezone_string\":\"-0700\",\"enableVisitTypeFilter_boolean\":false,\"ignoreConsent_boolean\":false}},\"time\":{\"actionTime\":\"2018
-08-28T10:07:13.866Z\"},\"outcome\":{\"result\":{\"result_boolean\":false,\"lastUpdatedDTTM_string\":\"2018-08-28T03:07:13\"}}},\"trackingId\":\"16dff1c1-91e9-49ca-b2a4-aca2bde04ef8\",\"auditLevel\":1
}\r",
      "@version" => "1"
}

Please help me to resolve this issue.

Hi @warkolm

Can you please help me to resolve it.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.