# Maximum size for index limit size **URL:** \ **Category:** Elasticsearch\ **Created:** [August 25, 2018, 1:35pm UTC](https://discuss.elastic.co/t/maximum-size-for-index-limit-size/146020 "2018-08-25T13:35:17Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![deepsing](https://avatars.discourse-cdn.com/v4/letter/d/d26b3c/32.png) [@deepsing](https://discuss.elastic.co/u/deepsing)\ **Post date:** [August 25, 2018, 1:35pm UTC](https://discuss.elastic.co/t/maximum-size-for-index-limit-size/146020/1 "2018-08-25T13:35:17Z")
Hi Elastic search Team I was facing below issue: java.lang.IllegalArgumentException: Limit of total fields [1000] in index [event-2018.08.24] has been exceeded at org.elasticsearch.index.mapper.MapperService.checkTotalFieldsLimit(MapperService.java:630) But I resolved it by increasing index size using below command: ```auto PUT [elasticsearch-hostname]:9200/event-*/_settings { index.mapping.total_fields.limit: 5000 } ``` I want to know that what could be the maximum value of total\_fields.limit? Is it dynamic or static? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 27, 2018, 12:30am UTC](https://discuss.elastic.co/t/maximum-size-for-index-limit-size/146020/2 "2018-08-27T00:30:57Z")
It's dynamic up to the 1000 limit. Beyond that it's static based on whatever you set. I don't know if there's a hard-hard limit in the code, but if you need to increase to a huge number then you may want to review your indexing strategy anyway. ---
**Author:** ![deepsing](https://avatars.discourse-cdn.com/v4/letter/d/d26b3c/32.png) [@deepsing](https://discuss.elastic.co/u/deepsing)\ **Post date:** [August 27, 2018, 5:24am UTC](https://discuss.elastic.co/t/maximum-size-for-index-limit-size/146020/3 "2018-08-27T05:24:38Z")
Hi There is a problem, user has to increase the limit size on every day basis, because below command increase the size of existing event-\* setting, not the new **event-%{+YYYY.MM.dd}** : ```auto PUT event-*/_settings { "index.mapping.total_fields.limit": 10000 } ``` Could we add below setting in template.json or somewhere else in elastic search conf? **index.mapping.total\_fields.limit: 5000** so we don't need to manually update the index limit. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 27, 2018, 7:47am UTC](https://discuss.elastic.co/t/maximum-size-for-index-limit-size/146020/4 "2018-08-27T07:47:54Z")
Use a template that contains it for your index pattern and you should be right. ---
**Author:** ![deepsing](https://avatars.discourse-cdn.com/v4/letter/d/d26b3c/32.png) [@deepsing](https://discuss.elastic.co/u/deepsing)\ **Post date:** [August 27, 2018, 9:47am UTC](https://discuss.elastic.co/t/maximum-size-for-index-limit-size/146020/5 "2018-08-27T09:47:04Z")
Hi Followed this link to update the template [Increase total fields limit via creation of index in logstash](https://discuss.elastic.co/t/increase-total-fields-limit-via-creation-of-index-in-logstash/98742) Below is the part of my updated template.json file. I tried to add "index.mapping.total\_fields.limit: 10000" under setting. But still it is not working ```auto { "template": "event-%{+YYYY.MM.dd}", "settings": { "index.refresh_interval" : "5s", "index.mapping.total_fields.limit": 10000 }, "mappings": { "logs": { "dynamic_templates": [ { "boolean_value": { "match": "*_boolean", "mapping": { "type": "boolean" } } }, ``` **Updated logstash.conf** ```auto output { elasticsearch { index => "event-%{+YYYY.MM.dd}" hosts => ["elasticsearch-machine"] template => "C:\apps\logstash-6.3.0\config\templates\template.json" template_name => 'event' # The default is logstash template_overwrite => true } ``` **Run GET event-_/\_settings command from kibana UI, There is no "index.mapping.total\_fields.limit": 10000 in the setting of "event-_"** ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/6/662750703e477a1561b98dcaeaf7c660a99b07bf.png) Error on Elastic Search cmd console ```auto [2018-08-28T02:44:44,268][DEBUG][o.e.a.a.i.m.p.TransportPutMappingAction] [TIsOAqH] failed to put mappings on indices [[[event-2018.08.28/lYtCv2TZSvikfNhXHklUuQ]]], type [doc] java.lang.IllegalArgumentException: Limit of total fields [1000] in index [event-2018.08.28] has been exceeded at org.elasticsearch.index.mapper.MapperService.checkTotalFieldsLimit(MapperService.java:630) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:463) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:356) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:288) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest(MetaDataMappingService.java:313) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.execute(MetaDataMappingService.java:230) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:630) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:267) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:197) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:132) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [elasticsearch-6.3.0.jar:6.3.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181] [2018-08-28T02:44:44,417][DEBUG][o.e.a.b.TransportShardBulkAction] [event-2018.08.28][1] failed to execute bulk item (index) BulkShardRequest [[event-2018.08.28][1]] containing [10] requests java.lang.IllegalArgumentException: Limit of total fields [1000] in index [event-2018.08.28] has been exceeded at org.elasticsearch.index.mapper.MapperService.checkTotalFieldsLimit(MapperService.java:630) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:463) ~[elasticsearch-6.3.0.jar:6.3.0] ``` ---
**Author:** ![deepsing](https://avatars.discourse-cdn.com/v4/letter/d/d26b3c/32.png) [@deepsing](https://discuss.elastic.co/u/deepsing)\ **Post date:** [August 27, 2018, 10:12am UTC](https://discuss.elastic.co/t/maximum-size-for-index-limit-size/146020/6 "2018-08-27T10:12:20Z")
**Logs on logstash console, I can see "index.mapping.total\_fields.limit": 10000 in the logs** ```auto C:\apps\logstash-6.3.0\bin>logstash -f ..\config\logstash.conf Sending Logstash's logs to C:/apps/logstash-6.3.0/logs which is now configured via log4j2.properties [2018-08-28T02:58:55,313][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2018-08-28T02:58:56,499][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"6.3.0"} [2018-08-28T02:59:09,582][INFO][logstash.pipeline] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>6, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50} [2018-08-28T02:59:11,250][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch-machine:9200/]}} [2018-08-28T02:59:11,273][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elasticsearch-machine:9200/, :path=>"/"} [2018-08-28T02:59:11,674][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http:/elasticsearch-machine:9200/"} [2018-08-28T02:59:11,826][INFO][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6} [2018-08-28T02:59:11,840][WARN][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6} [2018-08-28T02:59:11,877][INFO][logstash.outputs.elasticsearch] Using mapping template from {:path=>"C:\\apps\\logstash-6.3.0\\config\\templates\\template.json"} [2018-08-28T02:59:11,917][INFO][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"event-%{+YYYY.MM.dd}", "settings"=>{"index.number_of_shards"=>1, "index .refresh_interval"=>"5s", "index.mapping.total_fields.limit"=>10000}, "mappings"=>{"logs"=>{"dynamic_templates"=>[{"boolean_value"=>{"match"=>"*_boolean", "mapping"=>{"type"=>"boolean"}}}, {"string_va lue"=>{"match"=>"*_string", "mapping"=>{"type"=>"text"}}}, {"number_value"=>{"match"=>"*_number", "mapping"=>{"type"=>"text"}}}, {"boolean_strings"=>{"match_mapping_type"=>"boolean", "mapping"=>{"type "=>"text"}}}, {"date_strings"=>{"match_mapping_type"=>"date", "mapping"=>{"type"=>"text"}}}, {"double_strings"=>{"match_mapping_type"=>"double", "mapping"=>{"type"=>"text"}}}, {"long_strings"=>{"match _mapping_type"=>"long", "mapping"=>{"type"=>"text"}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "format"=>"yyyy-MM-dd'T'HH:mm:ss.SSSZ"}, "@version"=>{"type"=>"text"}, "event"=>{"type"=>"nested", "properties"=>{"resource"=>{"type"=>"nested", "properties"=>{"isSensitive"=>{"type"=>"boolean"}, "uri"=>{"type"=>"text"}, "id"=>{"type"=>"text"}}}, "subject"=>{"type"=>"nested", "properties"=>{"anony mousId"=>{"type"=>"text"}, "id"=>{"type"=>"text"}, "role"=>{"type"=>"text"}, "sessionId"=>{"type"=>"text"}}}, "action"=>{"type"=>"nested", "properties"=>{"name"=>{"type"=>"text"}, "arguments"=>{"type" =>"object"}, "outcome"=>{"type"=>"nested", "properties"=>{"result"=>{"type"=>"object"}, "error"=>{"type"=>"text"}}}, "actionTypes"=>{"type"=>"object"}, "time"=>{"type"=>"nested", "properties"=>{"actio nTime"=>{"type"=>"date", "format"=>"yyyy-MM-dd'T'HH:mm:ss.SSSZ"}}}}}, "environment"=>{"type"=>"nested", "properties"=>{"hostName"=>{"type"=>"text"}, "ip"=>{"type"=>"text"}, "serviceName"=>{"type"=>"te xt"}, "serviceInstance"=>{"type"=>"text"}}}, "trackingId"=>{"type"=>"text"}, "auditLevel"=>{"type"=>"integer"}}}}}}}} [2018-08-28T02:59:12,019][INFO][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/event [2018-08-28T02:59:12,228][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//elasticsearch-machine"]} [2018-08-28T02:59:13,895][INFO][logstash.pipeline] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#"} [2018-08-28T02:59:14,071][INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [2018-08-28T02:59:14,810][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600} ``` ```auto { "host" => "elasticsearch-machine", "@timestamp" => 2018-08-28T10:07:14.632Z, "event" => { "environment" => { "hostName" => "elasticsearch-machine", "serviceName" => "hello-service", "serviceInstance" => "hello-service" }, "subject" => { "sessionId" => "37e078e5-e112-48f0-b4af-63cce5b11327", "id" => "helloDeepak@example.com" }, "auditLevel" => 1, "resource" => { "isSensitive" => "false" }, "action" => { "arguments" => { "lastRenderedTime_string" => "2018-08-28T03:00:43", "preference" => { "enableVisitTypeFilter_boolean" => false, "timezone_string" => "-0700", "ignoreConsent_boolean" => false } }, "time" => { "actionTime" => "2018-08-28T10:07:13.866Z" }, "name" => "notification.checkForNewNotifications", "outcome" => { "result" => { "result_boolean" => false, "lastUpdatedDTTM_string" => "2018-08-28T03:07:13" } } }, "trackingId" => "16dff1c1-91e9-49ca-b2a4-aca2bde04ef8" }, "path" => "C:/Temp/Tomcat/log/hello-service-event.log", "message" => "[EVENT] 2018-08-28T03:07:13,877-0700 {\"environment\":{\"hostName\":\"elasticsearch-machine\",\"serviceName\":\"hello-service\",\"serviceInstance\":\"hello-service\"},\"resource\":{\"isSensit ive\":\"false\"},\"subject\":{\"id\":\"helloDeepak@example.com\",\"sessionId\":\"37e078e5-e112-48f0-b4af-63cce5b11327\"},\"action\":{\"name\":\"notification.checkForNewNotifications\",\"arguments\":{\"l astRenderedTime_string\":\"2018-08-28T03:00:43\",\"preference\":{\"timezone_string\":\"-0700\",\"enableVisitTypeFilter_boolean\":false,\"ignoreConsent_boolean\":false}},\"time\":{\"actionTime\":\"2018 -08-28T10:07:13.866Z\"},\"outcome\":{\"result\":{\"result_boolean\":false,\"lastUpdatedDTTM_string\":\"2018-08-28T03:07:13\"}}},\"trackingId\":\"16dff1c1-91e9-49ca-b2a4-aca2bde04ef8\",\"auditLevel\":1 }\r", "@version" => "1" } ``` Please help me to resolve this issue. ---
**Author:** ![deepsing](https://avatars.discourse-cdn.com/v4/letter/d/d26b3c/32.png) [@deepsing](https://discuss.elastic.co/u/deepsing)\ **Post date:** [August 28, 2018, 9:38am UTC](https://discuss.elastic.co/t/maximum-size-for-index-limit-size/146020/7 "2018-08-28T09:38:36Z")
Hi @warkolm Can you please help me to resolve it. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 25, 2018, 9:51am UTC](https://discuss.elastic.co/t/maximum-size-for-index-limit-size/146020/8 "2018-09-25T09:51:21Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.