Continuously getting - java.lang.IllegalArgumentException: Limit of total fields [1000] in index has been exceeded error

Elastic Stack Elasticsearch
1

I am running with ELK cluster 5.6.16.

I am getting continuously below error into elasticsearch logs since quite some time for only one index. Is there any method reduce no of fields Or I will have to increase no of fields ?

I am looking for solution to configure settings into configuration file.

[2019-06-28T05:14:26,727][DEBUG][o.e.a.b.TransportShardBulkAction] [29-121-IDC.justdial.com] [www-2019.06.27][0] failed to execute bulk item (index) BulkShardRequest [[www-2019.06.27][0]] containing [27] requests
java.lang.IllegalArgumentException: Limit of total fields [1000] in index [www-2019.06.27] has been exceeded
	at org.elasticsearch.index.mapper.MapperService.checkTotalFieldsLimit(MapperService.java:604) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:420) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:336) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:268) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest(MetaDataMappingService.java:311) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.execute(MetaDataMappingService.java:230) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.cluster.service.ClusterService.executeTasks(ClusterService.java:634) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.cluster.service.ClusterService.calculateTaskOutputs(ClusterService.java:612) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:571) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:576) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) ~[elasticsearch-5.6.16.jar:5.6.16]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) ~[elasticsearch-5.6.16.jar:5.6.16]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
2

Can anyone help me here ?

3

You can update limit of total fields after index has been created:

PUT my_index/_settings
{
"index.mapping.total_fields.limit": 2000
}

Note too many fields will lead to mapping explosion, which is a not good practice.

4

Agreed. So may be reducing the number of fields is the first thing you need to look at.
Why do you have so many?
Does some of them have the same meaning?

5

Yes. I am looking for solution reduce unwanted fields. I am not in favour to increase total fields.
Please see the given screenshot. There were 2.2k fields into index. I am talking about these wanted fields which are detected as fields.

These names are part of request url & i don't want to be in index.

Can I define selective fields for index into configuration ? Let me know if more info require. plz help.

unwanted_fields
unwanted_fields.png1368×589 45 KB

6

Why are you sending them to elasticsearch in the first place? If you don't need them, don't send them.

7

I am sending complete access log to ELK stack & storing below url pattrn under request field.

/webmain/autosuggest.php?cases=what&search=chem&city=Samastipur&area=&s=1&pg=index

Please see this access log :

14.143.7.170, 96.17.12.143, 96.17.16.38 - - [03/Jul/2019:17:03:14 +0530] "GET /webmain/autosuggest.php?cases=what&search=chem&city=Samastipur&area=&s=1&pg=index HTTP/1.0" 200 3598 "https://www.jamuna.com/" "Mozilla/5.0 (X11; Linux i686; rv:34.0) Gecko/20100101 Firefox/34.0" "REMOTE_ADDR : 96.17.16.38" "TRUE_CLIENT : 14.143.7.170" "AKAXFF : 14.143.7.170" www.jamuna.com 1.218 0.154 IN .

Logstash pattern :

(?<x_forwarded_for>%{IP}, .*|%{IP:xforwardedfor}|-) (%{NGUSER:ident}|-) (%{NGUSER:auth}|%{USERNAME:user}|-) [%{HTTPDATE:timestamp}] "(?:%{WORD:method} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:request})" %{NOTSPACE:response} (%{NOTSPACE:bytes_read}|-) (%{DATA:request_header_referer}|-) "(%{DATA:request_header_agent}|-)" "(REMOTE_ADDR : %{DATA:clientip}|-)" "(TRUE_CLIENT : %{DATA:http_true_client_ip}|-)" "(AKAXFF : %{DATA:http_akaxff}|-)" (?:%{HOSTNAME:http_host}|%{IP}|%{HOSTNAME:http_host}:%{POSINT}|-) (?:%{BASE10NUM:request_duration}|-) (?:%{BASE10NUM:upstream_request_duration}|-) (%{WORD:Country}|-)

8

If I'm not mistaken then request field just contains /webmain/autosuggest.php?cases=what&search=chem&city=Samastipur&area=&s=1&pg=index, right?

If so, and based only on the information you gave so far, which might be incomplete, you end up with a document like:

{
  "request": "/webmain/autosuggest.php?cases=what&search=chem&city=Samastipur&area=&s=1&pg=index"
}

This is not generating as many fields as you shown.

9

As you said, whatever data filled-up into request fields is correctly into ELK. My concern is only, I want to prevent to create below unwanted fields. That's the reason, I have raised this case.

&search=chem
&city=Samastipur

Hope you understand my concern.

10

You can always have a strict mapping with "dynamic": "strict" or "dynamic": false . See https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic.html

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.