Total Fields Limit setting

Elastic Stack Elasticsearch
1

I have the 5.0.0-alpha3 setup here, sat alongside the live 2.3.3 version (on Ubuntu). There is a feeder Logstash which delivers the same data to both live and beta. On the beta, I have an error the the "limit of total fields [1000] in index" has been exceeded.

Can anyone tell me how I set this limit? I've tried the setting "index.mapping.total_fields.limit: 2000" in the elasticsearch.yml, but this complains about an index setting in node settings.

Thanks

2

Hi Roger,

This was implemented in the following Github issue
The setting you referenced to was the correct one, but it's an index specific setting :slight_smile:
You can update it after the index has been created, though.

For example:

PUT my_index/_settings
{
  "index.mapping.total_fields.limit": 2000
}

Or during index creation:

PUT test
{
  "settings": {
    "index.mapping.total_fields.limit": 2000,
    "number_of_shards": 1,
    "number_of_replicas": 0
  },
  "mappings": {
    ...
  }
}

Hope this helps :smiley:

13 Likes
3

Thanks Byron

I've put that in the Kibana console and the new fields are coming through. I'm not sure how the index creation is done though, this is automatic, my indexes are logstash-*

4

Hi Roger,

If you're using Logstash for indexing data into Elasticsearch, you will need to change logstash's default index template. By default Logstash will create it's own mapping and add that to Elasticsearch. If you want to change that mapping, you will have to add some configuration to your logstash config as described here.

Here is an example of the default logtash template. You could copy that over, add the fields.limit to the settings part and then follow the instructions on the previous link on how to use it :slight_smile:

Hope this helps!

2 Likes
5

Is there a way to achieve the same from a Beat (e.g., via fields.yml)? I tried adding the following to the JSON template generated from fields.yml but it doesn't seem to affect anything.

@@ -29,7 +29,8 @@
   },
   "order": 0,
   "settings": {
-    "index.refresh_interval": "5s"
+    "index.refresh_interval": "5s",
+    "index.mapping.total_fields.limit" : 1000000000
   },
   "template": "openconfigbeat-*"
-}
\ No newline at end of file
+}
2 Likes
6

Hi. Got a Q on this.. I would like to have a template so whenever a new Twitter index gets created the ES system makes total fields limit to be 2000 by default.
Bit like below...
Anybody knows how to modify the below to have this setting included?

curl -XPUT localhost:9200/_template/twothousandfieldslimit -d '
{
"order" : 1,
"template" : "twitter*",
"settings" : {
"index" : {
"mapping.total_fields.limit" : "2000"
 }
}’
4 Likes