Limit of total fields [1000] in index [p_bs_api_abcd_2020_23] has been exceeded

Elastic Stack Logstash
1

I have followed the steps discussed in link Increase total fields limit via creation of index in logstash but not able to resolve it. Kindly find below few snippets of the configuration files.

output {
elasticsearch {
hosts => ["XX.XX.XX.XX:7001"]
index => "%{Service}_%{+YYYY_ww}"
manage_template => true
template => '/config/etc/logstash/templates/api_template.json'
template_name => 'api'
template_overwrite => true
}
}

Template json as follows:

{
"index_patterns" : "p_bs_api_abcd*",
"version" : 60001,
"settings" : {
"index.refresh_interval" : "5s",
"number_of_shards": 1,
"index.mapping.total_fields.limit": 2000
},
"mappings" : {
"dynamic_templates" : [ {
"message_field" : {
"path_match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text",
"norms" : false
}
}
}, {
"string_fields" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text", "norms" : false,
"fields" : {
"keyword" : { "type": "keyword", "ignore_above": 256 }
}
}
}
} ],
"properties" : {
"@timestamp": { "type": "date"},
"@version": { "type": "keyword"},
"geoip" : {
"dynamic": true,
"properties" : {
"ip": { "type": "ip" },
"location" : { "type" : "geo_point" },
"latitude" : { "type" : "half_float" },
"longitude" : { "type" : "half_float" }
}
}
}
}
}

When I restart the logstash still I am able to see error as below.

[2020-06-03T07:08:17,196][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>"/mnt/storage/software/logstash-7.0.0/config/etc/logstash/templates/api_template.json"}
[2020-06-03T07:08:17,293][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1, "index.mapping.total_fields.limit"=>2000, "index.lifecycle.name"=>"logstash-policy", "index.lifecycle.rollover_alias"=>"logstash"}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2020-06-03T07:08:17,321][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/api
[2020-06-03T07:12:16,988][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"p_bs_api_abcd_2020_23", :_type=>"_doc", :routing=>nil}, #LogStash::Event:0x4b6e8b56], :response=>{"index"=>{"_index"=>"p_bs_api_abcd_2020_23", "_type"=>"_doc", "_id"=>"GXjPeHIBIVr3QtguFT6g", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Limit of total fields [1000] in index [p_bs_api_abcd_2020_23] has been exceeded"}}}}

@theuntergeek @Christian_Dahlqvist - Have looped you in since you guys have solved the issue in the thread I followed so kindly pointing out where I have went wrong.

Cheers,
Maadavan

2

The index_pattern is wrong and what you provided in the template does not match your index name so does not get applied.

3

@Christian_Dahlqvist,

Thanks for reverting, but index pattern do match isn't it? In the template I have given it as

p_bs_api_abcd*

whereas in the error index name is p_bs_api_abcd_2020_23 in the error logs. Year & week number at the end of index name is dynamic and would be taken care of wildcard symbol (*) at the end of index name given in template.

Cheers,
Maadavan

4

hi, try increasing the digits here? not sure if that is the issue here

5

What is the output of _cat/templates ?

6

@Christian_Dahlqvist,

Kindly find the snippet of the template in ES. Still could see index patterns are as logstash-* instead of "api" which I have given it in template (snippet is present in my question post). Also I could see field limit has increased. So as you said it may be due to index pattern not matching. Where would I be giving that pattern?

{
"api" : {
"order" : 0,
"version" : 60001,
"index_patterns" : [
"logstash-*"
],
"settings" : {
"index" : {
"lifecycle" : {
"name" : "logstash-policy",
"rollover_alias" : "logstash"
},
"mapping" : {
"total_fields" : {
"limit" : "2000"
}
},
"refresh_interval" : "5s",
"number_of_shards" : "1"
}
}

@richylyq Thanks for your reply. I am able to see the updated field in template as I said above and the error is still only for 1000.

Cheers,
Maadavan

7

Look at the index_patterns field which does not match the name of the index you are using.

8

As per the documentation, the index_patterns in the index template needs to be an array of strings, not just a string as you have applied. So try setting that to "index_patterns":["p_bs_api_abcd*"]

See here.

9

Thanks Rahul.

I made the changes in template to array format , even then the _cat/templates shows index pattern to the default value logstash-*.

As per Logstash 7.0 does ignore index_patterns / template fields in template and overrules that with "logstash-*", seems there to be a bug in logstash 7.0. So have added ilm_enabled => false in my logstash output.

output {
elasticsearch {
hosts => ["localhost:61000"]
index => "%{capability}_%{+YYYY_ww}"
ilm_enabled => false
manage_template => true
template => '/config/etc/logstash/templates/increased_field_limit.json'
template_name => 'increased_field_limit'
template_overwrite => true
}
}

Post the above changes, in _cat/templates I am able to see the index pattern has been updated and matching with the index names too. But still I am receiving the error "reason"=>"Limit of total fields [1000] in index [XXXXX] has been exceeded"

Is it being caused by ilm_enabled => false or anything else to be checked?
Or a version upgrade would solve the issue?
Or should I reindex?

Any help much appreciated please.

Cheers,
Maadavan

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.