Regarding to logstash performance limit

Elastic Stack Logstash
1

Hi experts,

I have met a logstash performance limit issue, when I parse my log, logstash report this error log:

`

[2019-09-03T14:21:49,106][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2019.09.03", :_type=>"_doc", :_routing=>nil}, #LogStash::Event:0x1b6c342a], :response=>{"index"=>{"_index"=>"logstash-2019.09.03", "_type"=>"_doc", "_id"=>"4_PJ9WwB6iwMcN-ZT20T", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Limit of total fields [1000] in index [logstash-2019.09.03] has been exceeded"}}}}

`

That block the parsed data that can't be sent to elasticsearch. From the content, it seems a total fields limit, how can I enlarge this parameter？Or are there some method can solve this issue? Thank you.

2

That is an Elasticsearch limit and not managed through Logstash. How come you have so many fields?

3

The log is included a lot of data that I have added a lot of index to map them, therefore the index count is huge, are there some method can enlarge the mapping.total_fields.limit ?

I have retrieved the similar issue, here is a method to enlarge it, i will try to do it later.

curl -XPUT localhost:9200/_template/twothousandfieldslimit -d '
{
    "order" : 1,
    "template" : "twitter*",
    "settings" : {
        "index" : {
            "mapping.total_fields.limit" : "2000"
         }
    }
}'
4

The solution is:

curl -XPUT http://localhost:9200/lsh_nokia_nokiasbc-2019.09.*/_settings/?pretty -H 'Content-Type: application/json' -d ' { "index.mapping.total_fields.limit" : "2000" } '

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.