Limit Of Total Fields in index has been exceeded

[Akhilesh-Tiwari]

Hi Team,

when i am trying to push data from Filebeat to logstash and elasticsearch by using my log file,I am getting an exception.

"Java.lang.IllegalArgumentException:Limit Of Total Fields [1000] in index has been exceeded"

and then after i am getting

"Java.lang.OutOfMemeryError:java heap error"

can you please help me for this issue.

[Christian_Dahlqvist]

What does your mapping for that index look like? Which version are you using? What is the specification of your cluster?

[Akhilesh-Tiwari]

Hi @Christian_Dahlqvist,

I am using Filebeat 6.0.0,logstash 6.3.0 and Elasticsearch 6.3.0.

Filebeat.yml is

input_type: log
paths:
/app/DPlogs/Smart.2018-07-23.54.log
fields: {log_type: smart_logtype_original}
include_lines: ["< App Id : SMART-WEB-APPLICATION >"]

output.logstash:
output.logstash.hosts: ["172.21.153.176:5044"]
output.logstash.protocol: "http"
output.logstash.index: "beat_server_latest-%{+YYYY.MM.dd}"
output.logstash.template.name: "appstash"
output.logstash.template.path: "filebeat.template.json"
output.logstash.template.overwrite: false

Logstash.yml

input {
beats {
port => 5044
}
}
filter {
ruby {
code => "
fieldArray = event.get('message').split(' > < ')
for field in fieldArray
field = field.delete '< '
field = field.delete ' >'
result = field.split(' : ')
event.set(result[0], result[1])
end
"
}
}

output {

if [fields][log_type] == "smart_logtype_original"{

elasticsearch {

  hosts => "172.21.153.176:9200"
  index => "smart_index1"
  #user => "RonakB"
  #password => "June@2018"
  #ssl => true
  #ssl_certificate_verification => false
  #truststore => "/opt/READONLYREST/elasticsearch-6.3.0/config/keystore.jks"
  #truststore_password => "readonlyrest"

 }

}

if [fields][log_type] == "smart_logtype"{

elasticsearch {

  hosts => "172.21.153.176:9200"
  index => "smart_index"

 }

}

}

[Christian_Dahlqvist]

That is not what I asked for. What is the mapping of the index you had the error reported for? Use the get mapping API to retrieve it.