Hi there, I'm ingesting AWS WAF logs and it works fine for a few minutes but then stops with the following error: response=>{"index"=>{"_index"=>"waf-logs-2023.08.01", "_id"=>"rjCKGYoBsxYs-jwL007l", "status"=>400, "error"=>{"type"=>"document_parsing_exception", "reason"=>"[1:6817] failed to parse…

[image] feo13: What is the best approach to handle this? I would start by looking at the existing mapping on the index and see if there are any obvious issues.

[image] feo13: ruby { code => ' event.get("[httpRequest][headers]").each { |kv| event.set(name = kv["name"], value = kv["value"])} ' } I would assume that you can have a wide range of fields in the httpRequest.headers object and this ruby code would create …

Thank you very much for the reply! Should I just removed the ruby bit in the config? Thanks again!

You need to check if this is really the cause of the excessive number of fields. Assuming that the quantity of headers can be large and unknown, I would say that this is a probably cause, but you need to validate it. Then you would need to create a mapping for the httpRequest.headers field as flat…

Logstash stops processing AWS WAF logs when fields exceed 1000 (or any number)

Elastic Stack Logstash

system (system) Closed September 20, 2023, 4:41pm 6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Limit of total fields [num] in index [logstash-XX] has been exceeded Logstash	7	701	November 27, 2019
Regarding to logstash performance limit Logstash	4	407	October 8, 2019
How to found problematic messages? Logstash	2	384	March 4, 2018
Cloudtrail logs have too many fields Elasticsearch	1	457	April 23, 2020
reason"=>"Limit of total fields [1000] in index Logstash	4	450	June 23, 2020

Logstash stops processing AWS WAF logs when fields exceed 1000 (or any number)

Related topics