I am using logstash version 1.5.4 to push logs from a file to elasticsearch. We are using '-l' option and specifying a local file for storing logstash's logs.
I am not using --verbose or debug mode.
Now, in case of errors (code 400) received from elasticsearch, the logstash's log file grow very very quickly. I am seeing for 1 log entry of size <100 bytes its adding up 12 KB of logs in logstash log file, and within 2 hours, logstash logs grow upto 20 GB.
I have considered option to use --quiet but there the down side is that the log entry is lost (the actuals logs are rotated and logstash logs wont have it either). I need it in logstash logs, but not repeated several times with lot of noise.
Also implemented logrotate, but its frequency is set to 3 hours and I dont want to run it every 10 minutes or so to avoid extra load on the system just for deleting logs of a utility. Also, since we keep upto 7 files and rotate when size reaches 200 MB, running it more frequently would mean we would lose records every 7 runs of logrotate. If it runs every 3 hours we lose data after 24 hours, running it every 10 minutes would mean losing data every 70 minutes.
Is there any workaround to
a) log the actual message AND/OR log only once
b) reduce retrials for 400 error (or any error - 429, 503, anything) to 1 - tried to configure 'max_retries' to 5 doesnt help (not sure whats the purpose of this setting).
upgrading logstash to v5 is not an option for me (at least for now).
Content of logstash.log file (repeated for each log entry) -
{:timestamp=>"2017-08-28T11:34:18.540000+0000", :message=>"failed action with response of 400, dropping action: ["index", {:_id=>nil, :_index=>"xxxx-xxxx-xxxxx-2017.08.28", :_type=>"xxxx-xxxx-xxxxx", :_routing=>nil}, #<LogStash::Event:0x59cd9dc3 @metadata_accessors=#<LogStash::Util::Accessors:0x616d6818 @store={"path"=>"xxxx-xxxx-xxxxx.log", "retry_count"=>0}, @lut={"[path]"=>[{"path"=>"xxxx-xxxx-xxxxx.log", "retry_count"=>0}, "path"]}>, @cancelled=false, @data={"message"=>"xxxx-xxxx-xxxxx - - [2017/Aug/28:11:34:18 +0000] \"GET xxxx-xxxx-xxxxx HTTP/1.1\" 200 254 1 \"\" \"\" \"\"", "@version"=>"1", "@timestamp"=>"2017-08-28T11:34:18.000Z", "host"=>"xxxx-xxxx-xxxxx1-1", "path"=>"xxxx-xxxx-xxxxx.log", "type"=>"xxxx-xxxx-xxxxx", "tags"=>["xxxx-xxxx-xxxxx"], "clientip"=>"xxxx-xxxx-xxxxx", "user"=>"-", "auth"=>"-", "httpmethod"=>"GET", "url"=>"xxxx-xxxx-xxxxx", "httpversion"=>"1.1", "responsestatus"=>"200", "responselength"=>"254", "responsetime"=>"1"}, @metadata={"path"=>"xxxx-xxxx-xxxxx.log", "retry_count"=>0}, @accessors=#<LogStash::Util::Accessors:0x4d9147bd @store={"message"=>"xxxx-xxxx-xxxxx - - [2017/Aug/28:11:34:18 +0000] \"GET xxxx-xxxx-xxxxx HTTP/1.1\" 200 254 1 \"\" \"\" \"\"", "@version"=>"1", "@timestamp"=>"2017-08-28T11:34:18.000Z", "host"=>"xxxx-xxxx-xxxxx1-1", "path"=>"xxxx-xxxx-xxxxx.log", "type"=>"xxxx-xxxx-xxxxx", "tags"=>["xxxx-xxxx-xxxxx"], "clientip"=>"xxxx-xxxx-xxxxx", "user"=>"-", "auth"=>"-", "httpmethod"=>"GET", "url"=>"xxxx-xxxx-xxxxx", "httpversion"=>"1.1", "responsestatus"=>"200", "responselength"=>"254", "responsetime"=>"1"}, @lut={"host"=>[{"message"=>"xxxx-xxxx-xxxxx - - [2017/Aug/28:11:34:18 +0000] \"GET xxxx-xxxx-xxxxx HTTP/1.1\" 200 254 1 \"\" \"\" \"\"", "@version"=>"1", "@timestamp"=>"2017-08-28T11:34:18.000Z", "host"=>"xxxx-xxxx-xxxxx1-1", "path"=>"xxxx-xxxx-xxxxx.log", "type"=>"xxxx-xxxx-xxxxx", "tags"=>["xxxx-xxxx-xxxxx"], "clientip"=>"xxxx-xxxx-xxxxx", "user"=>"-", "auth"=>"-", "httpmethod"=>"GET", "url"=>"xxxx-xxxx-xxxxx", "httpversion"=>"1.1", "responsestatus"=>"200", "responselength"=>"254", "responsetime"=>"1"}, "host"], "path"=>[{"message"=>"xxxx-xxxx-xxxxx - - [2017/Aug/28:11:34:18 +0000] \"GET xxxx-xxxx-xxxxx HTTP/1.1\" 200 254 1 \"\" \"\" \"\"", "@version"=>"1", "@timestamp"=>"2017-08-28T11:34:18.000Z", "host"=>"xxxx-xxxx-xxxxx1-1", "path"=>"xxxx-xxxx-xxxxx.log", "type"=>"xxxx-xxxx-xxxxx", "tags"=>["xxxx-xxxx-xxxxx"], "clientip"=>"xxxx-xxxx-xxxxx", "user"=>"-", "auth"=>"-", "httpmethod"=>"GET", "url"=>"xxxx-xxxx-xxxxx", "httpversion"=>"1.1",
---------truncated-----
"responsetime"=>"1"}, "responsetime"], "@timestamp"=>[{"message"=>"xxxx-xxxx-xxxxx - - [2017/Aug/28:11:34:18 +0000] \"GET xxxx-xxxx-xxxxx HTTP/1.1\" 200 254 1 \"\" \"\" \"\"", "@version"=>"1", "@timestamp"=>"2017-08-28T11:34:18.000Z", "host"=>"xxxx-xxxx-xxxxx1-1", "path"=>"xxxx-xxxx-xxxxx.log", "type"=>"xxxx-xxxx-xxxxx", "tags"=>["xxxx-xxxx-xxxxx"], "clientip"=>"xxxx-xxxx-xxxxx", "user"=>"-", "auth"=>"-", "httpmethod"=>"GET", "url"=>"xxxx-xxxx-xxxxx", "httpversion"=>"1.1", "responsestatus"=>"200", "responselength"=>"254", "responsetime"=>"1"}, "@timestamp"]}>>]", :level=>:warn}