How to start parsing AWS load balancer logs from specified date

(Nitish) #1

AWS LB logs are stored in S3 bucket at AWSLogs/xxxxxxxxx/elasticloadbalancing/region/2018/10/16/<log_files>. I want to skip old logs and start parsing from start of this month.
Setting prefix AWSLogs/xxxxxxxxx/elasticloadbalancing/region/2018/10 will temporarily solve my problem but prefix will change at the start of next month. I am using exclude_pattern => "/*dev-test*" to skip dev and test logs. ELK version: 6.4.0

(Nitish) #2

I have written a shell script to update the prefix to current date and set a cron job to execute it every day after midnight and restart logstash.

Please let me know if there is any better solution.

(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.