AWS LB logs are stored in S3 bucket at AWSLogs/xxxxxxxxx/elasticloadbalancing/region/2018/10/16/<log_files>. I want to skip old logs and start parsing from start of this month.
Setting prefix AWSLogs/xxxxxxxxx/elasticloadbalancing/region/2018/10 will temporarily solve my problem but prefix will change at the start of next month. I am using exclude_pattern => "/*dev-test*" to skip dev and test logs. ELK version: 6.4.0
I have written a shell script to update the prefix to current date and set a cron job to execute it every day after midnight and restart logstash.
Please let me know if there is any better solution.