Logstash generate timely csv output

[Manoj_Hettiarachchi]

Hello,

In my system, I use the csv output plugin. The output block is as follows.

output {
csv {
               path => "/var/csv_reports/%{+YYYY}-%{+MM}-%{+dd}/transaction-report.csv"
               fields => ["timestamp","tid","api","response_time"]
       }
}

With this configuration, I am able to generate a separate CSV report for each day.
But the generated file size is too large because of this I need to generate separate files for 8-hour intervals.

I know that by adding %{+HH} we can generate hourly reports. But then there will be 24 reports of each day. So that is not my requirement.

Please advise how to achieve this requirement.

Thanks,

[Manoj_Hettiarachchi]

I have used a ruby base approach for this requirement. I have used a ruby code to generate custom file names checking the timestamp value in the log lines. My code is as follows to generate files for each 8 hour interval.

ruby {
        code => "
        require 'date'
        interval = 8
        date2 = DateTime.strptime(event.get('[@timestamp]').time.localtime.strftime('%Y-%m-%d_%H-%M-%S'), '%Y-%m-%d_%H-%M-%S')
        date = DateTime.now
        val = date2.hour/interval +1
        event.set('append_val', val) 
        event.set('index_day', event.get('[@timestamp]').time.localtime.strftime('%Y-%m-%d'))"
    }