How to store & get data with different time interval

Our team are currently focusing on log datas and visualizing them.
We are building a platform analyzing every audit log by log4j in our java project:

log -> filebeat -> logstash -> elasticsearch -> our platform


  • filebeat - 7.9.1
  • logstash -7.9.1
  • elasticsearch -7.9.1
  • Java - HighLevelRestClient

When in visualizing I had some problems. We have some log datas, and we need to visualize them in different time intervals, like recent 15 min, recent 30 min, recent 1 hour.

            .fetchSource(new String[]{"data_size_in","data_size_out", "server_id"},null)

Currently we are solving by aggregation with intervals, and change the interval based on time range we choose. Is there some more elegant solutions?

Here's my improvement thought:

  1. create more index with certain intervals(maybe use elasticsearch pipelines)

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.