How to store & get data with different time interval

Our team are currently focusing on log datas and visualizing them.
We are building a platform analyzing every audit log by log4j in our java project:

log -> filebeat -> logstash -> elasticsearch -> our platform

CURRENT TECHNIQUE STACK:

  • filebeat - 7.9.1
  • logstash -7.9.1
  • elasticsearch -7.9.1
  • Java - HighLevelRestClient

When in visualizing I had some problems. We have some log datas, and we need to visualize them in different time intervals, like recent 15 min, recent 30 min, recent 1 hour.

searchSourceBuilder
            .fetchSource(new String[]{"data_size_in","data_size_out", "server_id"},null)
            .query(QueryBuilders.rangeQuery("start_time")
                .lte("2020-09-25T02:50:00.000Z")
                .gte("2020-09-30T01:50:00.000Z"))
            .query(QueryBuilders.matchQuery("gateway_code","/memcached"))
            .aggregation(AggregationBuilders.dateHistogram("interval")
                .fixedInterval(DateHistogramInterval.DAY).field("start_time")
                .subAggregation(AggregationBuilders.terms("function")
                    .size(5).field("pub_item_name.keyword"))
            );

Currently we are solving by aggregation with intervals, and change the interval based on time range we choose. Is there some more elegant solutions?

Here's my improvement thought:

  1. create more index with certain intervals(maybe use elasticsearch pipelines)

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.