Our team are currently focusing on log datas and visualizing them.
We are building a platform analyzing every audit log by log4j in our java project:
log -> filebeat -> logstash -> elasticsearch -> our platform
CURRENT TECHNIQUE STACK:
- filebeat - 7.9.1
- logstash -7.9.1
- elasticsearch -7.9.1
- Java - HighLevelRestClient
When in visualizing I had some problems. We have some log datas, and we need to visualize them in different time intervals, like recent 15 min, recent 30 min, recent 1 hour.
searchSourceBuilder
.fetchSource(new String[]{"data_size_in","data_size_out", "server_id"},null)
.query(QueryBuilders.rangeQuery("start_time")
.lte("2020-09-25T02:50:00.000Z")
.gte("2020-09-30T01:50:00.000Z"))
.query(QueryBuilders.matchQuery("gateway_code","/memcached"))
.aggregation(AggregationBuilders.dateHistogram("interval")
.fixedInterval(DateHistogramInterval.DAY).field("start_time")
.subAggregation(AggregationBuilders.terms("function")
.size(5).field("pub_item_name.keyword"))
);
Currently we are solving by aggregation with intervals, and change the interval based on time range we choose. Is there some more elegant solutions?
Here's my improvement thought:
- create more index with certain intervals(maybe use elasticsearch pipelines)