Can I aggregate data in past intervals periodically in elasticsearch?

I need to develop a log component for the api gateway of my company.
The log component need to log all the api calls through the api gateway. The log contains the call time stamp, the name of the caller, the name of the called api, the name of the application containing the called api, success/failure flag, the response time of a successful call.
Requirement: Users should be able to see the total call times, successful times, failed times, successful ratio, average response time of successful time of the entire system, certain application, or a single api during a selected time span, e.g. last year, last week, etc.
Notes: The quantity of api calls could be very huge so it is not practical to calculate the data mentioned previously in time when a user select a time span. Therefore it is needed to add up the called times, successful times, failed times, total response time of a minimal timespan, e.g. a minute, periodically, and aggreate the minute data to hour data, hour data to day data, day data to week data/month data, so that there are aggregated sums of call times and other data for different time granularity such as minute, hour, day, week and month. Then when a user wants to see the statistics for the api calls of a particular period in the past, we can split the period to some appropriate timespan combinations and sum up the data of each timespan, e.g., 00:00:00a.m. on Sep. 30th, 2017 to 01:00:00a.m. on Nov. 1st, 2017 can be split into three timespans - a day interval (Sep. 30th, 2017), a month interval (Oct. 2017), and an hour interval (00:00a.m.-01:00a.m. Nov. 1st, 2017).

I want to know if Elastic stack can satisfy the above requirement?

Hi Qinsi,

Definitely yes. You can use Filebeat (light weight) or Logstash (powerful and full of plugins) to parse your log files and index in Elasticsearch. Them all you have to do is use Kibana to create the dashboards you mentioned.

Hope it helps.


This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.