In my system, I am using Filebeat, Logstash, Elasticsearch, and Kibana.
For every transaction in the system, a log is printed in the log file and it is saved in the Elasticsearch db.
Sample logline: TID: [-1234] [] [2018-08-02 09:53:36,629] INFO {org.apache.synapse.mediators.builtin.LogMediator} - API Request URL = /payment/v1/947032456/transactions/amount/, Request ID = 901fef70f925, API = Payment, Application Name = Payment, User Name = MMS_NEW {org.apache.synapse.mediators.builtin.LogMediator}
In Kibana I'm using this data to generate a data table visualization with has user, api, app, average tps, min tps and max tps.
The expectation of average tps is to get the average transactions per second considering the selected time interval.
The expectation of max tps is to get the maximum transactions happened in a second within the selected time interval.
The expectation of max tps is to get the minimum transactions to happen within the selected time interval.
But the results do not show correct results.
Ex: if we want to calculate the average tps within a 1 minute and i there are only 20 transactions within that time period we expect the average value to be 20/60
When I check the response form Elsticsearch the incorrect results are there. So I think the error is due to Elasticsearch neglects the time period with empty results when calculating the results. Even though I have used "gap_policy": "insert_zeros" the seconds with no any transactions were neglected when calculating the average, min, and the max.
Please help to find a solution.
The request set to Elasticsearch form kibana is as follows.
{
"size": 0,
"_source": {
"excludes": []
},
"aggs": {
"2": {
"terms": {
"field": "username.keyword",
"size": 100,
"order": {
"_term": "desc"
}
},
"aggs": {
"3": {
"terms": {
"field": "api.keyword",
"size": 100,
"order": {
"_term": "desc"
}
},
"aggs": {
"4": {
"terms": {
"field": "appname.keyword",
"size": 100,
"order": {
"_term": "desc"
}
},
"aggs": {
"1": {
"avg_bucket": {
"buckets_path": "1-bucket>_count",
"gap_policy": "insert_zeros"
}
},
"5": {
"max_bucket": {
"buckets_path": "5-bucket>_count"
}
},
"6": {
"min_bucket": {
"buckets_path": "6-bucket>_count"
}
},
"1-bucket": {
"date_histogram": {
"field": "@timestamp",
"interval": "5s",
"time_zone": "Asia/Katmandu",
"min_doc_count": 0
}
},
"5-bucket": {
"date_histogram": {
"field": "@timestamp",
"interval": "5s",
"time_zone": "Asia/Katmandu",
"min_doc_count": 0
}
},
"6-bucket": {
"date_histogram": {
"field": "@timestamp",
"interval": "5s",
"time_zone": "Asia/Katmandu",
"min_doc_count": 0
}
}
}
}
}
}
}
}
},
"version": true,
"stored_fields": [
""
],
"script_fields": {},
"docvalue_fields": [
"@timestamp"
],
"query": {
"bool": {
"must": [{
"match_all": {}
},
{
"match_all": {}
},
{
"range": {
"@timestamp": {
"gte": 1533181623220,
"lte": 1533182523220,
"format": "epoch_millis"
}
}
}
],
"filter": [],
"should": [],
"must_not": []
}
},
"highlight": {
"pre_tags": [
"@kibana-highlighted-field@"
],
"post_tags": [
"@/kibana-highlighted-field@"
],
"fields": {
"": {}
},
"fragment_size": 2147483647
}
}