In my system, I am using Filebeat, Logstash, Elasticsearch, and Kibana.
For every transaction in the system, a log is printed in the log file and it is saved in the Elasticsearch db.
Sample logline: TID: [-1234] [] [2018-08-02 09:53:36,629] INFO {org.apache.synapse.mediators.builtin.LogMediator} - API Request URL = /payment/v1/947032456/transactions/amount/, Request ID = 901fef70f925, API = Payment, Application Name = Payment, User Name = MMS_NEW {org.apache.synapse.mediators.builtin.LogMediator}
In Kibana I'm using this data to generate a data table visualization with has user, api, app, average tps, min tps and max tps.
-
The expectation of average tps is to get the average transactions per second considering the selected time interval.
-
The expectation of max tps is to get the maximum transactions happened in a second within the selected time interval.
-
The expectation of max tps is to get the minimum transactions to happen within the selected time interval.
But the results do not show correct results.
Ex: if we want to calculate the average tps within a 1 minute and i there are only 20 transactions within that time period we expect the average value to be 20/60
We suspect these invalid results are due to gaps in transactions ( no transactions available in every second ). The request sent by Kibana is as follows. Plese, help.
{
"size": 0,
"_source": {
"excludes": []
},
"aggs": {
"2": {
"terms": {
"field": "username.keyword",
"size": 100,
"order": {
"_term": "desc"
}
},
"aggs": {
"3": {
"terms": {
"field": "api.keyword",
"size": 100,
"order": {
"_term": "desc"
}
},
"aggs": {
"4": {
"terms": {
"field": "appname.keyword",
"size": 100,
"order": {
"_term": "desc"
}
},
"aggs": {
"1": {
"avg_bucket": {
"buckets_path": "1-bucket>_count",
"gap_policy": "insert_zeros"
}
},
"5": {
"max_bucket": {
"buckets_path": "5-bucket>_count"
}
},
"6": {
"min_bucket": {
"buckets_path": "6-bucket>_count"
}
},
"1-bucket": {
"date_histogram": {
"field": "@timestamp",
"interval": "5s",
"time_zone": "Asia/Katmandu",
"min_doc_count": 0
}
},
"5-bucket": {
"date_histogram": {
"field": "@timestamp",
"interval": "5s",
"time_zone": "Asia/Katmandu",
"min_doc_count": 0
}
},
"6-bucket": {
"date_histogram": {
"field": "@timestamp",
"interval": "5s",
"time_zone": "Asia/Katmandu",
"min_doc_count": 0
}
}
}
}
}
}
}
}
},
"version": true,
"stored_fields": [
""
],
"script_fields": {},
"docvalue_fields": [
"@timestamp"
],
"query": {
"bool": {
"must": [{
"match_all": {}
},
{
"match_all": {}
},
{
"range": {
"@timestamp": {
"gte": 1533181623220,
"lte": 1533182523220,
"format": "epoch_millis"
}
}
}
],
"filter": [],
"should": [],
"must_not": []
}
},
"highlight": {
"pre_tags": [
"@kibana-highlighted-field@"
],
"post_tags": [
"@/kibana-highlighted-field@"
],
"fields": {
"": {}
},
"fragment_size": 2147483647
}
}