Aggregation average value is not coming correct


(bagui) #1

Hi,

I'm using the below code to get the average value of cpu_usage using
aggregation. When I checked the output of cpu value individually and
calculate the avg, it is not matching with the aggregation avg value. I'm
using a boolquery along with rangeFilter here to get the data.

Please help to identify the issue.

Code :
public static SearchResponse searchResultWithAggregation(String es_index,
String es_type, List ipList) {
logger.debug("inside method searchResultWithAggregation...");
Client client = ESClientFactory.getInstance();
logger.debug("got the elasticsearch client connection");

BoolQueryBuilder bqb = QueryBuilders.boolQuery()
.mustNot(QueryBuilders.termQuery("address", "10.203.238.140"));

Iterator i = ipList.iterator();
logger.debug("got the ip list as :" + ipList);

while (i.hasNext()) {
bqb.should(QueryBuilders.termQuery("address", i.next()));
}

String time = "now-30m";
FilterBuilder fb = FilterBuilders.rangeFilter("@timestamp").from(time)
.to("now");

SearchResponse response = client
.prepareSearch(es_index)
.setTypes(es_type)
.setQuery(bqb)
.setPostFilter(fb)
.addAggregation(
AggregationBuilders.avg("cpu_average").field("value"))
.setSize(100).execute().actionGet();

System.out.println(response.toString());

return response;
}

Output :
{
"took" : 31,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 15,
"max_score" : 1.7314732,
"hits" : [ {
"_index" : "cpu_usage_metrics",
"_type" : "cpu_usage_metrics",
"_id" : "UQ9vquDGTQO8WedjgCcESA",
"_score" : 1.7314732, "_source" :
{"status":0,"occurrences":1,"value":"1","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:23:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
-s Aricloud.hostname -s.cpu_usage"}
}, {
"_index" : "cpu_usage_metrics",
"_type" : "cpu_usage_metrics",
"_id" : "EMT85ZKcS3OuoDmHgcSEjw",
"_score" : 1.7314732, "_source" :
{"status":0,"occurrences":1,"value":"3","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:25:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
-s Aricloud.hostname -s.cpu_usage"}
}, {
"_index" : "cpu_usage_metrics",
"_type" : "cpu_usage_metrics",
"_id" : "0Pf-XKZmTI-wpADuIVToFA",
"_score" : 1.7314714, "_source" :
{"status":0,"occurrences":1,"value":"3","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:21:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
-s Aricloud.hostname -s.cpu_usage"}
}, {
"_index" : "cpu_usage_metrics",
"_type" : "cpu_usage_metrics",
"_id" : "Pdn5h2gGRsK0hL2DKj0ZjA",
"_score" : 1.7314714, "_source" :
{"status":0,"occurrences":1,"value":"2","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:27:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
-s Aricloud.hostname -s.cpu_usage"}
}, {
"_index" : "cpu_usage_metrics",
"_type" : "cpu_usage_metrics",
"_id" : "5_mloLYMSgKRb_lnH7pqGQ",
"_score" : 1.7314714, "_source" :
{"status":0,"occurrences":1,"value":"3","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:33:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
-s Aricloud.hostname -s.cpu_usage"}
}, {
"_index" : "cpu_usage_metrics",
"_type" : "cpu_usage_metrics",
"_id" : "xjBgO2cXTH-DIQoNpIRnBA",
"_score" : 1.7314714, "_source" :
{"status":0,"occurrences":1,"value":"4","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:35:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
-s Aricloud.hostname -s.cpu_usage"}
}, {
"_index" : "cpu_usage_metrics",
"_type" : "cpu_usage_metrics",
"_id" : "0sclBpwcRQmfyKklXPJbow",
"_score" : 1.7314694, "_source" :
{"status":0,"occurrences":1,"value":"3","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:29:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
-s Aricloud.hostname -s.cpu_usage"}
}, {
"_index" : "cpu_usage_metrics",
"_type" : "cpu_usage_metrics",
"_id" : "O5CVCkYtRQGxJG3TJWDoNw",
"_score" : 1.7314694, "_source" :
{"status":0,"occurrences":1,"value":"4","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:39:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
-s Aricloud.hostname -s.cpu_usage"}
}, {
"_index" : "cpu_usage_metrics",
"_type" : "cpu_usage_metrics",
"_id" : "bEOcnsrRSS6eBdfn8UlG8w",
"_score" : 1.7314694, "_source" :
{"status":0,"occurrences":1,"value":"14","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:47:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
-s Aricloud.hostname -s.cpu_usage"}
}, {
"_index" : "cpu_usage_metrics",
"_type" : "cpu_usage_metrics",
"_id" : "Wp7kA_5QRYishSBGqJpNxA",
"_score" : 1.7314694, "_source" :
{"status":0,"occurrences":1,"value":"18","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:49:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
-s Aricloud.hostname -s.cpu_usage"}
}, {
"_index" : "cpu_usage_metrics",
"_type" : "cpu_usage_metrics",
"_id" : "vo6xLh8YRqqznXzwbap9mA",
"_score" : 1.7314656, "_source" :
{"status":0,"occurrences":1,"value":"3","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:31:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
-s Aricloud.hostname -s.cpu_usage"}
}, {
"_index" : "cpu_usage_metrics",
"_type" : "cpu_usage_metrics",
"_id" : "eZ2hacYhRKe_rsB5QZaJxQ",
"_score" : 1.7314656, "_source" :
{"status":0,"occurrences":1,"value":"13","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:45:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
-s Aricloud.hostname -s.cpu_usage"}
}, {
"_index" : "cpu_usage_metrics",
"_type" : "cpu_usage_metrics",
"_id" : "UedHm9r8S3CvQAOMzczZsQ",
"_score" : 1.7314491, "_source" :
{"status":0,"occurrences":1,"value":"2","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:37:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
-s Aricloud.hostname -s.cpu_usage"}
}, {
"_index" : "cpu_usage_metrics",
"_type" : "cpu_usage_metrics",
"_id" : "-Pz6wqikSxqN0HsjCso0tQ",
"_score" : 1.7314491, "_source" :
{"status":0,"occurrences":1,"value":"3","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:41:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
-s Aricloud.hostname -s.cpu_usage"}
}, {
"_index" : "cpu_usage_metrics",
"_type" : "cpu_usage_metrics",
"_id" : "B3pZx-xVQwiuCRJT8pzndQ",
"_score" : 1.7314491, "_source" :
{"status":0,"occurrences":1,"value":"5","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:43:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
-s Aricloud.hostname -s.cpu_usage"}
} ]
},
"aggregations" : {
"cpu_average" : {
"value" : 41.39343376342633
}
}
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/644333cf-8953-4a7f-9367-997b191bf054%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Alexander Reelsen) #2

Hey,

you are setting a post filter, which means, that the aggregations will work
without the range filter applied. You may want to use a filtered query and
move the filter inside the filter part of that particular query.

--Alex

On Thu, Jun 5, 2014 at 12:38 PM, Subhadip Bagui i.bagui@gmail.com wrote:

Hi,

I'm using the below code to get the average value of cpu_usage using
aggregation. When I checked the output of cpu value individually and
calculate the avg, it is not matching with the aggregation avg value. I'm
using a boolquery along with rangeFilter here to get the data.

Please help to identify the issue.

Code :
public static SearchResponse searchResultWithAggregation(String es_index,
String es_type, List ipList) {
logger.debug("inside method searchResultWithAggregation...");
Client client = ESClientFactory.getInstance();
logger.debug("got the elasticsearch client connection");

BoolQueryBuilder bqb = QueryBuilders.boolQuery()
.mustNot(QueryBuilders.termQuery("address", "10.203.238.140"));

Iterator i = ipList.iterator();
logger.debug("got the ip list as :" + ipList);

while (i.hasNext()) {
bqb.should(QueryBuilders.termQuery("address", i.next()));
}

String time = "now-30m";
FilterBuilder fb = FilterBuilders.rangeFilter("@timestamp").from(time)
.to("now");

SearchResponse response = client
.prepareSearch(es_index)
.setTypes(es_type)
.setQuery(bqb)
.setPostFilter(fb)
.addAggregation(
AggregationBuilders.avg("cpu_average").field("value"))
.setSize(100).execute().actionGet();

System.out.println(response.toString());

return response;
}

Output :
{
"took" : 31,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 15,
"max_score" : 1.7314732,
"hits" : [ {
"_index" : "cpu_usage_metrics",
"_type" : "cpu_usage_metrics",
"_id" : "UQ9vquDGTQO8WedjgCcESA",
"_score" : 1.7314732, "_source" :
{"status":0,"occurrences":1,"value":"1","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:23:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
-s Aricloud.hostname -s.cpu_usage"}
}, {
"_index" : "cpu_usage_metrics",
"_type" : "cpu_usage_metrics",
"_id" : "EMT85ZKcS3OuoDmHgcSEjw",
"_score" : 1.7314732, "_source" :
{"status":0,"occurrences":1,"value":"3","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:25:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
-s Aricloud.hostname -s.cpu_usage"}
}, {
"_index" : "cpu_usage_metrics",
"_type" : "cpu_usage_metrics",
"_id" : "0Pf-XKZmTI-wpADuIVToFA",
"_score" : 1.7314714, "_source" :
{"status":0,"occurrences":1,"value":"3","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:21:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
-s Aricloud.hostname -s.cpu_usage"}
}, {
"_index" : "cpu_usage_metrics",
"_type" : "cpu_usage_metrics",
"_id" : "Pdn5h2gGRsK0hL2DKj0ZjA",
"_score" : 1.7314714, "_source" :
{"status":0,"occurrences":1,"value":"2","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:27:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
-s Aricloud.hostname -s.cpu_usage"}
}, {
"_index" : "cpu_usage_metrics",
"_type" : "cpu_usage_metrics",
"_id" : "5_mloLYMSgKRb_lnH7pqGQ",
"_score" : 1.7314714, "_source" :
{"status":0,"occurrences":1,"value":"3","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:33:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
-s Aricloud.hostname -s.cpu_usage"}
}, {
"_index" : "cpu_usage_metrics",
"_type" : "cpu_usage_metrics",
"_id" : "xjBgO2cXTH-DIQoNpIRnBA",
"_score" : 1.7314714, "_source" :
{"status":0,"occurrences":1,"value":"4","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:35:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
-s Aricloud.hostname -s.cpu_usage"}
}, {
"_index" : "cpu_usage_metrics",
"_type" : "cpu_usage_metrics",
"_id" : "0sclBpwcRQmfyKklXPJbow",
"_score" : 1.7314694, "_source" :
{"status":0,"occurrences":1,"value":"3","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:29:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
-s Aricloud.hostname -s.cpu_usage"}
}, {
"_index" : "cpu_usage_metrics",
"_type" : "cpu_usage_metrics",
"_id" : "O5CVCkYtRQGxJG3TJWDoNw",
"_score" : 1.7314694, "_source" :
{"status":0,"occurrences":1,"value":"4","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:39:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
-s Aricloud.hostname -s.cpu_usage"}
}, {
"_index" : "cpu_usage_metrics",
"_type" : "cpu_usage_metrics",
"_id" : "bEOcnsrRSS6eBdfn8UlG8w",
"_score" : 1.7314694, "_source" :
{"status":0,"occurrences":1,"value":"14","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:47:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
-s Aricloud.hostname -s.cpu_usage"}
}, {
"_index" : "cpu_usage_metrics",
"_type" : "cpu_usage_metrics",
"_id" : "Wp7kA_5QRYishSBGqJpNxA",
"_score" : 1.7314694, "_source" :
{"status":0,"occurrences":1,"value":"18","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:49:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
-s Aricloud.hostname -s.cpu_usage"}
}, {
"_index" : "cpu_usage_metrics",
"_type" : "cpu_usage_metrics",
"_id" : "vo6xLh8YRqqznXzwbap9mA",
"_score" : 1.7314656, "_source" :
{"status":0,"occurrences":1,"value":"3","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:31:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
-s Aricloud.hostname -s.cpu_usage"}
}, {
"_index" : "cpu_usage_metrics",
"_type" : "cpu_usage_metrics",
"_id" : "eZ2hacYhRKe_rsB5QZaJxQ",
"_score" : 1.7314656, "_source" :
{"status":0,"occurrences":1,"value":"13","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:45:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
-s Aricloud.hostname -s.cpu_usage"}
}, {
"_index" : "cpu_usage_metrics",
"_type" : "cpu_usage_metrics",
"_id" : "UedHm9r8S3CvQAOMzczZsQ",
"_score" : 1.7314491, "_source" :
{"status":0,"occurrences":1,"value":"2","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:37:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
-s Aricloud.hostname -s.cpu_usage"}
}, {
"_index" : "cpu_usage_metrics",
"_type" : "cpu_usage_metrics",
"_id" : "-Pz6wqikSxqN0HsjCso0tQ",
"_score" : 1.7314491, "_source" :
{"status":0,"occurrences":1,"value":"3","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:41:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
-s Aricloud.hostname -s.cpu_usage"}
}, {
"_index" : "cpu_usage_metrics",
"_type" : "cpu_usage_metrics",
"_id" : "B3pZx-xVQwiuCRJT8pzndQ",
"_score" : 1.7314491, "_source" :
{"status":0,"occurrences":1,"value":"5","key":"Aricloud.vm.cpu_usage.cpu.usage","client":"vm.server2","@timestamp":"2014-06-05T15:43:13+05:30","check_name":"cpu_usage_metrics","address":"10.203.238.138","command":"cpu-usage-metrics.sh
-s Aricloud.hostname -s.cpu_usage"}
} ]
},
"aggregations" : {
"cpu_average" : {
"value" : 41.39343376342633
}
}
}

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/644333cf-8953-4a7f-9367-997b191bf054%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/644333cf-8953-4a7f-9367-997b191bf054%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM_R5T4BEO4XPBhmQszALNiyNEBDWPepYcm18ETVHQONNA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(system) #3