Aggregation document count issue

Kushal_Goyal · July 22, 2018, 12:21pm

Hi

I am new in ELK stack, need your advice.

I am using nested aggregation.

GET account_credit/person/_search
{
"aggs": {
"count_by_gender": {
"terms": {
"field": "creditcard_number.keyword"

  },
  "aggs": {
    "ip_address": {
      "terms": {
        "field": "ip_address.keyword"
      }
    }
  }
}

}
}

result is

{
"key": "3555539887215424",
"doc_count": 3,
"ip_address": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "6.202.255.220",
"doc_count": 3

but when I am using below DSL

GET account_credit/person/_search
{

"query" : {

			"term" : {"creditcard_number.keyword":"3555539887215424"}
	
},
"sort": [
  {
    "ip_address.keyword": {
      "order": "desc"
    }
  }
]

}

response

{
"took": 3,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 8,
"max_score": null,
"hits": [
{
"_index": "account_credit",
"_type": "person",
"_id": "3jupwWQBCxMvYbonq5dD",
"_score": null,
"_source": {
"id": 2000001,
"first_name": "Felicle",
"last_name": "Evitt",
"email": "fevitt0@ustream.tv",
"gender": "Female",
"ip_address": "75.73.29.197",
"creditcard_type": "switch",
"creditcard_number": "3555539887215424",
"balance": "96862289.14"
},
"sort": [
"75.73.29.197"
]
},
{
"_index": "account_credit",
"_type": "person",
"_id": "B76hwWQBR_kMIhVB6dfC",
"_score": null,
"_source": {
"id": 1,
"first_name": "Felicle",
"last_name": "Evitt",
"email": "fevitt0@ustream.tv",
"gender": "Female",
"ip_address": "75.73.29.197",
"creditcard_type": "switch",
"creditcard_number": "3555539887215424",
"balance": "96862289.14"
},
"sort": [
"75.73.29.197"
]
},
{
"_index": "account_credit",
"_type": "person",
"_id": "Db6qwWQBR_kMIhVBPtcE",
"_score": null,
"_source": {
"id": 2000001,
"first_name": "Felicle",
"last_name": "Evitt",
"email": "fevitt0@ustream.tv",
"gender": "Female",
"ip_address": "75.73.21.197",
"creditcard_type": "switch",
"creditcard_number": "3555539887215424",
"balance": "96862289.14"
},
"sort": [
"75.73.21.197"
]
},
{
"_index": "account_credit",
"_type": "person",
"_id": "9TutwWQBCxMvYbonBpey",
"_score": null,
"_source": {
"id": 222,
"first_name": "Felicle",
"last_name": "Evitt",
"email": "fevitt0@ustream.tv",
"gender": "Female",
"ip_address": "75.73.21.197",
"creditcard_type": "switch",
"creditcard_number": "3555539887215424",
"balance": "96862289.14"
},
"sort": [
"75.73.21.197"
]
},
{
"_index": "account_credit",
"_type": "person",
"_id": "-TutwWQBCxMvYbonZZfA",
"_score": null,
"_source": {
"id": 222,
"first_name": "Felicle",
"last_name": "Evitt",
"email": "fevitt0@ustream.tv",
"gender": "Female",
"ip_address": "75.73.20.197",
"creditcard_type": "switch",
"creditcard_number": "3555539887215424",
"balance": "96862289.14"
},
"sort": [
"75.73.20.197"
]
},
{
"_index": "account_credit",
"_type": "person",
"_id": "pr7yjGQBR_kMIhVBbaCL",
"_score": null,
"_source": {
"id": 222,
"first_name": "Darryl",
"last_name": "Cabrales",
"email": "dcabrales65@cafepress.com",
"gender": "Male",
"ip_address": "6.202.255.220",
"creditcard_type": "jcb",
"creditcard_number": "3555539887215424",
"balance": "7367624.70"
},
"sort": [
"6.202.255.220"
]
},
{
"_index": "account_credit",
"_type": "person",
"_id": "6DuCwWQBCxMvYbonP4_Y",
"_score": null,
"_source": {
"id": 222,
"first_name": "Darryl",
"last_name": "Cabrales",
"email": "dcabrales65@cafepress.com",
"gender": "Male",
"ip_address": "6.202.255.220",
"creditcard_type": "visa",
"creditcard_number": "3555539887215424",
"balance": "7367624.70"
},
"sort": [
"6.202.255.220"
]
},
{
"_index": "account_credit",
"_type": "person",
"_id": "2DuDwWQBCxMvYboncpPM",
"_score": null,
"_source": {
"id": 222,
"first_name": "Darryl",
"last_name": "Cabrales",
"email": "dcabrales65@cafepress.com",
"gender": "Male",
"ip_address": "6.202.255.220",
"creditcard_type": "visa",
"creditcard_number": "3555539887215424",
"balance": "7367624.70"
},
"sort": [
"6.202.255.220"
]
}
]
}
}

why in aggregation I am not getting all ip address.

Please help on this

dadoonet · July 22, 2018, 12:41pm

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.

I'm not sure if the first result is truncated or not. Could you paste it entirely?

Kushal_Goyal · July 22, 2018, 1:02pm

Hi

I am new in ELK stack, need your advice.

I am using nested aggregation.

  GET account_credit/person/_search
   {  "aggs": { 
"count_by_gender": {
"terms": {
"field": "creditcard_number.keyword"
 },
 "aggs": {
   "ip_address": {
     "terms": {
       "field": "ip_address.keyword"
     }
   }
 }
}
}
}

result is

"key": "3555539887215424",
"doc_count": 3,
"ip_address": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "6.202.255.220",
"doc_count": 3

but when I am using below DSL


GET account_credit/person/_search
{

"query" : {

			"term" : {"creditcard_number.keyword":"3555539887215424"}
	
},
"sort": [
  {
    "ip_address.keyword": {
      "order": "desc"
    }
  }
]
}

response is

{
"took": 3,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 8,
"max_score": null,
"hits": [
{
"_index": "account_credit",
"_type": "person",
"_id": "3jupwWQBCxMvYbonq5dD",
"_score": null,
"_source": {
"id": 2000001,
"first_name": "Felicle",
"last_name": "Evitt",
"email": "fevitt0@ustream.tv",
"gender": "Female",
"ip_address": "75.73.29.197",
"creditcard_type": "switch",
"creditcard_number": "3555539887215424",
"balance": "96862289.14"
},
"sort": [
"75.73.29.197"
]
},
{
"_index": "account_credit",
"_type": "person",
"_id": "B76hwWQBR_kMIhVB6dfC",
"_score": null,
"_source": {
"id": 1,
"first_name": "Felicle",
"last_name": "Evitt",
"email": "fevitt0@ustream.tv",
"gender": "Female",
"ip_address": "75.73.29.197",
"creditcard_type": "switch",
"creditcard_number": "3555539887215424",
"balance": "96862289.14"
},
"sort": [
"75.73.29.197"
]
},
{
"_index": "account_credit",
"_type": "person",
"_id": "Db6qwWQBR_kMIhVBPtcE",
"_score": null,
"_source": {
"id": 2000001,
"first_name": "Felicle",
"last_name": "Evitt",
"email": "fevitt0@ustream.tv",
"gender": "Female",
"ip_address": "75.73.21.197",
"creditcard_type": "switch",
"creditcard_number": "3555539887215424",
"balance": "96862289.14"
},
"sort": [
"75.73.21.197"
]
},
{
"_index": "account_credit",
"_type": "person",
"_id": "9TutwWQBCxMvYbonBpey",
"_score": null,
"_source": {
"id": 222,
"first_name": "Felicle",
"last_name": "Evitt",
"email": "fevitt0@ustream.tv",
"gender": "Female",
"ip_address": "75.73.21.197",
"creditcard_type": "switch",
"creditcard_number": "3555539887215424",
"balance": "96862289.14"
},
"sort": [
"75.73.21.197"
]
},
{
"_index": "account_credit",
"_type": "person",
"_id": "-TutwWQBCxMvYbonZZfA",
"_score": null,
"_source": {
"id": 222,
"first_name": "Felicle",
"last_name": "Evitt",
"email": "fevitt0@ustream.tv",
"gender": "Female",
"ip_address": "75.73.20.197",
"creditcard_type": "switch",
"creditcard_number": "3555539887215424",
"balance": "96862289.14"
},
"sort": [
"75.73.20.197"
]
},
{
"_index": "account_credit",
"_type": "person",
"_id": "pr7yjGQBR_kMIhVBbaCL",
"_score": null,
"_source": {
"id": 222,
"first_name": "Darryl",
"last_name": "Cabrales",
"email": "dcabrales65@cafepress.com",
"gender": "Male",
"ip_address": "6.202.255.220",
"creditcard_type": "jcb",
"creditcard_number": "3555539887215424",
"balance": "7367624.70"
},
"sort": [
"6.202.255.220"
]
},
{
"_index": "account_credit",
"_type": "person",
"_id": "6DuCwWQBCxMvYbonP4_Y",
"_score": null,
"_source": {
"id": 222,
"first_name": "Darryl",
"last_name": "Cabrales",
"email": "dcabrales65@cafepress.com",
"gender": "Male",
"ip_address": "6.202.255.220",
"creditcard_type": "visa",
"creditcard_number": "3555539887215424",
"balance": "7367624.70"
},
"sort": [
"6.202.255.220"
]
},
{
"_index": "account_credit",
"_type": "person",
"_id": "2DuDwWQBCxMvYboncpPM",
"_score": null,
"_source": {
"id": 222,
"first_name": "Darryl",
"last_name": "Cabrales",
"email": "dcabrales65@cafepress.com",
"gender": "Male",
"ip_address": "6.202.255.220",
"creditcard_type": "visa",
"creditcard_number": "3555539887215424",
"balance": "7367624.70"
},
"sort": [
"6.202.255.220"
]
}
]
}
}

why in aggregation I am not getting all ip address count correctly.

Please help on this

dadoonet · July 22, 2018, 7:41pm

Better. Sadly not well indented which makes it hard to read.

Anyway, could you answer the question I asked?

I'm not sure if the first result is truncated or not. Could you paste it entirely?

system · August 19, 2018, 7:41pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Elasticsearch aggregation sort issue Elasticsearch	2	891	July 5, 2017
Use nested doc_count in terms aggregation Elasticsearch	2	5015	April 24, 2017
Wrong doc_count on Aggregations Elasticsearch	7	3822	July 5, 2017
2nd Level Nested aggregation incorrect doc_count Elasticsearch	1	214	March 29, 2022
Getting different doc count for the same nested aggregation Elasticsearch	2	327	August 15, 2018

Aggregation document count issue

Related topics