Why my buckets's doc_count is limit to 3000?

11162 · August 12, 2014, 9:24am

this is my DSL

{

"query":{
"match_all":{}
},
        "aggs": {
            "range": {
                "date_range": {
                    "field": "@timestamp", "format": 
"yyy.MM.dd.HH.mm.ss",
"ranges": [{"from": "2014.08.12.09.18.45", "to":
"2014.08.12.09.20.50"}]
},
"aggs": {
"over_time": {
"date_histogram": {
"field": "@timestamp",
"interval": "1s",
"format": "yyy.MM.dd.HH.mm.ss"
},
"aggs": {
"total_sent": {
"sum": {"field": "bytes_sent"}
}
}
}
}
}
}
}

thus is my result

{

"took":6,
"timed_out":false,
"_shards":{
- "total":96,
- "successful":96,
- "failed":0
  },
"hits":{
- "total":258002,
- "max_score":0.0,
- "hits":[
  ]
  },
"aggregations":{
- "range":{
  - "buckets":[
    - {
      - "key":"2014.08.12.09.18.45-2014.08.12.09.20.50",
      - "from":1.407835125E12,
      - "from_as_string":"2014.08.12.09.18.45",
      - "to":1.40783525E12,
      - "to_as_string":"2014.08.12.09.20.50",
      - "doc_count":12000,
      - "over_time":{
        
        "buckets":[
        
        {
        
        "key_as_string":"2014.08.12.09.18.45",
        
        "key":1407835125000,
        
        "doc_count":3000,
        
        "total_sent":{
        
        "value":6.6126308E7
        }
        },
        
        {
        
        "key_as_string":"2014.08.12.09.18.47",
        
        "key":1407835127000,
        
        "doc_count":3000,
        
        "total_sent":{
        
        "value":9.286586E7
        }
        },
        
        {
        
        "key_as_string":"2014.08.12.09.18.49",
        
        "key":1407835129000,
        
        "doc_count":3000,
        
        "total_sent":{
        
        "value":1.21316184E8
        }
        },
        
        {
        
        "key_as_string":"2014.08.12.09.18.51",
        
        "key":1407835131000,
        
        "doc_count":3000,
        
        "total_sent":{
        
        "value":8.3529544E7
        }
        }
        ]
        }
        }
        ]
        }
        }

}

my soft flow is:
Nginx ==pipe==> syslogng ==udp==> logstash ==es_river==> elasticsearch

How can I break the 3000doc/sec ?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1fa43dda-31cd-4f10-95bc-0f31778d9663%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

jpountz · August 12, 2014, 1:27pm

I am not sure to understand if your question is about aggregations or
indexing speed?

On Tue, Aug 12, 2014 at 11:24 AM, 陈浩 humen1@gmail.com wrote:

this is my DSL

{
"query":{
"match_all":{}
},
        "aggs": {
            "range": {
                "date_range": {
                    "field": "@timestamp", "format":
"yyy.MM.dd.HH.mm.ss",
"ranges": [{"from": "2014.08.12.09.18.45", "to":
"2014.08.12.09.20.50"}]
},
"aggs": {
"over_time": {
"date_histogram": {
"field": "@timestamp",
"interval": "1s",
"format": "yyy.MM.dd.HH.mm.ss"
},
"aggs": {
"total_sent": {
"sum": {"field": "bytes_sent"}
}
}
}
}
}
}
}
thus is my result

{

"took":6,

"timed_out":false,

"_shards":{

"total":96,

"successful":96,

"failed":0
},

"hits":{

"total":258002,

"max_score":0.0,

"hits":[
]
},

"aggregations":{

"range":{

"buckets":[

{

"key":"2014.08.12.09.18.45-2014.08.12.09.20.50",

"from":1.407835125E12,

"from_as_string":"2014.08.12.09.18.45",

"to":1.40783525E12,

"to_as_string":"2014.08.12.09.20.50",

"doc_count":12000,

"over_time":{

"buckets":[

{

"key_as_string":"2014.08.12.09.18.45",

"key":1407835125000,

"doc_count":3000,

"total_sent":{

"value":6.6126308E7
}
},

{

"key_as_string":"2014.08.12.09.18.47",

"key":1407835127000,

"doc_count":3000,

"total_sent":{

"value":9.286586E7
}
},

{

"key_as_string":"2014.08.12.09.18.49",

"key":1407835129000,

"doc_count":3000,

"total_sent":{

"value":1.21316184E8
}
},

{

"key_as_string":"2014.08.12.09.18.51",

"key":1407835131000,

"doc_count":3000,

"total_sent":{

"value":8.3529544E7
}
}
]
}
}
]
}
}

}

my soft flow is:
Nginx ==pipe==> syslogng ==udp==> logstash ==es_river==> elasticsearch

How can I break the 3000doc/sec ?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1fa43dda-31cd-4f10-95bc-0f31778d9663%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/1fa43dda-31cd-4f10-95bc-0f31778d9663%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6kYQAU6ExwMFwxTuy3LQQx%2BLSh1_-wNO7cxTd9GbF-Ow%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

11162 · August 12, 2014, 2:24pm

indexing speed.

在 2014年8月12日，21:27，Adrien Grand adrien.grand@elasticsearch.com 写道：

I am not sure to understand if your question is about aggregations or indexing speed?

On Tue, Aug 12, 2014 at 11:24 AM, 陈浩 humen1@gmail.com wrote:
this is my DSL

{
"query":{
"match_all":{}
},
        "aggs": {
            "range": {
                "date_range": {
                    "field": "@timestamp", "format": "yyy.MM.dd.HH.mm.ss",
                    "ranges": [{"from": "2014.08.12.09.18.45", "to": "2014.08.12.09.20.50"}]
                },  
                "aggs": {
                    "over_time": {
                        "date_histogram": {
                            "field": "@timestamp",
                            "interval": "1s",
                            "format": "yyy.MM.dd.HH.mm.ss"
                        },
                        "aggs": {
                            "total_sent": {
                                "sum": {"field": "bytes_sent"}
                            }
                        }
                    }
                }
            }
        }
    }
thus is my result

{

"took":6,

"timed_out":false,
"_shards":{

"total":96,

"successful":96,
"failed":0

},
"hits":{

"total":258002,

"max_score":0.0,
"hits":[

]
},
"aggregations":{

"range":{

"buckets":[

{

"key":"2014.08.12.09.18.45-2014.08.12.09.20.50",

"from":1.407835125E12,
"from_as_string":"2014.08.12.09.18.45",

"to":1.40783525E12,

"to_as_string":"2014.08.12.09.20.50",

"doc_count":12000,
"over_time":{

"buckets":[

{

"key_as_string":"2014.08.12.09.18.45",

"key":1407835125000,
"doc_count":3000,

"total_sent":{

"value":6.6126308E7
}

},
{

"key_as_string":"2014.08.12.09.18.47",

"key":1407835127000,
"doc_count":3000,

"total_sent":{

"value":9.286586E7
}

},
{

"key_as_string":"2014.08.12.09.18.49",

"key":1407835129000,
"doc_count":3000,

"total_sent":{

"value":1.21316184E8
}

},
{

"key_as_string":"2014.08.12.09.18.51",

"key":1407835131000,
"doc_count":3000,

"total_sent":{

"value":8.3529544E7
}

}
]
}
}
]

}
}
}

my soft flow is:
Nginx ==pipe==> syslogng ==udp==> logstash ==es_river==> elasticsearch

How can I break the 3000doc/sec ?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1fa43dda-31cd-4f10-95bc-0f31778d9663%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/qj4BP7rEnTU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6kYQAU6ExwMFwxTuy3LQQx%2BLSh1_-wNO7cxTd9GbF-Ow%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/EFF0A61B-AB24-4C8B-ABE2-A1B2D5FCC5AC%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Topic		Replies	Views
Unable to get min_doc_count bucket for histogram aggs Elasticsearch	3	1467	June 28, 2018
Get sum of doc_count values in aggregation Elasticsearch	2	2133	August 25, 2018
Total hits count not equal to doc_count total Elasticsearch	1	444	March 19, 2021
Elasticsearch Range Aggregation is including upper limit excluding lower limit Elasticsearch	1	523	March 26, 2020
Buckets.doc_count > hits.total Elasticsearch	2	946	April 17, 2018

Why my buckets's doc_count is limit to 3000?

Related topics