Strange Fluent-Bit logs

Elastic Stack Elasticsearch
1

As a log collector, I installed fluent-bit.

It is installed on each node of the cluster. All right. He sends logs to elastic. I see them. All OK!

But if i look in the log of fluent-bit pod, then I see the following. Is this normal, or do I need to configure or configure something?

{"took":14959,"errors":true,"items":[{"index":{"_index":"kubernetes_cluster-2019.01.17","_type":"flb_type","_id":"U_cD3WgB3lQtmGeDeT7j","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":626,"_primary_term":2,"status":201}},{"index":{"_index":"kubernetes_cluster-2019.01.17","_type":"flb_type","_id":"VPcD3WgB3lQtmGeDeT7j","status":429,"error":{"type":"es_rejected_execution_exception","reason":"rejected execution of processing of [14672][indices:data/write/bulk[s][p]]: request: BulkShardRequest [[kubernetes_cluster-2019.01.17][4]] containing [24] requests, target allocation id: szv_pPlJRECoocKHvAa_Ng, primary term: 2 on EsThreadPoolExecutor[name = elasticsearch-master-2/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@15dfd52a[Running, pool size = 1, active threads = 1, queued tasks = 200, completed tasks = 2849]]"}}},{"index":{"_index":"kubernetes_cluster-2019.0

[2019/02/11 14:43:54] [error] [out_es] could not pack/validate JSON response

{"took":15028,"errors":true,"items":[{"index":{"_index":"kubernetes_cluster-2019.02.08","_type":"flb_type","_id":"VNUD3WgBIB2PCaLHefD4","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":12550,"_primary_term":2,"status":201}},{"index":{"_index":"kubernetes_cluster-2019.02.08","_type":"flb_type","_id":"VdUD3WgBIB2PCaLHefD4","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":12900,"_primary_term":2,"status":201}},{"index":{"_index":"kubernetes_cluster-2019.02.08","_type":"flb_type","_id":"VtUD3WgBIB2PCaLHefD4","status":429,"error":{"type":"es_rejected_execution_exception","reason":"rejected execution of processing of [14675][indices:data/write/bulk[s][p]]: request: BulkShardRequest [[kubernetes_cluster-2019.02.08][2]] containing [100] requests, target allocation id: lOBDNeN1QeuVZ1JGnkPxZQ, primary term: 2 on EsThreadPoolExecutor[name = elasticsearc

[2019/02/11 14:43:54] [ warn] [out_es] Elasticsearch error

{"took":15028,"errors":true,"items":[{"index":{"_index":"kubernetes_cluster-2019.02.08","_type":"flb_type","_id":"VNUD3WgBIB2PCaLHefD4","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":12550,"_primary_term":2,"status":201}},{"index":{"_index":"kubernetes_cluster-2019.02.08","_type":"flb_type","_id":"VdUD3WgBIB2PCaLHefD4","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":12900,"_primary_term":2,"status":201}},{"index":{"_index":"kubernetes_cluster-2019.02.08","_type":"flb_type","_id":"VtUD3WgBIB2PCaLHefD4","status":429,"error":{"type":"es_rejected_execution_exception","reason":"rejected execution of processing of [14675][indices:data/write/bulk[s][p]]: request: BulkShardRequest [[kubernetes_cluster-2019.02.08][2]] containing [100] requests, target allocation id: lOBDNeN1QeuVZ1JGnkPxZQ, primary term: 2 on EsThreadPoolExecutor[name = elasticsearch-master-2/write,

[2019/02/11 14:43:57] [error] [out_es] could not pack/validate JSON response

{"took":15015,"errors":true,"items":[{"index":{"_index":"kubernetes_cluster-2019.02.08","_type":"flb_type","_id":"4NUD3WgBIB2PCaLHhfZj","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":12704,"_primary_term":2,"status":201}},{"index":{"_index":"kubernetes_cluster-2019.02.08","_type":"flb_type","_id":"4dUD3WgBIB2PCaLHhfZj","status":429,"error":{"type":"es_rejected_execution_exception","reason":"rejected execution of processing of [14802][indices:data/write/bulk[s][p]]: request: BulkShardRequest [[kubernetes_cluster-2019.02.08][4]] containing [65] requests, target allocation id: VS1lozI6RVaC-nBnF5T3yw, primary term: 2 on EsThreadPoolExecutor[name = elasticsearch-master-2/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@15dfd52a[Running, pool size = 1, active threads = 1, queued tasks = 200, completed tasks = 2898]]"}}},{"index":{"_index":"kuber

[2019/02/11 14:43:57] [ warn] [out_es] Elasticsearch error

Thsnks

2

It looks like you are overloading your cluster, as the bulk threadpool is getting full and rejecting requests.

Do you have Monitoring enable?

3

Yes, I already googled it. But the fact is that I just installed it and it happens right at the start. I have a small cluster.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.