Error: failed to publish events: temporary bulk send failure

Elastic Stack Beats
1

I have issue about sending logs with filebeat. I have checked all the topics related this issue but i couldn't figure out it yet .

I have Elasticsearch v 7.10.2 and i am using filebeat 7.10.0.

I created my pipeline as below. when i trying to simulate pipeline with my logs, it is succesfull. ı thınk ı have no problem with my grok pattern .

 { - 
    "description": " XXXX",
    "processors": [ - 
      { - 
        "grok": { - 
          "field": "message",
          "patterns": [ - 
            "%{SYSLOGBASE2} trx_id:%{GREEDYDATA:trx_id} %{GREEDYDATA:program} trxid:%{GREEDYDATA:trxid}, appId:%{GREEDYDATA:appId}, serviceMethod:%{GREEDYDATA:serviceMethod}, serviceId:%{GREEDYDATA:serviceId}, accessedCountryCode2:%{WORD:accessedCountryCode2}, username:%{USERNAME:username}, schemaName:%{WORD:schemaName}, regionCode2:%{WORD:regionCode2}, clientip:%{IPORHOST:clientip}|%{GREEDYDATA:clientip}, failureText: %{WORD:failureText}, sessionId:%{GREEDYDATA:sessionId}, detailMessage:%{GREEDYDATA:detailMessage}"
          ]
        }
      },
      { - 
        "date": { - 
          "field": "timestamp8601",
          "formats": [ - 
            "yyyy-MM-dd HH:mm:ss.SSS"
          ]
        }
      },
      { - 
        "date_index_name": { - 
          "field": "timestamp8601",
          "index_name_prefix": "XXXXX-",
          "index_name_format": "yyyy-MM-dd-HH",
          "timezone": "UTC+3",
          "date_rounding": "d",
          "date_formats": [ - 
            "yyyy-MM-dd HH:mm:ss.SSS"
          ]
        }
      }
    ]
  }
}

my problem is starting with filebeat. When i started filebeat to send logs es, i could't send to all logs to es. (maybe %5 logs successfully sending)

filebeat yml file:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /......./logs/app.log 


  multiline.pattern: \d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}\.\d{3}
  multiline.negate: true
  multiline.match: after

filebeat.config.modules:
  path: 
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1


setup.ilm.enabled: false
output.elasticsearch:
  hosts: ["xx.xx.x.xxx", "xx.xx.x.xxx", "xx.xx.x.xxx"]
  protocol: "http"
  username: "xxx"
  password: "xxxx"
  ssl.verification_mode: none
  loadbalance: true
  pipeline: "xxxx"
  bulk_max_size: 256
  worker: 4
  compression_level: 0

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

here is the filebeatlogs:

2022-03-10T00:16:20.263+0300 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2022-03-10T00:16:20.263+0300 INFO [publisher] pipeline/retry.go:223 done
2022-03-10T00:16:20.421+0300 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2022-03-10T00:16:20.421+0300 INFO [publisher] pipeline/retry.go:223 done
2022-03-10T00:16:21.290+0300 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2022-03-10T00:16:21.290+0300 INFO [publisher] pipeline/retry.go:223 done
2022-03-10T00:16:21.986+0300 ERROR [publisher_pipeline_output] pipeline/output.go:180 failed to publish events: temporary bulk send failure
2022-03-10T00:16:22.290+0300 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2022-03-10T00:16:22.290+0300 INFO [publisher] pipeline/retry.go:223 done
2022-03-10T00:16:22.376+0300 ERROR [publisher_pipeline_output] pipeline/output.go:180 failed to publish events: temporary bulk send failure
2022-03-10T00:16:22.401+0300 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2022-03-10T00:16:22.401+0300 INFO [publisher] pipeline/retry.go:223 done
2022-03-10T00:16:22.401+0300 INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(Elasticsearch(http://xx.xx.x.xxx:9200))
2022-03-10T00:16:22.401+0300 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2022-03-10T00:16:22.401+0300 INFO [publisher] pipeline/retry.go:223 done
2022-03-10T00:16:22.401+0300 INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(Elasticsearch(http://xx.xx.x.xxx:9200))
2022-03-10T00:16:22.401+0300 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2022-03-10T00:16:22.401+0300 INFO [publisher] pipeline/retry.go:223 done
2022-03-10T00:16:22.402+0300 INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.10.0
2022-03-10T00:16:22.403+0300 INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.10.0
2022-03-10T00:16:22.406+0300 INFO template/load.go:97 Template filebeat-7.10.0 already exists and will not be overwritten.
2022-03-10T00:16:22.406+0300 INFO [index-management] idxmgmt/std.go:298 Loaded index template.
2022-03-10T00:16:22.407+0300 INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(Elasticsearch(http://xx.xx.x.xxx:9200)) established
2022-03-10T00:16:22.453+0300 INFO template/load.go:97 Template filebeat-7.10.0 already exists and will not be overwritten.
2022-03-10T00:16:22.453+0300 INFO [index-management] idxmgmt/std.go:298 Loaded index template.
2022-03-10T00:16:22.454+0300 INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(Elasticsearch(http://xx.xx.x.xxx:9200)) established
2022-03-10T00:16:22.643+0300 ERROR [publisher_pipeline_output] pipeline/output.go:180 failed to publish events: temporary bulk send failure

anyone any idea ?

thanks.

2

any idea ?

3

Hi @samettozbay

Could you please explain why you defined bulk_max_size and worker, was there any specific need for this?:

  hosts: ["xx.xx.x.xxx", "xx.xx.x.xxx", "xx.xx.x.xxx"]
  loadbalance: true
  bulk_max_size: 256
  worker: 4

Did you try to use default configuration? Do you see the same behavior?

According to: Configure the Elasticsearch output | Filebeat Reference [7.10] | Elastic - you are starting 12 workers, 4 for each instance - do you maybe see an increase of resource usage of Elasticsearch instances, that could cause connectivity issues?

regarding bulk_max_size:

However big batch sizes can also increase processing times, which might result in API errors, killed connections, timed-out publishing requests, and, ultimately, lower throughput.
4

Hi @Tetiana_Kravchenko

Thanks for reply. I have tried default configuration but nothing changed. Same result.

5

@samettozbay Did you check resource usage of Elasticsearch instances? could it be an issue?
Any relevant Elasticsearch logs?
Also did you try to connect to elasticsearch_ip:port? maybe some network limitations on the Elasticsearch instances?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.