The error message "output.go:92: ERR Failed to publish events: temporary bulk send failure"

Elastic Stack Beats
1

I have some problem, the error log is being outputted by filebeat endlessly. please see the following detail message.
2018-05-02T09:27:21.879178437Z 2018/05/02 09:27:21.879028 output.go:92: ERR Failed to publish events: temporary bulk send failure
2018-05-02T09:27:21.880330394Z 2018/05/02 09:27:21.880199 client.go:651: INFO Connected to Elasticsearch version 6.1.2
2018-05-02T09:27:21.882671683Z 2018/05/02 09:27:21.882543 load.go:73: INFO Template already exists and will not be overwritten.
2018-05-02T09:27:22.885488217Z 2018/05/02 09:27:22.885285 output.go:92: ERR Failed to publish events: temporary bulk send failure
2018-05-02T09:27:22.886610836Z 2018/05/02 09:27:22.886513 client.go:651: INFO Connected to Elasticsearch version 6.1.2
2018-05-02T09:27:22.888869339Z 2018/05/02 09:27:22.888749 load.go:73: INFO Template already exists and will not be overwritten.
2018-05-02T09:27:23.891780722Z 2018/05/02 09:27:23.891512 output.go:92: ERR Failed to publish events: temporary bulk send failure

and I collect some config in this environment. see below

  1. Beat version 6.1.1
  2. Beat config
    filebeat.prospectors:
    -type: log
    enabled: true
    paths:
    - "/var/log/containers/${SYSTEM_POD_NAME}${SYSTEM_NAMESPACE}${_SYSTEM_APP_NAME}-.log"
    symlinks: true
    pipeline: haproxyId
    scan_frequency: 1s
    fields:
    source: ${_SYSTEM_APP_NAME:default}
    host: ${HOSTNAME:default}
    tags:
    -${_SYSTEM_NAMESPACE:default}
    fields_under_root: true
    json.keys_under_root: true
    json.add_error_key: true
    json.message_key: log
    setup.template.name: "haproxy-%{[beat.version]}"
    setup.template.pattern: "haproxy-%{[beat.version]}-    "
    output.elasticsearch:
    hosts: ["elasticsearch-logging:9200"]
    enabled: true
    index: "haproxy-%{[beat.version]}-%{+yyyy.MM.dd}"
  3. Elasticsearch logs
    elasticseaarch version: 6.1.2
    no log output.
  4. reproduce step
    (1) run filebeat in docker, harvest log in json-file
    (2) put pipeline in elasticsearch, which named ‘haproxyId’
    (3) access haproxy stats, produce the access log in json-file, then the error log is being outputted by filebeat endlessly.
    Can someone help me solve this problem?
2

this problem have solved, which the reason is my elasticsearch pipeline config has error.
there is detailed error information in filebeat's debug info, we can analyse it careful.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.