The error message "output.go:92: ERR Failed to publish events: temporary bulk send failure"

I have some problem, the error log is being outputted by filebeat endlessly. please see the following detail message.
2018-05-02T09:27:21.879178437Z 2018/05/02 09:27:21.879028 output.go:92: ERR Failed to publish events: temporary bulk send failure
2018-05-02T09:27:21.880330394Z 2018/05/02 09:27:21.880199 client.go:651: INFO Connected to Elasticsearch version 6.1.2
2018-05-02T09:27:21.882671683Z 2018/05/02 09:27:21.882543 load.go:73: INFO Template already exists and will not be overwritten.
2018-05-02T09:27:22.885488217Z 2018/05/02 09:27:22.885285 output.go:92: ERR Failed to publish events: temporary bulk send failure
2018-05-02T09:27:22.886610836Z 2018/05/02 09:27:22.886513 client.go:651: INFO Connected to Elasticsearch version 6.1.2
2018-05-02T09:27:22.888869339Z 2018/05/02 09:27:22.888749 load.go:73: INFO Template already exists and will not be overwritten.
2018-05-02T09:27:23.891780722Z 2018/05/02 09:27:23.891512 output.go:92: ERR Failed to publish events: temporary bulk send failure

and I collect some config in this environment. see below

  1. Beat version 6.1.1
  2. Beat config
    filebeat.prospectors:
    -type: log
    enabled: true
    paths:
    - "/var/log/containers/${SYSTEM_POD_NAME}${SYSTEM_NAMESPACE}${_SYSTEM_APP_NAME}-.log"
    symlinks: true
    pipeline: haproxyId
    scan_frequency: 1s
    fields:
    source: ${_SYSTEM_APP_NAME:default}
    host: ${HOSTNAME:default}
    tags:
    -${_SYSTEM_NAMESPACE:default}
    fields_under_root: true
    json.keys_under_root: true
    json.add_error_key: true
    json.message_key: log
    setup.template.name: "haproxy-%{[beat.version]}"
    setup.template.pattern: "haproxy-%{[beat.version]}-
    "
    output.elasticsearch:
    hosts: ["elasticsearch-logging:9200"]
    enabled: true
    index: "haproxy-%{[beat.version]}-%{+yyyy.MM.dd}"
  3. Elasticsearch logs
    elasticseaarch version: 6.1.2
    no log output.
  4. reproduce step
    (1) run filebeat in docker, harvest log in json-file
    (2) put pipeline in elasticsearch, which named ‘haproxyId’
    (3) access haproxy stats, produce the access log in json-file, then the error log is being outputted by filebeat endlessly.
    Can someone help me solve this problem?

this problem have solved, which the reason is my elasticsearch pipeline config has error.
there is detailed error information in filebeat's debug info, we can analyse it careful.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.