What would happend if filebeat couldn't reach elasticsearch

Elastic Stack Beats
1

what would happend if filebeat couldn't reach elasticsearch?
filebeat couldn't reach elasticsearch for a while, and then my server which is running filebeat reported CPU alert to me, filebeat used about 60% CPU usage, how can i avoid it?
Did that means if elasticsearh down then all servers which running filebeat would get high cpu usage?

My filebeat version is 1.2.3

2

filebeat is trying to reconnect and send events every so often. Reconnect is subject to exponential backoff with maximum waiting time 1m. I would expect filebeat to be mostly blocked by the back-pressure from elasticsearch. You have any filebeat logs to share? How is filebeat configured?

4

the log:
2016-06-24T16:25:33+08:00 ERR Fail to publish event to REDIS: write tcp 10.117.2.59:35672->10.51.27.213:6379: write: connection reset by peer

the yml:

filebeat:
  prospectors:
    -
      paths:
        - /logs/web.log
      input_type: log
      fields:
        index_type: "yydb-web"
      fields_under_root: true
      document_type: web01
      scan_frequency: 10s
      harvester_buffer_size: 16384
      multiline:
        pattern: '^[[:digit:]]{4}-[[:digit:]]{2}-[[:digit:]]{2}'
        negate: true
        match: after
      tail_files: false
      backoff: 1s
      input_type: log
  spool_size: 2048
  idle_timeout: 5s
  registry_file: /etc/filebeat/registry
output:
  redis:
    host: "10.51.27.213"
    port: 6379
    save_topology: false
    index: "yydb-filebeat"
    db: 0
    db_topology: 1
    timeout: 5
    reconnect_interval: 1 
shipper:
logging:
  to_syslog: false
  to_files: true
  files:
    path: /etc/filebeat
    name: mybeat
    keepfiles: 7
  level: error
5

Do not use filebeat 1.x release with redis!!! The redis output in 1.x has been deprecated and is and totally rewritten for 5.x release doing proper error handling and backoff. The 5.0-alpha3 release redis output should behave much better.

Reason CPU with filebeat 1.x and redis output is high, due to 1.x redis module not correctly handling errors, basically discarding events and pushing your file as fast as possible into the void.

The rewrite also get's you:

  • load balancing to multiple redis endpoints
  • include TLS support if redis is deployed behind ssh tunnel (e.g. stunnel)
  • SOCKS5 proxy support
6

OK, thanks for your help!

7

This topic was automatically closed after 21 days. New replies are no longer allowed.