ERR Failed to Perform Many Index Operations

Elastic Stack Beats
1

We're trying to get filebeat to run on our logstash host to test the ingestion of logs but we keep getting the same error regardless of the config. I've pasted the config file we are using below which results in the error

output.go:287: ERR Failed to perform many index operations in a single API call: PerformRequest fails: Sending the request fails: Post http://localhost:5173/_bulk: EOF

Any ideas?

filebeat:

  prospectors:
    -
      paths:
        - "/var/log/messages"
      type: log
      scan_frequency: 10s
  registry_file: /var/lib/filebeat/registry

  config_dir: /etc/filebeat/prospector_configs



output:


  elasticsearch:
    enabled: false

  logstash:
    enabled: true
    hosts: ["localhost:5173"]

  file:
    enabled: false
    path: "/tmp/filebeat"
    filename: filebeat.log
    rotate_every_kb: 10000

shipper:
2

I'd check your ES logs, it may be rejecting things and if so it should mention something about threadpools and queue capacity.

3

Something is strange here. You config shows that the logstash output is enabled, but the log message indicates that the elasticsearch output is used. I suspect there's something wrong with the YAML file, can you try complete removing the elasticsearch section and see if it helps?

4

it even uses the hosts configured in logstash output when trying to connect to elasticsearch.

5

Thanks @tudor, I made some minor changes to the config. First, I did set up an input on the logstash server using the beats input plugin instead of lumberjack. I also changed the port from the lumberjack default port to 5044.

The input (on the same server) takes the filebeat input and dumps it to a lumberjack output which (as odd as this feels) sends it to a lumberjack input/redis output on the same server for shipping to redis. That feels like an extra step to me, but it works.

filebeat:
prospectors:
-
paths:
- "/var/log/messages"
type: log
scan_frequency: 10s
registry_file: var/lib/filebeat/registry
output:
elasticsearch:
enabled: false
logstash:
enabled: true
hosts: ["localhost:5044"]
shipper:

6

huh, so you config goes like this:

filebeat ---(lumberjack v2)---> logstash ----(lumberjack v1)----> logstash ----- (redis API) ----> redis

7

No lumberjack version 2 is the mix.

Our environment goes filebeat to a logstash-based filebeat input which we then output via to Redis via a Redis output.

If it matters, we then have dual-LS boxes in AWS where we pull from then Redis queue and shove into a 3 server ES cluster.