Target my ES cluster with ~40 FileBeat instances

[Yarden_Bar]

Hello,
Last year I found that Logstash doesn't perform load balancing between inputs and 'prefer' inputs that doesn't have pushback mechanism or that pushback is not relevant for them (like syslog input)

I'm thinking about pointing all of my FileBeat army, around 40 instances) to ship json-line formatted logs directly to our ES cluster using FileBeat's ES output.
Existing Logstash instances will still process logs that can't be shipped directly to ES.

The root cause is that there's a lot of contention on our Logstash instances which i'd like to minimize.

Can anyone comment on her/his experience with pointing FileBeat directly at ES cluster?

Thank you,
Yarden

[Yarden_Bar]

It is mentioned here that FileBeat specifically ignore the max_retries configuration option.

On one hand, its good that it'll keep trying to ship an event if ES returns a failure response.
On the other hand, what if the failed event has incorrect structure that doesn't match the index mapping? will it still try to ship the broken event or it will fail on the ES side with an error logged to file?

[ruflin]

Are you using JSON logs to be sent to elasticserach or just the basic default content? If the second, I would not expect any events that don't match the mapping. What is the structure of your event?

[Yarden_Bar]

My logs are formatted as valid Json(one per line) and the filter in Logstash that handles those events only renames some fields(Today I implemented the rename operation in our application so the filter is actually redundant).

The index template on the ES side has relevant mapping to facilitate the events I'm going to shoot at it).

Event example:
{"method":"GET","request":"/notifications","format":"js","controller":"user_notifications","action":"new_notification_count","status":200,"duration":187.01,"view":0.55,"db":8.61,"true_ip":"XXX.XXX.XXX.XXX","route":"user_notifications#new_notification_count","request_id":"SOME_REQUEST_ID","type":"rails","user_id":"123456","source":"unknown","tags":["request"],"@timestamp":"2016-08-01T16:28:22.171Z","@version":"1"}

Thank you,
Yarden

[steffens (Steffen Siering)]

It depends on status code by elasticsearch if event will be retried.

See status code check. If message is dropped the warning message Can not index event ... will be logged.

[Yarden_Bar]

Thank you @steffens,

I'll experiment by pointing 1,2,3,....N FileBeat instances to the ES cluster and report my findings

Yarden