Custom beat - re-use Filebeat log readers?


#1

I'm creating a new beat application, primarily to remove the logstash requirements for getting this data into Elastic Search. The file being read is new line delimited JSON, and gets rotated by logrotate.

Anyone reusing filebeat code for the "tailing" and "registry" tracking in their custom beat applications? Or should I just roll my own.

Thanks.


(Magnus B├Ąck) #2

primarily to remove the logstash requirements for getting this data into Elastic Search.

You are aware of Filebeat's ability to send data directly to ES, right?


#3

As JSON? Last I tried it sent to the JSON as text lines. I had to run it through Logstash to be added as JSON. Has this been changed?


(Steffen Siering) #4

Filebeat 5.0 alpha1 adds json per line support. The JSON is parsed and send as event to elasticsearch. Try console output with pretty: true to check out your events. Even better, Elasticsearch 5.0 will have Ingest-Node adding some filtering/parsing capabilities to Elasticsearch input.


(system) #5