I have ES standalone, Kibana standalone, logstash cluster, that collects data and Queue with Logstashes as a log "tailers".
Main purpose - decrease resources consumption of "endpoint" log harvesters, so i need to change my logstashes (green on schema) by the filebeats clients.
But there is some troubles:
- a filters (grok / mutate / date) on the LS clients.
- custom logs with my application
- i cant just move filter logic to LS cluster (it's already overburden)
- i think some troubles with queue
Q1: How can i do this with less pain? Use filebeat with CUSTOM module?
Q2: Is it correct that i can use GROK and other "tools" and directly output to the ES with my Filebeat`s custom module ?