Logstash to filebeat correct migration

Elastic Stack Beats
1

elk2
I have ES standalone, Kibana standalone, logstash cluster, that collects data and Queue with Logstashes as a log "tailers".

Main purpose - decrease resources consumption of "endpoint" log harvesters, so i need to change my logstashes (green on schema) by the filebeats clients.

But there is some troubles:

  • a filters (grok / mutate / date) on the LS clients.
  • custom logs with my application
  • i cant just move filter logic to LS cluster (it's already overburden)
  • i think some troubles with queue
  • ...

Q1: How can i do this with less pain? Use filebeat with CUSTOM module?

Q2: Is it correct that i can use GROK and other "tools" and directly output to the ES with my Filebeat`s custom module ?

2

@ep4sh

I will try to answer both of your questions, have you considered using Filebeat to send data to collectors Logstashes and send the events to the queue or to your queue directly? Filebeat support a few ouputs: ES, Logstash, Redis, Kafka In that case Filebeat will just acts as a simple log shipper and much of your infra will not change but it will still free up resources on the node.

Q1/Q2

You can send your data directly from Filebeat to ES and use the ingest node and a custom pipeline to do the parsing. Doing so will effectively increase the load on your ES nodes.

1 Like
3

So, "Ingest Node" called ES or Filebeat "node" (in my case)? If it correct, i see the flow looks like:

Filebeat (preprocces document) -> Queue (ex, Kafka) -> ES

Is it correct ?

Doing so will effectively increase the load on your ES nodes.
If it is possible, could you tell me, approximately, how much (in percent / 1 filebeat node)

4

@ep4sh Not exactly, it's either one of the following.

  1. Filebeat -> Logstash as collector of Filebeat data -> Kafka -> Logstash -> ES (also minimal changes with your infra)
  2. Filebeat -> kafka -> Logstash -> ES (minimal changes in your current infra)
  3. Filebeat -> ES (with a custom ingest node)

If it is possible, could you tell me, approximately, how much (in percent / 1 filebeat node )

It's hard to know without knowing the events rate and the work that need to be done and to be honest the only way to find out is to create an experimentation to test it.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.