Possible to use Beats processors instead of logstash for adding transforms / fields?

[mzmuda (Mike Zmuda)]

Hi all,

I'm wondering if it's possible to use the processors in beats / filebeats instead of logstash filters to add fields for data / visualization?

Right now I have filebeats configured to send data directly to elastic, and want to do some transforms on the message field of entries to generate some interesting data / trends/ dashboards / etc. Part of this would be adding fields to entries based on the message information, ie throwing regexes against the raw text to pull out machine names, different states etc.

My read through the beats docs seems to suggest that this is possible, if so are there any downsides to this approach vs having logstash in the overall pipeline?

[pierhugues (Pier-Hugues Pellerin)]

Actually we support three kind of transformation or manipulation:

• With Logsash using filters
• With Elasticsearch's ingest node.
• Or with beats processors.

I would say if you have already filebeat installed and one of the processors can allow you do to do the kind of manipulation you are expecting. I would go with that route because its the simplest to implement in your architecture.

Concerning downside, adding "works" on the edge like in filebeat could slow down the ingestion rate and use a bit more of cpu usage on the edge machine. But like in everything, nothing is free and if you do any work it will have impacts.