Filebeat lotstash output and 'exported fields'

[nathharp (Nathan Harper)]

Hi all,

I am planning our migration from our ageing rsyslog(RELP) > logtash > elasticsearch with a new set of infrastructure.

I have been experimenting with filebeat > logstash > elasticsearch but I'm not getting the message granularity that I was hoping for.

Using filebeat > elasticsearch I get lots of wonderful exported fields with loads of useful information. However, if I set filebeat with logstash output all this appears to be lost, and I just get everything lumped into the message field.

Is this expected behaviour? I know I can grok the output, but I was hoping save that for just additional insights or filtering

[andrewkroh]

You don't mention if you were using Filebeat modules when sending Filebeat -> Elasticsearch, but I suspect that you were based on the experience you describe.

Unfortunately FIlebeat modules don't yet work with Logstash because they use Ingest Node in Elasticsearch to parse. See the note here: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-modules-overview.html

An equivalent Logstash filter setup to the Filebeat system module is provided here. You can add this to your Logstash config.

[nathharp (Nathan Harper)]

Ah yes, I did miss that. You are correct, I am using filebeat modules.

Thank you for confirming, and particularly the logstash filter example, that will help out significantly.

[nathharp (Nathan Harper)]

Finally got a chance to test. This does exactly what I was after, many thanks