Filebeat output to logstash and elasticsearch both

Elastic Observability Logs
1

Hi

I am using Elasticstack 7.9 on-prem - I am wondering about a scenario if it's possible

I have 2 services on 1 VM - one is an haproxy service and the other is an nginx service

Both services generate log files on disk

I want to configure Filebeat so that :

  • haproxy logs are sent directly to Logstash (where I have some ruby code that reads from a DB and enriches the log data) and

  • nginx logs are sent directly to Elasticsearch since I using the nginx module to parse the logs correctly and that is sufficient

It seems that I can send either to logstash or elasticsearch - a workaround could be to check the event data and forward the nginx data as-is to elasticsearch - but I was wondering if there is a setting in filebeat for separate output destinations

Thanks

2

It is not possible, filebeat only supports one output.

What you can do in your case is run two filebeat instances one for the nginx logs and another for the haproxy log, or send both to logstash and do not apply any filter to the nginx logs.

3

What most people do is tag both, send both through logstash and the send to elasticsearch via 2 elasticsearch outputs

The one where you want to use the nginx module

Configured like this it will use the ngnix ingest pipeline

elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}" 
  }

The other output for the haproxy logs can have it's own elasticsearch output.

You can use conditionals based on those tags to choose which output you want to send the documents to.

Hope that makes sense

4

got it - thanks - I will use a tag to identify b/w haproxy and nginx and parse haproxy through the filter and forward nginx logs as-is

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.