[Question] Is output routing possible in Filebeat

Elastic Stack Beats
1

Hey there,

currently I'm working on configuring an ELK Stack.

  • Filebeat should put JSON oneliners ONLY into Logstash for parsing.
  • Filebeat should put /var/log/secure, /var/log/yum.log, /var/log/audit/audit.log directly into Elasticsearch.

With my current configuration, I only achieve that both prospectors where send to both outputs....
I thought pipelineing will help here but it didn't... Maybe my approach is wrong?

What I have so far:

filebeat.prospectors:
- input_type: log
  paths:
- /var/lib/docker/containers/*/*-json.log
  json.keys_under_root: true
  json.add_error_key: true
  fields:
type: "docker"
  tags: ["json"]

- paths:
- /var/log/secure
- /var/log/yum.log
- /var/log/audit/audit.log
  fields:
type: "syslog"
  tags: ["syslog"]

output.elasticsearch:
  hosts: ["host:port"]
  index: "syslog-%{+yyyy.MM}"
  pipelines:
    - pipeline: syslog_pipeline
      when.equals:
        fields.syslog: "syslog"

output.logstash:
  hosts: ["host:port"]
  pipelines:
    - pipeline: docker_pipeline
      when.equals:
        fields.type: "docker"
2

This kind of event routing is currently not possible with filebeat. The pipeline setting is for configuring the ingest node pipeline in Elasticsearch. This setting is not available for logstash.
Either use 2 filebeat instances or send all events to one output type only.

In 6.0 only one output can be selected, but the pipeline can be configured in the prospector. The pipeline setting is then forwarded to Logstash in the @metadata.pipeline setting. This would allow you to do filtering and forwarding to ES ingest node pipeline as well.

Indentation in your config files seems to be pretty off. This a copy'n paste error?

3

Yeah the wrong indentation comes from the quote method.

Whats the schedule on this feature?

From what I understood now is, that I've to define a logstash filter which selects a filter from the "field.type" or something else... Am I right?

4

Whats the schedule on this feature?

Which feature? Event routing?

5

Sure..

6

There is no active discussion on implementing event routing in beats. Feel free to open an enhancement request.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.