Ingest Pipeline with filebeat not working

Elastic Stack Kibana
1

Hi,

We have a few Linux hosts which monitor each other with Pacemaker. Here we would like to evaluate the pacemaker.log for this. With Filebeat the log arrives, but the message is not yet analyzable. Therefore I thought with an own ingest pipeline this would be best possible.

Each Message in our pacemaker log, looks like this:
Feb 23 18:48:33 host pacemaker-schedulerd[1669] (pcmk__log_transition_summary) notice: Calculated transition 3386, saving inputs in /var/lib/pacemaker/pengine/pe-input-374.bz2

Now I created an ingest pipeline, with the following information:
PUT _ingest/pipeline/filebeat-8.6.2-pacemaker { "description": "Filebeat pacemaker pipeline", "processors": [ { "set": { "field": "event.ingested", "value": "{{_ingest.timestamp}}" } }, { "gsub": { "field": "message", "pattern": "\\t", "replacement": "" } }, { "grok": { "field": "message", "patterns": [ "%{MONTH} %{MONTHDAY} %{TIME} %{HOSTNAME:hostname} %{DATA:process_name}\\[%{INT:process_id}\\] \\(%{WORD:log_type}\\) %{WORD:status}: %{GREEDYDATA:error_message}" ] } }, { "rename": { "field": "hostname", "target_field": "fileserver.host" } }, { "rename": { "field": "process_name", "target_field": "fileserver.process" } }, { "rename": { "field": "status", "target_field": "fileserver.status" } }, { "rename": { "field": "error_message", "target_field": "fileserver.error" } }, { "rename": { "field": "log_type", "target_field": "fileserver.log_type" } }, { "rename": { "field": "process_id", "target_field": "fileserver.process_id" } } ], "on_failure": [ { "set": { "field": "error.message", "value": "{{ _ingest.on_failure_message }}" } } ] }

The good news here, with the json file from an log entry, it works and the message can be decoded. But if im integrate it into my host in the filebeat.yml:

  • type: filestream
    id: my-filestream-id
    enabled: true

    paths:

    • /var/log/pacemaker/*.log
      pipeline: filebeat-8.6.2-pacemaker
      processors:
    • drop_event:
      when:
      not:
      equals:
      fileset.name: "pacemaker"

And with this setting I became no data and the filebeat process on the host is responding with success. Could anybody help me here?

Thanks in advance

2

Hi @heisenberg93 and welcome to the community!

the formatting of your config files makes it a little difficult to understand entirely, but from what i can tell you may need to move the pipeline from the path section to the output section. Take a look at the docs here for reference.

3

Okay, perfect, this works. But I got another service which generates logs and the message field got a different structure. Whats the best way here? Should I look out for a better ingest pipeline or is it possible to give every log directory an own ingest pipeline?

4

So I collect many data and the ingest pipeline is active even for my modules. But I only want it for the type filestream. Is there no chance?

5

You can use a pipeline on the input of the filestream if you want to specify different pipelines for each input. Also, if you have an input and an output pipeline specified, then the input pipeline is used.

Try moving the pipeline between the id and the enabled tags, something like this (as an example):

filebeat.inputs:
- type: filestream
  id: my-filestream-id
  pipeline: filebeat-8.6.2-pacemaker
  enabled: true
  paths:
    - /var/log/pacemaker/*.log
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.