Filebeat output elasticsearch pipeline not working

Elastic Stack Beats
1

Hi,
I am stuck on configuring filebeat to use a custom elasticsearch ingest pipeline.
For some reasons the pipeline is now applied for logs shipped by filebeat.
If a document is indexed manually, everything works as expected. Please see the details below.

Pipeline

    PUT _ingest/pipeline/test
    {
      "description": "Test",
      "processors": [
        {
          "set": {
            "field": "foo",
            "value": "bar"
          }
        }
      ]
    }

Create Index

   PUT test

Testdocument

    POST test/_doc?pipeline=test
    {
      "@timestamp" : "2020-07-07T14:21:52.000+02:00"
    }

Filebeat config

    output:
      elasticsearch:
        hosts:
        - https://elastic01-t-bfl.intra.realstuff.ch
        index: test
        password: xxxxxxxxxxx
        ssl.certificate_authorities:
        - /etc/pki/tls/certs/rs-ansible-ca.crt
        username: beats_writer
        pipelines:
          - pipeline: test

Index Query first document

    "hits" : [
        {
            "_index" : "test",
            "_type" : "_doc",
            "_id" : "E-xcKXMBiIdD4nFHVd6o",
            "_score" : 0.13353139,
            "_source" : {
              "@timestamp" : "2020-07-07T14:56:52.000+02:00",
              "foo" : "bar"
            }
        },
         {
            "_index" : "test",
            "_type" : "_doc",
            "_id" : "e-xcKXMBiIdD4nFHfN9m",
            "_score" : 1.0,
            "_source" : {
              "agent" : {
                "hostname" : "elastic01-t-bfl.intra.realstuff.ch",
                "name" : "elastic01-t-bfl.intra.realstuff.ch",
                "id" : "5f20d6ab-fc29-4b6d-90e7-5777c5381fa5",
                "type" : "filebeat",
                "ephemeral_id" : "73f7b3d0-51aa-47bd-9e9c-1d04d7cc9d07",
                "version" : "7.8.0"
              },
              "process" : {
                "name" : "journalbeat"
              },
              "log" : {
                "file" : {
                  "path" : "/var/log/messages"
                },
                "offset" : 244192259
              },
              "fileset" : {
                "name" : "syslog"
              },
              "message" : """2020-07-07T14:59:50.921+0200#011INFO#011[input]#011input/input.go:141#011journalbeat successfully published 1 events#011{"id": "cca32660-ba22-4298-87ac-2548992b4a3d"}""",
              "input" : {
                "type" : "log"
              },
              "@timestamp" : "2020-07-07T14:59:50.000+02:00",
              "system" : {
                "syslog" : { }
              },
              "ecs" : {
                "version" : "1.5.0"
              },
              "service" : {
                "type" : "system"
              },
              "host" : {
                "hostname" : "elastic01-t-bfl",
                "name" : "elastic01-t-bfl.intra.realstuff.ch"
              },
              "event" : {
                "timezone" : "+02:00",
                "module" : "system",
                "type" : "event",
                "dataset" : "system.syslog"
              }
            }
          }
    ]

Does anybody know the problem?

2

It looks like you just have one ingest pipeline that you want to apply to all logs ingested via Filebeat. In that case, is there any reason you are using the output.elasticsearch.pipelines setting instead of the output.elasticsearch.pipeline setting?

3

Hi @shaunak,
Thank you for your answer,
In my production config i would use a condition and a second pipeline later on. This is why I use the list. In the meantime I have changed it to pipeline as you propose but the result ist the same.
Regards Bernhard

4

I have done some further testing and it seems to be related to new 7.8 modules like checkpoint and fortinet. As soon as these modules are enabled the generic log processing fails and pipeline is not applied. Does this make any sense to you?

5

After a downgrade to 7.7.1 pipelines are working. Seems to be a bug of 7.8.

6

I just ran into the same issue, and switching back to 7.7.1 also resolved it for me. I'm not familiar enough with beats to dig further, but hopefully this is helpful for someone else.

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.