# Filebeat output elasticsearch pipeline not working

**URL:** <https://discuss.elastic.co/t/filebeat-output-elasticsearch-pipeline-not-working/240167>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [July 7, 2020, 1:17pm UTC](https://discuss.elastic.co/t/filebeat-output-elasticsearch-pipeline-not-working/240167 "2020-07-07T13:17:21Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![bernhard.fluehmann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bernhard.fluehmann/32/71755_2.png) [@bernhard.fluehmann](https://discuss.elastic.co/u/bernhard.fluehmann)\
**Post date:** [July 7, 2020, 1:17pm UTC](https://discuss.elastic.co/t/filebeat-output-elasticsearch-pipeline-not-working/240167/1 "2020-07-07T13:17:21Z")

</div>

Hi,  
I am stuck on configuring filebeat to use a custom elasticsearch ingest pipeline.  
For some reasons the pipeline is now applied for logs shipped by filebeat.  
If a document is indexed manually, everything works as expected. Please see the details below.

**Pipeline**

```auto
    PUT _ingest/pipeline/test
    {
      "description": "Test",
      "processors": [
        {
          "set": {
            "field": "foo",
            "value": "bar"
          }
        }
      ]
    }

```

**Create Index**

```auto
   PUT test

```

**Testdocument**

```auto
    POST test/_doc?pipeline=test
    {
      "@timestamp" : "2020-07-07T14:21:52.000+02:00"
    }

```

**Filebeat config**

```auto
    output:
      elasticsearch:
        hosts:
        - https://elastic01-t-bfl.intra.realstuff.ch
        index: test
        password: xxxxxxxxxxx
        ssl.certificate_authorities:
        - /etc/pki/tls/certs/rs-ansible-ca.crt
        username: beats_writer
        pipelines:
          - pipeline: test

```

**Index Query first document**

```auto
    "hits" : [
        {
            "_index" : "test",
            "_type" : "_doc",
            "_id" : "E-xcKXMBiIdD4nFHVd6o",
            "_score" : 0.13353139,
            "_source" : {
              "@timestamp" : "2020-07-07T14:56:52.000+02:00",
              "foo" : "bar"
            }
        },
         {
            "_index" : "test",
            "_type" : "_doc",
            "_id" : "e-xcKXMBiIdD4nFHfN9m",
            "_score" : 1.0,
            "_source" : {
              "agent" : {
                "hostname" : "elastic01-t-bfl.intra.realstuff.ch",
                "name" : "elastic01-t-bfl.intra.realstuff.ch",
                "id" : "5f20d6ab-fc29-4b6d-90e7-5777c5381fa5",
                "type" : "filebeat",
                "ephemeral_id" : "73f7b3d0-51aa-47bd-9e9c-1d04d7cc9d07",
                "version" : "7.8.0"
              },
              "process" : {
                "name" : "journalbeat"
              },
              "log" : {
                "file" : {
                  "path" : "/var/log/messages"
                },
                "offset" : 244192259
              },
              "fileset" : {
                "name" : "syslog"
              },
              "message" : """2020-07-07T14:59:50.921+0200#011INFO#011[input]#011input/input.go:141#011journalbeat successfully published 1 events#011{"id": "cca32660-ba22-4298-87ac-2548992b4a3d"}""",
              "input" : {
                "type" : "log"
              },
              "@timestamp" : "2020-07-07T14:59:50.000+02:00",
              "system" : {
                "syslog" : { }
              },
              "ecs" : {
                "version" : "1.5.0"
              },
              "service" : {
                "type" : "system"
              },
              "host" : {
                "hostname" : "elastic01-t-bfl",
                "name" : "elastic01-t-bfl.intra.realstuff.ch"
              },
              "event" : {
                "timezone" : "+02:00",
                "module" : "system",
                "type" : "event",
                "dataset" : "system.syslog"
              }
            }
          }
    ]

```

Does anybody know the problem?

---

<div class="post-metadata">

**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\
**Post date:** [July 7, 2020, 6:25pm UTC](https://discuss.elastic.co/t/filebeat-output-elasticsearch-pipeline-not-working/240167/2 "2020-07-07T18:25:48Z")

</div>

It looks like you just have one ingest pipeline that you want to apply to all logs ingested via Filebeat. In that case, is there any reason you are using the [`output.elasticsearch.pipelines`](https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#pipelines-option-es) setting instead of the [`output.elasticsearch.pipeline`](https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#pipeline-option-es) setting?

---

<div class="post-metadata">

**Author:** ![bernhard.fluehmann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bernhard.fluehmann/32/71755_2.png) [@bernhard.fluehmann](https://discuss.elastic.co/u/bernhard.fluehmann)\
**Post date:** [July 7, 2020, 7:55pm UTC](https://discuss.elastic.co/t/filebeat-output-elasticsearch-pipeline-not-working/240167/3 "2020-07-07T19:55:00Z")

</div>

Hi @shaunak,  
Thank you for your answer,  
In my production config i would use a condition and a second pipeline later on. This is why I use the list. In the meantime I have changed it to pipeline as you propose but the result ist the same.  
Regards Bernhard

---

<div class="post-metadata">

**Author:** ![bernhard.fluehmann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bernhard.fluehmann/32/71755_2.png) [@bernhard.fluehmann](https://discuss.elastic.co/u/bernhard.fluehmann)\
**Post date:** [July 8, 2020, 10:36am UTC](https://discuss.elastic.co/t/filebeat-output-elasticsearch-pipeline-not-working/240167/4 "2020-07-08T10:36:44Z")

</div>

I have done some further testing and it seems to be related to new 7.8 modules like checkpoint and fortinet. As soon as these modules are enabled the generic log processing fails and pipeline is not applied. Does this make any sense to you?

---

<div class="post-metadata">

**Author:** ![bernhard.fluehmann](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bernhard.fluehmann/32/71755_2.png) [@bernhard.fluehmann](https://discuss.elastic.co/u/bernhard.fluehmann)\
**Post date:** [July 17, 2020, 3:14pm UTC](https://discuss.elastic.co/t/filebeat-output-elasticsearch-pipeline-not-working/240167/5 "2020-07-17T15:14:28Z")

</div>

After a downgrade to 7.7.1 pipelines are working. Seems to be a bug of 7.8.

---

<div class="post-metadata">

**Author:** ![mike-twa](https://avatars.discourse-cdn.com/v4/letter/m/a183cd/32.png) [@mike-twa](https://discuss.elastic.co/u/mike-twa)\
**Post date:** [July 30, 2020, 12:08am UTC](https://discuss.elastic.co/t/filebeat-output-elasticsearch-pipeline-not-working/240167/6 "2020-07-30T00:08:09Z")

</div>

I just ran into the same issue, and switching back to 7.7.1 also resolved it for me. I'm not familiar enough with beats to dig further, but hopefully this is helpful for someone else.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 27, 2020, 2:08am UTC](https://discuss.elastic.co/t/filebeat-output-elasticsearch-pipeline-not-working/240167/7 "2020-08-27T02:08:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.