# Remote Filebeat cannot ship logs to elastic ERROR \[publisher\_pipeline\_output\]

**URL:** <https://discuss.elastic.co/t/remote-filebeat-cannot-ship-logs-to-elastic-error-publisher-pipeline-output/248893>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [September 17, 2020, 1:01am UTC](https://discuss.elastic.co/t/remote-filebeat-cannot-ship-logs-to-elastic-error-publisher-pipeline-output/248893 "2020-09-17T01:01:55Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Team\_K](https://avatars.discourse-cdn.com/v4/letter/t/e8c25b/32.png) [@Team\_K](https://discuss.elastic.co/u/Team_K)\
**Post date:** [September 17, 2020, 1:01am UTC](https://discuss.elastic.co/t/remote-filebeat-cannot-ship-logs-to-elastic-error-publisher-pipeline-output/248893/1 "2020-09-17T01:01:56Z")

</div>

I have been trying to ship logs from a remote server to my elk. Below is my conf file. I tested my conf file using `filebeat -c filebeat.yml test output` everything seems fine. Could you please help. Thank you

```auto
        elasticsearch: [<elastic_host>:9200](http://<elastic_host>/)...
        parse url... OK
        connection...
        parse host... OK
        dns lookup... OK
        addresses: <elastic_host>
        dial up... OK
        TLS... WARN secure connection disabled
        talk to server... OK
        version: 7.8.1

```

Below is the error in the filebeat logs

````
```220-09-16T17:31:28.991-0700	ERROR	[publisher_pipeline_output]	pipeline/output.go:154	Failed to connect to backoff(elasticsearch(http://10.176.140.123:9200)): Connection marked as failed because the onConnect callback failed: 1 error: Error loading pipeline for fileset system/auth: couldn't load pipeline: couldn't load json. Error: 400 Bad Request: {"error":{"root_cause":[{"type":"parse_exception","reason":"processor [set] doesn't support one or more provided configuration parameters [ignore_empty_value]","processor_type":"set"}],"type":"parse_exception","reason":"processor [set] doesn't support one or more provided configuration parameters [ignore_empty_value]","processor_type":"set"},"status":400}. Response body: {"error":{"root_cause":[{"type":"parse_exception","reason":"processor [set] doesn't support one or more provided configuration parameters [ignore_empty_value]","processor_type":"set"}],"type":"parse_exception","reason":"processor [set] doesn't support one or more provided configuration parameters [ignore_empty_value]","processor_type":"set"},"status":400} ```

````

My filebeat config file is:

```auto
                filebeat:
                  config:
                    modules:
                      path: /etc/filebeat/modules.d/*.yml
                      reload:
                        enabled: false
                  inputs:
                  - enabled: true
                    paths:
                    - /var/log/st2/st2actionrunner.gelf.log
                    - /var/log/st2/st2sensorcontainer.gelf.log
                    type: log
                logging:
                  files:
                    keepfiles: 7
                    name: filebeat
                    path: /var/log/filebeat
                    permissions: 420
                  level: warning
                  selectors:
                  - '*'
                  to_files: true
                output:
                  elasticsearch:
                    hosts:
                    - <elastic_ip_edited_for_confidentiality>:9200
                    index: ss-logs-%{[agent.version]}-%{+yyyy.MM.dd}
                path:
                  config: /etc/filebeat
                  data: /var/lib/filebeat
                  home: /usr/share/filebeat
                  logs: /var/log/filebeat
                processors:
                - add_host_metadata: null
                - add_cloud_metadata: null
                - add_docker_metadata: null
                - add_kubernetes_metadata: null
                - decode_json_fields:
                    fields:
                    - message
                setup:
                  ilm:
                    enabled: false
                  kibana: null
                  template:
                    fields: fields.yml
                    name: ss-logs
                    pattern: ss-logs-*
                    settings:
                      index:
                        number_of_shards: 1

```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 17, 2020, 1:05am UTC](https://discuss.elastic.co/t/remote-filebeat-cannot-ship-logs-to-elastic-error-publisher-pipeline-output/248893/2 "2020-09-17T01:05:03Z")

</div>

Welcome to our community! 😃 FYI it's not the best idea to just post an error and a config file without explaining what you are looking for.

However; Can you connect to your Elasticsearch instance?

---

<div class="post-metadata">

**Author:** ![Team\_K](https://avatars.discourse-cdn.com/v4/letter/t/e8c25b/32.png) [@Team\_K](https://discuss.elastic.co/u/Team_K)\
**Post date:** [September 17, 2020, 1:10am UTC](https://discuss.elastic.co/t/remote-filebeat-cannot-ship-logs-to-elastic-error-publisher-pipeline-output/248893/3 "2020-09-17T01:10:12Z")

</div>

Thank you @warkolm. Edited it. Yes,  
I can connect to my Elastic instance

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [September 17, 2020, 1:18pm UTC](https://discuss.elastic.co/t/remote-filebeat-cannot-ship-logs-to-elastic-error-publisher-pipeline-output/248893/4 "2020-09-17T13:18:50Z")

</div>

> [@Team\_K](#):
>
> `220-09-16T17:31:28.991-0700	ERROR	[publisher_pipeline_output]	pipeline/output.go:154	Failed to connect to backoff(elasticsearch(http://10.176.140.123:9200)): Connection marked as failed because the onConnect callback failed: 1 error: Error loading pipeline for fileset system/auth: couldn't load pipeline: couldn't load json. Error: 400 Bad Request: {"error":{"root_cause":[{"type":"parse_exception","reason":"processor [set] doesn't support one or more provided configuration parameters [ignore_empty_value]","processor_type":"set"}],"type":"parse_exception","reason":"processor [set] doesn't support one or more provided configuration parameters [ignore_empty_value]","processor_type":"set"},"status":400}. Response body: {"error":{"root_cause":[{"type":"parse_exception","reason":"processor [set] doesn't support one or more provided configuration parameters [ignore_empty_value]","processor_type":"set"}],"type":"parse_exception","reason":"processor [set] doesn't support one or more provided configuration parameters [ignore_empty_value]","processor_type":"set"},"status":400} `

You'll need Elasticsearch 7.9 to use this Filebeat 7.9 module. When Filebeat 7.9.2 comes out this requirement to have ES 7.9 will be relaxed. See [[Filebeat] backwards compatibility for set processor by leehinman · Pull Request #20908 · elastic/beats · GitHub](https://github.com/elastic/beats/pull/20908).

---

<div class="post-metadata">

**Author:** ![Team\_K](https://avatars.discourse-cdn.com/v4/letter/t/e8c25b/32.png) [@Team\_K](https://discuss.elastic.co/u/Team_K)\
**Post date:** [September 17, 2020, 6:41pm UTC](https://discuss.elastic.co/t/remote-filebeat-cannot-ship-logs-to-elastic-error-publisher-pipeline-output/248893/5 "2020-09-17T18:41:17Z")

</div>

> [@andrewkroh](#):
>
> You'll need Elasticsearch 7.9 to use this Filebeat 7.9 module. When Filebeat 7.9.2 comes out this requirement to have ES 7.9 will be relaxed. S

So downgrading to 7.8 should help right?

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [September 17, 2020, 7:00pm UTC](https://discuss.elastic.co/t/remote-filebeat-cannot-ship-logs-to-elastic-error-publisher-pipeline-output/248893/6 "2020-09-17T19:00:46Z")

</div>

Yes, using Filebeat 7.8 until the new release comes out should work as those Ingest Node pipelines are not using the `ignore_empty_value` option in the `set` processor that was introduced in Elasticsearch 7.9.

Or you could update your Elasticsearch cluster to 7.9.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 15, 2020, 9:00pm UTC](https://discuss.elastic.co/t/remote-filebeat-cannot-ship-logs-to-elastic-error-publisher-pipeline-output/248893/7 "2020-10-15T21:00:57Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.